Microsoft Service Fabric and logging

Question

I am investigating the best way to implement logging for my Service Fabric Stateless API and have been somewhat overwhelmed by the varying different solutions for what appears to be a relatively simple requirement.

I have implemented logging using the WebHostBuilder().ConfigureLogging and have successfully logged my trace messages to the Debug window and via Serilog.Extensions.Logging.File I have also managed to dump this log to a file, this all being controlled via a #if DEBUG directive and this I was happy with.

Then I needed to configure what would happen when deployed to a cluster within Azure and this is when I became overwhelmed!!!

I thought that I could register ServiceEventSource type logger in the same manner as I did with AddDebug however it was not this simple.

So I have managed to get my logs to appear within the diagnostic window using the ServiceEventSource.Current.Message but these logs are not integrated within the ASP.NET logging framework :/

My continued investigation has led me to understand that Service Fabric logging should be directed towards Application Insights albeit many, many articles having varying degrees of detail and applicability to the latest framework.

My current thinking is that I need to remove the ASP.NET logging and implement something such as EventFlow to allow my trace messages to be generated and subsequently piped through to Application Insights for interrogation at a later date, is my thinking correct??

Or am I currently going off at a tangent?

Answer 1

UPDATE 15/05/2019
After deploying this to Azure Service Fabric the log files were not populated, this appears to be an incompatibility between the Serilog.Sinks.AzureBlobStorage NUGET package and the .NET Core version 2.2.0 that my project was targeting.

I have posted a ticket on the GitHub page and await a response, in the short term you can download the source code and migrate the project to a Microsoft.NETCore.App 2.2.0 project and directly reference this and everything works perfectly.

ORIGINAL ANSWER
I seem to do this quite a lot, answering my own question but here goes again. It's taken me a day or two to get to the bottom of this so I thought I would share my findings and solution with the community in-case it might help somebody else in the future and/or somebody might have something to add or even contradict me which I'd welcome any input.

My development environment is as follows: -

Microsoft Visual Studio 15.9.11
Windows 10 Professional SDK: Microsoft.NETCore.App 2.2.0

I created a new Service Fabric Stateless Service the purpose of this service is to provide RESTful endpoints to a Angular 7 front end web application.

My requirement was to provide logging information in both my development environment via the Debug window and to also provide similar logging information whilst my apps are being hosted within a Service Fabric Cluster on Azure.

NUGET Package Installations
Microsoft.Extensions.Logging (2.2.0)
Serilog.AspNetCore (2.1.1)
Serilog.Enrichers.Environment (2.1.3)
Serilog.Settings.Configuration (3.0.1)
Serilog.Sinks.Debug (1.0.1)
Serilog.Sinks.AzureBlobStorage (1.3.0)

Controlling Development & Production Environments
I control the development & production environments using the DEBUG pre-processor directive to include either the appsettings.json or appsettings.Development.json file.

My appSettings.Development.json file is like this: -

{
  "AppSettings": {
     // My app settings not applicable to this
  },
  "Serilog": {
    "Using": [ "Serilog.Sinks.Debug" ],
    "MinimumLevel": {
      "Default": "Verbose",
      "Override": {
        "Microsoft": "Warning",
        "System": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "Debug",
        "Args": {
          "outputTemplate": "[{Timestamp:HH:mm:ss} {MachineName} {Level:u3}] {Message:lj}{NewLine}{Exception}"
        }
      }
    ],
    "Enrich": ["WithMachineName"]
  } 
}

My appSettings.json file is like this: -

{
  "AppSettings": {
     // My app settings not applicable to this
  },
  "Serilog": {
    "Using": [ "Serilog.Sinks.AzureBlobStorage" ],
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "System": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "AzureBlobStorage",
        "Args": {
          "outputTemplate": "[{Timestamp:HH:mm:ss} {MachineName} {Level:u3}] {Message:lj}{NewLine}{Exception}",
          "connectionString": "[Connection String]",
          "storageContainerName": "app",
          "storageFileName": "{yyyy}-{MM}-{dd}.log"
        }
      }
    ],
    "Enrich": [ "WithMachineName" ]
  }
}

As you can see from the above settings files I output to the Debug window when in development and I have chosen to output to Azure Blob Storage when deployed to a Service Fabric Cluster in Azure.

To implement the Serilog logging simple review my Stateless Service class implementation below, which shows how to toggle the two different appSettings.json files dependent upon the environment and also how the Serilog logger is inserted into the dependency injection system via the use of the UseSerilog extension method.

using System.Collections.Generic;
using System.Fabric;
using System.IO;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.ServiceFabric.Services.Communication.AspNetCore;
using Microsoft.ServiceFabric.Services.Communication.Runtime;
using Microsoft.ServiceFabric.Services.Runtime;
using Serilog;

namespace Caboodal.Manatee.ServiceFabric.Api.Identity
{
    internal sealed class Identity : StatelessService
    {
        public Identity(StatelessServiceContext context)
            : base(context)
        {
        }

        private string AppSettingsFilename
        {
            get
            {
#if DEBUG
                return "appsettings.Development.json";
#else
                return "appsettings.json";
#endif
            }
        }

        protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
        {
            var appSettings = GetAppSettings();

            Log.Logger = new LoggerConfiguration()
                .ReadFrom.Configuration(appSettings)
                .CreateLogger();

            return new[]
            {
                new ServiceInstanceListener(
                    serviceContext =>
                        new KestrelCommunicationListener(
                            serviceContext,
                            "ServiceEndpoint",
                            (url, listener) =>
                            {
                                ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting Kestrel on {url}");

                                return new WebHostBuilder()
                                    .UseKestrel()
                                    .ConfigureAppConfiguration(
                                        (builderContext, config) =>
                                        {
                                            config.AddJsonFile(AppSettingsFilename, false, true);
                                        })
                                    .ConfigureServices(
                                        services => services
                                            .AddSingleton(serviceContext))
                                    .UseContentRoot(Directory.GetCurrentDirectory())
                                    .UseSerilog()
                                    .UseStartup<Startup>()
                                    .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                                    .UseUrls(url)
                                    .Build();
                            }))
            };
        }

        private IConfigurationRoot GetAppSettings()
        {
            return new ConfigurationBuilder()
                .SetBasePath(Directory.GetCurrentDirectory())
                .AddJsonFile(AppSettingsFilename)
                .Build();
        }
    }
}

Using the Logger within a controller
Because the ILogger instance is configured as a Dependency Injected instance it can simply be accessed within your Controller classes like any other dependency eg

    [Authorize]
    [ApiController]
    [Route("[controller]")]
    public class UserController : ApiController
    {
        private readonly IUserService _userService;
        private readonly ILogger<UserController> _logger;

        public UserController(IUserService userService, ILogger<UserController> logger)
        {
            _userService = userService;
            _logger = logger;
        }

        [AllowAnonymous]
        [HttpPost("authenticate")]
        public IActionResult Authenticate([FromBody] DtoAuthenticateRequest request)
        {
            // Adding log entries
            _logger.Log(LogLevel.Debug, "Here is a log entry");

            // Some code in here
            return Ok(response);
        }
    }

I got very sidetracked with the ServiceEventSource.cs class but with the usage of Serilog I have now ignored this aspect of the project template.

If you wish to output your logs to other data consumers or simply into different formats then just review the Serilog website here for a complete list of the Sinks available, with Application Insights being one of the many.