Structured Logging with Serilog, Asp.NET 5.0 APIs, and Application Insights

This is a complete walk-through to set up Structured Logging with Serilog and Application Insights for Asp.NET APIs.

The Need

The ability to search through your logs and run queries over massive pile of events is a powerful tool that enables your organisation to build resilient and bug free software.

Traditionally what we used to do to implement logging in our applications was to write events as text lines using log management libraries such as Serilog or NLog.

As a result of that, we ended up with huge text-based log data stores that wouldn’t give us enough power to search through them and extract actionable insights that would help with the health of our assets long term.

To fix that, applications started onboarding Structured Logging, which would enable us to log queryable properties beside text. So as well as storing “An order with Id = 424 was submitted”, we would also store a field such as “OrderId: 424”, which then can be searched using a structured query language of choice.

Additionally what we used to do to implement logging in our applications was to create an interface like ICustomeLogger, and implement it with whatever we liked using NLog, Serilog, Log4Net or any other logging framework. We did that to minimise the change in our applications if we decided to change the logging framework later on.

Subsequently every application ended up repeating exactly the same pattern over and over again. Since then Microsoft came up with an ILogger interface (inside Microsoft.Extensions.Logging) which can be used as a consistent interface across all Apps and APIs.

We will then inject ILogger anywhere we need to log errors or any messages, and it will go straight to Application Insights in Azure or a local file when testing on your local machine.

By the way, this topic belongs to the series to set up an Asp.NET API for production use.

1. API Route Versioning
2. Configuration Management
3. Secret Management
4. Monitoring & Logging (NLog)
5. Monitoring & Structured Logging (Serilog)
6. Database
7. Documentation
8. CORS
9. Request Validation
10. Global Exception Handling
11. URL Rewriting
12. Deploy .NET API to Azure App Service
13. Call Other APIs
14. Distributed Caching
15. AutoMapper
16. API Gateways

1. Libraries

What we do in this post is to capture application logs using Serilog, append necessary queryable properties for some records as well as some diagnostic properties for all the records, and send the logs to a File (for local), and Azure Application Insights (for production). For that purpose, you will need the below packages:

• Serilog.AspNetCore
• Serilog.Enrichers.Environment
• Serilog.Enrichers.Process
• Serilog.Enrichers.Thread
• Serilog.Settings.Configuration
• Serilog.Sinks.ApplicationInsights
• Serilog.Sinks.Async
• Microsoft.ApplicationInsights.AspNetCore

1. Initialising Serilog

The first step in initialising and injecting Serilog into our Asp.NET 5.0 API is by calling UseSerilog on our host builder:

2. Cover Application Entry Point

Then we will need to cover the Main function, and log any potential fatal error that could happen there:

3. LoggingConfigurations.GetLogger()

I have extracted the log configuration into its own method, which I will cover below:

Consider the below points:

• Serilog Config Source: There are two ways to define Serilog behavior, in code using their FluentApi or from config files. As you can see in line 11, I am simply defining the config in my appSettings file, and read them all from there. I do this, because I can have different logging behaviour for each environment, such as excluding certain log records or properties in Production.
• Application Insights: Then I define another logging destination for production, which is Azure application insights, and I’d like to keep that just for production.
• Inject Logger: and finally I create an instance of the logger, and I inject it.

Note

Configuration, plumbing and troubleshooting your software foundation take a considerable amount of time in your product development. Consider using Pellerex which is a complete foundation for your enterprise software products, providing source-included Identity and Payment functions across UI (React), API (.NET), Pipeline (Azure DevOps) and Infrastructure (Kubernetes).

4. Configuration Behaviour

As I mentioned above, I define the behaviour of logging in my app settings as below:

Inserted at the root of my appSettings.json, is my Serilog configuration. Please note that I have deleted the default Log section that comes with Asp.NET templates as it was useless in this case. Here are the elements which I’d like to comment on:

• Write Tos (targets): they are the mediums which Serilog will write the logs to. I have two formats, one in json and one in simple log text.
• Minimum Levels: By default I have set Serilog to collect logs at information level, however I have overridden it for Microsoft and System logs, to be at Warning levels, so they won’t produce lots of noise.
• Application Insights: the instrumentation key for application insights, so it would know to which instance to send all the logs to.
• Enrich: A very important bit of configs is the Enrichers. They are components that help us enrich logs by injecting context around each log records. I have basically said I’d like you to enrich my log records and add MachineName, ProcessId, and ThreadId to the them. I’ll cover FromLogContext in the next section.

5. Log Contextual Information

There are multiple ways I can inject contextual values in the form of structured properties into log records:

5.1 Message Templates

Anywhere I am logging information using ILogger, I can inject properties.

logger.LogInformation('User {userId} placed a new order {orderId}', '27836478', 98733);

This will associate two searchable properties, like the json you saw in the beginning, with the log record.

5.2 Fixed Messages with Properties

If you like to have a fixed message, with properties attached to it, you could use the below model:

As you can see, this will not replace templates in the message, but it will add those properties to the log structure.

The down side with this approach is the use of ILogger from Serilog, and not Microsoft.Extensions.Logging. This will cause some challenges as over time replacing our logging framework will touch a lot of files.

5.3 LogContext.PushProperty

The next option is to use LogContext.PushProperty to inject more contextual properties around our logs. As an example, the below method will include CorrelationId for that LogInformation.

The downsides for this approach are that apart from using LogContext from Serilog in our application (which makes replacing the logging library harder in future), this approach is quite localised, meaning it will only inject CorrelationId for that specific LogInformation, and not the rest of our application.

6. Middleware

The approach I prefer and we have used in Pellerex ecosystem, is through Asp.NET Middleware. It means for every request we inject a fixed set of contextual and diagnostic information, which will make it far easier to debug our application in future.

Consider the below notes about this middleware:

• IDiagnosticContext: I haven’t used PushProperty here, as the nature of these additional properties are for diagnostic purposes.
• Localised: This approach is quite localised, and it enables me to keep the foot-print of Serilog limited in certain locations (and not all the classes), and in case I wanted to change that in future, I’ll only be touching a few places.

The way I use this middleware, is to call it in our pipeline after Authentication, as I need information such as UserId to be filled and present there.

6. Request Logging

To monitor the health and performance of my application over time, I also have chosen to log individual requests and the time taken to serve those requests. As you see in the above snippet, that is accomplished by calling app.UseSerilogRequestLogging();. The result of this would be log records like this:

HTTP GET /v1/products responded 200 in 1530.5491 ms

Note

Configuration, plumbing and troubleshooting your software foundation take a considerable amount of time in your product development. Consider using Pellerex which is a complete foundation for your enterprise software products, providing source-included Identity and Payment functions across UI (React), API (.NET), Pipeline (Azure DevOps) and Infrastructure (Kubernetes).

Pellerex: Identity Foundation for Your Next Enterprise Software

How are you building your current software today? Build everything from scratch or use a foundation to save on development time, budget and resources? For an enterprise software MVP, which might take 8–12 months with a small team, you might indeed spend 6 months on your foundation. Things like Identity, Payment, Infrastructure, DevOps, etc. they all take time, while contributing not much to your actual product. These features are needed, but they are not your differentiators.

Pellerex does just that. It provides a foundation that save you a lot development time and effort at a fraction of the cost. It gives you source-included Identity, Payment, Infrastructure, and DevOps to build Web, Api and Mobile apps all-integrated and ready-to-go on day 1.

Check out Pellerex and talk to our team today to start building your next enterprise software fast.