- To understand this article you should have basic knowledge of creating Web APIs in ASP.NET Core
- The code in this article has been tested on ASP.NET Core 2.2
- The source code for this article is available on GitHub
- Edit: Source code for ASP.NET Core 3.1 solution is now available on GitHub
In this article we will learn how to safely log API requests and responses in ASP.NET Core. But before that let’s look at why you would log calls to your API.
Why you should log API requests and responses
The main reason you should log calls to your API is to collect the usage statistics of your API. Collecting the statistics will allow you to understand which methods of your API are called by your consumers the most and how long each of your method takes to respond.
Other than these benefits, logging calls can also be helpful in debugging. Sometimes your customers might not hit your API endpoint due to the firewalls in their organisation blocking access. Other times they might send incorrect requests and wonder why they don’t get the correct response. If you don’t have an API Log it can be difficult to get to the root of these problems.
API gateway solutions like Kong and Apigee are available which provide API logging facilities and many more features. You should ideally look at utilising such a solution for your API. However, if using an API gateway is not feasible for your project you can implement API logging in your ASP.NET Core project itself. Let’s look at how you can do this.
Capturing API requests and responses
You can easily capture requests and responses by writing a middleware. The sample code for this project available on GitHub consists of the API logging middleware in ApiLoggingMiddleware.cs. This middleware is added to the pipeline in the Configure method of Startup.cs using the below line.
app.UseMiddleware<ApiLoggingMiddleware>();
Let’s have a look at our middleware code.
public async Task Invoke(HttpContext httpContext, ApiLogService apiLogService)
{
try
{
_apiLogService = apiLogService;
var request = httpContext.Request;
if (request.Path.StartsWithSegments(new PathString("/api")))
{
var stopWatch = Stopwatch.StartNew();
var requestTime = DateTime.UtcNow;
var requestBodyContent = await ReadRequestBody(request);
var originalBodyStream = httpContext.Response.Body;
using (var responseBody = new MemoryStream())
{
var response = httpContext.Response;
response.Body = responseBody;
await _next(httpContext);
stopWatch.Stop();
string responseBodyContent = null;
responseBodyContent = await ReadResponseBody(response);
await responseBody.CopyToAsync(originalBodyStream);
await SafeLog(requestTime,
stopWatch.ElapsedMilliseconds,
response.StatusCode,
request.Method,
request.Path,
request.QueryString.ToString(),
requestBodyContent,
responseBodyContent);
}
}
else
{
await _next(httpContext);
}
}
catch (Exception ex)
{
await _next(httpContext);
}
}
private async Task<string> ReadRequestBody(HttpRequest request)
{
request.EnableRewind();
var buffer = new byte[Convert.ToInt32(request.ContentLength)];
await request.Body.ReadAsync(buffer, 0, buffer.Length);
var bodyAsText = Encoding.UTF8.GetString(buffer);
request.Body.Seek(0, SeekOrigin.Begin);
return bodyAsText;
}
private async Task<string> ReadResponseBody(HttpResponse response)
{
response.Body.Seek(0, SeekOrigin.Begin);
var bodyAsText = await new StreamReader(response.Body).ReadToEndAsync();
response.Body.Seek(0, SeekOrigin.Begin);
return bodyAsText;
}
For every request to the application, the Invoke method of our middleware is called. This method has httpContext parameter which consists of the Request and Response properties. Request details like HTTP headers, request body, etc. are available in the Request property. All middleware which generate response write to the Response property.
We are only interested in logging the API requests and responses. So we use the if statement to only capture the request paths which start with /api. We will also log the time it takes to generate the response. So at the beginning we start a StopWatch and after the response is generated we stop it to get the elapsed time in milliseconds.
We read the body of the request using the ReadRequestBody method. In this method we read the body content from the stream and before returning from the method we reset the stream position back to the beginning of the stream so that the next middleware in the pipeline works normally. If you don’t understand this code you should learn more about Streams and Middleware.
Now let’s looks at how we can read the response body. Normally all middleware only write to the response stream and at the end of the pipeline the content of the stream is read and sent back to the client. As the response is only sent once this stream is designed to be only read once. If you try reading the response twice the second time it will return nothing. So if we read the content from response.Body in our middleware the client with always get an empty response!
To work around this problem with create a MemorySteam object and assign it to response.Body. So the further middleware in the pipeline will write to this memory steam instead of the original stream. We invoke the next middleware using await _next(httpContext). Once we return back to this middleware we read the response body from our memory stream using the ReadResponseBody method.
Next we copy this content into the original response stream so that it gets sent to the client.
After this we log the request and response using the SafeLog method described in the next section.
Safely logging requests and responses
Let’s look at the SafeLog method now.
private async Task SafeLog(DateTime requestTime,
long responseMillis,
int statusCode,
string method,
string path,
string queryString,
string requestBody,
string responseBody)
{
if (path.ToLower().StartsWith("/api/login"))
{
requestBody = "(Request logging disabled for /api/login)";
responseBody = "(Response logging disabled for /api/login)";
}
if (requestBody.Length > 100)
{
requestBody = $"(Truncated to 100 chars) {requestBody.Substring(0, 100)}";
}
if (responseBody.Length > 100)
{
responseBody = $"(Truncated to 100 chars) {responseBody.Substring(0, 100)}";
}
if (queryString.Length > 100)
{
queryString = $"(Truncated to 100 chars) {queryString.Substring(0, 100)}";
}
await _apiLogService.Log(new ApiLogItem
{
RequestTime = requestTime,
ResponseMillis = responseMillis,
StatusCode = statusCode,
Method = method,
Path = path,
QueryString = queryString,
RequestBody = requestBody,
ResponseBody = responseBody
});
}
The requests and responses can consist of sensitive data like passwords, secret tokens, etc. It is not secure to log such information. For demonstration purpose, our sample code has a dummy Login method which accepts a username and password and returns a secret token. We would not like to log the request body and response body for this method. In the SafeLog method we check if the path of request starts with /api/login and in such cases we replace the requestBody and responseBody values with text which says that the logging is disabled for this method.
Another thing to be taken care when logging is the size of the requests and responses. If we log requests without any limit on the size, a malicious user can send large number of requests with huge amount of data. This will fill our database and we can run out of disk space. So before logging the requests and responses we check that these are up to 100 characters in length. If these are greater than 100 characters then we only log the first 100 characters. You should replace 100 with a suitable value for your API.
To test the API logging I have provided a CustomerController, LoginController and LogsController. You can make calls to the CustomerController and LoginController actions and then open http://localhost:5000/Logs in your browser to view the logged data.
Summary
In this article we first discussed the benefits of logging requests and responses in APIs. If possible you should use API gateways solutions like Kong and Apigee which provide logging functionality as well as other features.
Later we looked at how middleware can be used to capture requests and responses in ASP.NET Core. Then we learnt how to safely log this information by excluding sensitive data from the logs and checking the size of the request and response before logging it.
