View POST request body in Application Insights

Question

Is it possible to view POST request body in Application Insights?

I can see request details, but not the payload being posted in application insights. Do I have to t

Answer 1

I can able to log the request message body in Application Insights using @yonisha method but I can't able to log the response message body. I am interested in logging the response message body. I am already logging the Post, Put, Delete Request message body using @yonisha method.

When I tried to access the response body in the TelemetryInitializer I keep getting an exception with an error message saying that "stream was not readable. When I researched more I found that AzureInitializer is running as part of HttpModule(ApplicationInsightsWebTracking) so by the time it gets control response object is disposed.

I got an idea from @Oskar answer. Why not have a delegate handler and record the response since the response object is not disposed at the stage of message handler. The message handler is part of the Web API life cycle i.e. similar to the HTTP module but confined to web API. When I developed and tested this idea, fortunately, It worked I recorded the response in the request message using message handler and retrieved it at the AzureInitializer (HTTP module whose execution happens later than the message handler). Here is the sample code.

public class AzureRequestResponseInitializer : ITelemetryInitializer
{
    public void Initialize(ITelemetry telemetry)
    {
        var requestTelemetry = telemetry as RequestTelemetry;
        if (requestTelemetry != null && HttpContext.Current != null && HttpContext.Current.Request != null)
        {
            if ((HttpContext.Current.Request.HttpMethod == HttpMethod.Post.ToString() 
                 || HttpContext.Current.Request.HttpMethod == HttpMethod.Put.ToString()) &&
                HttpContext.Current.Request.Url.AbsoluteUri.Contains("api"))
                using (var reader = new StreamReader(HttpContext.Current.Request.InputStream))
                {
                    HttpContext.Current.Request.InputStream.Position = 0;
                    string requestBody = reader.ReadToEnd();
                    if (requestTelemetry.Properties.Keys.Contains("requestbody"))
                    {
                        requestTelemetry.Properties["requestbody"] = requestBody;
                    }
                    else
                    {
                        requestTelemetry.Properties.Add("requestbody", requestBody);
                    }
                }
            else if (HttpContext.Current.Request.HttpMethod == HttpMethod.Get.ToString() 
                     && HttpContext.Current.Response.ContentType.Contains("application/json"))
            {
                var netHttpRequestMessage = HttpContext.Current.Items["MS_HttpRequestMessage"] as HttpRequestMessage;
                if (netHttpRequestMessage.Properties.Keys.Contains("responsejson"))
                {
                    var responseJson = netHttpRequestMessage.Properties["responsejson"].ToString();
                    if (requestTelemetry.Properties.Keys.Contains("responsebody"))
                    {
                        requestTelemetry.Properties["responsebody"] = responseJson;
                    }
                    else
                    {
                        requestTelemetry.Properties.Add("responsebody", responseJson);
                    }
                }
            }
        }

    }
}

config.MessageHandlers.Add(new LoggingHandler());

public class LoggingHandler : DelegatingHandler
{
    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        return base.SendAsync(request, cancellationToken).ContinueWith(task =>
        {
            var response = task.Result;
            StoreResponse(response);
            return response;
        });
    }


    private void StoreResponse(HttpResponseMessage response)
    {
        var request = response.RequestMessage;

        (response.Content ?? new StringContent("")).ReadAsStringAsync().ContinueWith(x =>
        {
            var ctx = request.Properties["MS_HttpContext"] as HttpContextWrapper;

            if (request.Properties.ContainsKey("responseJson"))
            {
                request.Properties["responsejson"] = x.Result;
            }
            else
            {
                request.Properties.Add("responsejson", x.Result);
            }
        });
    }
}

Answer 2

In Asp.Net core it looks like we dont have to use ITelemetryInitializer. We can use the middleware to log the requests to application insights. Thanks to @IanKemp https://github.com/microsoft/ApplicationInsights-aspnetcore/issues/686

 public async Task Invoke(HttpContext httpContext)
    {
        var requestTelemetry = httpContext.Features.Get<RequestTelemetry>();

        //Handle Request 
        var request = httpContext.Request;
        if (request?.Body?.CanRead == true)
        {
            request.EnableBuffering();

            var bodySize = (int)(request.ContentLength ?? request.Body.Length);
            if (bodySize > 0)
            {
                request.Body.Position = 0;

                byte[] body;

                using (var ms = new MemoryStream(bodySize))
                {
                    await request.Body.CopyToAsync(ms);

                    body = ms.ToArray();
                }

                request.Body.Position = 0;

                if (requestTelemetry != null)
                {
                    var requestBodyString = Encoding.UTF8.GetString(body);

                    requestTelemetry.Properties.Add("RequestBody", requestBodyString);
                }
            }
        }

        await _next(httpContext); // calling next middleware
    }

Answer 3

Few days back, I got a similar requirement to log the request Body in Application insights with filtering out sensitive input user data from the payload. So sharing my solution. The below solution is developed for ASP.NET Core 2.0 Web API.

ActionFilterAttribute

I've used ActionFilterAttribute from (Microsoft.AspNetCore.Mvc.Filters namespace) which provides the Model via ActionArgument so that by reflection, those properties can be extracted which are marked as sensitive.

public class LogActionFilterAttribute : ActionFilterAttribute
{
    private readonly IHttpContextAccessor httpContextAccessor;

    public LogActionFilterAttribute(IHttpContextAccessor httpContextAccessor)
    {
        this.httpContextAccessor = httpContextAccessor;
    }

    public override async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
    {
        if (context.HttpContext.Request.Method == HttpMethods.Post || context.HttpContext.Request.Method == HttpMethods.Put)
        {
            // Check parameter those are marked for not to log.
            var methodInfo = ((Microsoft.AspNetCore.Mvc.Controllers.ControllerActionDescriptor)context.ActionDescriptor).MethodInfo;
            var noLogParameters = methodInfo.GetParameters().Where(p => p.GetCustomAttributes(true).Any(t => t.GetType() == typeof(NoLogAttribute))).Select(p => p.Name);

            StringBuilder logBuilder = new StringBuilder();

            foreach (var argument in context.ActionArguments.Where(a => !noLogParameters.Contains(a.Key)))
            {
                var serializedModel = JsonConvert.SerializeObject(argument.Value, new JsonSerializerSettings() { ContractResolver = new NoPIILogContractResolver() });
                logBuilder.AppendLine($"key: {argument.Key}; value : {serializedModel}");
            }

            var telemetry = this.httpContextAccessor.HttpContext.Items["Telemetry"] as Microsoft.ApplicationInsights.DataContracts.RequestTelemetry;
            if (telemetry != null)
            {
                telemetry.Context.GlobalProperties.Add("jsonBody", logBuilder.ToString());
            }

        }

        await next();
    }
}

The 'LogActionFilterAttribute' is injected in MVC pipeline as Filter.

 services.AddMvc(options =>
 {
       options.Filters.Add<LogActionFilterAttribute>();
 });

NoLogAttribute

In above code, NoLogAttribute attribute is used which should be applied on Model/Model's Properties or method parameter to indicate that value should not be logged.

public class NoLogAttribute : Attribute
{
}

NoPIILogContractResolver

Also, NoPIILogContractResolver is used in JsonSerializerSettings during serialization process

internal class NoPIILogContractResolver : DefaultContractResolver
{
    protected override IList<JsonProperty> CreateProperties(Type type, MemberSerialization memberSerialization)
    {
        var properties = new List<JsonProperty>();

        if (!type.GetCustomAttributes(true).Any(t => t.GetType() == typeof(NoLogAttribute)))
        {
            IList<JsonProperty> retval = base.CreateProperties(type, memberSerialization);
            var excludedProperties = type.GetProperties().Where(p => p.GetCustomAttributes(true).Any(t => t.GetType() == typeof(NoLogAttribute))).Select(s => s.Name);
            foreach (var property in retval)
            {
                if (excludedProperties.Contains(property.PropertyName))
                {
                    property.PropertyType = typeof(string);
                    property.ValueProvider = new PIIValueProvider("PII Data");
                }

                properties.Add(property);
            }
        }

        return properties;
    }
}

internal class PIIValueProvider : IValueProvider
{
    private object defaultValue;

    public PIIValueProvider(string defaultValue)
    {
        this.defaultValue = defaultValue;
    }

    public object GetValue(object target)
    {
        return this.defaultValue;
    }

    public void SetValue(object target, object value)
    {

    }
}

PIITelemetryInitializer

To inject the RequestTelemetry object, I've to use ITelemetryInitializer so that RequestTelemetry can be retrieved in LogActionFilterAttribute class.

public class PIITelemetryInitializer : ITelemetryInitializer
{
    IHttpContextAccessor httpContextAccessor;

    public PIITelemetryInitializer(IHttpContextAccessor httpContextAccessor)
    {
        this.httpContextAccessor = httpContextAccessor;
    }

    public void Initialize(ITelemetry telemetry)
    {
        if (this.httpContextAccessor.HttpContext != null)
        {
            if (telemetry is Microsoft.ApplicationInsights.DataContracts.RequestTelemetry)
            {
                this.httpContextAccessor.HttpContext.Items.TryAdd("Telemetry", telemetry);
            }
        }
    }
}

The PIITelemetryInitializer is registered as

services.AddSingleton<ITelemetryInitializer, PIITelemetryInitializer>();

Testing feature

Following code demonstrates the usage of above code

Created a controller

[Route("api/[controller]")]
public class ValuesController : Controller
{
    private readonly ILogger _logger;

    public ValuesController(ILoggerFactory loggerFactory)
    {
        _logger = loggerFactory.CreateLogger<ValuesController>();
    }

    // POST api/values
    [HttpPost]
    public void Post([FromBody, NoLog]string value)
    {

    }

    [HttpPost]
    [Route("user")]
    public void AddUser(string id, [FromBody]User user)
    {

    }
}

Where User Model is defined as

public class User
{
    [NoLog]
    public string Id { get; set; }

    public string Name { get; set; }

    public DateTime AnneviseryDate { get; set; }

    [NoLog]
    public int LinkId { get; set; }

    public List<Address> Addresses { get; set; }
}

public class Address
{
    public string AddressLine { get; set; }

    [NoLog]
    public string City { get; set; }

    [NoLog]
    public string Country { get; set; }
}

So when API is invoked by Swagger tool

The jsonBody is logged in Request without sensitive data. All sensitive data is replaced by 'PII Data' string literal.

Answer 4

I implemented a middleware for this,

Invoke method does,

 if (context.Request.Method == "POST" || context.Request.Method == "PUT")
        {
            var bodyStr = GetRequestBody(context);
            var telemetryClient = new TelemetryClient();
            var traceTelemetry = new TraceTelemetry
            {
                Message = bodyStr,
                SeverityLevel = SeverityLevel.Verbose
            };
            //Send a trace message for display in Diagnostic Search. 
            telemetryClient.TrackTrace(traceTelemetry);
        }

Where, GetRequestBody is like,

private static string GetRequestBody(HttpContext context)
    {
        var bodyStr = "";
        var req = context.Request;

        //Allows using several time the stream in ASP.Net Core.
        req.EnableRewind();

        //Important: keep stream opened to read when handling the request.
        using (var reader = new StreamReader(req.Body, Encoding.UTF8, true, 1024, true))
        {
            bodyStr = reader.ReadToEnd();
        }

        // Rewind, so the core is not lost when it looks the body for the request.
        req.Body.Position = 0;
        return bodyStr;
    }

Answer 5

You can simply implement your own Telemetry Initializer:

For example, below an implementation that extracts the payload and adds it as a custom dimension of the request telemetry:

public class RequestBodyInitializer : ITelemetryInitializer
{
    public void Initialize(ITelemetry telemetry)
    {
        var requestTelemetry = telemetry as RequestTelemetry;
        if (requestTelemetry != null && (requestTelemetry.HttpMethod == HttpMethod.Post.ToString() || requestTelemetry.HttpMethod == HttpMethod.Put.ToString()))
        {
            using (var reader = new StreamReader(HttpContext.Current.Request.InputStream))
            {
                string requestBody = reader.ReadToEnd();
                requestTelemetry.Properties.Add("body", requestBody);
            }
        }
    }
}

Then add it to the configuration either by configuration file or via code:

TelemetryConfiguration.Active.TelemetryInitializers.Add(new RequestBodyInitializer());

Then query it in Analytics:

requests | limit 1 | project customDimensions.body

Answer 6

The solution provided by @yonisha is in my opinion the cleanest one available. However you still need to get your HttpContext in there and for that you need some more code. I have also inserted some comments which are based or taken from code examples above. It is important to reset the position of your request else you will lose its data.

This is my solution that I have tested and gives me the jsonbody:

public class RequestBodyInitializer : ITelemetryInitializer
{
    readonly IHttpContextAccessor httpContextAccessor;

    public RequestBodyInitializer(IHttpContextAccessor httpContextAccessor)
    {
        this.httpContextAccessor = httpContextAccessor;
    }

    public void Initialize(ITelemetry telemetry)
    {
        if (telemetry is RequestTelemetry requestTelemetry)
        {
            if ((httpContextAccessor.HttpContext.Request.Method == HttpMethods.Post ||
                 httpContextAccessor.HttpContext.Request.Method == HttpMethods.Put) &&
                httpContextAccessor.HttpContext.Request.Body.CanRead)
            {
                const string jsonBody = "JsonBody";

                if (requestTelemetry.Properties.ContainsKey(jsonBody))
                {
                    return;
                }

                //Allows re-usage of the stream
                httpContextAccessor.HttpContext.Request.EnableRewind();

                var stream = new StreamReader(httpContextAccessor.HttpContext.Request.Body);
                var body = stream.ReadToEnd();

                //Reset the stream so data is not lost
                httpContextAccessor.HttpContext.Request.Body.Position = 0;
                requestTelemetry.Properties.Add(jsonBody, body);
            }
        }
    }

Then also be sure to add this to your Startup -> ConfigureServices

services.AddSingleton<ITelemetryInitializer, RequestBodyInitializer>();

EDIT:

If you also want to get the response body I found it useful to create a piece of middleware (.NET Core, not sure about Framework). At first I took above approach where you log a response and a request but most of the time you want these together:

    public async Task Invoke(HttpContext context)
    {
        var reqBody = await this.GetRequestBodyForTelemetry(context.Request);

        var respBody = await this.GetResponseBodyForTelemetry(context);
        this.SendDataToTelemetryLog(reqBody, respBody, context);
    }

This awaits both a request and a response. GetRequestBodyForTelemetry is almost identical to the code from the telemetry initializer, except using Task. For the response body I have used the code below, I also excluded a 204 since that leads to a nullref:

public async Task<string> GetResponseBodyForTelemetry(HttpContext context)
{
    var originalBody = context.Response.Body;

    try
    {
        using (var memStream = new MemoryStream())
        {
            context.Response.Body = memStream;

            //await the responsebody
            await next(context);
            if (context.Response.StatusCode == 204)
            {
                return null;
            }

            memStream.Position = 0;
            var responseBody = new StreamReader(memStream).ReadToEnd();

            //make sure to reset the position so the actual body is still available for the client
            memStream.Position = 0;
            await memStream.CopyToAsync(originalBody);

            return responseBody;
        }
    }
    finally
    {
        context.Response.Body = originalBody;
    }
}