Check string content of response before retrying with Polly

Question

I'm working with a very flaky API. Sometimes I get 500 Server Error with Timeout, some other time I also get 500 Server Error because I gave it input that it can't handle SqlDateTime overflow. Must be between 1/1/1753 12:00:00 AM and 12/31/9999 11:59:59 PM..

Both of these cases give me HttpRequestException but I can look into the reply message from the server and determine the cause of the exception. If it is a timeout error, I should try again. If it is a bad input I should re-throw the exception, because no amount of retries will fix the problem of bad data.

What I'd like to do with Polly is to check on response message before attempting to retry. But all the samples I've seen so far only included type of exception.

I've come up with this so far:

        HttpResponseMessage response = null;
        String stringContent = null;
        Policy.Handle<FlakyApiException>()
             .WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
              async (exception, timeSpan, context) =>
            {
                response = await client.PostAsync(requestUri, new StringContent(serialisedParameters, Encoding.UTF8, "application/json"));
                stringContent = await response.Content.ReadAsStringAsync();

                if (response.StatusCode == HttpStatusCode.InternalServerError && stringContent.Contains("Timeout"))
                {
                    throw new FlakyApiException(stringContent);
                }
            });

Is there a better way to do this kind of checking?

Answer 1

In general, you can configure Polly policies to respond to the results of an execution (not just an exception), for example check an HttpResponseMessage.StatusCode with a predicate. Examples here in the Polly readme.

There is not however an in-built way to configure a single Polly policy to respond additionally to the content of the response message. This is because (as your example shows) obtaining that content requires a second async call, which may itself raise network errors.

This tl;dr engenders complications about how to express (in a simple syntax) a single policy which manages two different async steps with potentially different error handling for each step. Prior related discussion on Polly github: comment welcome.

As such, where a sequence requires two separate async calls, the Polly team currently recommends expressing this as two separate policies, similar to the example in the end of this answer.

---

The particular example in your question may not work because the onRetryAsync delegate (throwing FlakyApiException) is not itself guarded by the policy. A policy only guards the execution of delegates executed through .Execute/ExecuteAsync(...).

---

One approach could be to use two policies, a retry policy which retries all typical http exceptions and status codes including 500s; then inside that a Polly FallbackPolicy which traps the status code 500 representing SqlDateTime overflow, and excludes that from being retried by rethrowing as some distinguishing exception (CustomSqlDateOverflowException).

        IAsyncPolicy<HttpResponseMessage> rejectSqlError = Policy<HttpResponseMessage>
            .HandleResult(r => r.StatusCode == HttpStatusCode.InternalServerError)
            .FallbackAsync(async (delegateOutcome, context, token) =>
            {
                String stringContent = await delegateOutcome.Result.Content.ReadAsStringAsync(); // Could wrap this line in an additional policy as desired.
                if (delegateOutcome.Result.StatusCode == HttpStatusCode.InternalServerError && stringContent.Contains("SqlDateTime overflow"))
                {
                    throw new CustomSqlDateOverflowException(); // Replace 500 SqlDateTime overflow with something else.
                }
                else
                {
                    return delegateOutcome.Result; // render all other 500s as they were
                }
            }, async (delegateOutcome, context) => { /* log (if desired) that InternalServerError was checked for what kind */ });

        IAsyncPolicy<HttpResponseMessage> retryPolicy = Policy<HttpResponseMessage>
            .Handle<HttpRequestException>()
            .OrResult(r => r.StatusCode == HttpStatusCode.InternalServerError)
            .OrResult(r => /* condition for any other errors you want to handle */)
            .WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
                async (exception, timeSpan, context) =>
                {
                    /* log (if desired) retry being invoked */
                });

        HttpResponseMessage response = await retryPolicy.WrapAsync(rejectSqlError)
            .ExecuteAsync(() => client.PostAsync(requestUri, new StringContent(serialisedParameters, Encoding.UTF8, "application/json"), cancellationToken));