How to determine reason for poison queue message

Question

I have an azure function with a QueueTrigger that processes an image and outputs a blob as well as a table record.

Sometimes when processing several large images I have run into OutOfMemory exceptions, which can cause the queue item to be placed into a poison queue.

Sometimes there is a race condition and the table record being inserted errors out because there is already a record with that partition key and row key.

I could work around these issues myself within the functions, but my preferred way to handle this would be for the poison message to include a reason or the exception that caused the item to be placed in the poison queue. That way I can have another trigger listen on the poison queue and act accordingly after assessing what went wrong.

How should this be handled? I'm aware of the ErrorTrigger, which is nice to get the exception but I don't see how I can relate it back to the specific queue item that caused it.

Answer 1

There is no way to change a poison storage queue message queue after your Queue trigger fails.

Azure functions uses Web Jobs SDK. Using Web Jobs SDK directly it is possible to overwrite QueueProcessor.CopyMessageToPoisonQueueAsync. Some useful links:

https://github.com/Azure/azure-webjobs-sdk/issues/1204

https://github.com/Azure/azure-webjobs-sdk-samples/blob/master/BasicSamples/MiscOperations/CustomQueueProcessorFactory.cs

https://github.com/Azure/azure-webjobs-sdk/blob/ffae7c86ea87fd73748186bac2c38c5401b80f68/test/Microsoft.Azure.WebJobs.Host.EndToEndTests/AzureStorageEndToEndTests.cs

Answer 2

There is a hacky way to track the exception and the message that caused the message using a FunctionInvocation filter.

public static class QueueFunction
{
    [FunctionName("QueueFunction")]
    [TrackFailedMessages]
    public static void Run([QueueTrigger("myqueue")]string message)
    {
        throw new Exception("OMG");
    }
}

public class TrackFailedMessages : FunctionInvocationFilterAttribute
{
    private static readonly ConcurrentDictionary<Guid, object> QueueParamters = new ConcurrentDictionary<Guid, object>();
    public override Task OnExecutingAsync(FunctionExecutingContext executingContext, CancellationToken cancellationToken)
    {
        if (executingContext.Arguments.TryGetValue("message", out var data))
        {
            QueueParamters.AddOrUpdate(executingContext.FunctionInstanceId, data, (id, obj) => data);
        }
        return Task.CompletedTask;
    }

    public override Task OnExecutedAsync(FunctionExecutedContext executedContext, CancellationToken cancellationToken)
    {
        if (executedContext.FunctionResult.Exception != null &&
            QueueParamters.TryGetValue(executedContext.FunctionInstanceId, out var message))
        {
            executedContext.Logger.LogCritical(
                "The message {message} caused the exception {exception}",
                message,
                executedContext.FunctionResult.Exception.ToString());
        }
        QueueParamters.TryRemove(executedContext.FunctionInstanceId, out var _);
        return Task.CompletedTask;
    }
}

OnExecutinAsync will be called, when ever a function is invoked. Using the executingContext you have access to the parameters of the function. You can search for your QueueTrigger paramter and store the value in a dictionary, where the FunctionInstanceId is the key. FunctionInstanceId is a unique Id for every invocation.

In OnExecuted you can check if the function raised and exception and get the message from the dictionary, then store it somewhere else (database, log, etc.)

I am not sure if this catches the Table Storage Exception...