How to serve a file in ASP.NET Core while it's still being written

Question

I have a log file that's being continuously written to by a background service. Users need to be able to download the file so far. When I return an MVC FileResult , I get an InvalidOperationException due to Content-Length mismatch, presumably because some content has been written to the file while it has been served. There is a file served, and it's mostly OK, but it usually has an incomplete last line.

The background service is doing essentially this:

var stream = new FileStream(evidenceFilePath, FileMode.Append, FileAccess.Write, FileShare.Read);
while (true) // obviously it isn't actually this, but it does happen a lot!
{
    var content = "log content\r\n";
    stream.Write(Encoding.UTF8.GetBytes(content);
}

Here are some of the variations on the controller action (all have the same result):

public IActionResult DownloadLog1()
{
    return PhysicalFile("C:\\path\\to\\the\\file.txt", "text/plain", enableRangeProcessing: false); // also tried using true
}

public IActionResult DownloadLog2()
{
    var stream = new FileStream("C:\\path\\to\\the\\file.txt", FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
    return File(stream, "text/plain", enableRangeProcessing: false); // also tried true
}

Here's the exception I get when I try either of the above:

System.InvalidOperationException: Response Content-Length mismatch: too many bytes written (216072192 of 216059904).
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ThrowTooManyBytesWritten(Int32 count)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.VerifyAndUpdateWrite(Int32 count)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.WriteAsync(ReadOnlyMemory`1 data, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.Extensions.StreamCopyOperation.CopyToAsync(Stream source, Stream destination, Nullable`1 count, Int32 bufferSize, CancellationToken cancel)
   at Microsoft.AspNetCore.Mvc.Infrastructure.FileResultExecutorBase.WriteFileAsync(HttpContext context, Stream fileStream, RangeItemHeaderValue range, Int64 rangeLength)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeResultAsync(IActionResult result)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResultFilterAsync[TFilter,TFilterAsync]()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResultExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeResultFilters()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

I don't mind the exception too much, but I'd prefer it if it didn't happen. I do need to fix the incomplete last line problem though. The most obvious solution to me is to keep track of the number of bytes that have definitely been written to the file and somehow only serve those first n bytes. I don't see any easy way to do that with FileResult and the various helper methods that construct it though. The file can get pretty large (up to around 500MB), so it doesn't seem practical to buffer in memory.

Answer 1

I ended up writing a custom ActionResult and IActionResultExecutor to match, which are heavily based on the MVC FileStreamResult and FileStreamResultExecutor :

public class PartialFileStreamResult : FileResult
{
    Stream stream;
    long bytes;

    /// <summary>
    /// Creates a new <see cref="PartialFileStreamResult"/> instance with
    /// the provided <paramref name="fileStream"/> and the
    /// provided <paramref name="contentType"/>, which will download the first <paramref name="bytes"/>.
    /// </summary>
    /// <param name="stream">The stream representing the file</param>
    /// <param name="contentType">The Content-Type header for the response</param>
    /// <param name="bytes">The number of bytes to send from the start of the file</param>
    public PartialFileStreamResult(Stream stream, string contentType, long bytes)
        : base(contentType)
    {
        this.stream = stream ?? throw new ArgumentNullException(nameof(stream));
        if (bytes == 0)
        {
            throw new ArgumentOutOfRangeException(nameof(bytes), "Invalid file length");
        }
        this.bytes = bytes;
    }

    /// <summary>
    /// Gets or sets the stream representing the file to download.
    /// </summary>
    public Stream Stream
    {
        get => stream;
        set => stream = value ?? throw new ArgumentNullException(nameof(stream));
    }

    /// <summary>
    /// Gets or sets the number of bytes to send from the start of the file.
    /// </summary>
    public long Bytes
    {
        get => bytes;
        set
        {
            if (value == 0)
            {
                throw new ArgumentOutOfRangeException(nameof(bytes), "Invalid file length");
            }
            bytes = value;
        }
    }

    /// <inheritdoc />
    public override Task ExecuteResultAsync(ActionContext context)
    {
        if (context == null)
        {
            throw new ArgumentNullException(nameof(context));
        }
        var executor = context.HttpContext.RequestServices.GetRequiredService<IActionResultExecutor<PartialFileStreamResult>>();
        return executor.ExecuteAsync(context, this);
    }
}

public class PartialFileStreamResultExecutor : FileResultExecutorBase, IActionResultExecutor<PartialFileStreamResult>
{
    public PartialFileStreamResultExecutor(ILoggerFactory loggerFactory)
        : base(CreateLogger<PartialFileStreamResultExecutor>(loggerFactory))
    {
    }

    public async Task ExecuteAsync(ActionContext context, PartialFileStreamResult result)
    {
        if (context == null)
        {
            throw new ArgumentNullException(nameof(context));
        }

        if (result == null)
        {
            throw new ArgumentNullException(nameof(result));
        }

        using (result.Stream)
        {
            long length = result.Bytes;
            var (range, rangeLength, serveBody) = SetHeadersAndLog(context, result, length, result.EnableRangeProcessing);
            if (!serveBody) return;

            try
            {
                var outputStream = context.HttpContext.Response.Body;
                if (range == null)
                {
                    await StreamCopyOperation.CopyToAsync(result.Stream, outputStream, length, bufferSize: BufferSize, cancel: context.HttpContext.RequestAborted);
                }
                else
                {
                    result.Stream.Seek(range.From.Value, SeekOrigin.Begin);
                    await StreamCopyOperation.CopyToAsync(result.Stream, outputStream, rangeLength, BufferSize, context.HttpContext.RequestAborted);
                }
            }
            catch (OperationCanceledException)
            {
                // Don't throw this exception, it's most likely caused by the client disconnecting.
                // However, if it was cancelled for any other reason we need to prevent empty responses.
                context.HttpContext.Abort();
            }
        }
    }
}

I could have done some more work to add additional constructor overloads to set some of the optional parameters (eg download file name, etc) but this is adequate for what I need.

You need to add the IActionResultExecutor in Startup.ConfigureServices:

services.AddTransient<IActionResultExecutor<PartialFileStreamResult>, PartialFileStreamResultExecutor>();

My controller action therefore turned into:

[HttpGet]
public IActionResult DownloadLog()
{
    var (path, bytes) = GetThePathAndTheNumberOfBytesIKnowHaveBeenFlushed();

    var stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite); // this ensures that the file can be read while it's still being written
    return new PartialFileStreamResult(stream, "text/plain", bytes);
}

Answer 2

Files are unmanaged resources.

So when you access an unmanaged resource, like a file, it is opened via a handle. In case of file it is open_file_handle (recollecting from memory).

So, best way i can suggest (very generic) to write a log entry:

Open file,

Write file,

Close file,

Dispose if applicable

In nutshell don't keep the stream open.

Secondly for controller you can look at MSDN example to serve file via controller.

Answer 3

Well, you're going to likely have issues with file locking, in general, so you'll need to plan and compensate for that. However, your immediate issue here is easier to solve. The problem boils down to returning a stream. That stream is being written to as the response is being returned, so the content-length that was calculated is wrong by the time the response body is created.

What you need to do is capture the log in a point in time, namely by reading it into a byte[] . Then, you can return that, instead of the stream, and the content-length will be calculated properly because the byte[] will not change after it's been read to.

using (var stream = new FileStream("C:\\path\\to\\the\\file.txt", FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
using (var ms = new MemoryStream())
{
    await stream.CopyToAsync(ms);
    return File(ms.ToArray(), "text/plain");
}