MVC ASP.NET正在使用大量内存

Question

If I just browse some pages on the app, it sits at around 500MB. Many of these pages access the database but at this point in time, I only have roughly a couple of rows each for 10 tables, mostly storing strings and some small icons that are less than 50KB.

The real problem occurs when when I download a file. The file is roughly 140MB and is stored as a varbinary(MAX) in the database. The memory usage suddenly rises to 1.3GB for a split second and then falls back to 1GB. The code for that action is here:

public ActionResult DownloadIpa(int buildId)
{
    var build = _unitOfWork.Repository<Build>().GetById(buildId);
    var buildFiles = _unitOfWork.Repository<BuildFiles>().GetById(buildId);
    if (buildFiles == null)
    {
        throw new HttpException(404, "Item not found");
    }

    var app = _unitOfWork.Repository<App>().GetById(build.AppId);
    var fileName = app.Name + ".ipa";

    app.Downloads++;
    _unitOfWork.Repository<App>().Update(app);
    _unitOfWork.Save();

    return DownloadFile(buildFiles.Ipa, fileName);
}

private ActionResult DownloadFile(byte[] file, string fileName, string type = "application/octet-stream")
{
    if (file == null)
    {
        throw new HttpException(500, "Empty file");
    }

    if (fileName.Equals(""))
    {
        throw new HttpException(500, "No name");
    }

    return File(file, type, fileName);            
}

On my local computer, If I don't do anything, the memory usage stays at 1GB. If I then go back and navigate to some pages, it falls back down to 500MB.

On the deployment server, it stays at 1.6GB after the first download no matter what I do. I can force the memory usage to increase by continually downloading files until it reaches 3GB, where it drops back down to 1.6GB.

In every controller, I have overriden the Dispose() method as so:

protected override void Dispose(bool disposing)
{
    _unitOfWork.Dispose();
    base.Dispose(disposing);
}

This refers to:

public void Dispose()
{
    Dispose(true);
    GC.SuppressFinalize(this);
}

public void Dispose(bool disposing)
{
    if (!_disposed)
    {
        if (disposing)
        {
            _context.Dispose();
        }
    }

    _disposed = true;
}

So my unit of work should be disposed every time the controller is disposed. I am using Unity and I register the unit of work with a Heirarchical Lifetime Manager.

Here are a few of screenshots from the Profiler:

I believe this could be the problem or I am going down the wrong track. Why would Find() use 300MB?

EDIT:

Repository:

public class Repository<TEntity> : IRepository<TEntity> where TEntity : class
{
    internal IDbContext Context;
    internal IDbSet<TEntity> DbSet;

    public Repository(IDbContext context)
    {
        Context = context;
        DbSet = Context.Set<TEntity>();
    }

    public virtual IEnumerable<TEntity> GetAll()
    {            
        return DbSet.ToList();
    }

    public virtual TEntity GetById(object id)
    {
        return DbSet.Find(id);
    }

    public TEntity GetSingle(Expression<Func<TEntity, bool>> predicate)
    {
        return DbSet.Where(predicate).SingleOrDefault();
    }

    public virtual RepositoryQuery<TEntity> Query()
    {
        return new RepositoryQuery<TEntity>(this);
    }

    internal IEnumerable<TEntity> Get(
        Expression<Func<TEntity, bool>> filter = null,
        Func<IQueryable<TEntity>, IOrderedQueryable<TEntity>> orderBy = null,
        List<Expression<Func<TEntity, object>>> includeProperties = null)
    {
        IQueryable<TEntity> query = DbSet;

        if (includeProperties != null)
        {
            includeProperties.ForEach(i => query.Include(i));
        }

        if (filter != null)
        {
            query = query.Where(filter);
        }

        if (orderBy != null)
        {
            query = orderBy(query);
        }

        return query.ToList();
    }

    public virtual void Insert(TEntity entity)
    {
        DbSet.Add(entity);
    }

    public virtual void Update(TEntity entity)
    {
        DbSet.Attach(entity);
        Context.Entry(entity).State = EntityState.Modified;
    }

    public virtual void Delete(object id)
    {
        var entity = DbSet.Find(id);

        Delete(entity);
    }

    public virtual void Delete(TEntity entity)
    {
        if (Context.Entry(entity).State == EntityState.Detached)
        {
            DbSet.Attach(entity);
        }

        DbSet.Remove(entity);
    }
}

EDIT 2:

I ran dotMemory for a variety of scenarios and this is what I got.

The red circles indicate that sometimes there are multiple rises and drops happening on one page visit. The blue circle indicates download of a 40MB file. The green circle indicates download of 140MB file. Furthermore, a lot of the time, the memory usage keeps on increasing for a few more seconds even after the page has instantly loaded.

Answer 1

Because the file is large, it is allocated on the Large Object Heap, which is collected with a gen2 collection (which you see in your profile, the purple blocks is the large object heap, and you see it collected after 10 seconds).

On your production server, you most likely have much more memory than on your local machine. Because there is less memory pressure, the collections won't occur as frequently, which explains why it would add up to a higher number - there are several files on the LOH before it gets collected.

I wouldn't be surprised at all if, across different buffers in MVC and EF, some data gets copied around in unsafe blocks too, which explains the unmanaged memory growth (the thin spike for EF, the wide plateau for MVC)

Finally, a 500MB baseline is for a large project not completely surprising (madness! but true!)

So an answer to your question why it uses so much memory that is quite probable is "because it can", or in other words, because there is no memory pressure to perform a gen2 collection, and the downloaded files sit unused in your large object heap until collection evicts them because memory is abundant on your production server.

This is probably not even a real problem: if there were more memory pressure, there would be more collection, and you'd see lower memory usage.

As for what to do about it, I'm afraid you're out of luck with the Entity Framework. As far as I know it has no streaming API. WebAPI does allow streaming the response by the way, but that won't help you much if you have the whole large object sitting in memory anyway (though it might possibly help some with the unmanaged memory in the (by me) unexplored parts of MVC.

Answer 2

Add a GC.Collect() to the Dispose method for testing purposes. If the leak stays it is a real leak. If it vanishes it was just delayed GC.

You did that and said:

@usr Memory usage now hardly reaches 600MB. So really just delayed?

Clearly, there is no memory leak if GC.Collect removes the memory that you were worried about. If you want to make really sure, run your test 10 times. Memory usage should be stable.

Processing such big files in single chunks can lead to multiplied memory usage as the file travels through the different components and frameworks. It can be a good idea to switch to a streaming approach.

Answer 3

Apparently, that consists of System.Web and all it's children taking up around 200MB. This is quoted as the absolute minimum for your application pool.

Our web application using EF 6, with a model consisting of 220+ entities in .Net 4.0 starts up at around 480MB idle. We perform some AutoMapper operations at startup. Memory consumption peaks and then returns to around 500MB in daily use. We've just accepted this as the norm.

Now, for your file download spikes. The issue under web forms when using an ashx handler or the like was explored in this question: ASP.net memory usage during download

I don't know how that relates to the FileActionResult in MVC, but you can see that the buffer size needed to be controlled manually to minimise the memory spike. Try to apply the principles behind the answer from that question by:

Response.BufferOutput = false;
var stream = new MemoryStream(file);
stream.Position = 0;
return new FileStreamResult(stream, type); // Or just pass the "file" parameter as a stream

After applying this change, what does the memory behaviour look like?

See 'Debugging memory problems (MSDN)' for more details.

Answer 4

You may need to read the data in chunks and write to the output stream. Take a look at SqlDataReader.GetBytes http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqldatareader.getbytes(v=vs.110).aspx

Answer 5

This could be one of a few things:

As your file is rather large and is stored in your database and you are getting it via Entity Framework , you are caching this data in a few places. Each EF request caches that data until your context is disposed. When you return the file from the action, the data is then loaded again and then streamed to the client. All of this happens in ASP .NET as explained already.

A solution to this issue to not to stream large files directly from the database with EF and ASP .NET . A better solution is to use a background process to cache large files locally to the website and then have the client download them with a direct URL . This allows IIS to manage the streaming, saves your website a request and saves a lot of memory.

OR (less likely)

Seeing that you are using Visual Studio 2013 , this sounds awfully like a Page Inspector issue.

What happens is when you run your website with IIS Express from Visual Studio , Page Inspector caches all of the response data - including that of your file - causing a lot of memory to be used. Try adding:

<appSettings>
    <add key="PageInspector:ServerCodeMappingSupport" value="Disabled" />
</appSettings>

to your web.config to disable Page Inspector to see if that helps.

TL;DR

Cache the large file locally and let the client download the file directly. Let IIS handle the hard work for you.

Answer 6

I suggest trying Ionic.Zip library. I use it in one of our sites with a requirement to download multiple files into one unit.

I recently tested it with a group of files while one of the files is as large as 600MB:

• Total size of zipped/compressed folder: 260MB
• Total Size of unzipped folder: 630MB
• Memory usage spiked from 350MB to 650MB during download
• Total time: 1m 10s to download, no VPN