java.lang.OutOfMemoryError上传大文件

Question

Before Nick Wilson answer:

I'm developing a Java EE 6 web application under Glassfish 3. The main characteristic is to be able to upload big files (1Gb ore more bigger). When I try to upload a file (400MB of size) I receive the java.lang.OutOfMemoryError exception:

Caused by: java.lang.OutOfMemoryError
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:272)
at com.google.common.io.ByteStreams.read(ByteStreams.java:899)
at com.google.common.io.ByteStreams.readFully(ByteStreams.java:733)
at com.google.common.io.ByteStreams.readFully(ByteStreams.java:714)
at org.richfaces.request.BaseUploadedFile.getData(BaseUploadedFile.java:68)
at EJB.FileUploadBean.listener(FileUploadBean.java:147)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.el.parser.AstValue.invoke(AstValue.java:254)
at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:302)
at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105)
at org.richfaces.event.MethodExpressionEventListener.processEvent(MethodExpressionEventListener.java:125)

I already incremented the memory in Glassfish JVM Options:

-XX:MaxPermSize=256m
-XX:PermSize=128m
-XX:NewRatio=2
-XX:+UseParallelGC
-Xmx2048m

I tried the following code to upload file, but both give me the same error:

InputStream filecontent = null;
    int read = 0;
    final byte[] bytes = new byte[1024];

    OutputStream fop = null;
    File fout;
    if (file.getLength() > 0) {
        try {
            fout = new File(filename);
            fop = new FileOutputStream(fout);
            filecontent = item.getInputStream();
            if (!fout.exists()) {
                fout.createNewFile();
            }
            while ((read = filecontent.read(bytes)) != -1) {
                 fop.write(bytes, 0, read);
            }

and

 FileOutputStream fop = null;
    File fout;
    if (file.getLength() > 0) {
        try {
            fout = new File(filename);
            fop = new FileOutputStream(fout);

            if (!fout.exists()) {
                fout.createNewFile();
            }
            fop.write(item.getData());
            fop.flush();
            fop.close();

How can I solve the problem?

THanks

After Nick Wilson answer

I changed my code to:

public void listener(FileUploadEvent event) throws Exception {
    ufile = new UploadFile();
    InputStream filecontent = null;
    UploadedFile item = event.getUploadedFile();
    if (item instanceof UploadedFile25) {
        filecontent = ((UploadedFile25) item).getInputStream();
    } else {

        filecontent = item.getInputStream();
    }
    UploadedText file = new UploadedText();
    file.setLength(item.getData().length);
    file.setName(item.getName());
    Date date = new Date();
    String epoch = fDir + String.valueOf(date.getTime());
    File fPath = new File(epoch);
    fPath.mkdir();

    //file.setData(item.getData());
    files.add(file);
    String filename = epoch + '/' + item.getName();

    int read = 0;
    final byte[] bytes = new byte[1024];

    OutputStream fop = null;
    File fout;
    if (file.getLength() > 0) {
        try {
            fout = new File(filename);
            fop = new FileOutputStream(fout);

            if (!fout.exists()) {
                fout.createNewFile();
            }
            while ((read = filecontent.read(bytes)) != -1) {
                fop.write(bytes, 0, read);
            }

But I receive the same exception.

to be able to upload big files (1Gb ore more bigger). When I try to upload a file (400MB of size) I receive the java.lang.OutOfMemoryError exception:

Caused by: java.lang.OutOfMemoryError
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:272)
at com.google.common.io.ByteStreams.read(ByteStreams.java:899)
at com.google.common.io.ByteStreams.readFully(ByteStreams.java:733)
at com.google.common.io.ByteStreams.readFully(ByteStreams.java:714)
at org.richfaces.request.BaseUploadedFile.getData(BaseUploadedFile.java:68)
at EJB.FileUploadBean.listener(FileUploadBean.java:147)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.el.parser.AstValue.invoke(AstValue.java:254)
at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:302)
at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105)
at org.richfaces.event.MethodExpressionEventListener.processEvent(MethodExpressionEventListener.java:125)

I already incremented the memory in Glassfish JVM Options:

-XX:MaxPermSize=256m
-XX:PermSize=128m
-XX:NewRatio=2
-XX:+UseParallelGC
-Xmx2048m

I tried the following code to upload file, but both give me the same error:

InputStream filecontent = null;
    int read = 0;
    final byte[] bytes = new byte[1024];

    OutputStream fop = null;
    File fout;
    if (file.getLength() > 0) {
        try {
            fout = new File(filename);
            fop = new FileOutputStream(fout);
            filecontent = item.getInputStream();
            if (!fout.exists()) {
                fout.createNewFile();
            }
            while ((read = filecontent.read(bytes)) != -1) {
                 fop.write(bytes, 0, read);
            }

and

 FileOutputStream fop = null;
    File fout;
    if (file.getLength() > 0) {
        try {
            fout = new File(filename);
            fop = new FileOutputStream(fout);

            if (!fout.exists()) {
                fout.createNewFile();
            }
            fop.write(item.getData());
            fop.flush();
            fop.close();

How can I solve the problem?

THanks

After Nick Wilson answer

I changed my code to:

public void listener(FileUploadEvent event) throws Exception {
    ufile = new UploadFile();
    InputStream filecontent = null;
    UploadedFile item = event.getUploadedFile();
    if (item instanceof UploadedFile25) {
        filecontent = ((UploadedFile25) item).getInputStream();
    } else {

        filecontent = item.getInputStream();
    }
    UploadedText file = new UploadedText();
    file.setLength(item.getData().length);
    file.setName(item.getName());
    Date date = new Date();
    String epoch = fDir + String.valueOf(date.getTime());
    File fPath = new File(epoch);
    fPath.mkdir();

    //file.setData(item.getData());
    files.add(file);
    String filename = epoch + '/' + item.getName();

    int read = 0;
    final byte[] bytes = new byte[1024];

    OutputStream fop = null;
    File fout;
    if (file.getLength() > 0) {
        try {
            fout = new File(filename);
            fop = new FileOutputStream(fout);

            if (!fout.exists()) {
                fout.createNewFile();
            }
            while ((read = filecontent.read(bytes)) != -1) {
                fop.write(bytes, 0, read);
            }

But I receive the same exception.

Caused by: java.lang.OutOfMemoryError
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:272)
at com.google.common.io.ByteStreams.read(ByteStreams.java:899)
at com.google.common.io.ByteStreams.readFully(ByteStreams.java:733)
at com.google.common.io.ByteStreams.readFully(ByteStreams.java:714)
at org.richfaces.request.BaseUploadedFile.getData(BaseUploadedFile.java:68)
at EJB.FileUploadBean.listener(FileUploadBean.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.el.parser.AstValue.invoke(AstValue.java:254)
at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:302)
at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105)
at org.richfaces.event.MethodExpressionEventListener.processEvent(MethodExpressionEventListener.java:125)
... 38 more

I don't know how to solve :-(

Answer 1

您需要确保已给JVM适当的堆大小，检查配置，并确保-Xms设置为512，但-Xmx值比文件大小大得多，尤其是在加载整个文件时进入记忆。

Answer 2

You should be able to stream the data from the uploaded file rather than calling getData :

 public void listener(FileUploadEvent event) throws Exception {
 UploadedFile item = event.getUploadedFile();
 InputStream is = item.getInputStream();
 .....

Calls to item.getData() will cause the whole file to be loaded into memory.