Reading from javamail takes a long time

Question

I use javamail to read mails from an exchage account using IMAP protocol. Those mails are in plain format and its contents are XMLs.

Almost all those mails have short size (usually under 100Kb). However, sometimes I have to deal with large mails (about 10Mb-15Mb). For example, yesterday I received an email which was 13Mb size. It took more than 50min just to read it. Is it normal? Is there a way to increase its performance? The code is:

Session sesion = Session.getInstance(System.getProperties());
Store store = sesion.getStore("imap");
store.connect(host, user, passwd);
Folder inbox = store.getFolder("INBOX");
inbox.open(Folder.READ_WRITE);

Message[] messages = inbox.search(new FlagTerm(new Flags(Flags.Flag.SEEN), false));
for (int i = 0 ; i< messages.length ; i++){
    Object contents = messages[i].getContent();  // Here it takes 50 min on 13Mb mail
    // ...
}

Method that takes such a long time is messages[i].getContent() . What am I doing wrong? Any hint?

Thanks a lot and sorry for my english! ;)

Answer 1

I finally solved this issue and wanted to share.

The solution, at least the one that worked to me, was found in this site: http://www.oracle.com/technetwork/java/faq-135477.html#imapserverbug

So, my original code typed in my first post becomes to this:

Session sesion = Session.getInstance(System.getProperties());
Store store = sesion.getStore("imap");
store.connect(host, user, passwd);
Folder inbox = store.getFolder("INBOX");
inbox.open(Folder.READ_WRITE);

// Convert to MimeMessage after search 
MimeMessage[] messages = (MimeMessage[]) carpetaInbox.search(new FlagTerm(new Flags(Flags.Flag.SEEN), false));
for (int i = 0 ; i< messages.length ; i++){
    // Create a new message using MimeMessage copy constructor
    MimeMessage cmsg = new MimeMessage(messages[i]);
    // Use this message to read its contents 
    Object obj = cmsg.getContent(); 
// ....
}

The trick is, using MimeMessage() copy constructor, create a new MimeMessage and read its contents instead of original message. You should note that such object is not really connected to server, so any changes you make on it, like setting flags, won't take effect. Any change on message, have to be done on original message.

To sum up: This solution works reading large Plain Text mails (up to 15Mb) connecting to an Exchange Server using IMAP protocol. The times lowered from 51-55min to read a 13Mb mail, to 9seconds to read same mail. Unbelievable.

Hope this helps someone and sorry for English mistakes ;)

Answer 2

It would always be messages[i].getContent() that would be the slowest part of the code. The reason is normally IMAP server would not cache this part of message data. Nevertheless, you can try this:

    FetchProfile fp = new FetchProfile();
        fp.add(FetchProfile.Item.ENVELOPE);
        fp.add(FetchProfileItem.FLAGS);
        fp.add(FetchProfileItem.CONTENT_INFO);
    fp.add("X-mailer");

and after you have specified the fetch profile then you do your search/fetch of messages. 

Basically the concept is that the IMAP provider fetches the data for a message from the server only when necessary. (The javax.mail.FetchProfile is used to optimize this). The header and body structure information, once fetched, is always cached within the Message object. However, the content of a bodypart is not cached. So each time the content is requested by the client (either using getContent() or using getInputStream()), a new FETCH request is issued to the server. The reason for this is that the content of a message could be potentially large, and if we cache this content for a large number of messages, there is the possibility that the system may run out of memory soon since the garbage collector cannot free the referenced objects. Clients should be aware of this and must hold on to the retrieved content themselves if needed.

By using the above mentioned code snippet you could 'hope' for some speed improvement but it solely depends on your SMTP server if this would work or not. All the big SMTP server do not support this behaviour because of the load issue mentioned in the previous paragraph and hence you may not gain any speed.

Answer 3

Using the Folder.fetch method you can prefetch in one operation the metadata for multiple messages. That will reduce the time to process each message, but won't help that much with a huge message.

The handle huge message parts efficiently, you'll generally want to use the getInputStream method to process the data incrementally, rather than using the getContent method to read all the data in and create a huge String object with all the data.

You can also tune the fetching by specifying the "mail.imap.fetchsize" property, which defaults to 16384. If most of your messages are less than 100K, and you always need to read all of the data in the message, you might set the fetchsize to 100K. That will make small messages much faster and larger message more efficient.

Answer 4

I had a similar issue. Fetching mails via IMAP were very slow. Furthermore I had another issue downloading large attachments. After a look in the JavaMail FAQ I found the solution for the later issue in this question that advises to set the mail.imap.partialfetch (respectively mail.imaps.partialfetch ) to false . This not only fixes the download issue but the slow reading of the messages as well.

In the referenced JavaMail notes.txt it is said.

4. Due to a problem in the Microsoft Exchange IMAP server, insufficient number of bytes may be retrieved when reading big messages. There are two ways to workaround this Exchange bug:

   (a) The Exchange IMAP server provides a configuration option called "fast message retrieval" to the UI. Simply go to the site, server or recipient, click on the "IMAP4" tab, and one of the check boxes is "enable fast message retrieval". Turn it off and the octet counts will be exact. This is fully described athttp://support.microsoft.com/default.aspx?scid=kb;EN-US;Q191504

   (b) Set the "mail.imap.partialfetch" property to false. You'll have to set this property in the Properties object that you provide to your Session.

5. Certain IMAP servers do not implement the IMAP Partial FETCH functionality properly. This problem typically manifests as corrupt email attachments when downloading large messages from the IMAP server. To workaround this server bug, set the "mail.imap.partialfetch" property to false. You'll have to set this property in the Properties object that you provide to your Session.