Spring Integration IMAP : Failed to read attachment from mail due to virus scanning in Outlook 365

Question

I am using Spring Integration to read email from Outlook 365 (cloud) using IMAP inbound-channel-adapter .

Scenario: Target mailbox in Outlook 365 is doing virus scanning for new emails once arrived, during this scan outlook is detaching the attachment and attaching it again once virus scan is completed.

Problem: Attachment is missing in very few cases (1 mail out of 50 approx), this is because of those emails are read by inbound-channel-adapter when the attachment is not available in outlook ( detached by virus scanner).

Question:

How can ensure the attachment was read every time? If I make the thread waiting for 2 mins inside handleMessage method, then will it block the reading of next email just arrived? OR please let me know any other solution to handle this situation.

Spring-integration.xml:

<util:properties id="javaMailProperties">
    <prop key="mail.imap.socketFactory.class">javax.net.ssl.SSLSocketFactory</prop>
    <prop key="mail.imap.socketFactory.fallback">false</prop>
    <prop key="mail.store.protocol">imaps</prop>
    <prop key="mail.debug">${imap.debug}</prop>
    <prop key="mail.imaps.partialfetch">false</prop>
    <prop key="mail.imaps.fetchsize">102400</prop>   <!-- 100KB, default is 16KB -->  
</util:properties>

<mail:inbound-channel-adapter id="imapAdapter" 
                                  store-uri="${imap.uri}"                                     
                                  channel="recieveEmailChannel"                                          
                                  should-delete-messages="false"
                                  should-mark-messages-as-read="true"                                      
                                  auto-startup="true"
                                  simple-content="true"
                                  auto-close-folder="true"
                                  java-mail-properties="javaMailProperties">
    <int:poller fixed-delay="${imap.polling.interval}" time-unit="SECONDS"/>
</mail:inbound-channel-adapter>

<int:channel id="recieveEmailChannel">        
    <int:interceptors>
        <int:wire-tap channel="logger"/>
    </int:interceptors>
</int:channel>

<int:logging-channel-adapter id="logger" level="DEBUG"/>

<int:service-activator input-channel="recieveEmailChannel" ref="emailReceiver" method="handleMessage"/>

Answer 1

Yes, as long as you don't shift the work to a different thread, the SourcePollingChannelAdapter does block before the next poll. By default it is configured to poll only one message. Therefore, so far you are good.

Another way is probably to take a look into a custom search-term-strategy :

                <xsd:attribute name="search-term-strategy" type="xsd:string">
                    <xsd:annotation>
                        <xsd:appinfo>
                            <tool:annotation kind="ref">
                                <tool:expected-type
                                        type="org.springframework.integration.mail.SearchTermStrategy"/>
                            </tool:annotation>
                        </xsd:appinfo>
                        <xsd:documentation>
                            Reference to a custom implementation of
                            org.springframework.integration.mail.SearchTermStrategy
                            to use when retrieving email. Only permitted with 'imap' protocol or an 'imap' uri.
                            By default, the ImapMailReceiver will search for Messages based on the default
                            SearchTerm
                            which is "All mails that are RECENT (if supported), that are NOT ANSWERED, that are NOT
                            DELETED, that are NOT SEEN and have not
                            been processed by this mail receiver (enabled by the use of the custom USER flag or
                            simply NOT FLAGGED if not supported)".
                        </xsd:documentation>
                    </xsd:annotation>
                </xsd:attribute>

So, you won't poll the messages from the mail box until they satisfy the search term. See more info in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/mail.html#mail-namespace

Or probably the mail-filter-expression : https://docs.spring.io/spring-integration/docs/current/reference/html/mail.html#mail-filtering .

Anyway it would be great to have some flag on the message in that Outlook while it is scanned for viruses.

Answer 2

This solution is based on @Artem Bilan's answer & comment. Thanks for the direction Arten:-).

---

Problem: Failed to read attachment intermittently because of Outlook's virus scanning .

Idea: Delay email reading by some seconds, hoping that virus scan will be completed by that time.

Solution: Using custom SearchTermStrategy and OlderTerm .

SideEffect of this solution: As discussed in comments of @Artem's answer , while polling for new email JavaMail & IMAP has to search ENTIRE INBOX for each poll and find match based on custom searchTermStatregy. In my case, this mailbox receives ~1000 mails per day , so after some months/year INBOX will grow in size, email poller has to read million+ mails per poll. This will be a huge performance hit . So we need to tackle this issue as well.

=================================================================

Solution-Step1: Use custom searchTermStatregy to delay email reading:

Reading emails which are at least 10 mins old and not read already.

/**
 * 
 */
package com.abc.xyz.receiver;

import javax.mail.Flags;
import javax.mail.Folder;
import javax.mail.search.AndTerm;
import javax.mail.search.FlagTerm;
import javax.mail.search.SearchTerm;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.integration.mail.SearchTermStrategy;
import org.springframework.stereotype.Service;
import com.sun.mail.imap.OlderTerm;

/**
 * @author Sundararaj Govindasamy
 *
 */
@Service
public class MySearchTermStrategy implements SearchTermStrategy {
    
    Logger logger = LoggerFactory.getLogger(MySearchTermStrategy.class);

    @Override
    public SearchTerm generateSearchTerm(Flags supportedFlags, Folder folder) {
        logger.info("entering MySearchTermStrategy");

        SearchTerm older10MinsTerm = new OlderTerm(600);// in seconds
        SearchTerm flagTerm  = new FlagTerm(new Flags(Flags.Flag.SEEN), false);

        SearchTerm[] searchTermArray = new SearchTerm[2];
        searchTermArray[0] = older10MinsTerm;
        searchTermArray[1] = flagTerm;

        SearchTerm andTerm = new AndTerm(searchTermArray);
        return andTerm;
    }
}

In spring-integration.xml, configure custom searchTermStatregy as below.

<mail:inbound-channel-adapter id="imapAdapter" 
                                  store-uri="${imap.uri}"                                     
                                  channel="recieveEmailChannel"                                          
                                  should-delete-messages="false"
                                  should-mark-messages-as-read="true"                                      
                                  auto-startup="true"
                                  simple-content="true"
                                  auto-close-folder="false"
                                  search-term-strategy="mySearchTermStrategy"
                                  java-mail-properties="javaMailProperties">
    <int:poller fixed-delay="${imap.polling.interval}" time-unit="SECONDS"/>
</mail:inbound-channel-adapter>

======================================================================

Solution-Step2: Move emails which were read from INBOX to another folder daily at 10pm, so that email poller's load will be reduced.

Please note, Spring Integration framework don't have support for mail moving, When I tried with Spring Integration, it thrown below ClassCastException as discussed in this thread: https://stackoverflow.com/a/58033889/1401019

So I decided to use plain JavaMail APIs to move mails between folders as below. This Spring scheduler will run once per day and move all read mails from INBOX folder to SunTest folder as below.

/**
 * 
 */
package com.abc.xyz.receiver;

import java.util.Properties;
import javax.mail.Flags;
import javax.mail.Folder;
import javax.mail.Message;
import javax.mail.Session;
import javax.mail.Store;
import javax.mail.search.FlagTerm;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import com.sun.mail.imap.IMAPFolder;

/**
 * @author Sundararaj Govindasamy 
 * Move read email from INBOX to another folder
 *         to improve INBOX search performance.
 */
@Component
public class EmailMover {

    Logger logger = LoggerFactory.getLogger(EmailMover.class);

    /**
     * move all read mails everyday 10PM, after business hours.
     */
    @Scheduled(cron = "0 0 22 * * *")
    public void moveReadMails() {
        logger.info("entering: moveReadMails Scheduler");
        // create session object
        Session session = this.getImapSession();
        try {
            // connect to message store
            Store store = session.getStore("imaps");
            store.connect("outlook.office365.com", 993, "documents.local@mycompany.onmicrosoft.com",
                    "thisispassword");
            // open the inbox folder
            IMAPFolder inbox = (IMAPFolder) store.getFolder("INBOX");
            inbox.open(Folder.READ_WRITE);
            // fetch messages
            
            Message[] messageArr = inbox.search(new FlagTerm(new Flags(Flags.Flag.SEEN), true));
            logger.info("No. of read emails in INBOX:{}", messageArr.length);
            
            // read messages
            for (int i = 0; i < messageArr.length; i++) {
                Message msg = messageArr[i];
                String subject = msg.getSubject();
                logger.info("Subject:{}", subject);
            }
            
            Folder targetFolder = inbox.getStore().getFolder("SunTest");
            inbox.moveMessages(messageArr, targetFolder);
            logger.info("All read mails were moved successfully from INBOX to {} folder", targetFolder);
        } catch (Exception e) {
            logger.error("Exception while moving emails, exception:{}", e.getMessage());
        }
        logger.info("exiting: moveReadMails Scheduler");
    }
    /**
     * 
     * @return
     */
    private Session getImapSession() {
        Properties props = new Properties();
        props.setProperty("mail.store.protocol", "imaps");
        props.setProperty("mail.debug", "true");
        props.setProperty("mail.imap.host", "outlook.office365.com");
        props.setProperty("mail.imap.port", "993");
        props.setProperty("mail.imap.ssl.enable", "true");
        Session session = Session.getDefaultInstance(props, null);
        session.setDebug(true);
        return session;

    }
}