Gmail IMAP is sometimes returning bad results for fetch

Question

I'm using python to connect to gmail via IMAP.

When I do a fetch for a bunch of uids, the results sometimes contain a couple weird ones tacked on to the end.

This bug appears intermittent and started appearing sometime in the last week or so.

For example I do a fetch like:

>>>import imaplib
>>>conn = imaplib.IMAP4_SSL('imap.gmail.com')
>>># authenticate etc
>>>conn.uid('fetch', '12,13', '(X-GM-THRID)')

I sometimes get a result like:

>>>['1 (X-GM-THRID 123123123123 UID 12)', 
    '2 (X-GM-THRID 123123123123 UID 13)', 
    '365022 (UID 601722 FLAGS (\\Seen))']

(Line breaks added for readability, ids changed from original, normally I'm fetching far more than two.)

This is pretty weird. I haven't requested anything at all about the extra message. Sometimes it's there, sometimes not. No matter what fields I fetch, the extra result (I've only ever seen one or two) never contains them, and only contains the FLAGS info.

Any ideas why this might be happening?

Answer 1

The following was posted in the IMAP Protocol mailing list earlier this week:

"As of yesterday [Monday October 7th], Gmail now supports reporting flags changes anywhere we would report new/expunged messages (ie, most places its allowed by the protocol, but definitely during IDLE). Its only enabled for gmail.com users at the moment, Google Apps users will follow in a week or so, assuming we don't find any issues."

However, there are apparently issues with the new functionality because today they said it is being rolled back:

"We're seeing several reports of programs not expected extra FETCH responses, which we rolled out on Monday. We're rolling back soon, since older versions of a very popular client are having issues (though not to our knowledge with this part of things)."

-Rick

Answer 2

Gmail is sending you unsolicited FLAGS updates (because someone changed the message remotely). These aren't in response to your request, but IMAP allows the server to send you any* information at any time. Many servers will save these unsolicited responses for IDLE or NOOP responses, but apparently Gmail does not wish to wait.

However, until recently (apparently?) Gmail did not send flag updates at all, only EXPUNGEs.

*: There are a few rules about which responses can be sent when to avoid race conditions, but this isn't one of those.