Google使用什么协议用于Gmail？ （不是IMAP或POP）

Question

You can access gmail either using the web interface, Google's Android client or using IMAP. As far as I can tell, the web interface and the Android app uses a completely different protocol than IMAP -- they are not just interfaces on top of it. The reason I'm sure of that is because the Android app can without problem open a folder with 1m mail in < 3 seconds. No plain IMAP client can do that.

So my question is what is known about this secret protocol? Where is the reference documentation for it? Has it been reverse engineered? Does Google sanction its use?

arnt's answer provides an excellent method to test gmail's raw imap speed:

$ openssl s_client -host imap.gmail.com -port 993 -crlf 
...
* OK Gimap ready for requests from 12.34.56.78
$ a LOGIN ***@*** ***
a OK
$ c SELECT "[Gmail]/All mail" !!!!
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Flags permitted.
* OK [UIDVALIDITY 673376278] UIDs valid.
* 1142417 EXISTS
* 0 RECENT
* OK [UIDNEXT 1159771] Predicted next UID.
* OK [HIGHESTMODSEQ 8670601]
c OK [READ-WRITE] [Gmail]/All mail selected. (Success)

The command I've marked, c SELECT "[Gmail]/All mail" takes about 20 seconds to complete. Since that time is larger than it takes for the GMail app on my relatively underpowered Android phone to startup and load the All mail label which does it in less than 6 seconds even after I purged its caches. The web client is even faster.

Unless I'm missing something basic this proves "beyond reasonable doubt" that Google's GMail clients does not use IMAP since you never ever have to wait 20 seconds for any SELECT command to complete.

Answer 1

The Android app (at least the ones I've used) use IMAP. You can verify this by running Wireshark on the server.

As to why the Android app is so fast - what I know is that it uses the SEARCH command to select the most recent n messages. Desktop clients such as Thunderbird or Outlook are much more heavy-weight and download headers and metadata for every message in the folder, despite recommendations for them not to.

A smartphone does not have the resources to store and process millions of emails (although more modern ones might be getting there) so the SEARCH approach allowed quick mail access for handheld devices.

Anyhow, Wireshark can reveal a great deal about the behaviour of IMAP clients and servers. If you're really curious, give it a shot. You can't do this if the server is Gmail, but you can try it out on another server (eg hMailServer).

Answer 2

After more research, I've found that there exists an API for GMail: https://developers.google.com/gmail/api/ I don't think that API was released when I posted this question back in 2013.

Using that API, I have created a demo program that fetches the 100 last mails of a label: https://gist.github.com/bjourne/37a9d15b03862003008a1b0169190dbe

The relevant part of the program is:

resource = service.users().messages()
result = resource.list(userId = 'me', labelIds = [label]).execute()
mail_ids = [m['id'] for m in result['messages']]

start = time()
mails = []
batch = BatchHttpRequest()
cb = lambda req, res, exc: mails.append(to_mail(res))
for mail_id in mail_ids:
    get_request = resource.get(**headers_params(mail_id))
    batch.add(get_request, callback = cb)
result = batch.execute()
print('Took %.2f seconds' % (time() - start))

It lists the last 100 messages sorted by date in a label (folder in IMAP terminology) containing over 570k messages.

On my machine, this loop takes about 0.5 - 0.8 seconds. I can claim confidently that no pure IMAP client on the planet comes even close. Likely, IMAP won't ever get faster because it is a poor fit for how Google stores mail internally.

So I'll answer my own question. This is the API they are using and it wasn't exposed earlier.

Answer 3

"No other IMAP client can do this" is a pretty bold statement, but a million of messages is a pretty big number as well. I would encourage you to give Trojitá a try here. Chances are that the initial synchronization will be rather slow (it would transfer the flags for that million of messages for various technical reasons related to how the IMAP flags, SELECT , SEARCH and STATUS are specified), but the subsequent resynchronizaiton should be lightning fast thanks to ESEARCH , CONDSTORE and QRESYNC . I would be interested to hear how well Trojitá works with your setup -- the contact information are on the homepage.

To your question -- most webmails nowadays provide a private API for their own use. A typical architecture is to transfer messages about updated state via JSON, but there is no standard for this and the interface is prioprietary. Chances are that the GMail "app" uses the same (or similar) method. You don't have much options to verify this as it is likely using TLS. With a web interface, it is trivial to see the traffic with an appropriate browser plugin, but not so much with a standalone Android application.

Answer 4

You can test gmail's IMAP performance easily (if you have a million-message mailbox). Open an IMAP connection with

openssl s_client -connect imap.gmail.com:993 -crlf

then login and open your inbox.

a login yourname@gmail.com yourpassword
b select inbox

Or open your allmail box if inbox isn't big enough (name may vary depending on UI language):

c select "[Gmail]/All Mail"

If SELECT is fast but an IMAP client slow, then that's because the client sends additional/unneeded slow commands. Many choose to fill or update a data structure for the entire million messages even if they're going to display only 40 messages. That's client choice, not IMAP slowness.