Email Verification with DNS & SMTP Check from Heroku Rails App

Question

I'm thinking to use this gem https://github.com/kamilc/email_verifier to verify email addresses.

Basically this gem will check for MX record in DNS and connects with the smtp to check if an email address really exists or not. Does anyone have experience with this type of things please advice my concerns.

Does querying too many times to dns and smtp from my app, hurt my domain reputation?

I know that heroku switch ip addreses for all dynos everyday, is this same case for my ssl app as well?

Answer 1

Yes, I have experience (from a few years ago) of doing pretty much exactly that at a large scale. We were scanning the entire .SE domain ("we" in this case being the .SE registry), and tried to verify the addresses in the SOA records.

I strongly advice you not to bother with the SMTP part. The false-positive and false-negative rates will be plenty high enough to make your results meaningless, and your server will quickly get blacklisted by all major mail service providers.

Checking the DNS part makes sense and works well. Use a library with a good reputation to check the syntax of the email address (do not try to write it yourself, it's way more complex than you think), and if it passes check that the domain looks deliverable. "Looks deliverable" means that it either has MX records that can be resolved to IP addresses, or if it does not have any MX records, that it has A and/or AAAA record(s).

Don't try to contact the IP addresses you find. It takes a lot of time (relatively speaking), and it tells you nothing. If you can connect, start an SMTP transaction and the other end says "Sure, I can deliver that" you still don't know if the address you're testing is good, since most mail servers these days will say they accept everything and then silently drop invalid messages on the floor. They do this precisely to prevent spammers from doing exactly what you're trying to do (verifying addresses), so don't expect that behavior to change until the spammers go away. If the server says "Nope, can't deliver that now, try again later", it may be using greylisting. So you could maybe get a good answer -- if you try again in five or ten minutes. You probably don't want your Ruby method call to take ten minutes. And finally, if the server says "Nope, can't deliver that, ever", you still can't know if that means the address is invalid of if the server is having a bad day, and the address may start working at some point in the future.

So to sum up, if you try to do SMTP connections to validate mail addresses, you will spend a lot of time (both when writing the code and when running it), you'll piss off a lot of people, get put on a lot of blacklists (some of which are very, very much more eager to add things than to take them off again) and at the end of it all you still won't know any more than you did before you started. It's really not worth it.