How to solve Vertx blocking DNS issue

Question

TL;DR: is it possible to overcome blocking DNS issues with Vertx? Preferably with Vertx HttpClient?

I'm trying to use Vertx to scrape some URLs with different hosts. I'm currently using Vertx HttpClient to do so, I've recently encountered an issue with a blocking DNS call which sometimes blocks my HttpClient requests (and therefore also blocking the event loop). This issue is described here .

I can't use a blocking http client in an "executeBlocking" function since I went for the Vertx solution because it is asynchronous and should be able to handle the loads I'm expecting. Also I can't resolve the IP address myself because it only solves it for a specific host while I'm trying to fetch many different hosts

So my question is: is there any (relatively) elegant solution to this issue that doesn't require me to wait for future Netty versions? Preferably something that uses the native Vertx HttpClient.

Thanks in advance

Answer 1

其他选项是通过将-Dvertx.disableDnsResolver添加到命令行来禁用dns 解析器。

Answer 2

Ok so after digging about this issue for several days I found several possible solutions to this issue. I hope this might help someone

These are the possible solutions

1. Programmatically solving the DNS issue using an executeBlock - this requires first "fixing" the DNS issue using vertx.executeBlocking with a future handler and only when the handler is invoked invoking the actual HTTP client code. Some possible optimizations here might be:
   • Saving an in memory map of all the hosts that the DNS already resolved and if a URL with a resolved host arrives just scrap it asynchronously.
   • Setting ordered to false in the executeBlocking statement (if you really don't care about ordering) as it can improve the performance of that code
2. Waiting for Netty 4.1 to come out. I don't know exactly when it's going to happen but I obviously could not wait for it
3. Do nothing about blocking the event loop. This means the program will be sluggish until the DNS server starts getting cache hits. If you can accept some sluggishness, that just might be a good fit

All of these solutions are from the Vertx mailing list. You can see the full correspondence here (including some code examples)

Currently I'm running the program in a cluster and I didn't encounter this issue on it. If I will later need to solve this I will probably aim for the 1st solution (unless Netty 4.1 is released by then)

I hope I got all the solutions right, if someone sees this and has better ideas or corrections I'll highly appreciate it