Is there a clean way to resolve a DNS query (get IP by hostname) in Java asynchronously, in non-blocking way (ie state machine, not 1 query = 1 thread - I'd like to run tens of thousands queries simultaneously, but not run tens of thousands of threads)?
What I've found so far:
InetAddress.getByName()
implementation is blocking and looks like standard Java libraries lack any non-blocking implementations.Future
paradigm usage and, alas, very limited queue-only implementation. Fun
classAny other ideas/implementations I've missed?
Clarification . I have a fairly large (several TB per day) amount of logs. Every log line has a host name that can be from pretty much anywhere around the internet and I need an IP address for that hostname for my further statistics calculations. Order of lines doesn't really matter, so, basically, my idea is to start 2 threads: first to iterate over lines:
And a second thread that will:
epoll
/ kqueue
like technique)A simple model implementation in Perl using AnyEvent
shows me that my idea is generally correct and I can easily achieve speeds like 15-20K queries per second this way (naive blocking implementation gets like 2-3 queries per second - just the sake of comparison - so that's like 4 orders of magnitude difference). Now I need to implement the same in Java - and I'd like to skip rolling out my own DNS implementation ;)
It may be that the Apache Directory Services implementation of DNS on top of MINA is what you're looking for. The JavaDocs and other useful guides are on that page, in the left-hand side-bar.
在 netty 中有一些关于非阻塞 DNS 的工作,但它仍在进行中,可能只会在 5.0 中发布
我认为,您必须在使用基本套接字支持的原始 UDP 之上或使用 NIO 通道在 TCP 之上自己实现 DNS 客户端协议。
I don't have an answer to your question (I don't know if there is a DNS library that will operate in the async mode that you want) and this is too long for a comment.
But, you should be able to quickly produce an async one without having to write the full DNS handler yourself. Warning, I haven't done this so I could be all wrong.
Starting with the dnsjava code you ought to be able to implement your own resolver that will provide you both a sender and receiver method. Check out SimpleResolver and look at the send
method. You ought to be able to break up this method into two methods, one to send your request that runs up to the call to either the TCPClient or the UDPClient (you would handle the actual on the wire sending at this point, as you described, with your first thread), and, one to receive, which would be called by your second thread as a response to a socket read, and handle parsing the response. You may have to either copy all of the code from the SimpleResolver (lots of private methods that you'll need and licensing allows for it ), or, you could create your own version and simply load it ahead of the jared one in your classpath, or, your could reflect your way to the methods in question and set them accessible .
You can quickly build the network client side with either netty or mina . I prefer netty for the docs.
If you do go down this path and can/want to open source it, I can set aside some time to help if you get into trouble.
Linux has an asynchronous DNS lookup function: http://www.imperialviolet.org/2005/06/01/asynchronous-dns-lookups-with-glibc.html
If you are on Linux you just need to wrap that up in some JNI.
You have multiple options
Option 1: Java 5 Executors
Option 2: JMS with MessageListener
Option 2: Actor based framework
You can scale this well with this.Look at Akka .
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.