Winsock2 同步 IO ...等待 WSASend 在 WSAGetOverlappedResult 中使用 fWait==TRUE “完成”

Question

A coworker and I are having a disagreement on what constitutes "completion" of a WSASend overlapped IO request. He asserts using fWait as TRUE in the WSAGetOverlappedResult call only waits until the message is queued for sending. He believes waiting for the write/send operation to "complete" only means the message was successfully initiated. In my opinion that is far from a "completed" message to the other side of the socket...that would simple be a beginning of a send and not a completion. If the fWait of TRUE does not block until the bytes have been sent and been ACKed (or error returned), then this is far from synchronous...it would in fact be acting the same as asynchronous IO because it's just fire and forget.

I have been maintaining our company's communication library with my understanding of how to do, and what is, "synchronous" IO for decades so I'll be shocked if I'm indeed wrong in my understanding. But my coworker is a brilliant developer with TONS of TCP/IP experience and is adamant he's right. Says he even posed this question here on stackoverflow and was told he was right. I can't imagine how I could be misunderstanding "completion" of a send to mean anything other than the sending of the bytes requested were indeed sent and ACKed. But I've been wrong before LOL

So...who is right? What EXACTLY does it mean to wait for a WSASend request to be "complete"? Simply waiting until the message is queued for sending in the TCP/IP stack...or waiting for all the packets that constitute the message to be both sent and ACKed??? Or is the truth somewhere in-between?

Answer 1

You are wrong.

A send request is completed when the request is no longer being processed. Once the stack has taken over the operation, the request is no longer being processed and its resources can be used for some other purpose.

If you were right, overlapped I/O would be nearly useless. Who would want to have to wait until the other side ACKed a previous send in its entirety before they could queue any more data to the TCP connection?

How would overlapped I/O be useful if there was no way to determine when the queing process was complete? That's what the application needs to know so it can send another chunk of data.

You would always have dead time as the send queue would always have to get completely empty before more data could be queued on the sending side. Yuck!

And knowing when the ACK is received is useless anyway. It doesn't mean the application on the other end got the data. So it has no meaning at application layer. Knowing that the send has been queued does have meaning at application layer -- it means you can now queue more data to be sent and be guaranteed that all of the data from the previous send will be passed to the receive's application layer before any data from the next send. What's what the application needs to know.

A synchronous call to WASSend also completes when the send is queued. An asynchronous operation just means you don't have to wait for whatever you'd wait for in a synchronous operation. So that's another reason your understanding would be quite strange.

Lastly, there is no standard or common way to wait for a remote ACK with synchronous operations. So you would have to think that asynchronous operations default to providing a semantic that is not even really available with synchronous ones. That's also pretty strange.

Answer 2

David is correct.

WSASend will complete when that data is handed off to the TCPIP stack (buffered at their layer) to be sent whenever the transport will allow. If it's a blocking call, it will wait until it's ready to pend; just like if it's async, the OVERLAPPED I/O will complete once it pends.

Some may ask why is this even necessary? This is behavior is crucial for keeping as much data in-flight over a TCP connection. In order to keep as much data in-flight, a sender should call WSASend until it pends (recall if it's a synchronous WSASend call then that thread will just block until WSASend can complete; if it's asynchronous, the async completion will occur once that WSASend call can pend).

Why would WSASend ever pend, why not just complete immediately? WSASend will only complete once the transport (in kernel) is ready to buffer more data. So looping until WSASend pends will keep that connection flush with enough data to keep the maximum data in-flight.

Hope this makes sense.

Answer 3

My testing seems to show I am indeed wrong. Synchronous behavior does not exist for sending...just receiving. I see how this is a performance help, but I object to all documentation saying sending fWait of TRUE in WSAGetOverlappedResult will wait until the IO request is "complete". This word implies much more than just it being queued up. Well..I'm flabbergasted that my understanding was off...but TCP handles things well enough that it hasn't caused me issues before.

Thanks for the excellent replies and patience with my frustrated replies. I'm incredibly disappointed at how all the documentation is written using words that would imply my understanding was right...when all along waiting for IO to "end", "complete", "finish" absolutely does NOT wait.