2

I've recently begun to implement a UDP socket receiver with Registered I/O on Win32. I've stumbled upon the following issue: I don't see any way to cancel pending RIOReceive()/RIOReceiveEx() operations without closing the socket.

To summarize the basic situation:

During normal operation I want to queue quite a few RIOReceive()/RIOReceiveEx() operations in the request queue to ensure that I get the best possible performance when it comes to receiving the UDP packets.

However, at some point I may want to stop what I'm doing. If UDP packets are still arriving at that point, fine, I can just wait until all pending requests have been processed. Unfortunately, if the sender has also stopped sending UDP packets, I still have the pending receive operations.

That in and by itself is not a problem, because I can just keep going once operations start again.

However, if I want to reconfigure the buffers used in between, I run into an issue. Because the documentation states that it's an error to deregister a buffer with RIO while it's still in use, but as long as receive operations are still pending, the buffers are still officially in use, so I can't do that.

What I've tried so far related to cancellation of these requests:

  • CancelIo() on the socket (no effect)
  • CancelSynchronousIo(GetCurrentThread()) (no effect)
  • shutdown(s, SD_RECEIVE) (success, but no effect, the socket even receives packets afterwards -- though shutdown probably wouldn't have been helpful anyway)
  • WSAIoctl(s, SIO_FLUSH, ...) because the docs of RIOReceiveEx() mentioned it, but that just gives me WSAEOPNOTSUPP on UDP sockets (probably only useful for TCP and probably also only useful for sending, not receiving)
  • Just for fun I tried to set setsockopt(s, SOL_SOCKET, SO_RCVTIMEO, ...) with 1ms as the timeout -- and that doesn't appear to have any effect on RIO regardless of whether I call it before or after I queue the RIOReceive()/RIOReceiveEx() calls

Only closing the socket will successfully cancel the I/O.

I've also thought about doing RIOCloseCompletionQueue(), but there I wouldn't even know how to proceed afterwards, since there's no way of reassigning a completion queue to a request queue, as far as I can see, and you can only ever create a single request queue for a socket. (If there was something like RIOCloseRequestQueue() and that did cancel the pending requests, I'd be happy, but the docs only mention that closesocket() will free resources associated with the request queue.)

So what I'm left with is the following:

  • Either I have to write my logic so that the buffers that are being used are always fixed once the socket is opened, because I can't really ever change them in practice due to requests that could still be pending.

  • Or I have to close the socket and reopen it every time I want to change something here. But that is a race condition, because I'd have to bind the socket again, and I'd really like to avoid that if possible.

  • I've tested sending UDP packets to my own socket from a newly created different socket until all of the requests have been 'eaten up' -- and while that works in principle, I really don't like it, because if any kind of firewall rule decides to not allow this, the code would deadlock instantly.

On Linux io_uring I can just cancel existing operations, or even exit the uring, and once that's done, I'm sure that there are no receive operations still active, but the socket is still there and accessible. (And on Linux it's nice that the socket still behaves like a normal socket, on Windows if I create the socket with the WSA_FLAG_REGISTERED_IO flag, I can't use it outside of RIO except for operations such as bind().)

Am I missing something here or is this simply not possible with Registered I/O?

chris_se
  • 1,006
  • 1
  • 7
  • The TL;DR: Memory is cheap. Just allocate _maximum_ size buffers _once_. The problem goes away. Otherwise, with UDP you have to "know" when the sender is finished sending. See my recent answer: [thread function doesnt terminate until enter is pressed](https://stackoverflow.com/a/72341219) for a way to do this. A positive "I am done" message is usually preferable to trying to "divine" this with a timeout (which can produce intermittent/spurious closures if the sender has more to send but is delayed beyond the timeout). And, how do you handle packets that get squelched in network? – Craig Estey Jun 17 '22 at 20:33
  • are [CancelIoEx](https://learn.microsoft.com/en-us/windows/win32/fileio/cancelioex-func) have effect ? – RbMm Jun 18 '22 at 14:38
  • @CraigEstey it's not necessarily about pure memory usage, but also about the application logic. What you're suggesting with the buffers is basically my first bullet point -- and sure, that'll work, but I wanted to ask if there's a better way of doing this (w.r.t. timeouts: I agree, but I tried the 1ms timeout because nothing else seemed to work) @RbMm well, I can't pass any overlapped structure into `CancelIoEx()` because I don't have an overlapped structure to cancel; if I pass `NULL` I get the same result as with `CancelIo()`, i.e. `ERROR_NOT_FOUND` – chris_se Jun 20 '22 at 07:21

0 Answers0