The shared mutex problem in Rust (implementing AsyncRead/AsyncWrite for Arc<Mutex<IpStack>>)

Question

Suppose I have an userspace TCP/IP stack. It's natural that I wrap it in Arc<Mutex<>> so I can share it with my threads.

It's also natural that I want to implement AsyncRead and AsyncWrite for it, so libraries that expect impl AsyncWrite and impl AsyncRead like hyper can use it.

This is an example:

use core::task::Context;
use std::pin::Pin;
use std::sync::Arc;
use core::task::Poll;
use tokio::io::{AsyncRead, AsyncWrite};

struct IpStack{}

impl IpStack {
    pub fn send(self, data: &[u8]) {
        
    }
    
    //TODO: async or not?
    pub fn receive<F>(self, f: F) 
        where F: Fn(Option<&[u8]>){
        
    }
}

pub struct Socket {
    stack: Arc<futures::lock::Mutex<IpStack>>,
}

impl AsyncRead for Socket {
    fn poll_read(
        self: Pin<&mut Self>,
        cx: &mut Context<'_>,
        buf: &mut tokio::io::ReadBuf<'_>
    ) -> Poll<std::io::Result<()>> {
        //How should I lock and call IpStack::read here?
        Poll::Ready(Ok(()))
    }
}

impl AsyncWrite for Socket {
    fn poll_write(
        self: Pin<&mut Self>,
        cx: &mut Context<'_>,
        buf: &[u8],
    ) -> Poll<Result<usize, std::io::Error>> {
        //How should I lock and call IpStack::send here?
        Poll::Ready(Ok(buf.len()))
    }
    //poll_flush and poll_shutdown...
}

Playground

I don't see anything wrong with my assumptions and I don't see another better way to share a stack with multiple threads unless I wrap it in Arc<Mutex<>>

This is similar to try_lock on futures::lock::Mutex outside of async? which caught my interest.

How should I lock the mutex without blocking? Notice that once I got the lock, the IpStack is not async, it has calls that block. I would like to implement async to it too, but I don't know it the problem will get much harder. Or would the problem get simpler if it had async calls?

Answer 1

I found the tokio documentation page on tokio::sync::Mutex pretty helpful: https://docs.rs/tokio/1.6.0/tokio/sync/struct.Mutex.html

From your description it sounds you want:

• Non-blocking operations
• One big data structure that manages all the IO resources managed by the userspace TCP/IP stack
• To share that one big data structure across threads

I would suggest exploring something like an actor and use message passing to communicate with a task spawned to manage the TCP/IP resources. I think you could wrap the API kind of like the mini-redis example cited in tokio 's documentation to implement AsyncRead and AsyncWrite . It might be easier to start with an API that returns futures of complete results and then work on streaming. I think this would be easier to make correct. Could be fun to exercise it with loom .

I think if you were intent on synchronizing access to the TCP/IP stack through a mutex you'd probably end up with an Arc<Mutex<...>> but with an API that wraps the mutex locks like mini-redis . The suggestion the tokio documentation makes is that their Mutex implementation is more appropriate for managing IO resources rather than sharing raw data and that does fit your situation I think.

Answer 2

You should not use an asynchronous mutex for this. Use a standard std::sync::Mutex .

Asynchronous mutexes like futures::lock::Mutex and tokio::sync::Mutex allow locking to be awaited instead of blocking so they are safe to use in async contexts. They are designed to be used across await s. This is precisely what you don't want to happen! Locking across an await means that the mutex is locked for potentially a very long time and would prevent other asynchronous tasks wanting to use the IpStack from making progress.

ImplementingAsyncRead / AsyncWrite is straight-forward in theory: either it can be completed immediately, or it coordinates through some mechanism to notify the context's waker when the data is ready and returns immediately. Neither case requires extended use of the underlying IpStack , so its safe to use a non-asynchronous mutex.

use std::pin::Pin;
use std::sync::{Arc, Mutex};
use std::task::{Context, Poll};
use tokio::io::{AsyncRead, AsyncWrite};

struct IpStack {}

pub struct Socket {
    stack: Arc<Mutex<IpStack>>,
}

impl AsyncRead for Socket {
    fn poll_read(
        self: Pin<&mut Self>,
        cx: &mut Context<'_>,
        buf: &mut tokio::io::ReadBuf<'_>,
    ) -> Poll<std::io::Result<()>> {
        let ip_stack = self.stack.lock().unwrap();

        // do your stuff

        Poll::Ready(Ok(()))
    }
}

Answer 3

I don't see another better way to share a stack with multiple threads unless I wrap it in Arc<Mutex<>> .

A Mutex is certainly the most straightforward way to implement something like this, but I would suggest an inversion of control.

In the Mutex -based model, the IpStack is really driven by the Socket s, which consider the IpStack to be a shared resource. This results in a problem:

• If a Socket blocks on locking the stack, it violates the contract of AsyncRead by spending an unbounded amount of time executing.
• If a Socket doesn't block on locking the stack, choosing instead to use try_lock() , it may be starved because it doesn't remain "in line" for the lock. A fair locking algorithm, such as that provided by parking_lot , can't save you from starvation if you don't wait.

Instead, you might approach the problem the way the system network stack does. Sockets are not actors: the network stack drives the sockets, not the other way around.

In practice, this means that the IpStack should have some means for polling the sockets to determine which one(s) to write to/read from next. OS interfaces for this purpose, though not directly applicable, may provide some inspiration. Classically, BSD provided select(2) and poll(2) ; these days, APIs like epoll(7) (Linux) and kqueue(2) (FreeBSD) are preferred for large numbers of connections.

A dead simple strategy, loosely modeled on select / poll , is to repeatedly scan a list of Socket connections in round-robin fashion, handling their pending data as soon as it is available.

For a basic implementation, some concrete steps are:

• When creating a new Socket , a bidirectional channel (ie one bounded channel in each direction) is established between it and the IpStack .
• AsyncWrite on a Socket attempts to send data over the outgoing channel to the IpStack . If the channel is full, return Poll::Pending .
• AsyncRead on a Socket attempts to receive data over the incoming channel from the IpStack . If the channel is empty, return Poll::Pending .
• The IpStack must be driven externally (for instance, in an event loop on another thread) to continually poll the open sockets for available data, and to deliver incoming data to the correct sockets. By allowing the IpStack to control which sockets' data is sent, you can avoid the starvation problem of the Mutex solution.