GCC和MSVC之间的boost :: asio :: io_service行为的差异：无法取消已发布的作业

Question

When implementing a basic thread pool using boost::asio::io_service , I am observing some differences in how queued tasks are handled when stopping the io_service .

On MSVC 14 (MS Visual Studio 2015), for some reason the queued tasks which were not started yet are not dropped when stopping the io_service but are run nonetheless. These tasks are dropped when running this on Ubuntu 16.04 (GCC 5.4.0).

I have simplified and cleaned up the original tests and put them in a single file (also listed below) which only depends on boost and uses some sleeps to demonstrate the problem. You can build it with the CMakeLists.txt (also listed below) if you wish or use the online compilers linked below.
Notice that the thread pool uses only one worker thread so that the jobs are run sequentially.

The output with GCC is as expected ( Here on an online compiler ):

 checkAllWorkIsProcessedBeforeDestruction  
     passed.  
     passed.  
     passed.  
checkWorkCanBeCancelled
     passed.
     passed.
     passed.
checkWorkCanBeInterrupted
     passed.
     passed.
     passed.
checkUninterruptableWorkIsNotInterruptedButCanBeDropped
     passed.
     passed.
     passed.
     passed.

This is the output on MSVC 14 (Visual Studio 2015) ( Here on an online VC++ compiler ):

checkAllWorkIsProcessedBeforeDestruction
     passed.
     passed.
     passed.
checkWorkCanBeCancelled
     Error: functor 1 call expected: false current: true
     Error: functor 2 call expected: false current: true
     Error: running time expected: 150 current: 402
checkWorkCanBeInterrupted
     passed.
     passed.
     passed.
checkUninterruptableWorkIsNotInterruptedButCanBeDropped
     passed.
     Error: functor 2 call expected: false current: true
     passed.
     Error: running time expected: 250 current: 404

Am I doing something wrong?

I have also filled a bug to boost but got no response so far: #13317

---

Source code: ThreadPoolTests.cpp

// Copyright (c) 2017 Diego Barrios Romero <eldruin@gmail.com>

#include <iostream>
#include <string>
#include <memory>
#include <boost/asio.hpp>
#include <boost/bind.hpp>
#include <boost/thread.hpp>
#include <boost/chrono.hpp>

class ThreadPool
{
public:
  ThreadPool(const size_t threadCount = boost::thread::hardware_concurrency())
    : work(new boost::asio::io_service::work(service))
  {
    for (size_t i = 0; i < threadCount; ++i)
    {
      threads.create_thread(boost::bind(&boost::asio::io_service::run, &service));
    }
  }
  template<typename FunctionType>
  void post(FunctionType f)
  {
    service.post(f);
  }

  void interrupt()
  {
    threads.interrupt_all();
  }

  void cancel()
  {
    work.reset();
    service.stop();
  }

  ~ThreadPool()
  {
    work.reset();
    threads.join_all();
  }
private:
  boost::asio::io_service service;
  boost::thread_group threads;
  std::unique_ptr<boost::asio::io_service::work> work;
};


struct Functor
{
  void operator()()
  {
    boost::this_thread::sleep(boost::posix_time::milliseconds(200));
    boost::lock_guard<boost::mutex> lock(mutex);
    wasCalled_ = true;
  }
  bool wasCalled() const
  {
    boost::lock_guard<boost::mutex> lock(mutex);
    return wasCalled_;
  }

private:
  bool wasCalled_ = false;
  mutable boost::mutex mutex;
};

struct UninterruptableFunctor : public Functor
{
  void operator()()
  {
    boost::this_thread::disable_interruption disableInterruptions;
    Functor::operator()();
  }
};

template<typename F, typename T1, typename T2>
void check(F compare, T1 expected, T2 current, const std::string& msg)
{
  if (compare(expected, current))
  {
    std::cout << "\tpassed." << std::endl;
  }
  else
  {
    std::cout << std::boolalpha
              << "\tError: " << msg << " expected: " << expected
              << " current: " << current << std::endl;
  }
}

struct ThreadPoolTest
{
  boost::int_least64_t getRunningTimeInMS() const
  {
    auto executionTime = boost::chrono::high_resolution_clock::now() - start;
    return boost::chrono::duration_cast<boost::chrono::milliseconds>(executionTime).count();    
  }

  template<typename FunctorType, typename F>
  void runTest(F f, bool shouldFunctor1BeCalled, bool shouldFunctor2BeCalled)
  {
    FunctorType functor1, functor2;
    {
      ThreadPool pool(1);
      pool.post(boost::bind(&FunctorType::operator(), &functor1));
      pool.post(boost::bind(&FunctorType::operator(), &functor2));
      f(pool);
    }

    auto eq = [](bool a, bool b) { return a == b; };
    check(eq, shouldFunctor1BeCalled, functor1.wasCalled(), "functor 1 call");
    check(eq, shouldFunctor2BeCalled, functor2.wasCalled(), "functor 2 call");
  }

private:
  boost::chrono::high_resolution_clock::time_point start = boost::chrono::high_resolution_clock::now();
};

void doNothing(ThreadPool&) { }
void cancel(ThreadPool& pool)
{
  pool.cancel();
}
void waitForStartThenInterruptThenCancel(ThreadPool& pool)
{
  boost::this_thread::sleep(boost::posix_time::milliseconds(100));
  pool.interrupt();
  pool.cancel();
}

bool lessEq (const boost::int_least64_t a, const boost::int_least64_t b) { return a <= b; }
bool greaterEq (const boost::int_least64_t a, const boost::int_least64_t b) { return a >= b; }

void checkAllWorkIsProcessedBeforeDestruction()
{
  ThreadPoolTest test;
  std::cout << "checkAllWorkIsProcessedBeforeDestruction\n";
  test.runTest<Functor>(doNothing, true, true);
  check(lessEq, 350, test.getRunningTimeInMS(), "running time");
}

void checkWorkCanBeCancelled()
{
  ThreadPoolTest test;
  std::cout << "checkWorkCanBeCancelled\n";
  test.runTest<Functor>(cancel, false, false);
  check(greaterEq, 150, test.getRunningTimeInMS(), "running time");
}

void checkWorkCanBeInterrupted()
{
  ThreadPoolTest test;
  std::cout << "checkWorkCanBeInterrupted\n";
  test.runTest<Functor>(waitForStartThenInterruptThenCancel, false, false);
  check(greaterEq, 150, test.getRunningTimeInMS(), "running time");
}

void checkUninterruptableWorkIsNotInterruptedButCanBeDropped()
{
  ThreadPoolTest test;
  std::cout << "checkUninterruptableWorkIsNotInterruptedButCanBeDropped\n";
  test.runTest<UninterruptableFunctor>(waitForStartThenInterruptThenCancel, true, false);
  check(lessEq, 150, test.getRunningTimeInMS(), "running time");
  check(greaterEq, 250, test.getRunningTimeInMS(), "running time");
}

int main(int, char*[])
{
  checkAllWorkIsProcessedBeforeDestruction();
  checkWorkCanBeCancelled();
  checkWorkCanBeInterrupted();
  checkUninterruptableWorkIsNotInterruptedButCanBeDropped();
}

Here the CMakeLists.txt for compilation ease.

cmake_minimum_required (VERSION 2.8.11)
project (ThreadPoolTests)

set(CMAKE_CXX_STANDARD 11)

find_package(Boost COMPONENTS thread)
if (Boost_FOUND)
  include_directories(${Boost_INCLUDE_DIR})
else()
  message(FATAL_ERROR "No Boost found")
endif()

add_executable (ThreadPoolTests ThreadPoolTests.cpp)
target_link_libraries(ThreadPoolTests ${Boost_LIBRARIES})

---

Again the question: Am I doing something wrong?

Answer 1

At first I saw a problem with your tests.

Let's start with that first.

Cancel, Racy Tests

ThreadPool::cancel does two things:

• reset work (allowing io_service::run() to complete)
• stop the service

The problem is, resetting work doesn't affect any work already in progress. Neither does stop . So, any work already posted will be completed.

The only situation in which you get your "expected" behaviour is in the off chance that you stop the io_service even before the first thread in the pool started running one of the posted tasks.

The fact that you seem to get this behaviour on GCC consistently is a fluke. In reality it's a race, and that's easily demonstrated by adding the tiniest delay before cancel() .

In fact, even with the code as is, repeated runs showed spurious failures like:

checkWorkCanBeCancelled
    Error: functor 1 call expected: false actual: true
    passed (functor 2 call).
    Error: running time expected: 150 actual: 200

Enabling handler tracking confirms that only one handler was abandoned:

Even just enabling address sanitizer made the failure become pretty much always reproducible.

Digging Deeper: io_service::stop

If it's a race, and the tests are limited to a single service thread, surely the subtle race window would not be large enough to allow the second task to run even though io_service::stop() had been called?

I struggled with this one, so I instrumented the Functor function with some more timing sampling. Let's also record the invocation of a task so we can distinguish tasks that were never started from those that merely didn't complete:

struct Functor {
    void operator()() {
        {
            boost::lock_guard<boost::mutex> lock(mutex);
            state_.invocation = Clock::now();
        }
        boost::this_thread::sleep(boost::posix_time::milliseconds(200));
        {
            boost::lock_guard<boost::mutex> lock(mutex);
            state_.completion = Clock::now();
        }
    }

    struct State {
        TP start, invocation, completion;

        friend std::ostream& operator<<(std::ostream& os, State const& s) {
            return os << "[" << relative(s.invocation, s.start) << "," << relative(s.completion, s.start) << "]";
        }
    };

    State sample() const {
        boost::lock_guard<boost::mutex> lock(mutex);
        return state_;
    }

  private:
    State state_ = { TP::min(), TP::min(), TP::min() };
    mutable boost::mutex mutex;
};


struct Functor {
    void operator()() {
        {
            boost::lock_guard<boost::mutex> lock(mutex);
            state_.wasInvoked = true;
        }
        boost::this_thread::sleep(boost::posix_time::milliseconds(200));
        {
            boost::lock_guard<boost::mutex> lock(mutex);
            state_.wasCompleted = true;
        }
    }

    struct State {
        bool wasInvoked, wasCompleted;

        friend std::ostream& operator<<(std::ostream& os, State const& s) {
            if (s.wasInvoked && s.wasCompleted) return os << "[invoked,completed]";
            if (s.wasInvoked) return os << "[invoked]";
            assert(!s.wasCompleted);
            return os << "[]";
        }
    };

    State sample() const {
        boost::lock_guard<boost::mutex> lock(mutex);
        return state_;
    }

  private:
    State state_ = { false, false };
    mutable boost::mutex mutex;
};

Now, runTests can be extended to keep timings of all tasks as well as those of the actions/pool shutdown:

struct ThreadPoolTest {
    boost::int_least64_t getRunningTimeInMS() const { return relative(Clock::now(), start); }

    template <typename FunctorType, typename ScenarioAction>
    void runTest(ScenarioAction action, bool shouldFunctor1BeCalled, bool shouldFunctor2BeCalled) {
        struct Task {
            std::string name;
            Task(std::string name) : name(name) {}

            FunctorType functor;
            Functor::State before, after, final_;
        } tasks[] = { {"functor1"}, {"functor2"} };

        TP before_action, after_action, pool_final;

        {
            ThreadPool pool(1);
            for (auto& task : tasks) pool.post(std::ref(task.functor));

            for (auto& task : tasks) task.before = task.functor.sample();

            before_action = Clock::now();
            action(pool);
            after_action = Clock::now();

            for (auto& task : tasks) task.after = task.functor.sample();
        }

        pool_final = Clock::now();
        for (auto& task : tasks) task.final_ = task.functor.sample();

        // aids in pretty printing
        for (auto& task : tasks) for (auto sample : { &Task::before, &Task::after, &Task::final_ }) {
            (task.*sample).start = start;
        }

        for (auto& task : tasks)
            std::cout << "DEBUG: " << task.name << " before:" << task.before << " after:" << task.after << " final:" << task.final_ << "\n";
        std::cout << "DEBUG: pool/action before:" << relative(before_action, start) << " after:" << relative(after_action, start) << " final:" << relative(pool_final, start) << "\n";

        check(std::equal_to<>{}, shouldFunctor1BeCalled, is_set(tasks[0].final_.completion), "functor 1 call");
        check(std::equal_to<>{}, shouldFunctor2BeCalled, is_set(tasks[1].final_.completion), "functor 2 call");
    }

  private:
    TP start = Clock::now();
};

And our GCC run prints:

checkWorkCanBeCancelled
DEBUG: functor1 before:[-1,-1] after:[-1,-1] final:[0,200]
DEBUG: functor2 before:[-1,-1] after:[-1,-1] final:[-1,-1]
DEBUG: pool/action before:0 after:0 final:200
    Error: functor 1 call expected: false actual: true
    passed (functor 2 call).
    Error: running time expected: 150 actual: 200

It shows that the action function ( cancel in this case) got invoked at the same time that our service thread was invoking functor1 . And functor2 was never invoked.

On MSVC , the same test prints:

checkWorkCanBeCancelled
DEBUG: functor1 before:[-1,-1] after:[-1,-1] final:[2,198]
DEBUG: functor2 before:[-1,-1] after:[-1,-1] final:[198,401]
DEBUG: pool/action before:0 after:0 final:404
    Error: functor 1 call expected: false actual: true
    Error: functor 2 call expected: false actual: true
    Error: running time expected: 150 actual: 405

Like with GCC, the cancel action ran at time 0ms, but strangely both tasks completed even though they got invoked after that action ran.

This indicates that on Windows, Asio will not prevent existing tasks being dispatched to threads if io_service::stop() is invoked. Increasing the load to 9 tasks shows consistent results:

 DEBUG: functor1 before:[-1,-1] after:[-1,-1] final:[2,195] DEBUG: functor2 before:[-1,-1] after:[-1,-1] final:[195,398] DEBUG: functor3 before:[-1,-1] after:[-1,-1] final:[399,602] DEBUG: functor4 before:[-1,-1] after:[-1,-1] final:[602,821] DEBUG: functor5 before:[-1,-1] after:[-1,-1] final:[821,1024] DEBUG: functor6 before:[-1,-1] after:[-1,-1] final:[1024,1228] DEBUG: functor7 before:[-1,-1] after:[-1,-1] final:[1228,1431] DEBUG: functor8 before:[-1,-1] after:[-1,-1] final:[1431,1634] DEBUG: functor9 before:[-1,-1] after:[-1,-1] final:[1634,1837] DEBUG: pool/action before:0 after:0 final:1838 

Interrupt

Interruptions work fine on Linux/GCC and MSVC .

Uninterruptables

The last scenario is effectively identical to the second (because the tasks are immune to interruption).

Summary:

On windows, the behaviour of io_service::stop() contradicts the documentation, so that would be a bug:

Here's a much simpler reproducer:

#include <boost/asio.hpp>
#include <thread>
#include <iostream>
using namespace std::chrono_literals;

int main() {
    boost::asio::io_service s;

    s.post([] { std::this_thread::sleep_for(5ms); std::cout << "1\n"; });
    s.post([] { std::this_thread::sleep_for(5ms); std::cout << "2\n"; });
    s.post([] { std::this_thread::sleep_for(5ms); std::cout << "3\n"; });

    std::thread th([&] { s.run(); });

    std::this_thread::sleep_for(1ms);
    s.stop();

    th.join();
}

Which prints 1 on GCC and 1 2 3 on MSVC.