R并行写SEXP结构

Question

I am working on a data processing module in R using C/C++ code, mainly for speed reasons. Here is a list of facts of my problem.

• The final outcome data is a list of string vectors and takes between 20MB to 200MB of memory.
• The data processing can be fit into single-producer/multiple-consumer model.
• It takes significant amount of time by wrap to convert vector<vector<string> > to List for my data.

Therefore I intend to work directly in SEXP structures, by which I could possibly save the time for the final conversion. My main function looks like this.

boost::atomic<bool> done(false);
SEXP myfun(...) {
    ...
    SEXP sdataStr;
    PROTECT(sdataStr=allocVector(VECSXP, nElem));
    vector<SEXP> dataStr(nElem);
    for (int i=0; i<nElem; ++i) {
         dataStr[i]=SET_VECTOR_ELT(sdataStr, i, allocVector(STRSXP, n));
    }
    Producer producer(&queue);
    Consumer consumer1(dataStr, nElem, &queue);
    Consumer consumer2(dataStr, nElem, &queue);

    boost::thread produce(producer);
    boost::thread consume1(consumer1);
    boost::thread consume2(consumer2);

    produce.join();
    done=true;
    consume1.join();
    consume2.join();
    UNPROTECT(1);
    return sdataStr;
}

My consumer class looks like this

class Consumer {
    vector<SEXP>& m_dataStr;
    boost::lockfree::queue<buffer>* m_queue;
    buffer m_buffer;

    public:
    Consumer(vector<SEXP>& dataStr, boost::lockfree::queue<buffer>* queue) : m_dataStr(dataStr), m_queue(queue) {}

    void operator()() {
        while (!done) {
            while (m_queue->pop(m_buffer)) {
                process_item();
            }
        }
        while (m_queue->pop(m_buffer)) {
            process_item();
        }
    }

    private:
    process_item() {
        ...
        // for some 0<=idx<nElem, 0<=i<n, some char* f and integer len
        SET_STRING_ELT(m_dataStr[idx], i, mkCharLen(f,len));
        ...
    }
}

These are the only places I use Rinternals. The logic of the program ensures that writing to the same place by different threads never happens, ie the idx and i combination in Consumer class can at most occur once. I encountered various strange problems, such as "stack imbalance", or "snapping into wrong generation", and etc. Is there something I am missing? Or calling SET_STRING_ELT in multiple threads is not recommended? Thank you very much!

Answer 1

C/R API functions should not be called in threads unless you know what you are doing, for example mkCharLen might modify the internal hash table that is used for all R strings, so you can't call this in a thread. SET_STRING_ELT is probably also not useable in a thread, especially if the write barrier is on.