Write to dataframe with %dopar% in foreach

Question

I want to use a foreach each loop running with a doParallel backend for getting tweets from a MySQL database with the RMySql package.

I create a connection to the database for every user id I want to query, then I get every tweet from that user by 200 batches. If the batch size is 0 (so there are no further tweets) I query next user id.

I want to store the information in a dataframe called tweets, which has columns for the number of hashtags in a tweet and a column with dates. For every tweet I want to find out how many hashtags it has and in which month it was created. Then I want to increase the number in the dataframe by 1.

So how can I write the results for every tweet in the dataframe?

My dataframe in the beginning:

| dates    | zero_ht | one_ht | two_ht | three_ht | four_ht | five_ht |
|----------|---------|--------|--------|----------|---------|---------|
| 01/01/13 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/02/13 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/03/13 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/04/13 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/05/13 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/06/13 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/07/13 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/08/13 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/09/13 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/10/13 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/11/13 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/12/13 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/01/14 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/02/14 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/03/14 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/04/14 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/05/14 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/06/14 | 0       | 0      | 0      | 0        | 0       | 0       |
| 01/07/14 | 0       | 0      | 0      | 0        | 0       | 0       |

My code:

x<- foreach(i=1:nrow(ids) ,.packages=c("DBI", "RMySQL"),.combine=rbind ) %dopar% {

con <- dbConnect(MySQL(), *CREDENTIALS*)

start <- 0

length <- 1
while(length > 0)
{
query <- *QUERY*
data <- dbGetQuery(con, query)

length <- nrow(data)

#print(paste("Starting at ",start,sep=""))

for(j in 1:length)
{   
    if(length==0)
    {

    }
    else{ 

    #get the number of hashtags used
    number <-   nchar((gsub("[^#]","",data$message[j])))

    #get the date the tweet was created
    date <- paste(format(as.Date(data$created_at[j]), "%Y-%m"),"-01",sep="")
    # just use it when there are less than 5 hashtags
    if(number < 5)
    {

        if(number==0)
        {


        tweets[tweets$dates==date,2] <- tweets[tweets$dates==date,2]+1


        }
        else{
            tweets[tweets$dates==date,number+1] <- tweets[tweets$dates==date,number+1]+1


        }

    }

}    
}
#increase the start by 200; to get the next 200 tweets
start <- start + 200

}
data.frame(date=date,number=number)
dbDisconnect(con) 
}

Answer 1

Thanks to the comments I could solve the problem: The reason for the list with just "TRUE"s in it, was that the last command in the foreach loop was

dbDisconnect(con) 

And when the database connection was closed successfully it returns a "TRUE".

So I just had to swap the last two lines and make

data.frame(date=date,number=number)

and everything worked fine.

Regards