拆分用R和Quanteda标记语料库

Question

I am working on a project for NLP. I need to take some blogs, news and tweets (you have probably heard of this capstone already) in .txt files and create n-grams frequencies.

I did experiments on the steps to take the txt files to a frequencies data frame for analysis:

Read > Conver to corpus > Clean corpus > Tokenize > Convert to dfm > Convert to df

The bottle necks in the process were the tokenize and convert to dfm steps (over 5x more time).

I had two choices:

1. Split the cleaned corpus to tokenize by piece
2. Split-read the .txt files from the beginning

No. 1 seemed the best, but so far I have not found a function or package that can do this in a way I want. So I will write a long code to split-read from the beginning in 20 chunks (due to my computing constraints).

Is there a way I can split a corpus ("corpus" "list") created with the quanteda package in chunks (defined lines by me) so I can tokenize and turn to dfm in a "streaming" kinda way?

Answer 1

I think the package you will find most useful currently is the tm package. It is a pretty complex but thorough package even though its still in an experimental state at version .7.1. Without more detail I can't give you more exact usage info because it all depends on your sources, how you want to process the corpus and other factors. The gist of what you'll need to do is first create a reader object dependant on your source material. It can handle web input, plain texts, pdf and others. Then you can use one of the Corpus creation functions depending on whether you want to keep the whole thing in memory etc. You can then use the various 'tidying' functions to operate on the entire corpus as though each document were an element in a vector. You can do the same with tokenizing. With a few more specifics we can give you more specific answers.

Answer 2

Since this question hasn't been directly answered, I am reposting related content from the article I wrote in 2016 as a Community Mentor for the JHU Capstone, Capstone n-grams: how much processing power is required?

Overview

Students in the Johns Hopkins University Data Science Specialization Capstone course typically struggle with the course project because of the amount of memory consumed by the objects needed to analyze text. The question asks about the best approach for processing the 4+ million documents that are in the raw data files. The short answer to this question is that it depends on the amount of RAM on one's machine. Since R objects must reside in RAM, one must understand the amount of RAM consumed by the objects being processed.

A machine with 16Gb of RAM is required to process all of the data from the three files without processing it in smaller chunks or processing a random sample of data. My testing indicates that the working memory needed to process the files is approximately 1.5 - 3 times the size of the object output by the quanteda::tokens_ngrams() function from quanteda version 0.99.22, and therefore a 1 Gb tokenized corpus and consumes 9 Gb of RAM to generate a 4 Gb n-gram object. Note that quanteda automatically uses multiple threads if your computer has multiple cores / threads.

To help reduce the guesswork in the memory utilization, here is a summary of the amount of RAM consumed by objects required to analyze the files for the Swift Key sponsored capstone: predicting text.

Raw data

There are three raw data files used in the Capstone project. Once loaded into memory using a text processing function such as readLines() or readr::read_lines() , the resulting object sizes are as follows.

1. en_US.blogs.txt: 249 Mb
2. en_US.news.txt: 250 Mb
3. en_US.twitter.txt: 301 Mb

These files must be joined into a single object and converted to a corpus. Together they consume about 800 Mb of RAM.

When converted to a corpus with quanteda::corpus() the resulting file size is 1.1 Gb in size.

N-gram object sizes

To maximize the amount of RAM available for n-gram processing, once the corpus is generated, one must remove all objects from memory other than the tokenized corpus used as input to tokens_ngrams() . The object sizes for various n-grams is as follows.

1. 2-grams: 6.3 Gb
2. 3-grams: 6.5 Gb
3. 4-grams: 6.5 Gb
4. 5-grams: 6.3 Gb
5. 6-grams: 6.1 Gb

Working with less memory

I was able to process a 25% sample of the capstone data on a MacBook Pro with 8 Gb of RAM, and a 5% sample on an HP Chromebook running Ubuntu Linux with 4 Gb of RAM. Adding to Ken Benoit's comment to the original question, one can assign a numeric group (eg repeating IDs of 1 - 20 to split to 20 groups) and then use the corpus_segment() function to split the corpus on group ID. Then the groups can be processed individually via an apply() function to generate n-grams. A general process to generate all of the required n-grams is represented in the following pseudocode.

 for each group in the corpus
     for each size n-gram 
           1. generate n-grams
           2. write to file
           3. rm() n-gram object

This pseudocode can be implemented with a couple of apply() functions.

Object sizes: 25% sample

1. 2-grams: 2.0 Gb
2. 3-grams: 2.9 Gb
3. 4-grams: 3.6 Gb
4. 5-grams: 3.9 Gb
5. 6-grams: 4.0 Gb

Object sizes: 5% sample

1. 2-grams: 492 Mb
2. 3-grams: 649 Mb
3. 4-grams: 740 Mb
4. 5-grams: 747 Gb
5. 6-grams: 733 Gb