如何从csv文件中的一行读取JSON字符串？

Question

I'm new to MapReduce and MRjob, I am trying to read a csv file that I want to process using MRjob in python. But it has about 5 columns with JSON strings(eg. {}) or an array of JSON strings (eg. [{},{}]), some of them are nested.

My mapper so far looks as follows:

from mrjob.job import MRJob
import csv
from io import StringIO

class MRWordCount(MRJob):
    def mapper(self, _, line):
        l = StringIO(line)
        reader = csv.reader(l) # returns a generator.

        for cols in reader:
            columns = cols

        yield None, columns

I get the error -

_csv.Error: field larger than field limit (131072)

But that seems to happen because my code separates the JSON strings into separate columns as well (because of the commas inside).

How do I make this, so that the JSON strings are not split? Maybe I'm overlooking something?

Alternatively, is there any other ways I could read this file with MRjob that would make this process simpler or cleaner?

Answer 1

Your JSON string is not surrounded by quote characters so every comma in that field makes the csv engine think its a new column. take a look here what you are looking for is quotechar change your data so that you json is surrounded with a special character (The default is " ) and adjust your csv reader accordingly