从CSV文件中删除新行

Question

我想删除CSV文件字段数据中的新行字符。 SO /其他地方的多个人也会问同样的问题。 但是，提供的解决方案是脚本编写的。 我正在寻找像PYTHON这样的编程语言或Spark中的解决方案（不仅仅是这两个），因为我有很大的文件。

以前就同一主题提出的问题：

• 从CSV文件的字符串列中删除新行字符

• 用空格替换双引号之间的换行符

• 从CSV文件的字符串列中删除新行

• https://unix.stackexchange.com/questions/222049/how-to-detect-and-remove-newline-character-within-a-column-in-a-csv-file

我有一个大小约为1GB的CSV文件，想要删除字段数据中的新行字符。 CSV文件的架构动态变化，因此我无法对架构进行硬编码。 换行符并不总是出现在逗号之前，即使在字段中也会随机出现。

样本数据：

playerID,yearID,gameNum,gameName,teamName,lgID,GP,startingPos
gomezle01,1933,1,Cricket,Team1,NYA,AL,1
ferreri01,1933,2,Hockey,"This is 
Team2",BOS,AL,1
gehrilo01,1933,3,"Game name is 
Cricket" 
,Team3,NYA,AL,1
gehrich01,1933,4,Hockey,"Here it is 
Team4",DET,AL,1
dykesji01,1933,5,"Game name is 
Hockey"
,"Team name 
Team5",CHA,AL,1

预期产出：

playerID,yearID,gameNum,gameName,teamName,lgID,GP,startingPos
gomezle01,1933,1,Cricket,Team1,NYA,AL,1
ferreri01,1933,2,Hockey,"This is Team2",BOS,AL,1
gehrilo01,1933,3,"Game name is Cricket" ,Team3,NYA,AL,1
gehrich01,1933,4,Hockey,"Here it is Team4",DET,AL,1
dykesji01,1933,5,"Game name is Hockey","Team name Team5",CHA,AL,1

换行符可以在任何字段的数据中。

编辑：根据代码截图：

Answer 1

如果您正在使用pyspark，那么我建议您使用sparkContext的wholeTextFiles函数来读取文件， 因为您的文件需要作为整个文本读取以进行适当的解析 。

在使用wholeTextFiles读取它wholeTextFiles ，您应该通过替换行尾字符来解析，并执行一些额外的格式化，以便将整个文本分解为八个字符串的组。

import re
rdd = sc.wholeTextFiles("path to your csv file")\
    .map(lambda x: re.sub(r'(?!(([^"]*"){2})*[^"]*$),', ' ', x[1].replace("\r\n", ",").replace(",,", ",")).split(","))\
    .flatMap(lambda x: [x[k:k+8] for k in range(0, len(x), 8)])

你应该得到输出

[u'playerID', u'yearID', u'gameNum', u'gameName', u'teamName', u'lgID', u'GP', u'startingPos']
[u'gomezle01', u'1933', u'1', u'Cricket', u'Team1', u'NYA', u'AL', u'1']
[u'ferreri01', u'1933', u'2', u'Hockey', u'"This is Team2"', u'BOS', u'AL', u'1']
[u'gehrilo01', u'1933', u'3', u'"Game name is Cricket"', u'Team3', u'NYA', u'AL', u'1']
[u'gehrich01', u'1933', u'4', u'Hockey', u'"Here it is Team4"', u'DET', u'AL', u'1']
[u'dykesji01', u'1933', u'5', u'"Game name is Hockey"', u'"Team name Team5"', u'CHA', u'AL', u'1']

如果您想将所有数组rdd行转换为行的字符串，则可以添加

.map(lambda x: ", ".join(x))

你应该得到

playerID, yearID, gameNum, gameName, teamName, lgID, GP, startingPos
gomezle01, 1933, 1, Cricket, Team1, NYA, AL, 1
ferreri01, 1933, 2, Hockey, "This is Team2", BOS, AL, 1
gehrilo01, 1933, 3, "Game name is Cricket", Team3, NYA, AL, 1
gehrich01, 1933, 4, Hockey, "Here it is Team4", DET, AL, 1
dykesji01, 1933, 5, "Game name is Hockey", "Team name Team5", CHA, AL, 1

Answer 2

您可以使用re ， pandas和io模块，如下所示：

import re
import io
import pandas as pd

with open('data.csv','r') as f:
    data = f.read()
df = pd.read_csv(io.StringIO(re.sub('"\s*\n','"',data)))

for col in df.columns: #To replace all line breaks in all textual columns
    if df[col].dtype == np.object_:
        df[col] = df[col].str.replace('\n','');

In [78]: df
Out[78]:
    playerID    yearID  gameNum gameName               teamName        lgID GP  startingPos
0   gomezle01   1933    1       Cricket                Team1           NYA  AL  1
1   ferreri01   1933    2       Hockey                 This is Team2   BOS  AL  1
2   gehrilo01   1933    3       Game name is Cricket   Team3           NYA  AL  1
3   gehrich01   1933    4       Hockey  Here it is     Team4           DET  AL  1
4   dykesji01   1933    5       Game name is Hockey    Team name Team5 CHA  AL  1

如果您希望此DataFrame作为输出CSV文件使用：

df.to_csv('./output.csv')

Answer 3

这是一个基本的，在通过csv读取之前进行简单的预处理。

import csv

def simple_sanitize(data):
    result = []
    for i, a in enumerate(data):
        if i + 1 != len(data) and data[i + 1][0] == ',':
            a = a.replace('\n', '')
            result.append(a + data[i + 1])
        elif a[0] != ',':
            result.append(a)
    return result

data = [line for line in open('test.csv', 'r')]
sdata = simple_sanitize(data)

with open('out.csv','w') as f:
    for row in sdata:
        f.write(row)

result = [list(val.replace('\n', '') for val in line) for line in csv.reader(open('out.csv', 'r'))]

print(result)

结果：

[['playerID', 'yearID', 'gameNum', 'gameName', 'teamName', 'lgID', 'GP', 'startingPos'], 
['gomezle01', '1933', '1', 'Cricket', 'Team1', 'NYA', 'AL', '1'], 
['ferreri01', '1933', '2', 'Hockey', 'This is Team2', 'BOS', 'AL', '1'], 
['gehrilo01', '1933', '3', 'Game name is Cricket ', 'Team3', 'NYA', 'AL', '1'], 
['gehrich01', '1933', '4', 'Hockey', 'Here it is Team4', 'DET', 'AL', '1'], 
['dykesji01', '1933', '5', 'Game name is Hockey', 'Team name Team5', 'CHA', 'AL', '1']]

Answer 4

它可以使用一点清洁，但这里有一些代码可以做你想要的。 适用于字段内和逗号之前的换行符。 如果需要更多要求，可以进行一些调整：

import csv

with open('data.csv', 'r') as csvfile:
    reader = csv.reader(csvfile, delimiter=',', quotechar='"')
    actual_rows = [next(reader)]
    length = len(actual_rows[0])
    real_row = []
    for row in reader:
        if len(row) < length:
            if real_row:
                real_row[-1] += row[0]
                real_row += row[1:]
            else:
                real_row = row
        else:
            real_row = row
        if len(real_row) == length:
            real_row = map(lambda s: s.replace('\n', ' '), real_row)
            # store real_row or use it as needed
            actual_rows.append(list(real_row))
            real_row = []

    print(actual_rows)

我将更正的行存储在actual_rows但如果您不想加载到内存中，只需在注释中指出每个循环中使用real_row变量

Answer 5

此解决方案的基本思想是使用石斑鱼配方获得固定长度的块（长度等于第一行中的列数）。 由于它不会立即读取整个文件，因此不会破坏大文件的内存使用量。

$ cat a.py
import csv,itertools as it,operator as op

def grouper(iterable,n):return it.zip_longest(*[iter(iterable)]*n)

with open('in.csv') as inf,open('out.csv','w',newline='') as outf:
 r,w=csv.reader(inf),csv.writer(outf)
 hdr=next(r)
 w.writerow(hdr)
 for row in grouper(filter(bool,map(op.methodcaller('replace','\n',''),it.chain.from_iterable(r))),len(hdr)):
  w.writerow(row)

$ python3 a.py
$ cat out.csv
playerID,yearID,gameNum,gameName,teamName,lgID,GP,startingPos
gomezle01,1933,1,Cricket,Team1,NYA,AL,1
ferreri01,1933,2,Hockey,This is Team2,BOS,AL,1
gehrilo01,1933,3,Game name is Cricket ,Team3,NYA,AL,1
gehrich01,1933,4,Hockey,Here it is Team4,DET,AL,1
dykesji01,1933,5,Game name is Hockey,Team name Team5,CHA,AL,1

这里做出的一个假设是输入csv中没有空单元。