将 tsv 文件转换为 pandas dataframe
[英]transform a tsv file into a pandas dataframe
问题描述
自上周以来，我的 tsv 遇到了一个问题，我想修改并转换为 pandas dataframe。
我的文件如下所示：
'NC_011745.1_islands.csv': [['PAI 1 EaaA, EibA : 3.1'],
['PAI 2 EaaA : 7.75'],
['PAI 3 Capsule : 4.428571428571429'],
['PAI 4 EaaA : 7.75'],
['PAI 5 ipaH : 7.75'],
['PAI 6 IreA, IrgA homolog adhesin (Iha) : '
'0.96875'],
['PAI 7 IrgA homolog adhesin (Iha), Aerobactin : '
'0.8157894736842105'],
['PAI 8 MsbB2, VirK : 2.8181818181818183'],
['PAI 9 Antigen 43, AIDA-I type : '
'1.3478260869565217']],
'NC_017632_islands.csv': [['PAI 1 Capsule : 15.857142857142858'],
['PAI 2 AAI/SCI-II, direct heme uptake system, '
'Colibactin, Colibactin : 1.819672131147541'],
['PAI 3 F9-like fimbriae, Type 1 fimbriae : '
'3.3636363636363638'],
['PAI 4 Ferrous iron transport : 5.045454545454546'],
['PAI 5 Cah, AIDA-I type, Salmochelin, S fimbriae : '
'2.707317073170732'],
['PAI 6 ECP, Tsh : 13.875'],
['PAI 7 ACE/AEC T6SS : 9.25'],
['PAI 8 Tia/Hek, P fimbriae, F17-like fimbriae, '
'AAI/SCI-II, CNF-1, Alpha-hemolysin, '
'hemagglutinin-like adhesin : 1.088235294117647']],
'NC_017646_islands.csv': [['PAI 1 Allantion utilization : 5.285714285714286'],
['PAI 2 direct heme uptake system : 4.44'],
['PAI 3 ipaH : 27.75'],
['PAI 4 P fimbriae, Aerobactin, Sat, IrgA homolog '
'adhesin (Iha), K1 capsule, K1 capsule, T2SS : '
'1.3058823529411765'],
['PAI 5 P fimbriae, Tia/Hek : 5.842105263157895'],
['PAI 6 VirK, MsbB2 : 10.090909090909092']]}
我想将其修改并导出为 pandas dataframe ，如下所示：
\ EaaA, EibA EaaA Capsule ipaH IreA, IrgA homolog adhesin (Iha) ...
NC_011745.1 3.1 7.75 4.4285.. 7.75 0.96875
NC_017632 NA NA 15.8574 NA NA
对我来说主要问题是把它写成 dataframe，我试过：
df = pd.DataFrame([dict]).T
df.to_tsv()
但它说这个功能不适用于 tsv 而是 csv
2 个解决方案
解决方案1
1 已采纳 2020-06-12 14:52:52
您不能使用 pandas 开箱即用地执行此操作 - pandas 很好，但这不是魔术。 在您的数据准备好用于所需格式的 dataframe 之前，您需要进行大量操作。 尝试这样的事情：
_dict={'NC_011745.1_islands.csv': [['PAI 1 EaaA, EibA : 3.1'],
['PAI 2 EaaA : 7.75'],
['PAI 3 Capsule : 4.428571428571429'],
['PAI 4 EaaA : 7.75'],
['PAI 5 ipaH : 7.75'],
['PAI 6 IreA, IrgA homolog adhesin (Iha) : '
'0.96875'],
['PAI 7 IrgA homolog adhesin (Iha), Aerobactin : '
'0.8157894736842105'],
['PAI 8 MsbB2, VirK : 2.8181818181818183'],
['PAI 9 Antigen 43, AIDA-I type : '
'1.3478260869565217']],
'NC_017632_islands.csv': [['PAI 1 Capsule : 15.857142857142858'],
['PAI 2 AAI/SCI-II, direct heme uptake system, '
'Colibactin, Colibactin : 1.819672131147541'],
['PAI 3 F9-like fimbriae, Type 1 fimbriae : '
'3.3636363636363638'],
['PAI 4 Ferrous iron transport : 5.045454545454546'],
['PAI 5 Cah, AIDA-I type, Salmochelin, S fimbriae : '
'2.707317073170732'],
['PAI 6 ECP, Tsh : 13.875'],
['PAI 7 ACE/AEC T6SS : 9.25'],
['PAI 8 Tia/Hek, P fimbriae, F17-like fimbriae, '
'AAI/SCI-II, CNF-1, Alpha-hemolysin, '
'hemagglutinin-like adhesin : 1.088235294117647']],
'NC_017646_islands.csv': [['PAI 1 Allantion utilization : 5.285714285714286'],
['PAI 2 direct heme uptake system : 4.44'],
['PAI 3 ipaH : 27.75'],
['PAI 4 P fimbriae, Aerobactin, Sat, IrgA homolog '
'adhesin (Iha), K1 capsule, K1 capsule, T2SS : '
'1.3058823529411765'],
['PAI 5 P fimbriae, Tia/Hek : 5.842105263157895'],
['PAI 6 VirK, MsbB2 : 10.090909090909092']]}
f = {}
for key, a in _dict.items():
e = {}
for b in a:
for c in b:
d = c.split(" : ")
d[0] = d[0].replace("PAI ", "")[2:]
d = {d[0]:d[1]}
e = {**e, **d}
f[key] = e
df = pd.DataFrame.from_dict(f, 'index')
您需要制定一个强大的方法来解析您的字符串 - 可能是正则表达式 - 但这应该可以帮助您入门。
解决方案2
0 2020-06-13 05:22:02
循环答案已被@bm13563 的字典格式接受。 我用“熊猫”回应。
- 由于数据是列表格式，因此创建一个数据框。
- 删除括号并按“：”拆分列。
- 垂直连接到一个空的数据框。
- 按文件名分组并格式化。
import pandas as pd
lst_a = [['PAI 1 EaaA, EibA : 3.1'],['PAI 2 EaaA : 7.75'],['PAI 3 Capsule : 4.428571428571429'],['PAI 4 EaaA : 7.75'],['PAI 5 ipaH : 7.75'],['PAI 6 IreA, IrgA homolog adhesin (Iha) : ' '0.96875'],['PAI 7 IrgA homolog adhesin (Iha), Aerobactin : ' '0.8157894736842105'],['PAI 8 MsbB2, VirK : 2.8181818181818183'],['PAI 9 Antigen 43, AIDA-I type : ' '1.3478260869565217']]
lst_b = [['PAI 1 Capsule : 15.857142857142858'],['PAI 2 AAI/SCI-II, direct heme uptake system, Colibactin, Colibactin : 1.819672131147541'],['PAI 3 F9-like fimbriae, Type 1 fimbriae : 3.3636363636363638'],['PAI 4 Ferrous iron transport : 5.045454545454546'],['PAI 5 Cah, AIDA-I type, Salmochelin, S fimbriae : 2.707317073170732'],['PAI 6 ECP, Tsh : 13.875'],['PAI 7 ACE/AEC T6SS : 9.25'],['PAI 8 Tia/Hek, P fimbriae, F17-like fimbriae, AAI/SCI-II, CNF-1, Alpha-hemolysin, hemagglutinin-like adhesin : 1.088235294117647']]
lst_c = [['PAI 1 Allantion utilization : 5.285714285714286'],['PAI 2 direct heme uptake system : 4.44'],['PAI 3 ipaH : 27.75'],['PAI 4 P fimbriae, Aerobactin, Sat, IrgA homolog adhesin (Iha), K1 capsule, K1 capsule, T2SS : 1.3058823529411765'],['PAI 5 P fimbriae, Tia/Hek : 5.842105263157895'],['PAI 6 VirK, MsbB2 : 10.090909090909092']]
all_df = pd.DataFrame(index=[], columns=['col_name', 'value', 'file_name'])
filenames = ['NC_011745.1','NC_017632','NC_017646']
d_lists = [lst_a,lst_b,lst_c]
for k in range(len(d_lists)):
df = pd.DataFrame({filenames[k]:d_lists[k]})
df = df.astype(str)
df[filenames[k]] = df[filenames[k]].str.replace("^\['|'\]$", "")
df = df[filenames[k]].str.split(' : ', expand=True)
df.columns = ['col_name','value']
df['col_name'] = df['col_name'].apply(lambda x: x[6:])
df['file_name'] = filenames[k]
all_df = all_df.append(df, ignore_index=True)
i += 1
all_df['value'] = all_df['value'].astype('float')
all_df.groupby(['file_name','col_name'])['value'].sum().unstack()
col_name AAI/SCI-II, direct heme uptake system, Colibactin, Colibactin ACE/AEC T6SS Allantion utilization Antigen 43, AIDA-I type Cah, AIDA-I type, Salmochelin, S fimbriae Capsule ECP, Tsh EaaA EaaA, EibA F9-like fimbriae, Type 1 fimbriae Ferrous iron transport IreA, IrgA homolog adhesin (Iha) IrgA homolog adhesin (Iha), Aerobactin MsbB2, VirK P fimbriae, Aerobactin, Sat, IrgA homolog adhesin (Iha), K1 capsule, K1 capsule, T2SS P fimbriae, Tia/Hek Tia/Hek, P fimbriae, F17-like fimbriae, AAI/SCI-II, CNF-1, Alpha-hemolysin, hemagglutinin-like adhesin VirK, MsbB2 direct heme uptake system ipaH
file_name
NC_011745.1 NaN NaN NaN 1.347826 NaN 4.428571 NaN 15.5 3.1 NaN NaN 0.96875 0.815789 2.818182 NaN NaN NaN NaN NaN 7.75
NC_017632 1.819672 9.25 NaN NaN 2.707317 15.857143 13.875 NaN NaN 3.363636 5.045455 NaN NaN NaN NaN NaN 1.088235 NaN NaN NaN
NC_017646 NaN NaN 5.285714 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 1.305882 5.842105 NaN 10.090909 4.44 27.75
将 pandas dataframe 从 tsv 转换为 parquet
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.