[英]How to create a pandas dataframe from a txt file with comments?
I need to create a pandas dataframe based on 4 txt files with comments (to skip while reading) based on the following structure:我需要创建一个 pandas dataframe 基于 4 个带有评论的 txt 文件(阅读时跳过)基于以下结构:
# Moteur conçu par le Poly Propulsion Lab (PPL)
nom=Tondeuse
# Propriétés générales
hauteur=0.5
masse=20.0
prix=110.00
# Propriétés du moteur
impulsion specifique=80
and和
# Moteur conçu par le Poly Propulsion Lab (PPL)
nom=Civic VTEC
# Propriétés générales
hauteur=2.0
masse=3000.0
prix=2968.00
# Propriétés du moteur
impulsion specifique=205
and和
# Moteur conçu par le Poly Propulsion Lab (PPL)
nom=VelociRAPTOR
# Propriétés générales
hauteur=4.0
masse=2000.0
prix=6000.00
# Propriétés du moteur
impulsion specifique=250
and和
# Moteur conçu par le Poly Propulsion Lab (PPL)
nom=La Puissance
# Propriétés générales
hauteur=12.0
masse=15000.0
prix=39000.00
# Propriétés du moteur
impulsion specifique=295
That's the result I need to have:这就是我需要的结果:
nom hauteur masse prix impulsion specifique
0 Tondeuse 0.5 20.0 110.0 80
1 Civic VTEC 2.0 3000.0 2968.0 205
2 VelociRAPTOR 4.0 2000.0 6000.0 250
3 La Puissance 12.0 15000.0 39000.0 295
I don't know if it's possible, but that's what i was asked to do我不知道这是否可能,但这就是我被要求做的
Your data files look very close to configuration files.您的数据文件看起来非常接近配置文件。 You can use configparser to generate a dictionary from each file:您可以使用configparser从每个文件生成字典:
from pathlib import Path
from configparser import ConfigParser
data = []
for file in Path("data").glob("*.txt"):
parser = ConfigParser()
# INI file requires a section header. Yours don't have one.
# So let's give it one called DEFAULT
parser.read_string("[DEFAULT]\n" + file.read_text())
data.append(dict(parser.items("DEFAULT")))
df = pd.DataFrame(data)
welcome to Stackoverflow: :)欢迎来到 Stackoverflow::)
If your txt files have their content like you just showed, you could read them in using pandas as a CSV file.如果您的 txt 文件具有您刚才显示的内容,您可以使用 pandas 作为 CSV 文件读取它们。
The pandas.read_csv function has multiple things that will help you here: pandas.read_csv function 有很多东西可以帮助你:
comment
input argument, with which you can define lines that are to be ignored有一个comment
输入参数,您可以使用它定义要忽略的行=
sign as a separator, which will make you able to split up your data in the wanted sections您可以使用=
号作为分隔符,这样您就可以将数据拆分到所需的部分Now, let's try to read one of your files using the read_csv
function:现在,让我们尝试使用read_csv
function 读取您的文件之一:
import pandas as pd
df = pd.read_csv(file, comment='#', sep='=', header=None)
df
nom Tondeuse
0 hauteur 0.5
1 masse 20.0
2 prix 110.0
3 impulsion specifique 80.0
We're not completely there yet.我们还没有完全做到这一点。 We want to remove that index column that gives no info, and we want to transpose the dataframe (rows <-> columns) to be able to concatenate all dataframes together.我们想要删除没有提供任何信息的索引列,并且我们想要转置 dataframe(行 <-> 列)以便能够将所有数据帧连接在一起。 Let's do it!我们开始做吧!
import pandas as pd
df = pd.read_csv(file, comment='#', sep='=', header=None, index_col=0).T
df
0 nom hauteur masse prix impulsion specifique
1 Tondeuse 0.5 20.0 110.00 80
That's looking way better!这样看起来好多了! Putting index_col=0
makes the lefternmost column be the index column, and the .T
at the end transposes your dataframe. Now we just need to put this inside of a loop and make a complete script out of it!设置index_col=0
使最左边的列成为索引列,最后的.T
转置您的 dataframe。现在我们只需要将它放在一个循环中并从中制作一个完整的脚本!
import pandas as pd
import glob
import os
files = glob.glob(os.path.join(path, '*.csv'))
all_dfs = []
for file in files:
current_df = pd.read_csv(file, comment='#', sep='=', header=None, index_col=0).T
all_dfs.append(current_df)
total_df = pd.concat(all_dfs)
total_df
0 nom hauteur masse prix impulsion specifique
1 La Puissance 12.0 15000.0 39000.00 295
1 Civic VTEC 2.0 3000.0 2968.00 205
1 VelociRAPTOR 4.0 2000.0 6000.00 250
1 Tondeuse 0.5 20.0 110.00 80
Notice that you still have that lefternmost column with the index number, I did not clean it out because I wasn't sure of what you wanted there.请注意,您仍然有最左边的带有索引号的列,我没有清除它,因为我不确定您在那里想要什么。
Also, importantly, you need to be aware that if there is a slight difference in the names of the columns in your files (eg impulsion specifique
vs impulsion spécifique
) this will bring errors.此外,重要的是,您需要注意,如果文件中列的名称略有不同(例如impulsion specifique
与impulsion spécifique
),这将带来错误。 You will need to create error handling procedures for these.您将需要为这些创建错误处理程序。 Or maybe enforcing a certain schema, but that is out of the scope of this question.或者可能强制执行某种模式,但这超出了这个问题的 scope。
I hope this helps!我希望这有帮助!
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.