[英]reading a complicated CSV file with python
I didn't know how to title this question so if it should be renamed please inform me and I will. 我不知道如何为这个问题加上标题,因此,如果应该重命名,请通知我,我会的。
I'm reading a csv
file that I saved off from a piece of equipment I used in taking some measurements. 我正在读取一个
csv
文件,该文件是我从进行一些测量的设备中保存下来的。 The data and various other key information is all being saved off. 数据和其他各种关键信息都被保存下来。 I've worked all day long on this and I cannot figure out how to retrieve every piece of information from this file properly.
我整天都在为此工作,我无法弄清楚如何从该文件中正确检索每条信息。 I need to access each piece of data/information and after doing so plot the data and read in the other various information like the data/timestamp/timezone/modelnumber/serialnumber and so on... I apologize if this question is too generic but I'm at lost on how to resolve this.
我需要访问每条数据/信息,然后绘制数据并读取其他各种信息,例如data / timestamp / timezone / modelnumber / serialnumber等等。如果这个问题过于笼统,我深表歉意,但我不知道如何解决这个问题。
I've typed up several versions of code so I'll list only what I've been able to get to work. 我已经输入了多个版本的代码,所以我只列出了我可以使用的代码。 I have no idea why I have to use
sep=delimiter
. 我不知道为什么我必须使用
sep=delimiter
。 I would think delimiter=',' would work but it doesn't. 我认为delimiter =','可以工作,但是不行。 I've found from research that
header=none
because my file doesn't have headings. 我从研究中发现
header=none
因为我的文件没有标题。
Canopy is telling me that the engine 'C' will not work so as a result I specify 'python'. Canopy告诉我引擎“ C”将无法工作,因此我指定了“ python”。 From the output of this code it seems to capture everything.
从此代码的输出看来,它捕获了所有内容。 But its telling me I only have one column and have no idea how to separate out all this information.
但是它告诉我,我只有一栏,不知道如何分离所有这些信息。
Here is part of my csv file. 这是我的csv文件的一部分。
! FILETYPE CSV
! VERSION 1.0 1
! TIMESTAMP Friday 15 April 2016 04:50:05
! TIMEZONE (GMT+08:00) Kuala Lumpur Singapore
! NAME Keysight Technologies
! MODEL N9917A
! SERIAL US52240515
! FIRMWARE_VERSION A.08.05
! CORRECTION
! Application SA
! Trace TIMESTAMP: 2016-04-15 04:50:05Z
! Trace GPS Info...
! GPS Latitude:
! GPS Longitude:
! GPS Seconds Since Last Read: 0
! CHECKSUM 1559060681
! DATA Freq SA Max Hold SA Blank SA Blank SA Blank
! FREQ UNIT Hz
! DATA UNIT dBm
BEGIN
2000000000,-62.6893499803169,0,0,0
2040000000,-64.1528386206532,0,0,0
2080000000,-63.7751897198055,0,0,0
2120000000,-63.663056855945,0,0,0
2160000000,-64.227155790167,0,0,0
2200000000,-63.874804848758,0,0,0
END
Here is my code: 这是我的代码:
import pandas as pd
df = pd.read_csv('/Users/_XXXXXXXXX_/Desktop/TP1_041416_C.csv',
sep='delimter',
header=None,
engine='python')
UPDATE: - this version will read and parse the header part first, it will generate info
dictionary from the parsed header and will prepare skiprows
list for the pd.read_csv()
function 更新: -此版本将首先读取并解析标题部分,它将从已解析的标题生成
info
字典,并将为pd.read_csv()
函数准备skiprows
列表
from collections import defaultdict
import io
import re
import pandas as pd
import pprint as pp
fn = r'D:\temp\.data\36671176.data'
def parse_header(filename):
"""
parses useful (check `header_flt` variable) information from the header
and saves it into `defaultdict`,
generates `skiprow` list for `pd.read_csv()`,
breaks after parsing the header, so the data block will NOT be read
returns: parsed info as defaultdict obj, skiprows list
"""
# useful header information that will be saved into defaultdict
header_flt = r'TIMESTAMP|TIMEZONE|NAME|MODEL|SERIAL|FIRMWARE_VERSION|Trace TIMESTAMP:'
with open(fn) as f:
d = defaultdict(list)
i = 0
skiprows = []
for line in f:
line = line.strip()
if line.startswith('!'):
skiprows.append(i)
# parse: `key` (first RegEx group)
# and `value` (second RegEx group)
m = re.search(r'!\s+(' + header_flt + ')\s+(.*)\s*', line)
if m:
# print(m.group(1), '->', m.group(2))
# save parsed `key` and `value` into defaultdict
d[m.group(1)] = m.group(2)
elif line.startswith('BEGIN'):
skiprows.append(i)
else:
# stop loop if line doesn't start with: '!' or 'BEGIN'
break
i += 1
return d, skiprows
info, skiprows = parse_header(fn)
# parses data block, the header part will be skipped
# NOTE: `comment='E'` - will skip the footer row: "END"
df = pd.read_csv(fn, header=None, usecols=[0,1], names=['freq', 'dBm'],
skiprows=skiprows, skipinitialspace=True, comment='E',
error_bad_lines=False)
print(df)
print('-' * 60)
pp.pprint(info)
OLD version: - reads the whole file in memory and parses it. OLD版本: -读取内存中的整个文件并进行解析。 It might cause problems with big files because it will allocate memory for the whole file content plus the resulting DF:
这可能会导致大文件出现问题,因为它将为整个文件内容以及生成的DF分配内存。
from collections import defaultdict
import io
import re
import pandas as pd
import pprint as pp
fn = r'D:\temp\.data\36671176.data'
header_pat = r'(TIMESTAMP|TIMEZONE|NAME|MODEL|SERIAL|FIRMWARE_VERSION)\s+([^\r\n]*?)\s*[\r\n]+'
def parse_file(filename):
with open(fn) as f:
txt = f.read()
m = re.search(r'BEGIN\s*[\r\n]+(.*)[\n\r]+END', txt, flags=re.DOTALL|re.MULTILINE)
if m:
data = m.group(1)
df = pd.read_csv(io.StringIO(data), header=None, usecols=[0,1], names=['freq', 'dBm'])
else:
df = pd.DataFrame()
d = defaultdict(list)
for m in re.finditer(header_pat, txt, flags=re.S|re.M):
d[m.group(1)] = m.group(2)
return df, d
df, info = parse_file(fn)
print(df)
pp.pprint(info)
Output: 输出:
freq dBm
0 2000000000 -62.689350
1 2040000000 -64.152839
2 2080000000 -63.775190
3 2120000000 -63.663057
4 2160000000 -64.227156
5 2200000000 -63.874805
defaultdict(<class 'list'>,
{'FIRMWARE_VERSION': 'A.08.05',
'MODEL': 'N9917A',
'NAME': 'Keysight Technologies',
'SERIAL': 'US52240515',
'TIMESTAMP': 'Friday 15 April 2016 04:50:05',
'TIMEZONE': '(GMT+08:00) Kuala Lumpur Singapore'})
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.