[英]Extracting text data into a meaningful table for analysis using Python (or R)
我正在從事一個工程項目,我正在使用來自檔案的機器性能數據。 機器大約每5秒產生一個數據集,然后這個數據在許多.txt
文件中按日期提供,每個文件包含以下格式的數據。 下面顯示的數據來自2013_04_17.txt
文件,該文件包含該特定日期的所有性能數據。
2013-04-27 00:00:05.011
V_1 100 V_2 26695 V_3 33197 V_4 c681 V_5 29532
V_6 4600 V_7 4606 V_8 4f55 V_9 5a V_10 8063 V_11 4300 V_12 4700
V_13 4504 V_14 4400 V_15 4202 V_16 255 V_17 4300 V_18 91 V_19 6f
V_20 300 V_21 14784
V_22 5.085 V_23 7.840 V_24 -8.061 V_25 36.961
2013-04-27 00:00:10.163
V_1 100 V_2 26695 V_3 33199 V_4 c681 V_5 29872
V_6 4600 V_7 4606 V_8 4f55 V_9 5a V_10 8063 V_11 4300 V_12 4700
V_13 4504 V_14 4400 V_15 4202 V_16 255 V_17 4300 V_18 91 V_19 6f
V_20 300 V_21 14790
V_22 5.085 V_23 7.840 V_24 -8.061 V_25 37.961
..........
我需要以表格格式或CSV格式查看此數據,以便能夠生成性能圖並檢測任何異常。 但是,我沒有足夠的Python編程經驗來解析這個文本文件。
我已經查看了大熊貓和正則表達式的一些想法,但未能達到預期的結果,我希望以表格形式或CSV文件的形式將數據作為變量Date,Time, V_1
, V_2
, V_3
等以及隨后的行作為每5秒獲得的所有值。
編輯 :您可以在沒有正則表達式的情況下獲得相同的結果,如下所示:注意,我們假設文件格式始終相同,因此我們期望文件開頭的日期和時間
# reading data from a file for example log.txt
with open('log.txt', 'r') as f:
data = f.read()
data = string.split()
v_readings = dict()
v_readings['date'] = data.pop(0)
v_readings['time' ]= data.pop(0)
i=0
while i < len(data):
v_readings[data[i]] = data[i+1]
i += 2
導出到csv文件:
csv = '\n'
csv += ','.join(v_readings.keys())
csv += '\n'
csv += ','.join(v_readings.values())
print(csv)
with open('out.csv', 'w') as f:
f.write(csv)
輸出:
date,time,V_1,V_2,V_3,V_4,V_5,V_6,V_7,V_8,V_9,V_10,V_11,V_12,V_13,V_14,V_15,V_16,V_17,V_18,V_19,V_20,V_21,V_22,V_23,V_24,V_25
2013-04-27,00:00:05.011,100,26695,33197,c681,29532,4600,4606,4f55,5a,8063,4300,4700,4504,4400,4202,255,4300,91,6f,300,14784,5.085,7.840,-8.061,36.961
使用正則表達式:這是使用python中的變量和字典中的正則表達式提取這些數據的方法
這是一個起點,然后你可以隨心所欲地做任何你喜歡的事
import re
string = """
2013-04-27 00:00:05.011 V_1 100 V_2 26695 V_3 33197 V_4 c681 V_5 29532 V_6 4600 V_7 4606 V_8 4f55 V_9 5a V_10 8063 V_11 4300 V_12 4700 V_13 4504 V_14 4400 V_15 4202 V_16 255 V_17 4300 V_18 91 V_19 6f V_20 300 V_21 14784 V_22 5.085 V_23 7.840 V_24 -8.061 V_25 36.961
"""
# extract date
match = re.search(r'\d{4}-\d\d-\d\d', string)
my_date = match.group()
# extract time
match = re.search(r'\d\d:\d\d:\d\d\.\d+', string)
my_time = match.group()
#getting V's into a dictionary
match = re.findall(r'V_\d+ \d+', string)
v_readings = dict()
for item in match:
k, v = item.split()
v_readings[k] = v
# print output
print(my_date)
print(my_time)
print(v_readings)
輸出:
2013-04-27
00:00:05.011
{'V_1': '100', 'V_2': '26695', 'V_3': '33197', 'V_5': '29532', 'V_6': '4600', 'V_7': '4606', 'V_8': '4', 'V_9': '5', 'V_10': '8063', 'V_11': '4300', 'V_12': '4700', 'V_13': '4504', 'V_14': '4400', 'V_15': '4202', 'V_16': '255', 'V_17': '4300', 'V_18': '91', 'V_19': '6', 'V_20': '300', 'V_21': '14784', 'V_22': '5', 'V_23': '7', 'V_25': '36'}
您可以從文件中一次讀取一個令牌開始:
with open('2013_04_17.txt') as infile:
for line in infile:
for token in line.split():
print(token)
之后,您只需要創建一個狀態機來記住您所在的部分,並在找到結束時處理每個部分:
def process_record(timestamp, values):
"""print CSV format"""
print(','.join([timestamp] + values))
with open('t.txt') as infile:
timestamp = None
values = []
for line in infile:
line = line.strip()
if timestamp is None:
timestamp = line
elif not line: # blank line is separator
process_record(timestamp, values)
timestamp = None
values = []
else:
values.extend(line.split()[1::2])
if timestamp is not None: # process last record, no separator after it
process_record(timestamp, values)
這給你CSV輸出:
2013-04-27 00:00:05.011,100,26695,33197,c681,29532,4600,4606,4f55,5a,8063,4300,4700,4504,4400,4202,255,4300,91,6f,300,14784,5.085,7.840,-8.061,36.961
2013-04-27 00:00:10.163,100,26695,33199,c681,29872,4600,4606,4f55,5a,8063,4300,4700,4504,4400,4202,255,4300,91,6f,300,14790,5.085,7.840,-8.061,37.961
有一個更簡單的方法。 假設此數據出現在.txt文件的列中(即數據是固定寬度格式 ),您可以使用pandas函數pandas.read_fwf()並傳入包含每個固定寬度字段范圍的元組。線。
import pandas
colspecs = [(0,10), (11, 23), (28,31), (37, 42), (48, 54), (59, 63), (70, 75), ...]
data = pandas.read_fwf(TXT_PATH, colspecs = colspecs, header=None)
data.columns = ['date', 'time', 'V_1', 'V_2', 'V_3', 'V_4', 'V_5', ...]
print(data)
date time V_1 V_2 V_3 V_4 V_5
0 2013-04-27 00:00:05.011 100 26695 33197 c681 29532
1 2013-04-27 00:00:10.163 100 26695 33199 c681 29872
從那里,您可以使用該命令將格式化的數據保存到文件中
data.to_csv('filename.csv', index=False)
在R中,這將非常特定於您的情況,您可以嘗試將所有.txt文件放入一個新文件夾,例如將其稱為date_data。 假設所有文件都采用相同的格式,請嘗試運行此文件。
library(purrr)
library(tidyverse)
setwd(./date_data)
odd_file_reader <- function(x){
as.data.frame(matrix(scan(x, what="character", sep=NULL), ncol = 52, byrow = TRUE)[,-seq(3,51,2)])
}
binded_data <- tibble(filenames = list.files()) %>%
mutate(yearly_sat = map(filenames, odd_file_reader)) %>%
unnest()
試試我的簡單代碼,我用過熊貓
import pandas as pd
with open('2013_04_17.txt', 'r') as f:
large_list = [word for line in f for word in line.split() if 'V_' not in word]
print(large_list)
col_titles = ('date','time','v1','v2','vN','vN','vN','vN','vN','vN','vN','vN'
,'vN','vN','vN','vN','vN','vN','vN','vN','vN','vN','vN','vN','vN','vN','vN')
data = pd.np.array(large_list).reshape((len(large_list) // 27, 27))
pd.DataFrame(data, columns=col_titles).to_csv("output3.csv", index=False)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.