Python脚本从文件读取数据并将其存储在mysql表中

Question

I am trying to write a python script that would read records from a text file and insert into a mysql table .

For a particular time stamp, eg:(1438084440) and servername (r01vb01) - 6 column values are generated.

I need to store these 8 entities into mysql table .

As timestamp and servername are repeated in all 6 rows, I am unable to find a best way to extract and insert it into the table.

The end result should be like :

servername    date_time   cpu_number     cpu_user     cpu_nice  cpu_system   cpu_wait   cpu_idle
r01vb01     1438084440   40.0              0.0         0.0         0.0       0.0          NaN
r01vb01     1438084445   40.0              0.0         0.0         0.0       0.0          NaN
r01vb01     1438084450   40.0              0.0         0.0         0.0       0.0          NaN

Table structure :

CREATE TABLE `cpu_util_all` (
`servername` VARCHAR(45) NOT NULL,
`date_time` DATETIME NOT NULL,
`cpu_number` DECIMAL(10,2) NULL,
`cpu_user` DECIMAL(10,2) NULL,
`cpu_nice` DECIMAL(10,2) NULL,
`cpu_system` DECIMAL(10,2) NULL,
 'cpu_wait` DECIMAL(10,2) NULL,
`cpu_idle` DECIMAL(10,2) NULL,
 PRIMARY KEY (`servername`, `date_time`));



Text file :

 cpunumber  1438084440  r01vb01 40.0
 cpunumber  1438084445  r01vb01 40.0
 cpunumber  1438084450  r01vb01 40.0
 cpunice    1438084440  r01vb01 0.0
 cpunice    1438084445  r01vb01 0.0
 cpunice    1438084450  r01vb01 0.0
 cpusystem  1438084440  r01vb01 0.0
 cpusystem  1438084445  r01vb01 0.0
 cpusystem  1438084450  r01vb01 0.0
 cpuwait    1438084440  r01vb01 0.0
 cpuwait    1438084445  r01vb01 0.0
 cpuwait    1438084450  r01vb01 0.0
 cpuuser    1438084440  r01vb01 0.0
 cpuuser    1438084445  r01vb01 0.0
 cpuuser    1438084450  r01vb01 0.0
 cpudile    1438084440  r01vb01 NaN
 cpudile    1438084445  r01vb01 NaN
 cpudile    1438084450  r01vb01 NaN

I tried reading and storing data from text file into array matrix. But not sure how to turn into a data base table.

 with open("test.txt") as textFile:
     lines = [line.split() for line in textFile]

  [['cpunumber', '1438084440', 'r01vb01', '40.0'], ['cpunumber',          '1438084445', 'r01vb01', '40.0'], 
  ['cpunumber', '1438084450', 'r01vb01', '40.0'], ['cpunice', '1438084440', 'r01vb01', '0.0'], 
  ['cpunice', '1438084445', 'r01vb01', '0.0'], ['cpunice', '1438084450', 'r01vb01', '0.0'], 
  ['cpusystem', '1438084440', 'r01vb01', '0.0'], ['cpusystem', '1438084445', 'r01vb01', '0.0'], 
  ['cpusystem', '1438084450', 'r01vb01', '0.0'], ['cpuwait', '1438084440', 'r01vb01', '0.0'], 
  ['cpuwait', '1438084445', 'r01vb01', '0.0'], ['cpuwait', '1438084450', 'r01vb01', '0.0'], 
  ['cpuuser', '1438084440', 'r01vb01', '0.0'], ['cpuuser', '1438084445', 'r01vb01', '0.0'], 
  ['cpuuser', '1438084450', 'r01vb01', '0.0'], ['cpudile', '1438084440', 'r01vb01', 'NaN'], 
  ['cpudile', '1438084445', 'r01vb01', 'NaN'], ['cpudile', '1438084450', 'r01vb01', 'NaN']   

Answer 1

You should create a dictionary that is keyed by servername and datetime

Example: from collections import defaultdict

lines = defaultdict(dict)
with open("file_parse.txt") as f:
    for line in f:
        parts = line.split()
        key = tuple(parts[1:3]) # unique key with servername and datetime
        lines[key][parts[0]] = parts[3]
        lines[key]["servername"] = parts[1]
        lines[key]["datetime"] = parts[3]
res = list(lines.values())
print(res)
# [
#    {'cpudile': 'NaN', 'cpunice': '0.0', 'cpunumber': '40.0', 'cpusystem': '0.0', 'cpuuser': '0.0', 'cpuwait': '0.0', 'datetime': 'NaN', 'servername': '1438084445'},
#    {'cpudile': 'NaN', 'cpunice': '0.0', 'cpunumber': '40.0', 'cpusystem': '0.0', 'cpuuser': '0.0', 'cpuwait': '0.0', 'datetime': 'NaN', 'servername': '1438084440'},
#    {'cpudile': 'NaN', 'cpunice': '0.0', 'cpunumber': '40.0', 'cpusystem': '0.0', 'cpuuser': '0.0', 'cpuwait': '0.0', 'datetime': 'NaN', 'servername': '1438084450'}
# ]