来自多个传感器的数据保存到导入到熊猫的 txt 文件中

Question

Good day everyone.

I was hoping someone here could help me with a bit of a problem. I've run an experiment, where data has been gathered from 6 separate sensors simultaneously. The data has then been exported to a single shared txt file. Now I need to import the data to python to analyze it.

I know I can do this by taking each of the lines and simply copy&pasting data output from each sensor into a separate document, and then import those in a loop - but that is a lot of work and brings in a high potential of human error.

But is there no way of using readline with specific lines read, and porting that to pandas DataFrame? There is a fixed header spacing, and line spacing between each sensor.

I tried:

f=open('OR0024622_auto3200.txt')
lines = f.readlines()

base = 83
sensorlines = 6400

Sensor=[]
Sensor = lines[base:sensorlines+base]

df_sens = pd.DataFrame(Sensor)
df_sens

but the output isn't very useful: Snip from of Output

-- Here's the file i am importing: link.

Any suggestions ?

Answer 1

Looks like a tab separated data.

use

>>> df = pd.read_csv('OR0024622_auto3200.txt', delimiter=r'\t', skiprows=83, header=None, nrows=38955-84)
>>> df.tail()
          0                   1              2
38686  6397   3.1980000000e+003   9.28819e-009
38687  6398   3.1985000000e+003   9.41507e-009
38688  6399   3.1990000000e+003   1.11703e-008
38689  6400   3.1995000000e+003   9.64276e-009
38690  6401   3.2000000000e+003   8.92203e-009
>>> df.head()
   0                   1              2
0  1   0.0000000000e+000   6.62579e+000
1  2   5.0000000000e-001   3.31289e+000
2  3   1.0000000000e+000   2.62362e-011
3  4   1.5000000000e+000   1.51130e-011
4  5   2.0000000000e+000   8.35723e-012

Answer 2

abhilb's answer is to the point and correct, but there is a lot to be said regarding loading/reading files. A quick browser search will take you a long way (I encourage you to read up on this!), but I'll add a few details here:

If you want to load multiple files that match a pattern you can do so iteratively via glob:

import pandas as pd
from glob import glob as gg
filePattern = "/path/to/file/*.txt"

for fileName in gg(filePattern):
    df = pd.read_csv('OR0024622_auto3200.txt', delimiter=r'\t')

This will load each file one-by-one. What if you want to put all data into a single dataframe? Do this:

masterDF = pd.Dataframe()

for fileName in gg(filePattern):
    df = pd.read_csv('OR0024622_auto3200.txt', delimiter=r'\t')
    masterDF = pd.concat([masterDF, df], axis=0)

This works great for pandas, but what if you want to read into a numpy array?

import numpy as np

# using previous imports
base = 83
sensorlines = 6400

# create an empty array that has three columns    
masterArray = np.full((0, 3), np.nan)

for fileName in gg(filePattern):
    # open the file (NOTE: this does not read the file, just puts it in a buffer)
    with open(fileName, "r") as tmp:
        # now read the file and split each line by the carriage return (could be "\r\n")
        # you now have a list of strings
        data = tmp.read().split("\n")

        # keep only the "data" portion of the file
        data = data[base:sensorlines + base]

        # convert list of strings to an array of floats
        # here, I use a "list comprehension" for speed and simplicity
        data = np.array([r.split("\t") for r in data]).astype(float)

        # stack your new data onto your master array
        masterArray = np.vstack([masterArray, data])

Opening a file via the "with open(fileName, "r")" syntax is handy because Python automatically closes the file when you are done. If you don't use "with" then you must manually close the file (eg tmp.close()).

These are just some starting points to get you on your way. Feel free to ask for clarification.