Enthought Python和MatPlotLib中的乱码.csv时间序列数据

Question

EDIT:

Thank you for your prompt reply, Jonathan.

As you suggest below, I tried using numpy.loadtxt . Unfortunately, a similar error appears. The output of data = numpy.loadtxt("MyData.csv", skiprows = 39, delimiter = ",") is

Traceback (most recent call last):
  File "/Users/aleksnavratil/Desktop/sandbox.py", line 23, in <module>
    data = numpy.loadtxt("MyData.csv", skiprows = 39, delimiter = ",")
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/lib/npyio.py", line 805, in loadtxt
    items = [conv(val) for (conv, val) in zip(converters, vals)]
ValueError: could not convert string to float: ﾿ÒﾭóﾈÀëÐٟﾡB.AØME84ﾊﾓ

The same error is thrown for arbitrary skiprows kwargs. Perhaps this lends credence to the character encoding problem hypothesis. I'm still at a loss for a solution.

/EDIT

I have a .csv datafile produced by a scientific instrument (a CETR Universal Media Tester UMT-2). The data represents a time series of measurements. The file behaves strangely when I access it from Python, but is well behaved when accessed via cat, Nano, TextEdit, etc. This phenomenon persists across Windows 7 and Snow Leopard machines, though both are using the Enthought Scientific Python distribution.

The output of

f = codecs.open("MyData.csv",encoding="ascii")
data = f.xreadlines()
for line in data:
    print line

is

****
?****************************************
?****************************************

ÿÿÿZ
Ðí0


þÿÿî
üÿÿð
éí0
óÿÿí
ôí0

etc….

This smells like an encoding problem, so I investigated a bit:

The output of file -i "MyData.csv" is

MyData.csv: text/plain; charset=us-ascii

Using the CharDet module; the output of chardetect.py "MyData.csv" is

MyData.csv: ascii with confidence 1.0

Using the Codecs package, I tried several common encodings to no avail. Also, I tried using Matplotlib's csv2rec . The output of

r = mlab.csv2rec(codecs.open("MyData.csv", 'rU',),skiprows=39, delimiter=",")

is

Traceback (most recent call last):
  File "/Volumes/AVN2109/Raw Data/CETR_Plotter.py", line 40, in <module>
    r = mlab.csv2rec(codecs.open("MyData.csv", 'rU',),skiprows=39, delimiter=",")
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-        packages/matplotlib/mlab.py", line 2181, in csv2rec
    process_skiprows(reader)
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-    packages/matplotlib/mlab.py", line 2176, in process_skiprows
    for i, row in enumerate(reader):
Error: line contains NULL byte

This is also true for arbitrary skiprows kwargs.

Furthermore, the instrument has the option to produce .txt's (as well as .csv's) as its output. The behavior is identical in both cases. Perhaps I'm missing something obvious. Does anyone know how to persuade this data to play nice with Python?

Answer 1

The package that you need to use to load your data is numpy and the function that I recommend you start with is loadtxt . It is the simplest and works great in your case since your data is homogeneous:

import numpy
data = numpy.loadtxt("MyData.csv", skiprows = 39, delimiter = ",")

Of course your file is a little more complex with seemingly 2 arrays and some stuff at the top you may or may not want to throw away. You can refine this, loading the metadata first and then the numerical values. You may also be interested in retaining the headers for the columns to create a "structured array" also called a "record array". To do that I recommend you build the dtype with a first pass by opening the file and detecting the names of the columns and their unit, and passing this dtype to loadtxt afterwards like above.