Dropping a "garbage" column from pandas dataframe

Question

i am trying my hardest to plot some data i have from a particularly badly formatted file (I can not change the format of the files, so i have to build around the issues i meet). I am trying to import the data from the file, and removing some garbage data i do not need, such as error messages, but i am struggling.

Here i have a function that allows me to open the file i want to work with, and some workarounds for formatting it into the dataframe:

headers = ['Date','Time','Pressure','Temperature','Bias','RefTemp', 'Garbage']
def plotDigitalFunction():
    infile=askopenfilename()

    df = pd.read_csv(infile,sep="\t",names=headers, skiprows=1, parse_dates=[['Date','Time']])
    df = df.drop('Garbage', axis=1)

the top of my file looks something like this:

Date    Time    Pressure    Temperature Bias    Error
06.02.12    13:42:19:549         -2689      895524     1842052        27.0  ERROR: T1B1

So, here i have 6 headers and 7 columns. I am skipping the first row, and setting my own headers and combining Date and time so i have 6 (i need the date and time stamp in the same column).

I have tried to work with this code in jupyter notebook, and it works flawlessly. I get something like this:

        Date_Time   Pressure    Temperature Bias    RefTemp
    0   06.02.12 13:42:19:549   -2689   895524  1842052 27.0
    1   06.02.12 13:42:20:546   -2689   895467  1841921 27.0
    2   06.02.12 13:42:21:544   -2689   895388  1841817 27.0
    3   06.02.12 13:42:22:543   -2691   895287  1841672 27.0

But when i am running the same code in python 3.6.2 it seems as if just the column header gets deleted and the data beneath it gets shifted and placed under the column to the left of it. This wont work, and i am struggling to figure out what i am doing wrong.

I had a solution before to open the file and then creating a temporary csv file wich i then read from, that worked. But it is quite an amount of data to run through so it takes double the amount of time to process.

I hope this is explained well enough and dont hesitate to ask questions if i need to elaborate.

Thanks in advance!

edit: i just tried the same code in the console, using the filename hardcoded instead of using "infile" with tkinter's askopenfilename(). that worked fine. Could the problem be because of that?

Answer 1

Probably it's about python version that you use. Check what python version you are using with jupyter and use the same one.