converting a dataframe to a csv file

Question

I am working with a data Adult that I have changed and would like to save it as a csv. however after saving it as a csv and re-loading the data to work with again, the data is not converted properly. The headers are not preserved and some columns are now combined. I have looked through the page and online, but what I have tried is not working. I load the data in with the following code:

import numpy as np ##Import necassary packages
import pandas as pd
import matplotlib.pyplot as plt
from pandas.plotting import scatter_matrix
from sklearn.preprocessing import *
url2="http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data" #Reading in Data from a freely and easily available source on the internet
Adult = pd.read_csv(url2, header=None, skipinitialspace=True) #Decoding data by removing extra spaces in cplumns with skipinitialspace=True
##Assigning reasonable column names to the dataframe
Adult.columns = ["age","workclass","fnlwgt","education","educationnum","maritalstatus","occupation",  
                 "relationship","race","sex","capitalgain","capitalloss","hoursperweek","nativecountry",
                 "less50kmoreeq50kn"]

After inserting missing values and changing the data frame as desired I have tried:

df = Adult

df.to_csv('file_name.csv',header = True)

df.to_csv('file_name.csv')

and a few other variations. How can I save the file to a CSV and preserve the correct format for the next time I read the file in?

When re-loading the data I use the code:

import pandas as pd
df = pd.read_csv('file_name.csv')

when running df.head the output is:

<bound method NDFrame.head of        Unnamed: 0  Unnamed: 0.1  age  ... Black  Asian-Pac-Islander Other
0               0             0   39  ...     0                   0     0
1               1             1   50  ...     0                   0     0
2               2             2   38  ...     0                   0     0
3               3             3   53  ...     1                   0     0

and print(df.loc[:,"age"].value_counts()) the output is:

36    898
31    888
34    886
23    877
35    876

which should not have 2 columns

Answer 1

If you pickle it like so:

Adult.to_pickle('adult.pickle')

You will, subsequently, be able to read it back in using read_pickle as follows:

original_adult = pd.read_pickle('adult.pickle')

Hope that helps.

Answer 2

If you want to preserve the output column order you can specify the columns directly while saving the DataFrame:

import pandas as pd

url2 = "http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data" 
df = pd.read_csv(url2, header=None, skipinitialspace=True)

my_columns = ["age", "workclass", "fnlwgt", "education", "educationnum", "maritalstatus", "occupation",
             "relationship","race","sex","capitalgain","capitalloss","hoursperweek","nativecountry",
             "less50kmoreeq50kn"]
df.columns = my_columns

# do the computation ...

df[my_columns].to_csv('file_name.csv') 

You can add parameter index=False to the to_csv('file_name.csv', index=False) function if you are not interested in saving the DataFrame row index. Otherwise, while reading the csv file again you'd need to specify the index_col parameter.

---

According to the documentation value_counts() returns a Series object - you see two columns because the first one is the index - Age (36, 31, ...), and the second is the count (898, 888, ...).

Answer 3

I replicated your code and it works for me. The order of the columns is preserved.

Let me show what I tried. Tried this batch of code:

import numpy as np ##Import necassary packages
import pandas as pd
import matplotlib.pyplot as plt
from pandas.plotting import scatter_matrix
from sklearn.preprocessing import *

url2="http://archive.ics.uci.edu/ml/machine-learning- 
databases/adult/adult.data" #Reading in Data from a freely and easily 
available source on the internet

Adult = pd.read_csv(url2, header=None, skipinitialspace=True) #Decoding data 
by removing extra spaces in cplumns with skipinitialspace=True

##Assigning reasonable column names to the dataframe
Adult.columns =["age","workclass","fnlwgt","education","educationnum","maritalstatus","occupation",  
             "relationship","race","sex","capitalgain","capitalloss","hoursperweek","nativecountry",
             "less50kmoreeq50kn"]

This worked perfectly. Then

df = Adult 

This also worked. Then I saved this data frame to a csv file. Make sure you are providing the absolute path to the file even if is is being saved in the same folder as this script.

df.to_csv('full_path_to_the_file.csv',header = True)
# so someting like
#df.to_csv('Users/user_name/Desktop/folder/NameFile.csv',header = True)

Load this csv file into a new_df. It will generate a new column for keeping track of index. It is unnecessary and you can drop it like following:

new_df = pd.read_csv('Users/user_name/Desktop/folder/NameFile.csv', index_col = None)
new_df= new_df.drop('Unnamed: 0', axis =1)

When I compare the columns of the new_df from the original df, with this line of code

new_df.columns == df.columns

I get

array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
    True,  True,  True,  True,  True,  True])

You might not have been providing the absolute path to the file or saving the file twice as here. You only need to save it once.

df.to_csv('file_name.csv',header = True)

df.to_csv('file_name.csv')

Answer 4

When you save the dataframe in general, the first column is the index, and you sould load the index when reading the dataframe, also whenever you assign a dataframe to a variable make sure to copy the dataframe:

df = Adult.copy()
df.to_csv('file_name.csv',header = True)

And to read:

df = pd.read_csv('file_name.csv', index_col=0)

The first columns from print(df.loc[:,"age"].value_counts()) is the index column which is shown if you query the datframe, to save this to a list, use the to_list method:

print(df.loc[:,"age"].value_counts().to_list())