使用熊猫从.txt文件导入数据，并在各列之间使用换行符

Question

I am importing data from Boston Housing Data into a pandas dataframe. The last 3 items for every row is separated into the next row. Is there a way to import the data using pd.read_csv to include these off items? Here is my code:

import pandas as pd
path = '/Users/Main/Desktop/boston.txt'
df = pd.read_csv(path, skiprows=21, sep='\s+', header=None)

This provides me with a dataframe with 11 columns, but I need 14 columns. Also, is there a better way to skip all the text at the top of the file without manually counting each row?

Answer 1

First of all, you can just use the boston housing dataset from scikit-learn. http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html . If you still want to use the text file, unfortunately I think you will have to do some processing on the text file, to remove the line breaks. I have tried to give an example of the kind of processing needed.

# read the file, and separate the lines.
with open('boston.txt', 'r') as f:
    text = [line for line in f.readlines()]

# starting from first row of data, remove \n from even numbered rows,
# and append the next row to it.
start_row = 22
new_rows = []
for i,l in enumerate(text[start_row:]):
    if not i%2:
        newl = l.strip('\n')+text[start_row+i+1]
        new_rows.append(newl)

new_data = ''.join(new_rows)

# finally save the data.
with open('boston_new.txt', 'w') as f:
    f.write(new_data)

Now you can read the data easily. The delim_whitespace is similar to using sep='\\s+'.

col_names = ['CRIM', 'ZN', 'INDUS', 'CHAS','NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV']
pd.read_csv('boston_new.txt', delim_whitespace=True, header=None, names=col_names)

After doing this once, you should save the data in a proper .csv format that is readable by pandas without giving so many parameters.

pd.to_csv('boston_final.csv')

Answer 2

I ended up trying the same idea, appending each overflow line to the line before it.

boston = pd.read_csv("FILE_LOCATION", sep='\s+', header = None)

oklist = []

for row in range(1012):

    if row % 2 == 0:
        rowa = boston.iloc[row,]
        row = row + 1
        rowb = boston.iloc[row,]

        new_row = rowa.append(rowb)
        clean_list = new_row.iloc[0:14].tolist()
        oklist.append(clean_list)

pd.DataFrame(oklist)