Pandas 在从 CSV 读取的 DataFrame 的最后一列中添加额外的逗号

Question

How could I remove the extra comma that pandas adds to the last column of my dataset please?

Here's a sample of how the data looks in the CSV file: Sample of the data in the CSV file

Here's the syntax that I'm using to import it:

import pandas as pd
df = pd.read_csv (r'*path*', sep='|')
print (df)

And here's how it appears in Spyder: Sample of the output in Spyder's console

---

Adding lambda functions as following produced an error in Spyder's console:

import pandas as pd
df = pd.read_csv (r'C:\Users\mohamed.a.eshra\Downloads\us_cities_states_counties.csv', sep='|')
df["City alias,"] = df["City alias,"].apply(lambda x: x if x[-1]!="," else x[:-1])
df = df.rename(columns={"City alias,": "City alias"})
print (df)

Error after adding the previous functions:

Traceback (most recent call last): File "C:\Users\mohamed.a.eshra.spyder-py3\Scripts\Opening a CSV file using Pandas.py", line 9, in df["City alias,"] = df["City alias,"].apply(lambda x: x if x[-1],=":" else x[:-1]) File "C.\ProgramData\Anaconda3\lib\site-packages\pandas\core\series,py", line 3848. in apply mapped = lib,map_infer(values, f. convert=convert_dtype) File "pandas_libs\lib,pyx", line 2329. in pandas._libs.lib.map_infer
File "C:\Users\mohamed.a.eshra.spyder-py3\Scripts\Opening a CSV file using Pandas.py", line 9, in df["City alias,"] = df["City alias,"].apply(lambda x: x if x[-1],=":" else x[:-1]) TypeError: 'float' object is not subscriptable

I would appreciate your help in solving the issue.

Thank you!

Answer 1

You can apply a function to the last column in order to remove the commas, and then rename the column:

Edited to handle missing values:

import numpy as np
import pandas as pd

def clean_city_name(x):
   if pd.isnull(x):
        return np.NaN
   else:
       if x[-1]==',':
           return x[:-1]
   return x

df["City alias,"] = df["City alias,"].apply(clean_city_name)

df = df.rename(columns={"City alias,": "City alias"})

This way you will get rid of the commas.

It would be however interesting to investigate why these commas appears, so that you can avoid running into this again.

Usually the comma is the default separator in CSV files, so this comma at the end could be the sign that an empty column was present at the end in the original CSV before the separator was changed to "|"( if this happened at some point, I don't know how your dataset was created so that stays hypothetical! ).

Answer 2

So I deleted several preceding columns in the source CSV file using Excel and the data frame loaded correctly.

Dataframe print in Spyder

Seems that there was an empty column in the underlying CSV. However, not sure how the dataset was loading in the first place as mentioned by "A Co".

Thank you.