Dataframe Replace not working - encoding = 'ISO-8859-1', Python 3.6

Question

I have imported some CSV file to Data frame

Data = pd.read_csv(filePath, encoding = 'ISO-8859-1', dtype=object)

I am replacing column "Indicator" with some values

DataT['Indicator'] = DataT['Indicator'].str.replace('export(us$ mil)', 'exports (in us$ mil)')
DataT['Indicator'] = DataT['Indicator'].str.replace('import(us$ mil)', 'imports (in us$ mil)')

But replacement not working due to encoding issue.

Please suggest how to solve this?

File Downloaded from: http://wits.worldbank.org/data/public/cp/wits_en_trade_summary_allcountries_allyears.zip

Code to import all the csv files:-

for i, file in os.listdir(sourcePath):
    if file.upper().endswith('.CSV'):
    filePath = os.path.join(sourcePath, file)
    Data = pd.read_csv(filePath, encoding = 'ISO-8859-1', dtype=object) 

    Data['FileName'] = file
    DataAll = pd.concat([DataAll, Data], sort=False)

Answer 1

loading a sample from your data, I noticed the values for "Indicator" column are not all lower case - ie 'Export(US$ Mil)' rather than 'export(us$ mil)' . you need to either use the correct value, or alternatively:

DataT['Indicator'] = DataT['Indicator'].str.lower().replace('export(us$ mil)',
                                                            'exports (in us$ mil)')

you can always check the unique values for a column using df[col].unique()

Answer 2

After lot of trial, i got into the below solution, Just import re module.

However you can simplified your code as:

import pandas as pd
import glob
import re
for f in glob('/your_Dir_path/somefiles*.csv'):
    Data = pd.read_csv(f, encoding = 'ISO-8859-1', dtype=object)

Dataset:

>>> Data['Indicator'].head()
0     GDP (current US$ Mil)
1    No. Of Export partners
2    No. Of Export products
3    No. Of Import partners
4    No. Of Import products
Name: Indicator, dtype: object
>>> Data['Indicator'].head(100)
0                     GDP (current US$ Mil)
1                    No. Of Export partners
2                    No. Of Export products
3                    No. Of Import partners
4                    No. Of Import products
5                   No. Of Tariff Agreement
6           Trade Balance (current US$ Mil)
7      Trade (US$ Mil)-Top 5 Export Partner
8      Trade (US$ Mil)-Top 5 Export Partner
9      Trade (US$ Mil)-Top 5 Export Partner
10     Trade (US$ Mil)-Top 5 Export Partner
11     Trade (US$ Mil)-Top 5 Import Partner
12     Trade (US$ Mil)-Top 5 Export Partner
13     Trade (US$ Mil)-Top 5 Import Partner
14     Trade (US$ Mil)-Top 5 Export Partner
15     Trade (US$ Mil)-Top 5 Import Partner
16     Trade (US$ Mil)-Top 5 Export Partner
17     Trade (US$ Mil)-Top 5 Export Partner
18     Trade (US$ Mil)-Top 5 Import Partner

Result:

>>> Data['Indicator'].str.replace(re.escape("Trade (US$ Mil)"), "IN Trade (US$ Mil)").head(100)
0                       GDP (current US$ Mil)
1                      No. Of Export partners
2                      No. Of Export products
3                      No. Of Import partners
4                      No. Of Import products
5                     No. Of Tariff Agreement
6             Trade Balance (current US$ Mil)
7     IN Trade (US$ Mil)-Top 5 Export Partner
8     IN Trade (US$ Mil)-Top 5 Export Partner
9     IN Trade (US$ Mil)-Top 5 Export Partner
10    IN Trade (US$ Mil)-Top 5 Export Partner
11    IN Trade (US$ Mil)-Top 5 Import Partner
12    IN Trade (US$ Mil)-Top 5 Export Partner
13    IN Trade (US$ Mil)-Top 5 Import Partner
14    IN Trade (US$ Mil)-Top 5 Export Partner
15    IN Trade (US$ Mil)-Top 5 Import Partner
16    IN Trade (US$ Mil)-Top 5 Export Partner
17    IN Trade (US$ Mil)-Top 5 Export Partner
18    IN Trade (US$ Mil)-Top 5 Import Partner
19    IN Trade (US$ Mil)-Top 5 Import Partner
20    IN Trade (US$ Mil)-Top 5 Import Partner
21    IN Trade (US$ Mil)-Top 5 Export Partner
22    IN Trade (US$ Mil)-Top 5 Export Partner
23    IN Trade (US$ Mil)-Top 5 Export Partner
24    IN Trade (US$ Mil)-Top 5 Export Partner
25    IN Trade (US$ Mil)-Top 5 Export Partner
26    IN Trade (US$ Mil)-Top 5 Export Partner
27    IN Trade (US$ Mil)-Top 5 Export Partner
28    IN Trade (US$ Mil)-Top 5 Import Partner
29    IN Trade (US$ Mil)-Top 5 Export Partner
                       ...
70      Partner share(%)-Top 5 Export Partner
71      Partner share(%)-Top 5 Import Partner
72      Partner share(%)-Top 5 Export Partner
73      Partner share(%)-Top 5 Import Partner
74      Partner share(%)-Top 5 Export Partner
75      Partner share(%)-Top 5 Export Partner
76      Partner share(%)-Top 5 Import Partner
77      Partner share(%)-Top 5 Import Partner
78      Partner share(%)-Top 5 Import Partner
79      Partner share(%)-Top 5 Export Partner
80      Partner share(%)-Top 5 Export Partner
81      Partner share(%)-Top 5 Export Partner
82      Partner share(%)-Top 5 Export Partner
83      Partner share(%)-Top 5 Export Partner
84      Partner share(%)-Top 5 Export Partner
85      Partner share(%)-Top 5 Export Partner
86      Partner share(%)-Top 5 Import Partner
87      Partner share(%)-Top 5 Export Partner
88      Partner share(%)-Top 5 Import Partner
89      Partner share(%)-Top 5 Export Partner
90                         Country Growth (%)
91           Duty Free Tariff Lines Share (%)
92                    Export Product share(%)
93                    Export Product share(%)
94                    Export Product share(%)
95                    Export Product share(%)
96                    Export Product share(%)
97                    Export Product share(%)
98                    Export Product share(%)
99                    Export Product share(%)
Name: Indicator, Length: 100, dtype: object

For your example You should try below:

import re

DataT['Indicator'] = DataT['Indicator'].str.replace(re.escape('export(us$ mil)'), 'exports (in us$ mil)')
DataT['Indicator'] = DataT['Indicator'].str.replace(re.escape('import(us$ mil)'), 'imports (in us$ mil)')