无论正则表达式标志如何，str.contains() 都会返回错误的答案

Question

I have a pandas series with dtype object that contains curency strings, like '$ 50000'. Not all of them are in dollars, some for example are 'GBP 43000' etc. I am trying to use Pandas to figure out which contain $ and which don't.

The series is called valid_usa_gross . To find out which rows in it contain $, first I tried this

valid_usa_gross_USD = valid_usa_gross.str.contains('$')

the series it returned had True everywhere, so I assumed all rows had $, but I was wrong. By inspecting the CSV file, I found entries with different currencies. I then discovered that, unless I specify regex=False , the string I pass to contains() will be interpreted as regex, and God knows what $ may mean in that case. So I tried

valid_usa_gross_USD = valid_usa_gross.str.contains('$', regex=False)

which resulted in a series with False everywhere. That is also incorrect, because the vast majority of the entries in valid_usa_gross_USD do contain a $ symbol. I even tried to escape the $ or add an 'r' in front of the search string, the result is always wrong (either all True or all False ).

What am I doing wrong?

Answer 1

Looks like the problem was the encoding. I was reading the whole CSV as

imdb_raw = pd.read_csv(dataset_path)

and that didn't throw any errors, so I had no reason to suspect anything. However, reading it as

imdb_raw = pd.read_csv(dataset_path, encoding='ISO-8859-1)

results in the $ correctly displayed.