为什么PyCharm显示无效的unicode字符？

Question

I want to process a unicode UTF-8 text in PyCharm. The characters in the csv file are saved in Excel 2016 as UTF-8 unicode. However, when i try to write in unicode on the code input area or when i output the result in the console, it gives me invalid characters. Also I am not able to write in unicode characters on PyCharm, How can i fix this?

import pandas as pd

data1=pd.read_csv('Feb2018_Payroll.csv',encoding='utf-8')
print(data1)

When i copy paste the console output here (below), it looks fine. However, in the console it's invalid characters. What could be the reason and how can i solve it?

   ተ.ቁ.      የሠራተኛ ስም       የሰሩበት ቀን         የወር ደሞዝ         የኃላፊነት አበል  
0    1.0      ገ/መድህን     30.0            4000.00          250.0   
1    2.0     በቀለ       30.0            2000.00            NaN   
2    3.0     ሽፈራው      30.0            3000.00            NaN   

Answer 1

So as for now, PyCharm for Mac uses only 16 bits for the Unicode character code. So this PyCharm for Mac only shows characters in the Basic Multilingual Plane (BMP). As a result, characters that use more than 16 bits are shown as blanks.

With that being said, there are other Intellij derivatives that already display the full range of Unicode characters (32-bit). One example is the Android Studio for Mac.

Answer 2

Solution: If Hindi Chars are not showing in pycharm go to

File>Settings>Editor>FileEncoding

Select UTF-8 Everywhere and at last select with BOM option.

[File Encoding in PyCharm Community Edition 2018.3 1