需要使用 Pyspark 处理 2 亿条记录

Question

I am cleaning the data ( approx 200 Million rows) using python Pyspark but getting the error:

ValueError: unichr() arg not in range(0x10000) (narrow Python build)

The size of the csv file is 21 GB and I added config('spark.driver.memory','8g') and I am using Macbook Pro 16GB.

When I try to process the same file by reducing the size to 1GB, it executes successfully.

Any recommendations!!

I am using Apache 2.4

Expected results: To process the raw_data.csv file Actual results: base =

base[:pos] + unichr(char) + base[pos:]
ValueError: unichr() arg not in range(0x10000) (narrow Python build)

Answer 1

This error could be related to an unescaped unicode character in your char variable. Could you try using:

base = base[:pos] + char.decode('unicode-escape') + base[pos:]

Including your code would make it easier to help debug the issue.