BigQuery STRING 列中允许使用哪些字符（出现“UDF 内存不足”错误）

Question

I have a dataframe containing receipt-data. The column text in my dataframe contains the text from the receipt and seems to be an issue when I try to upload the data to BigQuery using df.to_gbq(...) since it produces the error

GenericGBQException: Reason: 400 Resources exceeded during query execution: UDF out of memory.; Failed to read Parquet file /some/file. 
This might happen if the file contains a row that is too large,
 or if the total size of the pages loaded for the queried columns is too large.

According to the error-message it seems to be an "memory error", but I have tried to convert all characters in each text to an "a" (to see if the strings contained to many characters) but that worked fine ie I doubt it is that.

I have tried converting all characters to utf8 by

df["text"] = df["text"].str.encode('utf-8') (since according to the docs they should be so) but that failed. I have tried to replace "\n" with " " but that fails aswell.

It seems like there's some values in my receipt-text that causes some troubles, but It's very difficult to figure out what (and since I have ~3 mio rows, it takes a while to try each and every row at a time) - are there any values that are not allowed in a big-query table?

Answer 1

It turns out that chunksize in to_gbq does not split up the chunks in the way I thought it did. Manually looping over the dataframe in chunks like

CHUNKSIZE = 100_000
for i in range(0,df.shape[0]//CHUNKSIZE):
    print(i)
    df_temp = dataframe.iloc[i*CHUNKSIZE:(i+1)*CHUNKSIZE]
    df_temp.to_gbq(destination_table="Dataset.my_table",
    project_id = "my-project",
    if_exists="append",
    )

did the trick (setting chunksize=100_000 did not work)