使用 to_dataframe() 作为 BigQuery 管理员角色时出现 BigQuery 权限错误

Question

I'm having a similar problem to the one stated in Google BigQuery query in Python works when using result(), but Permission issue when using to_dataframe() in that.result() works fine but to_dataframe() throws an error, but my problem shouldn't be connected to lack of permissions if my role is BigQuery Admin. I also have a role Editor, could that perhaps obstruct the way permissions work?

Answer 1

The answer turned out to be quite simple, though not obvious. One should set create_bqstorage_client to False .

.to_dataframe(create_bqstorage_client=False)

As in the linked issue, the problem was in regards to bqstorage, however uninstalling google-cloud-bigquery-storage was not enough, as _validate_bqstorage function is checking both parameters and returns False only if both are either None or False .

using_bqstorage_api = bqstorage_client or create_bqstorage_client
if not using_bqstorage_api:
    return False

On the side note, I've created a custom function that takes virtually the same amount of time. If, for some reason, reader is still unable to use .to_dataframe , this function may be helpful

def sql_to_df(query):
    query_job = client.query(query) # API request
    results = query_job.result() # Waits for query to finish
    data = []
    for i, row in enumerate(results):
        if i==0:
            columns = list(row.keys())
        data.append(list(row.values()))
    df = pd.DataFrame(data=data, columns=columns)
    df.index += 1
    return df