有什么方法可以将数据从谷歌电子表格传递到 clickhouse？

Question

There is problem that our people fill every day in google spreadsheet some data and I need with a certain frequency (eg once a day) to send these tables in clickhouse

(it located on our aws servers)

it doesn't matter whether clickhouse writes only new data from tables or all tables every time

please tell me a working method how to do it

from the Toolkit are python ,can in theory work with sqlalchemy and airflow DAG but for the development of dag in airflow I have not yet found a guide how to write in python a script to transfer data from googlespreadsheet

the second option is with owox extension for google spreadsheet - but there you need to work with Google BigQuery, and this will breed a zoo, and I would not like to pay for BQ yet

Do you have any ideas how to use scripts to upload tables to Clickhouse from google spreadsheets?

Answer 1

I found the Python library pygsheets - it is easier to access spreadsheets using the api than directly

official pygsheets dock - https://pygsheets.readthedocs.io/en/stable/

in addition I found more libraries: gspread and oauth2client which can also be used to work on Python with the api step by step guide https://towardsdatascience.com/accessing-google-spreadsheet-data-using-python-90a5bc214fd2 official documentation for gspread https://gspread.readthedocs.io/en/latest/

than i can make dag at airflow and manage etl process