从气流 DAG 连接到 ClickHouse

Question

I write this code:

from airflow import DAG
from airflow.hooks.clickhouse_hook import ClickHouseHook
from airflow.operators.python_operator import PythonOperator
from airflow.utils.dates import days_ago
from datetime import datetime 

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2020, 10, 18)    
}

def _data_from_clickhouse():
    
    
    ch_hook = ClickHouseHook(clickhouse_conn_id='ClickHouse_rnd_conn')   
    ch_hook.get_records('select * from cpa_traffic_costs')


with DAG(dag_id='data_to_clickhouse',          
         default_args=default_args) as dag:  
        
    get_data_from_clickhouse = PythonOperator(
        task_id='get_data_from_clickhouse',
        python_callable=_data_from_clickhouse,
    )
    
    get_data_from_clickhouse

But I have an error when I start this dag in Web UI

clickhouse_driver.errors.SocketTimeoutError: Code: 209. (85.***.***.***:8123)

This error happens only in airflow dag. I set up a connection "ClickHouse_rnd_conn" via the web UI of airflow. Maybe the problem is that the airflow does not have a connection type "clickhouse" and i used type "HTTP"

Answer 1

You may use the wrong port, see this github issue :

This driver use native protocol (port 9000). Port 8123 is used for HTTP protocol.

Answer 2

clickhouse_driver.errors.SocketTimeoutError: Code: 209. (85. . .***:8123)

Your Web UI host does not have permission to access 85. . .***:8123 ? Firewall is in between CH server and Web UI host.