How to check if query is pushed down from databricks to snowflake?

Question

I'm trying to use query pushdown from databricks to Snowflake. I'm reading data from snowflake(data source) to databricks, creating dataframes and applying joins, filter and aggregate functions. Code is running fine but not able to find if the query is pushed down to snowflake. How to check if query has run on snowflake or spark(databricks) cluster?

Answer 1

At least these 2 ways:

1. Use the History from Snowflake UI to see if the query was run in the Snowflake side.

2. Enable debug mode on Spark connector by setting the context to DEBUG:

   sc.setLogLevel('DEBUG')