Error while trying to preview data in DataFlow in Azure Synaspe

Question

I have a data pipeline in Azure Synaspe. The Pipeline has a data flow activity that is reading data from a Spark database in Azure Data Lake Storage Gen 2 data lake. The data then sinks into another spark database.

The other day when I tried to preview from the source spark database, I got the following error(data source name is masked).

at Source '<>': org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:null);

This was working perfectly fine previously.

Please note: There were no code changes to this pipeline.

Any thoughts

Answer 1

Make sure Autoscale of Spark pool configuration is Enable.

You should try by increasing Number of nodes.

Also try with more Node Size.

Reference - https://docs.microsoft.com/en-us/answers/questions/706694/unable-to-run-sql-queries-in-azure-synapse-error-o.html