Synapse Spark Pool Error HTTP Request failed to Authenticate

Question

I am getting the below error, It use to run fine but suddenly started getting the below error

Error code 1

LIVY_JOB_STATE_DEAD

Message

[plugins.synapse-ent-tekura.sparkpoolsmall.180 WorkspaceType: CCID:] [Monitoring] Livy Endpoint=[https://hubservice.australiasoutheast.azuresynapse.net:8001/api/v1.0/publish/1567bc8e-206c-475f-bf1f-7c550c17a6d6]. Livy Id=[23] Job failed during run time with state=[dead].

Source

Unknown

Sparl logs is here:

Copy

Caused by: org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator$HttpException: HTTP Error -1CustomTokenProvider getAccessToken threw java.io.IOException: POST failed with 'Unauthorized' (401) and message: {"result":"UserError","errorId":"Unauthorized","errorMessage":"Invalid JobSessionToken: IDX10249: X509SecurityKey validation failed. The associated certificate has expired. ValidTo (UTC): '1/28/2023 7:18:14 AM', Current time (UTC): '1/30/2023 7:06:05 PM'.. TraceId: f51d91bd-f786-4225-a223-e9401cf58527. Error Component: TokenService"} at org.apache.hadoop.fs.azurebfs.oauth2.CustomTokenProviderAdapter.refreshToken(CustomTokenProviderAdapter.java:91) at org.apache.hadoop.fs.azurebfs.oauth2.AccessTokenProvider.getToken(AccessTokenProvider.java:50) at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAccessToken(AbfsClient.java:1055) at org.apache.883 02989938088.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:256)... 26 more 23/01/30 19:06:05 INFO ShutdownHookManager: Shutdown hook called 23/01/30 19:06:05 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-16f17c6d-3914-46f6-979c-9da7c8149d86 23/01/30 19:06:05 INFO MetricsSystemImpl: Stopping azure-file-system metrics system... 23/01/30 19:06:05 INFO MetricsSystemImpl: azure-file-system metrics system stopped. 23/01/30 19:06:05 INFO MetricsSystemImpl: azure-file-system metrics system shutdown complete.

Failed to find which authentication is failed

Answer 1

Most likely it is because of expired SSL certificate.

ValidTo (UTC): '1/28/2023 7:18:14 AM', 
Current time (UTC): '1/30/2023 7:06:05 PM'

If this is something managed by you then ask your infra team to renew the SSL certificates or wait for new certificate to be added if you do not know who manages these certificates.