How to get the Source Data When Ingestion Failure in KUSTO ADX

Question

1. I have a base table in ADX Kusto DB.

.create table base (info:dynamic)

2. I have written a function which parses( dynamic column) the base table and greps a few columns and stores it in another table whenever the base table gets data(from EventHub). Below function and its update policy

.create function extractBase() { base | evaluate bag_unpack(info) | project tostring(column1), toreal(column2), toint(column3), todynamic(column4) } .alter table target_table policy update @'[{"IsEnabled": true, "Source": "base", "Query": "extractBase()", "IsTransactional": false, "PropagateIngestionProperties": true}]'

suppose if the base table does not contain the expected column, ingestion error happens. how do I get the source(row) for the failure? When using .show ingestion failures , it displays the failure message. there is a column called IngestionSourcePath . when I browse the URL, getting an exception as Resource Not Found .

If ingestion failure happens, I need to store the particular row of base table into IngestionFailure Table. for further investigation

Answer 1

In this case, your source data cannot "not have" a column defined by its schema. If no value was ingested for some column in some row, a null value will be present there and the update policy will not fail.

Here the update policy will break if the original table row does not contain enough columns. Currently the source data for such errors is not emitted as part of the failure message.

In general, the source URI is only useful when you are ingesting data from blobs. In other cases the URI shown in the failed ingestion info is a URI on an internal blob that was created on the fly and no one has access to.

However, there is a command that is missing from documentation (we will make sure to update it) that allows you to duplicate (dump to storage container you provide) the source data for the next failed ingestion into a specific table.

The syntax is: .dup-next-failed-ingest into TableName to h@'Path to Azure blob container'

Here the path to Azure Blob container must include a writeable SAS. The required permission to run this command is DB admin.