将自定义日志发送到 Azure 使用 Azure 数据工厂进行日志分析

Question

I am trying to ingest custom logs in to the Azure log analytics using Azure Data factory. HTTP Data collector is the API that Microsoft provided to ingest custom logs to Azure log analytics. I have created a pipeline with a Web Activity in Azure Data factory to post some sample log to Log analytics. Below are the settings for the Web Activity.

URL: https://xxloganalyticsworkspaceIDxx.ods.opinsights.azure.com/api/logs?api-version=2016-04-01 Method: POST Headers: Authorization - SharedKey xxloganalyticsworkspaceIDxx: Log-Type - ADFRecord Content-Type - application/json x-ms-date - <Signature created using PowerShell RFC 1123 format> Body - [{"StringValue":"MyString1","NumberValue":42,"BooleanValue":true,"DateValue":"2019-09-12T20:00:00.625Z","GUIDValue":"9909ED01-A74C-4874-8ABF-D2678E3AE23D"}]

I am aware that that Authorization and x-ms-date headers should be provided dynamically however for testing I have created these headers using the PowerShell code provided by Microsoft in the article https://docs.microsoft.com/en-us/azure/azure-monitor/logs/data-collector-api#request-headers and used the same headers in the web activity. The pipeline fails with the error below:

{"Error":"InvalidAuthorization","Message":"An invalid signature was specified in the Authorization header"}

I have tried pushing the same data with same headers using postman and the REST call was successful implying the header signature generated using PowerShell works. I can see the customs logs are pushed to Log analytics.

Would appreciate any help on this to fix the problem and also please let me know how can I provide Authorization headers in Web activity dynamically.

Answer 1

The data factory web activity seem to have some issue with sending custom log to log analytics workspace, after some try-and-error I have found the way that managed to work.

1. Make sure you try with alternate approach eg python/C#/etc and get the request working. (I used python for example https://docs.microsoft.com/en-us/azure/azure-monitor/logs/data-collector-api#python-sample )

2. In data factory web activity body section, wrap the request with [] to make it an array, ie {"A":6} would become [{"A":6}] (not sure the exact reason but seem to work, there may be some process around how they process the body)

3. Remove all redundant space as seems azure would compress your body before it sent, ie [{"A": 6, "B": 7}] should become [{"A":6,"B":7}]

4. Now use that processed body string's content_length for building your signature, you may test it using python first and copy(hard-coding) the value of Authorization(signature) and x-ms-date to data factory web activity for testing and it should work now.

Please share if there is a proper solution for this, thanks:)