堆棧內存溢出
  簡體   English   中英

Kusto - 最新版本行的物化視圖

[英]Kusto - Materialized View on latest version of rows

 原文  2022-03-02 10:40:55  7   1    azure-data-explorer/ kql/ kusto-explorer

問題描述

我有一個名為“元數據”的表，其中包含由 TestId 分區的參數和參數值列表。 每次更改測試時，測試都將重新攝取到 Azure Data Explorer，且版本為 never。

我的總體目標是：

  1. 定義一個 Function (GetTestsFromSearch)，它采用鍵值對的參數（動態），這讓我可以查詢所有測試（最新版本）以查找鍵/值對的匹配項：
( {{"Search param1", "Search value1"},{"Search param2", "Search value2"}}

例子

GetTestsFromSearch({{"ProjectId", "SturnProject"},{"Product Name", "Nacelle "}})

應該返回

測試編號 版本
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 3個
ea5b688c-b61f-4c5b-bb87-af2eac94d454 1個

來自下面的示例 metadata.table

  1. 另一個目標是創建一個只包含每個測試的最新元數據的物化視圖（下表解釋）

metadata.table 示例

測試編號 測試名稱 范圍 參數值 版本
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的測試 項目編號 轉向計划 1個
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的測試 產品分類 2兆瓦 1個
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的測試 項目開始日期 2022-02-03 1個
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的測試 項目編號 轉向計划 2個
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的測試 產品分類 2兆瓦 2個
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的測試 項目開始日期 2022-02-03 2個
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的測試 項目編號 轉向計划 3個
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的測試 產品分類 2兆瓦 3個
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的測試 項目開始日期 2022-02-03 3個
ea5b688c-b61f-4c5b-bb87-af2eac94d454 我的測試 項目編號 轉向計划 1個
ea5b688c-b61f-4c5b-bb87-af2eac94d454 我的測試 項目 State 打開 1個
ea5b688c-b61f-4c5b-bb87-af2eac94d454 我的測試 產品名稱 機艙 1個

隨着時間的推移，將有數以千計的測試在幾個不同的版本中，因此我預計，創建一個物化視圖是個好主意，它只維護每個測試的最新版本——我試圖將視圖創建為：

metadata
| summarize arg_max(Version,*) by TestId

但這只為每個 TestId/Version 提供了一個參數和參數值，而不是測試的整個結果集。

  • 誰能為我指明這個物化視圖的正確方向？

我已經包含了一個 metadata.table 作為 DataTable 的示例，它可以直接在 kusto 中使用。

元數據表作為數據表

datatable (TestId: string, Name: string, Parameter: string, ParameterValue: string, Version: int) [
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Report, DMS number","1234-231",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project name","Thor3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","GTRS reference","gtrs",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Product Category","2MW",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project number","TE-12321",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DUT responsible person","ANFRB3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test execution person","ANFRB3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project Manager","ANFRB3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVPR, DMS number","1234-1234",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVRE, DMS number","1231-1213",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Start Date","2022-02-23T00:00:00.0000000Z",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Category","Verification safety",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","GTRS reference","gtrs",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project number","TE-12321",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DUT responsible person","ANFRB2",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test execution person","ANFRB2",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project Manager","ANFRB2",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVPR, DMS number","1234-1234",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVRE, DMS number","1231-1213",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Start Date","2022-02-23T00:00:00.0000000Z",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Category","Verification safety",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Product Category","2MW",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project name","Thor3",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Report, DMS number","1234-231",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Category","Verification safety",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project Manager","ANFRB",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","GTRS reference","gtrs",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Product Category","2MW",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project name","Thor3",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project number","TE-12321",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DUT responsible person","ANFRB",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test execution person","ANFRB",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVPR, DMS number","1234-1234",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Report, DMS number","1234-231",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVRE, DMS number","1231-1213",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Start Date","2022-02-23T00:00:00.0000000Z",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","GTRS reference","gtrs232",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Product Category","4MW",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Project name","Myproject",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Project number","43324534",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","DUT responsible person","ANFRB",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test execution person","ANFRB",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Project Manager","ANFRB",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","DVPR, DMS number","435123454",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test Report, DMS number","123123123",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","DVRE, DMS number","12312312312",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test Start Date","2022-03-01T00:00:00.0000000Z",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test Category","Verification functionality",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test facility","CHE",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test rig","rig23",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Sample ID","1",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Link to test data","asdfsafdsdfa",int(1)
]

謝謝

1 個解決方案

解決方案1
 1 已采納  2022-03-02 12:04:09

如果大多數測試包含相同的屬性（如您的示例），您可以考慮將架構更改為寬架構，其中TestId和Name的每次運行（ Version ）都是一條記錄。 結果架構類似於以下 output：

**datatable**
| extend pack(Parameter, ParameterValue)
| summarize make_bag(Column1) by TestId, Name, Version
| evaluate bag_unpack(bag_Column1)

然后，您可以使用以下聚合設置物化視圖，它將提供您正在尋找的 IIUC：

T | summarize arg_max(Version, *) by TestId, Name

要從示例中的架構切換到建議的架構，您可以將攝取管道更改為以新的架構格式攝取，或者使用更新策略進行轉換。 如果您選擇后者，請避免在更新策略 function 中使用bag_unpack插件。相反，顯式投影您需要的列，以避免非確定性架構。

另一種選擇是將所有屬性保留在單個動態列中，結果如下：

**datatable**
| extend pack(Parameter, ParameterValue)
| summarize make_bag(Column1) by TestId, Name, Version

並使用與上面相同的物化視圖定義。

對於第一個問題 - 使用第二個建議的架構，您可以嘗試如下操作：

let GetTestsFromSearch = (Filter:dynamic)
{
    T
    | extend pack(Parameter, ParameterValue)
    | summarize Properties = make_bag(Column1) by TestId, Name, Version  
    | summarize arg_max(Version, *) by TestId, Name
    | extend Filter
    | mv-apply Filter on 
    ( 
        extend key=tostring(bag_keys(Filter)[0]) 
        | extend expected = tostring(Filter[key]), actual = tostring(Properties[key])
        | summarize count(), countif(actual == expected)
        | where count_  == countif_
    )
};
GetTestsFromSearch(dynamic({"Test Category" : "Verification safety", "Project name" : "Thor3"}));
測試編號 名稱 版本 特性 數數_ countif_
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 ANFRB-FILEVIEW-測試 3個 {
"檢測報告，DMS編號": "1234-231",
“項目名稱”：“雷神3”，
“GTRS參考”：“gtrs”，
“產品類別”：“2MW”，
“項目編號”：“TE-12321”，
“DUT負責人”：“ANFRB3”，
“測試執行人”：“ANFRB3”，
“項目經理”：“ANFRB3”，
"DVPR, DMS 號碼": "1234-1234",
"DVRE, DMS 號碼": "1231-1213",
“測試開始日期”：“2022-02-23T00:00:00.0000000Z”,
《測試類別》：《驗證安全》
}		 2個 2個

暫無
暫無

聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

相關問題 Kusto - 向現有物化視圖添加列 截斷並加載 Kusto 表而不是物化視圖，以便它可以用於連續導出 在不在 Kusto 中序列化數據的情況下在物化視圖中按用戶名和時間戳查找下一條記錄 STRUCT 的 BigQuery 具體化視圖 刷新postgres物化視圖的性能 大查詢物化視圖； 使用（內部）連接創建物化視圖 BigQuery 中的視圖與具體化視圖 不支持的編碼：使用 Kusto Spark 連接器或使用 Spark 版本 < 3.3.0 的 Kusto 導出從 Kusto 讀取時的 DELTA_BYTE_ARRAY BigQuery 實體化視圖 - ARRAY_AGG 問題 BigQuery 實體化視圖：查詢限制與創建查詢限制
相關標簽
 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM