堆栈内存溢出
  繁体   English   中英

Kusto - 最新版本行的物化视图

[英]Kusto - Materialized View on latest version of rows

 原文  2022-03-02 10:40:55  0   1    azure-data-explorer/ kql/ kusto-explorer

问题描述

我有一个名为“元数据”的表，其中包含由 TestId 分区的参数和参数值列表。 每次更改测试时，测试都将重新摄取到 Azure Data Explorer，且版本为 never。

我的总体目标是：

  1. 定义一个 Function (GetTestsFromSearch)，它采用键值对的参数（动态），这让我可以查询所有测试（最新版本）以查找键/值对的匹配项：
( {{"Search param1", "Search value1"},{"Search param2", "Search value2"}}

例子

GetTestsFromSearch({{"ProjectId", "SturnProject"},{"Product Name", "Nacelle "}})

应该返回

测试编号 版本
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 3个
ea5b688c-b61f-4c5b-bb87-af2eac94d454 1个

来自下面的示例 metadata.table

  1. 另一个目标是创建一个只包含每个测试的最新元数据的物化视图（下表解释）

metadata.table 示例

测试编号 测试名称 范围 参数值 版本
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的测试 项目编号 转向计划 1个
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的测试 产品分类 2兆瓦 1个
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的测试 项目开始日期 2022-02-03 1个
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的测试 项目编号 转向计划 2个
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的测试 产品分类 2兆瓦 2个
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的测试 项目开始日期 2022-02-03 2个
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的测试 项目编号 转向计划 3个
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的测试 产品分类 2兆瓦 3个
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 我的测试 项目开始日期 2022-02-03 3个
ea5b688c-b61f-4c5b-bb87-af2eac94d454 我的测试 项目编号 转向计划 1个
ea5b688c-b61f-4c5b-bb87-af2eac94d454 我的测试 项目 State 打开 1个
ea5b688c-b61f-4c5b-bb87-af2eac94d454 我的测试 产品名称 机舱 1个

随着时间的推移，将有数以千计的测试在几个不同的版本中，因此我预计，创建一个物化视图是个好主意，它只维护每个测试的最新版本——我试图将视图创建为：

metadata
| summarize arg_max(Version,*) by TestId

但这只为每个 TestId/Version 提供了一个参数和参数值，而不是测试的整个结果集。

  • 谁能为我指明这个物化视图的正确方向？

我已经包含了一个 metadata.table 作为 DataTable 的示例，它可以直接在 kusto 中使用。

元数据表作为数据表

datatable (TestId: string, Name: string, Parameter: string, ParameterValue: string, Version: int) [
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Report, DMS number","1234-231",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project name","Thor3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","GTRS reference","gtrs",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Product Category","2MW",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project number","TE-12321",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DUT responsible person","ANFRB3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test execution person","ANFRB3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project Manager","ANFRB3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVPR, DMS number","1234-1234",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVRE, DMS number","1231-1213",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Start Date","2022-02-23T00:00:00.0000000Z",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Category","Verification safety",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","GTRS reference","gtrs",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project number","TE-12321",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DUT responsible person","ANFRB2",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test execution person","ANFRB2",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project Manager","ANFRB2",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVPR, DMS number","1234-1234",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVRE, DMS number","1231-1213",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Start Date","2022-02-23T00:00:00.0000000Z",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Category","Verification safety",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Product Category","2MW",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project name","Thor3",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Report, DMS number","1234-231",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Category","Verification safety",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project Manager","ANFRB",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","GTRS reference","gtrs",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Product Category","2MW",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project name","Thor3",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project number","TE-12321",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DUT responsible person","ANFRB",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test execution person","ANFRB",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVPR, DMS number","1234-1234",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Report, DMS number","1234-231",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVRE, DMS number","1231-1213",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Start Date","2022-02-23T00:00:00.0000000Z",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","GTRS reference","gtrs232",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Product Category","4MW",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Project name","Myproject",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Project number","43324534",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","DUT responsible person","ANFRB",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test execution person","ANFRB",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Project Manager","ANFRB",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","DVPR, DMS number","435123454",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test Report, DMS number","123123123",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","DVRE, DMS number","12312312312",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test Start Date","2022-03-01T00:00:00.0000000Z",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test Category","Verification functionality",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test facility","CHE",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test rig","rig23",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Sample ID","1",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Link to test data","asdfsafdsdfa",int(1)
]

谢谢

1 个解决方案

解决方案1
 1 已采纳  2022-03-02 12:04:09

如果大多数测试包含相同的属性（如您的示例），您可以考虑将架构更改为宽架构，其中TestId和Name的每次运行（ Version ）都是一条记录。 结果架构类似于以下 output：

**datatable**
| extend pack(Parameter, ParameterValue)
| summarize make_bag(Column1) by TestId, Name, Version
| evaluate bag_unpack(bag_Column1)

然后，您可以使用以下聚合设置物化视图，它将提供您正在寻找的 IIUC：

T | summarize arg_max(Version, *) by TestId, Name

要从示例中的架构切换到建议的架构，您可以将摄取管道更改为以新的架构格式摄取，或者使用更新策略进行转换。 如果您选择后者，请避免在更新策略 function 中使用bag_unpack插件。相反，显式投影您需要的列，以避免非确定性架构。

另一种选择是将所有属性保留在单个动态列中，结果如下：

**datatable**
| extend pack(Parameter, ParameterValue)
| summarize make_bag(Column1) by TestId, Name, Version

并使用与上面相同的物化视图定义。

对于第一个问题 - 使用第二个建议的架构，您可以尝试如下操作：

let GetTestsFromSearch = (Filter:dynamic)
{
    T
    | extend pack(Parameter, ParameterValue)
    | summarize Properties = make_bag(Column1) by TestId, Name, Version  
    | summarize arg_max(Version, *) by TestId, Name
    | extend Filter
    | mv-apply Filter on 
    ( 
        extend key=tostring(bag_keys(Filter)[0]) 
        | extend expected = tostring(Filter[key]), actual = tostring(Properties[key])
        | summarize count(), countif(actual == expected)
        | where count_  == countif_
    )
};
GetTestsFromSearch(dynamic({"Test Category" : "Verification safety", "Project name" : "Thor3"}));
测试编号 名称 版本 特性 数数_ countif_
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0 ANFRB-FILEVIEW-测试 3个 {
"检测报告，DMS编号": "1234-231",
“项目名称”：“雷神3”，
“GTRS参考”：“gtrs”，
“产品类别”：“2MW”，
“项目编号”：“TE-12321”，
“DUT负责人”：“ANFRB3”，
“测试执行人”：“ANFRB3”，
“项目经理”：“ANFRB3”，
"DVPR, DMS 号码": "1234-1234",
"DVRE, DMS 号码": "1231-1213",
“测试开始日期”：“2022-02-23T00:00:00.0000000Z”,
《测试类别》：《验证安全》
}		 2个 2个

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 Kusto - 向现有物化视图添加列 截断并加载 Kusto 表而不是物化视图，以便它可以用于连续导出 在不在 Kusto 中序列化数据的情况下在物化视图中按用户名和时间戳查找下一条记录 STRUCT 的 BigQuery 具体化视图 刷新postgres物化视图的性能 大查询物化视图； 使用（内部）连接创建物化视图 BigQuery 中的视图与具体化视图 不支持的编码：使用 Kusto Spark 连接器或使用 Spark 版本 < 3.3.0 的 Kusto 导出从 Kusto 读取时的 DELTA_BYTE_ARRAY BigQuery 实体化视图 - ARRAY_AGG 问题 BigQuery 实体化视图：查询限制与创建查询限制
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM