使用 Azure 数据工厂过滤 JSON 数据

Question

I currently have some log files from a chatbot built on Microsoft's Bot Framework that I'm trying to process using Azure Data Factory. I'm running a daily process to create a daily JSON file of the logs and now I want to build on this to create a process to create another JSON file that only includes some of the logs. For example, I just want to include the logs that are of type "trace" and "message". I've tried looking at how I can use features such as lookup and filter but just can't work out how to accomplish what I'm aiming for.

Ideally I want to create a process where I isolate the "message" logs that include the phrase "learning list" and capture the "trace" log that appears before it so I can capture the message that was sent to the chatbot that resulted in the "learning list" response.

I've included a sample of the logs below:

{"type":"message","timestamp":"2020-04-29T14:02:28.808Z","text":"teams"}
{"type":"trace","timestamp":"2020-04-29T14:02:32.345Z","value":{"message":{"text":"teams","textFormat":"plain","type":"message","timestamp":"2020-04-29T14:02:28.808Z","localTimestamp":"2020-04-29T14:02:28.808Z","id":"1588168948793","channelId":"msteams","serviceUrl":"https://smba.trafficmanager.net/emea/","from":{"id":"29:1Tgnjrvpvsg9voZ7PGpR4lzSB3VOLeUa7Hnl9oxOEEflevU6O346pAD_GGwagLA6BnH1Ir8kjIse5FSOjznhFnw","name":"Sean Betts","aadObjectId":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","role":"user"},"conversation":{"conversationType":"personal","tenantId":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37","id":"a:1sfqZBXIxkjxMv9nMwQsWT_Vl9lTo8OTipwPdmOKFiGEgKDTqvu0rA1o4TKcc71tkiMHVmgg3YZgKhnhzU9kXqvWkDEP2KZXACcxd8YViOz6uthdEPCJrbsWKcDN9xJRE"},"recipient":{"id":"28:dfa7e609-37b7-442f-b0b7-f11d5bbc73d8","name":"SearchBot"},"entities":[{"locale":"en-GB","country":"GB","platform":"Mac","type":"clientInfo"}],"channelData":{"tenant":{"id":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37"}},"locale":"en-GB"},"queryResults":[{"questions":["How do I use Teams"],"answer":"[Here's](https://xxx.pdf) our guide to Teams.","score":0.805,"id":13,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}},{"questions":["I'm looking for the People Team Handbook"],"answer":"[Here](https://xxx.pdf) is the People Team Handbook.","score":0.39189999999999997,"id":3,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}}],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"message","timestamp":"2020-04-29T14:00:10.157Z","text":"[Here](https://xxx.pdf) is the OMG UK New Starter's Handbook."}
{"type":"delay","timestamp":"2020-04-29T14:02:31.799Z","value":2000}
{"type":"trace","timestamp":"2020-04-29T14:00:09.895Z","value":{"message":{"text":"new starters","textFormat":"plain","type":"message","timestamp":"2020-04-29T14:00:06.656Z","localTimestamp":"2020-04-29T14:00:06.656Z","id":"1588168806641","channelId":"msteams","serviceUrl":"https://smba.trafficmanager.net/emea/","from":{"id":"29:1Tgnjrvpvsg9voZ7PGpR4lzSB3VOLeUa7Hnl9oxOEEflevU6O346pAD_GGwagLA6BnH1Ir8kjIse5FSOjznhFnw","name":"Sean Betts","aadObjectId":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","role":"user"},"conversation":{"conversationType":"personal","tenantId":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37","id":"a:1sfqZBXIxkjxMv9nMwQsWT_Vl9lTo8OTipwPdmOKFiGEgKDTqvu0rA1o4TKcc71tkiMHVmgg3YZgKhnhzU9kXqvWkDEP2KZXACcxd8YViOz6uthdEPCJrbsWKcDN9xJRE"},"recipient":{"id":"28:dfa7e609-37b7-442f-b0b7-f11d5bbc73d8","name":"omgsearchbot"},"entities":[{"locale":"en-GB","country":"GB","platform":"Mac","type":"clientInfo"}],"channelData":{"tenant":{"id":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37"}},"locale":"en-GB"},"queryResults":[{"questions":["Where is the OMG UK New Starter's Handbook"],"answer":"[Here](https://xxx.pdf) is the OMG UK New Starter's Handbook.","score":0.7702,"id":38,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}}],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"typing","timestamp":"2020-04-29T14:02:31.799Z"}
{"type":"message","timestamp":"2020-04-29T14:02:32.539Z","text":"[Here's](https://xxx.pdf) our guide to Teams."}
{"type":"typing","timestamp":"2020-04-29T14:04:05.874Z"}
{"type":"message","timestamp":"2020-04-29T14:04:03.177Z","text":"help"}
{"type":"message","timestamp":"2020-04-29T14:04:06.410Z","text":"You can ask me to find lots of different types of documents for you, such as:\n\n - Info on Bankside \n\n - Specialist Service summaries\n\n - IT guides \n\n - Lots of different People Guides \n\nJust type in what you're looking for and I'll try and find it for you. \n\n🤓"}
{"type":"trace","timestamp":"2020-04-29T14:04:06.158Z","value":{"message":{"text":"help","textFormat":"plain","type":"message","timestamp":"2020-04-29T14:04:03.177Z","localTimestamp":"2020-04-29T14:04:03.177Z","id":"1588169043168","channelId":"msteams","serviceUrl":"https://smba.trafficmanager.net/emea/","from":{"id":"29:1Tgnjrvpvsg9voZ7PGpR4lzSB3VOLeUa7Hnl9oxOEEflevU6O346pAD_GGwagLA6BnH1Ir8kjIse5FSOjznhFnw","name":"Sean Betts","aadObjectId":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","role":"user"},"conversation":{"conversationType":"personal","tenantId":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37","id":"a:1sfqZBXIxkjxMv9nMwQsWT_Vl9lTo8OTipwPdmOKFiGEgKDTqvu0rA1o4TKcc71tkiMHVmgg3YZgKhnhzU9kXqvWkDEP2KZXACcxd8YViOz6uthdEPCJrbsWKcDN9xJRE"},"recipient":{"id":"28:dfa7e609-37b7-442f-b0b7-f11d5bbc73d8","name":"SearchBot"},"entities":[{"locale":"en-GB","country":"GB","platform":"Mac","type":"clientInfo"}],"channelData":{"tenant":{"id":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37"}},"locale":"en-GB"},"queryResults":[{"questions":["Help","What can you do"],"answer":"You can ask me to find lots of different types of documents for you, such as:\n\n - Info on Bankside \n\n - Specialist Service summaries\n\n - IT guides \n\n - Lots of different People Guides \n\nJust type in what you're looking for and I'll try and find it for you. \n\n🤓","score":1,"id":30,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}}],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"message","timestamp":"2020-04-29T14:04:56.946Z","text":"widget"}
{"type":"delay","timestamp":"2020-04-29T14:04:05.874Z","value":2000}
{"type":"typing","timestamp":"2020-04-29T14:04:59.425Z"}
{"type":"delay","timestamp":"2020-04-29T14:04:59.425Z","value":2000}
{"type":"trace","timestamp":"2020-04-29T14:04:59.762Z","value":{"message":{"text":"widget","textFormat":"plain","type":"message","timestamp":"2020-04-29T14:04:56.946Z","localTimestamp":"2020-04-29T14:04:56.946Z","id":"1588169096931","channelId":"msteams","serviceUrl":"https://smba.trafficmanager.net/emea/","from":{"id":"29:1Tgnjrvpvsg9voZ7PGpR4lzSB3VOLeUa7Hnl9oxOEEflevU6O346pAD_GGwagLA6BnH1Ir8kjIse5FSOjznhFnw","name":"Sean Betts","aadObjectId":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","role":"user"},"conversation":{"conversationType":"personal","tenantId":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37","id":"a:1sfqZBXIxkjxMv9nMwQsWT_Vl9lTo8OTipwPdmOKFiGEgKDTqvu0rA1o4TKcc71tkiMHVmgg3YZgKhnhzU9kXqvWkDEP2KZXACcxd8YViOz6uthdEPCJrbsWKcDN9xJRE"},"recipient":{"id":"28:dfa7e609-37b7-442f-b0b7-f11d5bbc73d8","name":"SearchBot"},"entities":[{"locale":"en-GB","country":"GB","platform":"Mac","type":"clientInfo"}],"channelData":{"tenant":{"id":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37"}},"locale":"en-GB"},"queryResults":[],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"message","timestamp":"2020-04-29T14:04:59.966Z","text":"I'm sorry, I don't have an answer to that question yet, but I'll add it to my learning list!"}
{"type":"message","timestamp":"2020-04-29T14:08:36.024Z","text":"widget"}
{"type":"typing","timestamp":"2020-04-29T14:08:38.518Z"}
{"type":"trace","timestamp":"2020-04-29T14:08:38.775Z","value":{"message":{"text":"widget","textFormat":"plain","type":"message","timestamp":"2020-04-29T14:08:36.024Z","localTimestamp":"2020-04-29T14:08:36.024Z","id":"1588169316004","channelId":"msteams","serviceUrl":"https://smba.trafficmanager.net/emea/","from":{"id":"29:1Tgnjrvpvsg9voZ7PGpR4lzSB3VOLeUa7Hnl9oxOEEflevU6O346pAD_GGwagLA6BnH1Ir8kjIse5FSOjznhFnw","name":"Sean Betts","aadObjectId":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","role":"user"},"conversation":{"conversationType":"personal","tenantId":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37","id":"a:1sfqZBXIxkjxMv9nMwQsWT_Vl9lTo8OTipwPdmOKFiGEgKDTqvu0rA1o4TKcc71tkiMHVmgg3YZgKhnhzU9kXqvWkDEP2KZXACcxd8YViOz6uthdEPCJrbsWKcDN9xJRE"},"recipient":{"id":"28:dfa7e609-37b7-442f-b0b7-f11d5bbc73d8","name":"SearchBot"},"entities":[{"locale":"en-GB","country":"GB","platform":"Mac","type":"clientInfo"}],"channelData":{"tenant":{"id":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37"}},"locale":"en-GB"},"queryResults":[],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"delay","timestamp":"2020-04-29T14:08:38.518Z","value":2000}
{"type":"message","timestamp":"2020-04-29T14:08:39.000Z","text":"I'm sorry, I don't have an answer to that question yet, but I'll add it to my learning list!"}
{"type":"delay","timestamp":"2020-04-29T14:08:49.250Z","value":2000}
{"type":"message","timestamp":"2020-04-29T14:08:46.809Z","text":"what is teams?"}
{"type":"typing","timestamp":"2020-04-29T14:08:49.250Z"}
{"type":"message","timestamp":"2020-04-29T14:08:49.743Z","text":"[Here's](https://xxx.pdf) our guide to Teams."}
{"type":"trace","timestamp":"2020-04-29T14:08:49.515Z","value":{"message":{"text":"what is teams?","textFormat":"plain","type":"message","timestamp":"2020-04-29T14:08:46.809Z","localTimestamp":"2020-04-29T14:08:46.809Z","id":"1588169326794","channelId":"msteams","serviceUrl":"https://smba.trafficmanager.net/emea/","from":{"id":"29:1Tgnjrvpvsg9voZ7PGpR4lzSB3VOLeUa7Hnl9oxOEEflevU6O346pAD_GGwagLA6BnH1Ir8kjIse5FSOjznhFnw","name":"Sean Betts","aadObjectId":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","role":"user"},"conversation":{"conversationType":"personal","tenantId":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37","id":"a:1sfqZBXIxkjxMv9nMwQsWT_Vl9lTo8OTipwPdmOKFiGEgKDTqvu0rA1o4TKcc71tkiMHVmgg3YZgKhnhzU9kXqvWkDEP2KZXACcxd8YViOz6uthdEPCJrbsWKcDN9xJRE"},"recipient":{"id":"28:dfa7e609-37b7-442f-b0b7-f11d5bbc73d8","name":"SearchBot"},"entities":[{"locale":"en-GB","country":"GB","platform":"Mac","type":"clientInfo"}],"channelData":{"tenant":{"id":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37"}},"locale":"en-GB"},"queryResults":[{"questions":["How do I use Teams"],"answer":"[Here's](xxx.pdf) our guide to Teams.","score":0.7812,"id":13,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}},{"questions":["Where is the season ticket loan form"],"answer":"The season ticket loan form is [here](https://xxx.docx).","score":0.311,"id":33,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}}],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"delay","timestamp":"2020-04-29T14:08:59.621Z","value":2000}
{"type":"message","timestamp":"2020-04-29T14:08:57.323Z","text":"what is sharepoint?"}
{"type":"typing","timestamp":"2020-04-29T14:08:59.621Z"}
{"type":"message","timestamp":"2020-04-29T14:09:00.082Z","text":"Try this [guide](https://xxx.pdf)."}
{"type":"message","timestamp":"2020-04-29T14:11:55.926Z","text":"travel loan"}
{"type":"trace","timestamp":"2020-04-29T14:08:59.892Z","value":{"message":{"text":"what is sharepoint?","textFormat":"plain","type":"message","timestamp":"2020-04-29T14:08:57.323Z","localTimestamp":"2020-04-29T14:08:57.323Z","id":"1588169337303","channelId":"msteams","serviceUrl":"https://smba.trafficmanager.net/emea/","from":{"id":"29:1Tgnjrvpvsg9voZ7PGpR4lzSB3VOLeUa7Hnl9oxOEEflevU6O346pAD_GGwagLA6BnH1Ir8kjIse5FSOjznhFnw","name":"Sean Betts","aadObjectId":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","role":"user"},"conversation":{"conversationType":"personal","tenantId":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37","id":"a:1sfqZBXIxkjxMv9nMwQsWT_Vl9lTo8OTipwPdmOKFiGEgKDTqvu0rA1o4TKcc71tkiMHVmgg3YZgKhnhzU9kXqvWkDEP2KZXACcxd8YViOz6uthdEPCJrbsWKcDN9xJRE"},"recipient":{"id":"28:dfa7e609-37b7-442f-b0b7-f11d5bbc73d8","name":"SearchBot"},"entities":[{"locale":"en-GB","country":"GB","platform":"Mac","type":"clientInfo"}],"channelData":{"tenant":{"id":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37"}},"locale":"en-GB"},"queryResults":[{"questions":["How do I use SharePoint"],"answer":"Try this [guide](https://xxx.pdf).","score":0.95,"id":14,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}},{"questions":["What are the OMG benefits"],"answer":"You can find details of all our benefits [here](https://xxx.pdf).","score":0.4697,"id":4,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}}],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"typing","timestamp":"2020-04-29T14:11:58.563Z"}
{"type":"delay","timestamp":"2020-04-29T14:11:58.563Z","value":2000}
{"type":"trace","timestamp":"2020-04-29T14:11:58.844Z","value":{"message":{"text":"travel loan","textFormat":"plain","type":"message","timestamp":"2020-04-29T14:11:55.926Z","localTimestamp":"2020-04-29T14:11:55.926Z","id":"1588169515916","channelId":"msteams","serviceUrl":"https://smba.trafficmanager.net/emea/","from":{"id":"29:1Tgnjrvpvsg9voZ7PGpR4lzSB3VOLeUa7Hnl9oxOEEflevU6O346pAD_GGwagLA6BnH1Ir8kjIse5FSOjznhFnw","name":"Sean Betts","aadObjectId":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","role":"user"},"conversation":{"conversationType":"personal","tenantId":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37","id":"a:1sfqZBXIxkjxMv9nMwQsWT_Vl9lTo8OTipwPdmOKFiGEgKDTqvu0rA1o4TKcc71tkiMHVmgg3YZgKhnhzU9kXqvWkDEP2KZXACcxd8YViOz6uthdEPCJrbsWKcDN9xJRE"},"recipient":{"id":"28:dfa7e609-37b7-442f-b0b7-f11d5bbc73d8","name":"SearchBot"},"entities":[{"locale":"en-GB","country":"GB","platform":"Mac","type":"clientInfo"}],"channelData":{"tenant":{"id":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37"}},"locale":"en-GB"},"queryResults":[{"questions":["Where is the personal loan form"],"answer":"The personal loan form is [here](https://xxx.docx).","score":0.6198,"id":32,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}},{"questions":["Where is the season ticket loan form"],"answer":"The season ticket loan form is [here](https://xxx.docx)","score":0.5587,"id":33,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}}],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"message","timestamp":"2020-04-29T14:12:44.255Z","text":"travel loan"}
{"type":"message","timestamp":"2020-04-29T14:11:59.118Z","text":"The personal loan form is [here](https://xxx.docx)"}
{"type":"delay","timestamp":"2020-04-29T14:12:46.693Z","value":2000}
{"type":"message","timestamp":"2020-04-29T14:12:47.196Z","text":"The season ticket loan form is [here](xxx.docx)"}
{"type":"trace","timestamp":"2020-04-29T14:12:46.947Z","value":{"message":{"text":"travel loan","textFormat":"plain","type":"message","timestamp":"2020-04-29T14:12:44.255Z","localTimestamp":"2020-04-29T14:12:44.255Z","id":"1588169564239","channelId":"msteams","serviceUrl":"https://smba.trafficmanager.net/emea/","from":{"id":"29:1Tgnjrvpvsg9voZ7PGpR4lzSB3VOLeUa7Hnl9oxOEEflevU6O346pAD_GGwagLA6BnH1Ir8kjIse5FSOjznhFnw","name":"Sean Betts","aadObjectId":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","role":"user"},"conversation":{"conversationType":"personal","tenantId":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37","id":"a:1sfqZBXIxkjxMv9nMwQsWT_Vl9lTo8OTipwPdmOKFiGEgKDTqvu0rA1o4TKcc71tkiMHVmgg3YZgKhnhzU9kXqvWkDEP2KZXACcxd8YViOz6uthdEPCJrbsWKcDN9xJRE"},"recipient":{"id":"28:dfa7e609-37b7-442f-b0b7-f11d5bbc73d8","name":"SearchBot"},"entities":[{"locale":"en-GB","country":"GB","platform":"Mac","type":"clientInfo"}],"channelData":{"tenant":{"id":"f3e0ea3d-dd96-4574-83b5-7f35c71cde37"}},"locale":"en-GB"},"queryResults":[{"questions":["Where is the season ticket loan form","Where is the travel loan form"],"answer":"The season ticket loan form is [here](https://xxx.docx?d=w422ff4c734754bfe9f0a03e602b67989&csf=1&web=1&e=DgdTBd).","score":0.95,"id":33,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}},{"questions":["Where is the personal loan form"],"answer":"The personal loan form is [here](https://xxx.docx?d=wda346b28670040518308b74d36f672f0&csf=1&web=1&e=aYHsmf).","score":0.5345,"id":32,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}}],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"typing","timestamp":"2020-04-29T14:12:46.693Z"}
{"type":"trace","timestamp":"2020-04-29T08:34:51.019Z","value":{"message":{"type":"message","id":"4RbdyZGhJblDsnWwZjZUfB-a|0000000","timestamp":"2020-04-29T08:34:48.092Z","serviceUrl":"https://webchat.botframework.com/","channelId":"webchat","from":{"id":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","name":"You","role":"user"},"conversation":{"id":"4RbdyZGhJblDsnWwZjZUfB-a"},"recipient":{"id":"omgsearchbot@8LBpMqDWWGw","name":"omgsearchbot"},"textFormat":"plain","locale":"en-US","text":"hi","entities":[{"type":"ClientCapabilities","requiresBotState":true,"supportsListening":true,"supportsTts":true}],"channelData":{"clientActivityID":"1588149288082pbqyejo0nqb","clientTimestamp":"2020-04-29T08:34:48.082Z"}},"queryResults":[],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"message","timestamp":"2020-04-29T08:34:48.537Z","text":"Welcome to BhavBot! Ask me a question and I will try to answer it."}
{"type":"typing","timestamp":"2020-04-29T08:34:57.525Z"}
{"type":"conversationUpdate","timestamp":"2020-04-29T08:34:37.687Z"}
{"type":"delay","timestamp":"2020-04-29T08:34:50.615Z","value":2000}
{"type":"message","timestamp":"2020-04-29T08:34:51.084Z","text":"I'm sorry, I don't have an answer to that question yet, but I'll add it to my learning list!"}
{"type":"message","timestamp":"2020-04-29T08:34:48.092Z","text":"hi"}
{"type":"message","timestamp":"2020-04-29T08:34:55.394Z","text":"Teams"}
{"type":"message","timestamp":"2020-04-29T08:35:04.330Z","text":"Try this guide - https://xxx.pdf"}
{"type":"typing","timestamp":"2020-04-29T08:35:04.073Z"}
{"type":"message","timestamp":"2020-04-29T08:34:58.049Z","text":"Here's our guide to Teams - https://xxx.pdf"}
{"type":"message","timestamp":"2020-04-29T08:35:01.938Z","text":"Sharepoint"}
{"type":"delay","timestamp":"2020-04-29T08:35:04.073Z","value":2000}
{"type":"conversationUpdate","timestamp":"2020-04-29T09:19:43.903Z"}
{"type":"delay","timestamp":"2020-04-29T08:34:57.525Z","value":2000}
{"type":"trace","timestamp":"2020-04-29T08:34:57.955Z","value":{"message":{"type":"message","id":"4RbdyZGhJblDsnWwZjZUfB-a|0000003","timestamp":"2020-04-29T08:34:55.394Z","serviceUrl":"https://webchat.botframework.com/","channelId":"webchat","from":{"id":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","name":"You","role":"user"},"conversation":{"id":"4RbdyZGhJblDsnWwZjZUfB-a"},"recipient":{"id":"omgsearchbot@8LBpMqDWWGw","name":"omgsearchbot"},"textFormat":"plain","locale":"en-US","text":"Teams","channelData":{"clientActivityID":"1588149295371d9xy4sw90yp","clientTimestamp":"2020-04-29T08:34:55.371Z"}},"queryResults":[{"questions":["How do I use Teams"],"answer":"Here's our guide to Teams - https://xxx.pdf","score":0.81349999999999989,"id":13,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}}],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"trace","timestamp":"2020-04-29T08:35:04.252Z","value":{"message":{"type":"message","id":"4RbdyZGhJblDsnWwZjZUfB-a|0000005","timestamp":"2020-04-29T08:35:01.938Z","serviceUrl":"https://webchat.botframework.com/","channelId":"webchat","from":{"id":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","name":"You","role":"user"},"conversation":{"id":"4RbdyZGhJblDsnWwZjZUfB-a"},"recipient":{"id":"omgsearchbot@8LBpMqDWWGw","name":"omgsearchbot"},"textFormat":"plain","locale":"en-US","text":"Sharepoint","channelData":{"clientActivityID":"15881493019184oofaqrynuo","clientTimestamp":"2020-04-29T08:35:01.918Z"}},"queryResults":[{"questions":["How do I use SharePoint"],"answer":"Try this guide - https://xxx.pdf","score":0.95,"id":14,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}}],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"typing","timestamp":"2020-04-29T09:20:27.881Z"}
{"type":"delay","timestamp":"2020-04-29T09:20:27.881Z","value":2000}
{"type":"message","timestamp":"2020-04-29T09:19:53.074Z","text":"Here's our guide to Teams - https://xxx.pdf"}
{"type":"delay","timestamp":"2020-04-29T09:19:52.713Z","value":2000}
{"type":"message","timestamp":"2020-04-29T09:19:44.098Z","text":"Welcome to SearchBot! Ask me a question and I will try to answer it."}
{"type":"message","timestamp":"2020-04-29T09:20:25.737Z","text":"annalect"}
{"type":"typing","timestamp":"2020-04-29T09:19:52.713Z"}
{"type":"trace","timestamp":"2020-04-29T09:19:53.011Z","value":{"message":{"type":"message","id":"ACjTjAWTJtvJeh6szBjzie-a|0000001","timestamp":"2020-04-29T09:19:50.525Z","serviceUrl":"https://webchat.botframework.com/","channelId":"webchat","from":{"id":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","name":"You","role":"user"},"conversation":{"id":"ACjTjAWTJtvJeh6szBjzie-a"},"recipient":{"id":"omgsearchbot@8LBpMqDWWGw","name":"omgsearchbot"},"textFormat":"plain","locale":"en-US","text":"Teams","entities":[{"type":"ClientCapabilities","requiresBotState":true,"supportsListening":true,"supportsTts":true}],"channelData":{"clientActivityID":"1588151990502bgbcx3nbrkb","clientTimestamp":"2020-04-29T09:19:50.502Z"}},"queryResults":[{"questions":["How do I use Teams"],"answer":"Here's our guide to Teams - https://xxx.pdf","score":0.81349999999999989,"id":13,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}}],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"message","timestamp":"2020-04-29T09:19:50.525Z","text":"Teams"}
{"type":"typing","timestamp":"2020-04-29T08:37:42.242Z"}
{"type":"trace","timestamp":"2020-04-29T08:37:42.604Z","value":{"message":{"type":"message","id":"K1XxtCn6qeEDuW2ASjWv6R-a|0000000","timestamp":"2020-04-29T08:37:28.594Z","serviceUrl":"https://webchat.botframework.com/","channelId":"webchat","from":{"id":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","name":"You","role":"user"},"conversation":{"id":"K1XxtCn6qeEDuW2ASjWv6R-a"},"recipient":{"id":"omgsearchbot@8LBpMqDWWGw","name":"omgsearchbot"},"textFormat":"plain","locale":"en-US","text":"hi","entities":[{"type":"ClientCapabilities","requiresBotState":true,"supportsListening":true,"supportsTts":true}],"channelData":{"clientActivityID":"1588149448577tc6dant0tj9","clientTimestamp":"2020-04-29T08:37:28.577Z"}},"queryResults":[],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"message","timestamp":"2020-04-29T08:37:40.215Z","text":"Welcome to SearchBot! Ask me a question and I will try to answer it."}
{"type":"message","timestamp":"2020-04-29T09:20:28.280Z","text":"Here's the Annalect one-pager - https:xxx.pdf"}
{"type":"delay","timestamp":"2020-04-29T08:37:42.242Z","value":2000}
{"type":"conversationUpdate","timestamp":"2020-04-29T08:37:25.793Z"}
{"type":"trace","timestamp":"2020-04-29T09:20:28.218Z","value":{"message":{"type":"message","id":"ACjTjAWTJtvJeh6szBjzie-a|0000003","timestamp":"2020-04-29T09:20:25.737Z","serviceUrl":"https://webchat.botframework.com/","channelId":"webchat","from":{"id":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","name":"You","role":"user"},"conversation":{"id":"ACjTjAWTJtvJeh6szBjzie-a"},"recipient":{"id":"omgsearchbot@8LBpMqDWWGw","name":"omgsearchbot"},"textFormat":"plain","locale":"en-US","text":"annalect","channelData":{"clientActivityID":"1588152025726lxcqw61814d","clientTimestamp":"2020-04-29T09:20:25.726Z"}},"queryResults":[{"questions":["I want to find out more about Annalect"],"answer":"Here's the Annalect one-pager - https://xxx.pdf?csf=1&web=1&e=dGEBZ5","score":0.7965000000000001,"id":16,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}}],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"message","timestamp":"2020-04-29T08:37:28.594Z","text":"hi"}
{"type":"message","timestamp":"2020-04-29T08:37:42.698Z","text":"I'm sorry, I don't have an answer to that question yet, but I'll add it to my learning list!"}
{"type":"message","timestamp":"2020-04-29T08:37:53.695Z","text":"Benefits"}
{"type":"typing","timestamp":"2020-04-29T08:37:48.440Z"}
{"type":"typing","timestamp":"2020-04-29T08:37:55.843Z"}
{"type":"trace","timestamp":"2020-04-29T08:37:48.611Z","value":{"message":{"type":"message","id":"K1XxtCn6qeEDuW2ASjWv6R-a|0000003","timestamp":"2020-04-29T08:37:46.327Z","serviceUrl":"https://webchat.botframework.com/","channelId":"webchat","from":{"id":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","name":"You","role":"user"},"conversation":{"id":"K1XxtCn6qeEDuW2ASjWv6R-a"},"recipient":{"id":"omgsearchbot@8LBpMqDWWGw","name":"omgsearchbot"},"textFormat":"plain","locale":"en-US","text":"Teams","channelData":{"clientActivityID":"1588149466306udb6svuls1","clientTimestamp":"2020-04-29T08:37:46.306Z"}},"queryResults":[{"questions":["How do I use Teams"],"answer":"Here's our guide to Teams - https://xxx.pdf","score":0.81349999999999989,"id":13,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}}],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"delay","timestamp":"2020-04-29T08:37:55.843Z","value":2000}
{"type":"message","timestamp":"2020-04-29T08:37:48.674Z","text":"Here's our guide to Teams - https://xxx.pdf"}
{"type":"delay","timestamp":"2020-04-29T08:37:48.440Z","value":2000}
{"type":"message","timestamp":"2020-04-29T08:37:46.327Z","text":"Teams"}
{"type":"trace","timestamp":"2020-04-29T08:37:56.014Z","value":{"message":{"type":"message","id":"K1XxtCn6qeEDuW2ASjWv6R-a|0000005","timestamp":"2020-04-29T08:37:53.695Z","serviceUrl":"https://webchat.botframework.com/","channelId":"webchat","from":{"id":"de4b2545-f94a-4a86-8ab8-e639889a2cf8","name":"You","role":"user"},"conversation":{"id":"K1XxtCn6qeEDuW2ASjWv6R-a"},"recipient":{"id":"omgsearchbot@8LBpMqDWWGw","name":"omgsearchbot"},"textFormat":"plain","locale":"en-US","text":"Benefits","channelData":{"clientActivityID":"1588149473675fitfj50t5w8","clientTimestamp":"2020-04-29T08:37:53.675Z"}},"queryResults":[{"questions":["What are the OMG benefits"],"answer":"You can find details of all our benefits here - https://xxx.pdf","score":0.88,"id":4,"source":"Editorial","metadata":[],"context":{"isContextOnly":false,"prompts":[]}}],"knowledgeBaseId":"078a1d0a-8d56-4f47-8836-34d1e818f2cc","scoreThreshold":0.3,"top":3,"strictFilters":[],"metadataBoost":[],"context":{}}}
{"type":"message","timestamp":"2020-04-29T08:37:56.120Z","text":"You can find details of all our benefits here - https://xxx.pdf"}

Answer 1

Since the filter data you are looking for is inside the JSON file, you'll probably have to use Data Flow to solve this problem. Inside the Data Flow, use a Filter row modifier to select only the rows that meet your criteria. Since it looks like the row schemas are different based on the type, so you could branch multiple Filters for each type and Select only the desired columns. For the final output, you could either aggregate the branches back together using a Union, or have multiple Sinks (one for each type).