如何在结构化流中保留和重置状态？

Question

I have a requirement where I need to show the aggregated count of a specific category starting from the onset of current day till current time.

I am using Structure Streaming to do the grouping. As window does not persist state of a dataframe, I am not sure how to implement the logic where I can persist its state and increment the counter on previous state. Also how will I reset the state on the onset of a new day.

Input Record:

{"Floor_Id" : "Shop Floor 1",
"HaltRecord" : {
    "HaltReason" : "Danahydraulic Error",
    "Severity" : "Low",
    "FaultErrorCategory" : "Docked",
    "NonFaultErrorCategory" : null
},
"Description" : "Forklift",
"Category" : {
    "Type" : "Halt",
    "End_time" : NumberLong(2018-02-13T12:00:01),
    "Start_time" : NumberLong(2018-02-13T12:00:00)
},
"Asset_Id" : 123,
"isError" : "y",
"Timestamp": 2018-02-13T12:00:01}

Output Response:

{
    "Floor_Id": "Shop Floor 1",
    "Error_Category": [
        {
            "Category": "Operator Error",
            "DataPoints": 
                {
                    "NumberOfErrors": 20,
                    "Date": 2018-02-13
                }
        },
        {
            "Category": "Danahydraulic Error",
            "DataPoints": {
                    "NumberOfErrors": 15,
                    "Date": 2018-02-13
                }
        }
    ]
}

Any help is much appreciated.

Answer 1

我没有使用过结构化流的状态函数，但我知道它是mapGroupWithState函数，可以提供持久状态和计数逻辑的功能。