在电力自动化中对大型阵列进行排序

Question

I have around 700 resources of which I have metrics array containing 1300+ values, float values

I am trying to compute 80th percentile of this large array of values in Power automate.

My issue is its taking 10-15 min per case, also I have to construct a separate array as my source array sometimes has non existing metric property

my source array looks like this (having 1392 elements)

{
"data":[
{
"timestamp":"2022-11-11T07:00:00Z",
"average":34.4
},
 {
 "timestamp": "2022-12-10T06:00:00Z"
}
]}

I tried various ways

1. I made another array first, wherever average was not present, I added a zero to result array
2. then sorted this array ( via sort expression)
3. applied formula for 80th percentile to get right index
4. extracted that index value from array.

Step 1 is taking a lot of time unfortunately, Apply to each is being used with max concurrency.

Please advise how we can make this faster.

Thanks

Answer 1

I think you should look at using the Advanced Data Operations connector.

There's an expert function that allows you to run an SQL statement over your dataset. You could filter this data further to everything above the 80th percentile by simply adding a WHERE clause at the end.

This is the flow I tested with...

... so in that single step, it will do everything you want it to.

This is the SQL statement supplied in the screenshot above...

SELECT *, PERCENT_RANK() OVER(ORDER BY [Average]) AS [Percentile] 
FROM (
    SELECT * FROM [Data] WHERE [Average] IS NOT NULL
) AS [NewData] 
ORDER BY [Percentile] DESC

Note: To make sure it's producing the correct result, I've removed all items that have a null average value.

This is the resulting flow which took a mere 6 seconds to run...

... and here is the resulting JSON array with the data...

[
    {
        "average": 22.674375,
        "timeStamp": "2022-11-11T07:30:00",
        "Percentile": 1
    },
    {
        "average": 22.1976666666667,
        "timeStamp": "2022-11-11T19:30:00",
        "Percentile": 0.985915492957746
    },
    {
        "average": 22.1189166666667,
        "timeStamp": "2022-11-11T07:00:00",
        "Percentile": 0.971830985915493
    },
    {
        "average": 21.85075,
        "timeStamp": "2022-11-11T08:30:00",
        "Percentile": 0.957746478873239
    },
    {
        "average": 21.2718020833333,
        "timeStamp": "2022-11-12T00:00:00",
        "Percentile": 0.943661971830986
    },
    {
        "average": 21.2456145833333,
        "timeStamp": "2022-11-11T16:30:00",
        "Percentile": 0.929577464788732
    },
    {
        "average": 21.1545520833333,
        "timeStamp": "2022-11-11T09:30:00",
        "Percentile": 0.915492957746479
    },
    {
        "average": 21.09884375,
        "timeStamp": "2022-11-11T15:00:00",
        "Percentile": 0.901408450704225
    },
    {
        "average": 21.0786145833333,
        "timeStamp": "2022-11-12T09:30:00",
        "Percentile": 0.887323943661972
    },
    {
        "average": 20.9709270833333,
        "timeStamp": "2022-11-11T19:00:00",
        "Percentile": 0.873239436619718
    },
    {
        "average": 20.7876979166667,
        "timeStamp": "2022-11-12T06:30:00",
        "Percentile": 0.859154929577465
    },
    {
        "average": 20.7539270833333,
        "timeStamp": "2022-11-12T05:00:00",
        "Percentile": 0.845070422535211
    },
    {
        "average": 20.724375,
        "timeStamp": "2022-11-12T16:30:00",
        "Percentile": 0.830985915492958
    },
    {
        "average": 20.7124479166667,
        "timeStamp": "2022-11-11T15:30:00",
        "Percentile": 0.816901408450704
    },
    {
        "average": 20.7032708333333,
        "timeStamp": "2022-11-12T13:30:00",
        "Percentile": 0.802816901408451
    },
    {
        "average": 20.6885104166667,
        "timeStamp": "2022-11-11T08:00:00",
        "Percentile": 0.788732394366197
    },
    {
        "average": 20.6832083333333,
        "timeStamp": "2022-11-11T13:30:00",
        "Percentile": 0.774647887323944
    },
    {
        "average": 20.6540208333333,
        "timeStamp": "2022-11-11T10:30:00",
        "Percentile": 0.76056338028169
    },
    {
        "average": 20.6439166666667,
        "timeStamp": "2022-11-12T04:30:00",
        "Percentile": 0.746478873239437
    },
    {
        "average": 20.59803125,
        "timeStamp": "2022-11-12T07:00:00",
        "Percentile": 0.732394366197183
    },
    {
        "average": 20.5738125,
        "timeStamp": "2022-11-11T06:30:00",
        "Percentile": 0.71830985915493
    },
    {
        "average": 20.5018645833333,
        "timeStamp": "2022-11-12T00:30:00",
        "Percentile": 0.704225352112676
    },
    {
        "average": 20.500875,
        "timeStamp": "2022-11-11T14:30:00",
        "Percentile": 0.690140845070423
    },
    {
        "average": 20.46678125,
        "timeStamp": "2022-11-11T16:00:00",
        "Percentile": 0.676056338028169
    },
    {
        "average": 20.43015625,
        "timeStamp": "2022-11-12T07:30:00",
        "Percentile": 0.661971830985916
    },
    {
        "average": 20.4273020833333,
        "timeStamp": "2022-11-11T11:00:00",
        "Percentile": 0.647887323943662
    },
    {
        "average": 20.4161041666667,
        "timeStamp": "2022-11-11T21:30:00",
        "Percentile": 0.633802816901409
    },
    {
        "average": 20.3894791666667,
        "timeStamp": "2022-11-11T10:00:00",
        "Percentile": 0.619718309859155
    },
    {
        "average": 20.3787604166667,
        "timeStamp": "2022-11-11T17:30:00",
        "Percentile": 0.605633802816901
    },
    {
        "average": 20.3724270833333,
        "timeStamp": "2022-11-11T09:00:00",
        "Percentile": 0.591549295774648
    },
    {
        "average": 20.3398333333333,
        "timeStamp": "2022-11-12T12:30:00",
        "Percentile": 0.577464788732394
    },
    {
        "average": 20.2841145833333,
        "timeStamp": "2022-11-12T08:30:00",
        "Percentile": 0.563380281690141
    },
    {
        "average": 20.2529449152542,
        "timeStamp": "2022-11-11T13:00:00",
        "Percentile": 0.549295774647887
    },
    {
        "average": 20.2286875,
        "timeStamp": "2022-11-11T17:00:00",
        "Percentile": 0.535211267605634
    },
    {
        "average": 20.22403125,
        "timeStamp": "2022-11-11T14:00:00",
        "Percentile": 0.52112676056338
    },
    {
        "average": 20.2209375,
        "timeStamp": "2022-11-11T11:30:00",
        "Percentile": 0.507042253521127
    },
    {
        "average": 20.1904895833333,
        "timeStamp": "2022-11-12T06:00:00",
        "Percentile": 0.492957746478873
    },
    {
        "average": 20.1900729166667,
        "timeStamp": "2022-11-11T12:30:00",
        "Percentile": 0.47887323943662
    },
    {
        "average": 20.148125,
        "timeStamp": "2022-11-12T01:30:00",
        "Percentile": 0.464788732394366
    },
    {
        "average": 20.129625,
        "timeStamp": "2022-11-11T23:30:00",
        "Percentile": 0.450704225352113
    },
    {
        "average": 20.0616666666667,
        "timeStamp": "2022-11-12T14:30:00",
        "Percentile": 0.436619718309859
    },
    {
        "average": 20.0449895833333,
        "timeStamp": "2022-11-12T10:00:00",
        "Percentile": 0.422535211267606
    },
    {
        "average": 20.0333645833333,
        "timeStamp": "2022-11-11T12:00:00",
        "Percentile": 0.408450704225352
    },
    {
        "average": 19.9994583333333,
        "timeStamp": "2022-11-11T21:00:00",
        "Percentile": 0.394366197183099
    },
    {
        "average": 19.9905,
        "timeStamp": "2022-11-12T11:00:00",
        "Percentile": 0.380281690140845
    },
    {
        "average": 19.9578125,
        "timeStamp": "2022-11-12T09:00:00",
        "Percentile": 0.366197183098592
    },
    {
        "average": 19.94271875,
        "timeStamp": "2022-11-12T05:30:00",
        "Percentile": 0.352112676056338
    },
    {
        "average": 19.9379895833333,
        "timeStamp": "2022-11-12T10:30:00",
        "Percentile": 0.338028169014084
    },
    {
        "average": 19.91659375,
        "timeStamp": "2022-11-11T18:00:00",
        "Percentile": 0.323943661971831
    },
    {
        "average": 19.8985104166667,
        "timeStamp": "2022-11-12T01:00:00",
        "Percentile": 0.309859154929577
    },
    {
        "average": 19.8849479166667,
        "timeStamp": "2022-11-12T14:00:00",
        "Percentile": 0.295774647887324
    },
    {
        "average": 19.8658229166667,
        "timeStamp": "2022-11-12T15:00:00",
        "Percentile": 0.28169014084507
    },
    {
        "average": 19.85403125,
        "timeStamp": "2022-11-11T18:30:00",
        "Percentile": 0.267605633802817
    },
    {
        "average": 19.8325,
        "timeStamp": "2022-11-11T22:30:00",
        "Percentile": 0.253521126760563
    },
    {
        "average": 19.7983958333333,
        "timeStamp": "2022-11-12T04:00:00",
        "Percentile": 0.23943661971831
    },
    {
        "average": 19.7926145833333,
        "timeStamp": "2022-11-12T15:30:00",
        "Percentile": 0.225352112676056
    },
    {
        "average": 19.780875,
        "timeStamp": "2022-11-11T20:30:00",
        "Percentile": 0.211267605633803
    },
    {
        "average": 19.7798333333333,
        "timeStamp": "2022-11-11T23:00:00",
        "Percentile": 0.197183098591549
    },
    {
        "average": 19.76359375,
        "timeStamp": "2022-11-12T11:30:00",
        "Percentile": 0.183098591549296
    },
    {
        "average": 19.7552604166667,
        "timeStamp": "2022-11-12T08:00:00",
        "Percentile": 0.169014084507042
    },
    {
        "average": 19.695125,
        "timeStamp": "2022-11-12T13:00:00",
        "Percentile": 0.154929577464789
    },
    {
        "average": 19.50509375,
        "timeStamp": "2022-11-11T20:00:00",
        "Percentile": 0.140845070422535
    },
    {
        "average": 19.4972291666667,
        "timeStamp": "2022-11-12T16:00:00",
        "Percentile": 0.126760563380282
    },
    {
        "average": 19.4948541666667,
        "timeStamp": "2022-11-11T22:00:00",
        "Percentile": 0.112676056338028
    },
    {
        "average": 19.4910104166667,
        "timeStamp": "2022-11-12T12:00:00",
        "Percentile": 0.0985915492957746
    },
    {
        "average": 19.4601979166667,
        "timeStamp": "2022-11-12T03:00:00",
        "Percentile": 0.0845070422535211
    },
    {
        "average": 19.2794375,
        "timeStamp": "2022-11-12T03:30:00",
        "Percentile": 0.0704225352112676
    },
    {
        "average": 15.9129791666667,
        "timeStamp": "2022-11-12T02:30:00",
        "Percentile": 0.0563380281690141
    },
    {
        "average": 15.20909375,
        "timeStamp": "2022-11-12T02:00:00",
        "Percentile": 0.0422535211267606
    },
    {
        "average": 11.3621875,
        "timeStamp": "2022-11-12T17:00:00",
        "Percentile": 0.028169014084507
    },
    {
        "average": 9.31830208333333,
        "timeStamp": "2022-11-12T17:30:00",
        "Percentile": 0.0140845070422535
    },
    {
        "average": 9.08600961538462,
        "timeStamp": "2022-11-12T18:00:00",
        "Percentile": 0
    }
]

It should be noted that the Expert function is a part of the Standard pricing model. It's very affordable but whether you deem that so or not is very much relative.