如何在大数据平台中比较大文件？

Question

Here are some big files coming in a day, not very frequent, 2-3 every single day, and they are converted into JSON format.

The file's content looks like:

[
    {
        "spa_ref_data": {
            "approval_action": "New",
            "spa_ref_no": "6500781413",
            "begin_date": null,
            "end_date": "20191009",
            "doc_file_name": "LEN_SPA_6500781413.json",
            "LEN_V": "v1",
            "version_no": null,
            "spa_ref_id": null,
            "spa_ref_notes": "MC00020544",
            "vend_code": "LEN"
        },
        "cust_data": [
            {
                "cust_name": null,
                "cust_no": null,
                "cust_type": "E",
                "state": null,
                "country": null
            },
            {
                "cust_name": null,
                "cust_no": null,
                "cust_type": "C",
                "state": null,
                "country": null
            }
        ],
        "product_data": [
            {
                "mfg_partno": "40AH0135US",
                "std_price": null,
                "rebate_amt": "180",
                "max_spa_qty": null,
                "rebate_type": null,
                "min_spa_qty": null,
                "min_cust_qty": null,
                "max_cust_qty": null,
                "begin_date": "20180608",
                "end_date": null
            },
            {
                "mfg_partno": "40AJ0135US",
                "std_price": null,
                "rebate_amt": "210",
                "max_spa_qty": null,
                "rebate_type": null,
                "min_spa_qty": null,
                "min_cust_qty": null,
                "max_cust_qty": null,
                "begin_date": "20180608",
                "end_date": null
            }
        ]
    },
    {
        "spa_ref_data": {
            "approval_action": "New",
            "spa_ref_no": "5309745006",
            "begin_date": null,
            "end_date": "20190426",
            "doc_file_name": "LEN_SPA_5309745006.json",
            "LEN_V": "v1",
            "version_no": null,
            "spa_ref_id": null,
            "spa_ref_notes": "MC00020101",
            "vend_code": "LEN"
        },
        "cust_data": [
            {
                "cust_name": null,
                "cust_no": null,
                "cust_type": "E",
                "state": null,
                "country": null
            },
            {
                "cust_name": null,
                "cust_no": null,
                "cust_type": "C",
                "state": null,
                "country": null
            }
        ],
        "product_data": [
            {
                "mfg_partno": "10M8S0HU00",
                "std_price": null,
                "rebate_amt": "698",
                "max_spa_qty": null,
                "rebate_type": null,
                "min_spa_qty": null,
                "min_cust_qty": null,
                "max_cust_qty": null,
                "begin_date": "20180405",
                "end_date": null
            },
            {
                "mfg_partno": "20K5S0CM00",
                "std_price": null,
                "rebate_amt": "1083",
                "max_spa_qty": null,
                "rebate_type": null,
                "min_spa_qty": null,
                "min_cust_qty": null,
                "max_cust_qty": null,
                "begin_date": "20180405",
                "end_date": null
            }
        ]
    }
]

This is a mock data file.In fact, it is a array with length 30000+.

My target is to compare the coming one with the latest one. And get the changed data.

The leader says I must use the big data techs. And the performance must be good.

We use Apache NIFI and hadoop big data tools to do it.

Is there some advice ?

Answer 1

For example, you can use ExecuteScript processor with js scrpit to compare jsons. It works fast. Also you can split your big array json with SplitRecord processor and compare each one by executeScript proccessor. It also works good.