将结果添加到字典的最快方法

Question

I am trying to deal with the Freshservice v1 API which returns paginated results, but does not include an EOF key in the last page. I am therefore trying to construct a dict with the results and work out when we have reached the last page.

At the moment I have the following code:

assets = dict()

page = 1
request_string = 'https://domain.freshservice.com/cmdb/items.json?page=%s' % page
returned_assets = requests.get(request_string, auth=(user, pw)).json()
assets.update(returned_assets)

if len(returned_assets) == 50: # if there are 50 assets, there either more pages or this is a full page
    while len(returned_assets) != 0:
        page += 1
        request_string = 'https://domain.freshservice.com/cmdb/items.json?page=%s' % page
        returned_assets = requests.get(request_string, auth=(user, pw)).json()
        assets.update(returned_assets)

Currently when I run the code it takes an age to update assets with returned_assets .

This is an example response from the API:

[
    {
        "agent_id": null,
        "asset_tag": "",
        "assigned_on": null,
        "ci_type_id": 6000806999,
        "created_at": "2017-07-11T15:29:22+01:00",
        "department_id": null,
        "depreciation_id": null,
        "description": "Test",
        "display_id": 2,
        "id": 6002201649,
        "impact": 3,
        "location_id": null,
        "name": "Test entry",
        "salvage": null,
        "updated_at": "2018-07-05T17:39:20+01:00",
        "usage_type": 1,
        "user_id": null,
        "department_name": null,
        "used_by": null,
        "business_impact": "High",
        "agent_name": null,
        "levelfield_values": {
            "state_6000742951": "Operational",
            "bandwidth_6000742951": "1Gbit/s",
            "network_range_6000742951": null,
            "carrier_6000806999": "Test",
            "supplier_circuit_id_6000806999": "AAAAAAAAA",
            "carrier_tail_provider_6000806999": "Test",
            "carrier_tail_order_id_onea_6000806999": null,
            "a_end_address_6000806999": "Test",
            "a_end_termination_details_6000806999": null,
            "b_end_address_6000806999": "Test",
            "b_end_termination_details_6000806999": "Test",
            "a_end_vlan_id_6000806999": 0,
            "b_end_vlan_id_6000806999": "0",
            "notes_6000806999": null
        },
        "ci_type_name": "Circuit",
        "location_name": null,
        "product_name": null,
        "vendor_name": null,
        "state_name": null
    }
]

How can I do this in a more efficient way? Would it be better to store the results in a list?

Answer 1

As suggested by various posters I tried adding my results to a list instead, and processing time was massively faster. I used list.extend(new_list_to_add) to add the new entries to the list.

The new code looks as so:

assets = []

page = 1
request_string = 'https://domain.freshservice.com/cmdb/items.json?page=%s' % page
returned_assets = requests.get(request_string, auth=(user, pw)).json()
assets = returned_assets

if len(assets) == 50: # if there are 50 assets, there either more pages or this is a full page
    while len(returned_assets) != 0:
        page += 1
        request_string = 'https://domain.freshservice.com/cmdb/items.json?page=%s' % page
        returned_assets = requests.get(request_string, auth=(user, pw)).json()
        assets.extend(returned_assets)