How to export Azure Prices REST API to CSV

Question

I would like to save the whole Azure Prices REST API to CSV.

In order to do so I have to query the endpoint https://prices.azure.com/api/retail/prices which ends with a:

"NextPageLink":"https://prices.azure.com:443/api/retail/prices?$skip=100","Count":100}

I wrote a Python scripts that could help me grab that NextPageLink and loop it into a function:

import requests
import json
import pandas as pd 
from timeit import default_timer as timer
from datetime import timedelta

start = timer()
NextPageLink = "https://prices.azure.com/api/retail/prices"

def GetJSON(NextPageLink):
    wjdata = requests.get(NextPageLink).json()
    df = pd.DataFrame(wjdata)
    df.to_csv("test.csv", index=False)
    if 'NextPageLink' in wjdata:
        print (timer(), wjdata['NextPageLink'])
        NextPageLink = wjdata['NextPageLink']
        return NextPageLink

GetJSON(NextPageLink) 

The script is quite simple but it just saves the first page and doesn't query the NextPageLink .

What am I doing wrong?

Answer 1

To get all data from the API you can try (there are 4558 requests total):

import requests
import pandas as pd

url = "https://prices.azure.com/api/retail/prices"

all_data = []
while True:
    print(url)
    data = requests.get(url).json()
    all_data.extend(data["Items"])
    if data["NextPageLink"]:
        url = data["NextPageLink"]
    else:
        break

df = pd.DataFrame(all_data)
print(df.head().to_markdown(index=False))
df.to_csv("data.csv", index=False)

Prints:

......

https://prices.azure.com:443/api/retail/prices?$skip=455600
https://prices.azure.com:443/api/retail/prices?$skip=455700
https://prices.azure.com:443/api/retail/prices?$skip=455800

currencyCode	tierMinimumUnits	retailPrice	unitPrice	armRegionName	location	effectiveStartDate	meterId	meterName	productId	skuId	availabilityId	productName	skuName	serviceName	serviceId	serviceFamily	unitOfMeasure	type	isPrimaryMeterRegion	armSkuName	reservationTerm	effectiveEndDate
USD	0	3	3	southindia	IN South	2016-12-01T00:00:00Z	ff92c14c-af83-412e-9144-a2542dfe0b4f	Certificate Renewal Request	DZH318Z0BQG0	DZH318Z0BQG0/0001		Key Vault	Premium	Key Vault	DZH3157JCZ2M	Security	1	Consumption	False		nan	nan
USD	0	3	3	southcentralus	US South Central	2016-12-01T00:00:00Z	ff92c14c-af83-412e-9144-a2542dfe0b4f	Certificate Renewal Request	DZH318Z0BQG0	DZH318Z0BQG0/002Q		Key Vault	Standard	Key Vault	DZH3157JCZ2M	Security	1	Consumption	False		nan	nan
USD	0	3	3	jioindiawest	IN West Jio	2021-04-15T00:00:00Z	ff92c14c-af83-412e-9144-a2542dfe0b4f	Certificate Renewal Request	DZH318Z0BQG0	DZH318Z0BQG0/004V		Key Vault	Premium	Key Vault	DZH3157JCZ2M	Security	1	Consumption	False		nan	nan
USD	0	3	3	germanywestcentral	DE West Central	2016-12-01T00:00:00Z	ff92c14c-af83-412e-9144-a2542dfe0b4f	Certificate Renewal Request	DZH318Z0BQG0	DZH318Z0BQG0/003Q		Key Vault	Premium	Key Vault	DZH3157JCZ2M	Security	1	Consumption	False		nan	nan
USD	0	3	3	westeurope	EU West	2016-12-01T00:00:00Z	ff92c14c-af83-412e-9144-a2542dfe0b4f	Certificate Renewal Request	DZH318Z0BQG0	DZH318Z0BQG0/001Z		Key Vault	Premium	Key Vault	DZH3157JCZ2M	Security	1	Consumption	False		nan	nan

and saves data.csv (screenshot from LibreOffice):