将数据从 JSON 转换为 pandas dataframe

Question

I would like to extract some data from the following line of text. The data of interest are: 'exchange_symbol under both market_pair_base and 'market_pair_quote , all the data under exchange_reported . Any help would be great.

{'status': {'timestamp': '2019-10-04T02:57:39.238Z', 'error_code': 0, 'error_message': None, 'elapsed': 14, 'credit_count': 1, 'notice': None}, 'data': {'id': 112, 'name': 'Liquid', 'slug': 'liquid', 'num_market_pairs': 190, 'market_pairs': [{'market_pair_base': {'exchange_symbol': 'ETH', 'currency_id': 1027, 'currency_symbol': 'ETH', 'currency_type': 'cryptocurrency'}, 'market_pair_quote': {'exchange_symbol': 'JPY', 'currency_id': 2797, 'currency_symbol': 'JPY', 'currency_type': 'fiat'}, 'quote': {'exchange_reported': {'price': 18522.8757, 'volume_24h_base': 12901.4143123, 'volume_24h_quote': 238971293.660934, 'last_updated': '2019-10-04T02:57:04.000Z'}, 'USD': {'price': 173.503768779353, 'volume_24h': 2238444.00576794, 'last_updated': '2019-10-04T02:57:04.000Z'}}, 'market_id': 4431, 'market_pair': 'ETH/JPY', 'category': 'spot', 'fee_type': 'percentage'}, {'market_pair_base': {'exchange_symbol': 'XRP', 'currency_id': 52, 'currency_symbol': 'XRP', 'currency_type': 'cryptocurrency'}, 'market_pair_quote': {'exchange_symbol': 'JPY', 'currency_id': 2797, 'currency_symbol': 'JPY', 'currency_type': 'fiat'}, 'quote': {'exchange_reported': {'price': 26.55199, 'volume_24h_base': 8223150.63965144, 'volume_24h_quote': 218341013.552519, 'last_updated': '2019-10-04T02:56:04.000Z'}, 'USD': {'price': 0.248712479001935, 'volume_24h': 2045200.18079406, 'last_updated': '2019-10-04T02:56:04.000Z'}}, 'market_id': 16254, 'market_pair': 'XRP/JPY', 'category': 'spot', 'fee_type': 'percentage'}]}}

I tried the following code but this send an error:

    File "proc_manual_comp.py", line 112, in <module>
     for item in data['data'][currency]['market_pairs']
  TypeError: 'int' object is not subscriptable

the code:

c_price = []
url = 'https://pro-api.coinmarketcap.com/v1/exchange/market-pairs/latest'
parameters = {
   'id':'112',
   'start':'1',
   'limit':'91'
 }
 headers = {
   'Accepts': 'application/json',
   'X-CMC_PRO_API_KEY': 'XXXXXXXXXXXXXXXXXXXXXXX',
 }
 session = Session()
 session.headers.update(headers)
 response = session.get(url, params=parameters)
 data = json.loads(response.text)
 for currency in  data['data']:
   used_list = [ 
        item['market_pairs']['market_pair_base'] 
        for item in data['data'][currency]['market_pairs']
   ]
   price = pd.DataFrame.from_records(used_list)
   print(price)
   price['timestamp'] = pd.to_datetime(price['timestamp'])
   price['timestamp'] = price['timestamp'].astype(str).str[:-6]
   price_c = price.set_index('timestamp').close
   c_price.append(price_c.rename(currency))
   print(c_price)
  c_price = pd.concat(c_price, axis=1)

Expected output:

market_pair_base  market_pair_quoted   last_updated             price
ETH               JPY                  2019-10-04T02:57:04.000Z 18522.8757

Answer 1

it seems like a problem with your for loop.

for currency in  data['data']:

here since data['data'] is a dict hence values of currency is going to be keys of dict, an alternate way can be something like this

for market_pairs in data['data']['market_pairs']:
    used_list = [market_pair['market_pair_base'] for market_pair in market_pairs]
    price = pd.DataFrame.from_records(used_list)

this is based on the fact that you wanted to put all market_pair_base in a Dataframe.

Answer 2

Assumed JSON as given in d variable

Try below snippet:

target_df=pd.DataFrame(columns=['market_pair_base','market_pair_quote','price','last_updated'])
target=dict()

usedlist=d['data']['market_pairs']

for i in range(len(usedlist)):
    target['market_pair_base']=[usedlist[i]['market_pair_base']['exchange_symbol']]
    target['market_pair_quote'] = [usedlist[i]['market_pair_quote']['exchange_symbol']]
    target['price'] = [usedlist[i]['quote']['exchange_reported']['price']]
    target['last_updated'] = [usedlist[i]['quote']['exchange_reported']['last_updated']]
    target_df=pd.concat([target_df, pd.DataFrame(target)], ignore_index=True)

This print(target_df) will give desired output as below:

market_pair_base market_pair_quote        price              last_updated
0              ETH               JPY  18522.87570  2019-10-04T02:57:04.000Z
1              XRP               JPY     26.55199  2019-10-04T02:56:04.000Z