Troubleshooting format of JSON get request

问题

I'm running into an issue with trying to convert a get request response into my desired format. The structure of the JSON response is a little more complicated and I'm having trouble with getting it into the right format.

Here's my code for converting the json script into a pandas dataframe:

data = pd.DataFrame.from_dict(resp_isrc.json())
data.head()

This is JSON structure that I am getting is below:

{"country":"US",
 "artist_name":"Frank Ocean",
 "title_name":"Provider",
 "release_date":"2017-08-25",
 "core_genre":"R&B/Hip-Hop",
 "metrics":[{"name":"ISRC w/SES On-Demand",
             "value":[{"name":"tp","value":2810},
                      {"name":"lp","value":2450},
                      {"name":"ytd","value":2740},
                      {"name":"atd","value":554267}]},

回答1:

A bit of preprocessing may be needed to achieve what you want. This is fairly typical for json results from requests. In some cases a more general approach may be used. But here, since the structure is fairly simple, metrics can be flattened fairly easily.

def flatten2df(resp_isrc, df=None):
    metrics = resp_isrc.pop('metrics')
    flattened_json = []
    for item in metrics:
        metric = item['name']
        values = item['value']
        new_entry = {entry['name']: entry['value'] for entry in values}
        new_entry.update({'metric': metric})
        new_entry.update(resp_isrc)
        flattened_json.append(new_entry)
    new_df = pandas.DataFrame(flattened_json)
    if df is None:
        return new_df
    return df.append(new_df, ignore_index=True)

If there are multiple requests, you can loop it over

df = pandas.DataFrame()

for i in lst:
    response = requests.get(...)
    df = flatten2df(response.json(), df)

Faster:

Appends all flattened results to a list first and then input into dataframe.

def flatten2list(resp_isrc, flattened_json):
    metrics = resp_isrc.pop('metrics')
    for item in metrics:
        metric = item['name']
        values = item['value']
        new_entry = {entry['name']: entry['value'] for entry in values}
        new_entry.update({'metric': metric})
        new_entry.update(resp_isrc)
        flattened_json.append(new_entry)

If there are multiple requests, you can loop it over

results = []

for i in lst:
    response = requests.get(...)
    flatten2list(response.json(), results)

df = pandas.DataFrame(results)

回答2:

Here how you can extract name, ip lp data from json dictionary:

import pandas as pd

data = []
for item in raw_data.get('metrics'):
    tp = 0
    lp = 0
    for name in item.get('value'):
        if 'tp' in name.values():
            tp = name.get('value')
        if 'lp' in name.values():
            lp = name.get('value')
    data.append([item.get('name'), tp, lp])

df = pd.DataFrame(data, columns=['name', 'ip', 'lp'])

Output:

                         name      ip      lp
0        ISRC w/SES On-Demand    2810    2450
1  ISRC w/SES On-Demand Audio    2735    2384
2              Digital Tracks      21      19
3   Streaming On-Demand Total  395705  348276
4   Streaming On-Demand Audio  367707  323399
5   Streaming On-Demand Video   27998   24877
6  Streaming Programmed Total     187     153

来源：https://stackoverflow.com/questions/60102370/troubleshooting-format-of-json-get-request

标签

python

json

pandas

get

request