Expand nested list of dictionaries in a pandas dataframe column

问题

I have this dataframe called "leads" I got from saving the output of an SFDC SOQL into a dataframe. I have been trying to expand column "Leads__r.record"

Company    Month     Amount   Leads__r.done   Leads__r.record                      Leads__r.totalSize
0   A1  September   500000    True          [{u'Id': u'Q500, u'Company': u'...                1.0
1   B1  December    16200     True          [{u'Id': u'Q600', u'Company': u'...               1.0
2   C1  December    35000     True          [{u'Id': u'Q700', u'Company': u'...               1.0
3   D1  December    16200     True          [{u'Id': u'Q800', u'Company': u'...               1.0
4   E1  December    40000     True          [{u'Id': u'Q900', u'Company': u'...               1.0
5   F1  December    54000     True          [{u'Id': u'Q950', u'Company': u'...               1.0
6   G1  August      67700     True          [{u'Id': u'Q1000', u'Company': u'...              1.0

This is the output of leads['Leads__r.record']:

         leads['Leads__r.record']

0       [{u'Id': u'Q500', u'Company': u'...
1       [{u'Id': u'Q600', u'Company': u'...
2       [{u'Id': u'Q700', u'Company': u'...
3       [{u'Id': u'Q800', u'Company': u'...
4       [{u'Id': u'Q900', u'Company': u'...
5       [{u'Id': u'Q950', u'Company': u'...

output of leads['Leads__r.record'][0]

[{u'Company': u'A1',
  u'CurrencyIsoCode': u'AUD',
  u'Date_Submitted__c': u'2018-09-12',
  u'Deal_Amount__c': 500000,
  u'Id': u'Q500',
  u'DEAL_ID': u'a78989',
  u'PrimaryProductFamily__c': u'Cloud',
  u'Primary_Product_Family__c': u'Cloud',
  u'Product_interest__c': None,
  u'attributes': {u'type': u'Lead',
  u'url': u'/services/data/v24.0/sobjects/Lead/00Qf200001Zls5CEAR'}}]

I have tried everything, :

pd.concat([pd.DataFrame(x) for x in leads['Leads__r.record']], 
    keys=leads['Leads__r.totalSize']).reset_index(level=1, drop=True).reset_index()

The above used to work but for some reason I keep getting error, ValueError: DataFrame constructor not properly called!

leads['Leads__r.records'].reset_index().drop('index',1)

This is the closest I've come:

pd.concat([pd.DataFrame(x) for x in leads['Leads__r.records'][0:200]], keys=leads.index).reset_index(level=1, drop=True).reset_index()

This works but only for the first 200 rows

回答1:

Try json_normalize as follows:

from pandas.io.json import json_normalize

# Convert the column of single-element lists to a list of dicts
records = [x[0] for x in df['Leads__r.record']]

# normalize the records and concat with the original df
res = pd.concat([df.drop('Leads__r.record', axis=1), json_normalize(records)], axis=1)

来源：https://stackoverflow.com/questions/55093729/expand-nested-list-of-dictionaries-in-a-pandas-dataframe-column

标签

python

pandas

dataframe

nested