Python extract elements from Json string

Question

I have a Json string from which I\'m able to extract few components like formatted_address,lat,lng, but I\'m unable to extract feature(val

Answer 1

You need to understand schema of your data.

Error in json_st['results']['address_components']

because json_st['results'] is an array

Check it here http://jsoneditoronline.org

Here is some sample

for result in data['results']:
    print type(result)
    for address_component in result['address_components']:
        print type(address_component)
        print address_component['long_name']
        print address_component['short_name']
        for _type in address_component['types']:
            print _type

Answer 2

I would go for json_normalize, thought of one line answer but I dont think its possible i.e (Here I did only for px_val and py_val you can do similar things for other columns)

from pandas.io.json import json_normalize

import pandas as pd
import json

with open('dat.json') as f:
    data = json.load(f)

result = json_normalize(data,'results')

result['px_val'] = result['geometry'].apply(json_normalize).apply(lambda x : x['location.lat'])
result['py_val'] = result['geometry'].apply(json_normalize).apply(lambda x : x['location.lng'])

print(result[['formatted_address','px_val','py_val']])

                              formatted_address        px_val      py_val
0  Satya Niwas, Kanti Nagar, J B Nagar, Andheri E...  19.109659  72.867471
1  82, Bamanpuri, Ajit Nagar, J B Nagar, Andheri ...  19.109749  72.867249
2  Todi Building, Sheth Bhavanidas Benani Marg, K...  19.109827  72.867789
3  KASI APARTMENTS, Shriniwas Bagarka Rd, Bamanpu...  19.109334  72.867052
4  Silver Line Apts., Bamanpuri, J.B. Nagar, J B ...  19.109108  72.867078
5  Gokul panch chs, 81-B, 81-B, Sheth Bhavanidas ...  19.109871  72.867057
6  Ajit Nagar, J B Nagar, Andheri East, Mumbai, M...  19.110316  72.868073
7  Bamanpuri, J B Nagar, Andheri East, Mumbai, Ma...  19.108435  72.865749
8  J B Nagar, Andheri East, Mumbai, Maharashtra 4...  19.111062  72.865592
9           Andheri East, Mumbai, Maharashtra, India  19.115491  72.872695

I try to parse political certainly not proud of this solution i.e

pol = []
for i in result['address_components'].apply(json_normalize):
     pol.append(','.join(i.apply(lambda x : x['long_name'] if 'political' in x['types'] else np.nan,1).dropna()))

result['political'] = pol

Output result['political']

0    Kanti Nagar,J B Nagar,Andheri East,Mumbai,Mumb...
1    Bamanpuri,Ajit Nagar,J B Nagar,Andheri East,Mu...
2    Kanti Nagar,J B Nagar,Andheri East,Mumbai,Mumb...
3    Bamanpuri,Kanti Nagar,J B Nagar,Andheri East,M...
4    Bamanpuri,J.B. Nagar,J B Nagar,Andheri East,Mu...
5    Bamanpuri,Ajit Nagar,J B Nagar,Andheri East,Mu...
6    Ajit Nagar,J B Nagar,Andheri East,Mumbai,Mumba...
7    Bamanpuri,J B Nagar,Andheri East,Mumbai,Mumbai...
8    J B Nagar,Andheri East,Mumbai,Mumbai Suburban,...
9    Andheri East,Mumbai,Mumbai Suburban,Maharashtr...
Name: political, dtype: object

To convert it to a method we can do

def get_cols(st):
    pol = []
    for i in result['address_components'].apply(json_normalize):
         pol.append(','.join(i.apply(lambda x : x['long_name'] if st in x['types'] else np.nan,1).dropna()))

   return  pol

result['political'] = get_cols('political') 
# This will assign the new column political with data. 

Answer 3

This is rather a pretty broad question...

To help you to start:

record_path = ['address_components']

meta= [
  'formatted_address',
  ['geometry','location','lat'],
  ['geometry','location','lng'],  
]

x = pd.io.json.json_normalize(d['results'], record_path, meta)

Result:

In [20]: pd.options.display.max_rows = 15

In [21]: x
Out[21]:
          long_name       short_name                                          types  \
0       Satya Niwas      Satya Niwas    [establishment, point_of_interest, premise]
1       Kanti Nagar      Kanti Nagar  [political, sublocality, sublocality_level_3]
2         J B Nagar        J B Nagar  [political, sublocality, sublocality_level_2]
3      Andheri East     Andheri East  [political, sublocality, sublocality_level_1]
4            Mumbai           Mumbai                          [locality, political]
5   Mumbai Suburban  Mumbai Suburban       [administrative_area_level_2, political]
6       Maharashtra               MH       [administrative_area_level_1, political]
..              ...              ...                                            ...
83            India               IN                           [country, political]
84           400047           400047                                  [postal_code]
85     Andheri East     Andheri East  [political, sublocality, sublocality_level_1]
86           Mumbai           Mumbai                          [locality, political]
87  Mumbai Suburban  Mumbai Suburban       [administrative_area_level_2, political]
88      Maharashtra               MH       [administrative_area_level_1, political]
89            India               IN                           [country, political]

                                    formatted_address  geometry.location.lat  geometry.location.lng
0   Satya Niwas, Kanti Nagar, J B Nagar, Andheri E...              19.109659              72.867471
1   Satya Niwas, Kanti Nagar, J B Nagar, Andheri E...              19.109659              72.867471
2   Satya Niwas, Kanti Nagar, J B Nagar, Andheri E...              19.109659              72.867471
3   Satya Niwas, Kanti Nagar, J B Nagar, Andheri E...              19.109659              72.867471
4   Satya Niwas, Kanti Nagar, J B Nagar, Andheri E...              19.109659              72.867471
5   Satya Niwas, Kanti Nagar, J B Nagar, Andheri E...              19.109659              72.867471
6   Satya Niwas, Kanti Nagar, J B Nagar, Andheri E...              19.109659              72.867471
..                                                ...                    ...                    ...
83  J B Nagar, Andheri East, Mumbai, Maharashtra 4...              19.111062              72.865592
84  J B Nagar, Andheri East, Mumbai, Maharashtra 4...              19.111062              72.865592
85           Andheri East, Mumbai, Maharashtra, India              19.115491              72.872695
86           Andheri East, Mumbai, Maharashtra, India              19.115491              72.872695
87           Andheri East, Mumbai, Maharashtra, India              19.115491              72.872695
88           Andheri East, Mumbai, Maharashtra, India              19.115491              72.872695
89           Andheri East, Mumbai, Maharashtra, India              19.115491              72.872695

[90 rows x 6 columns]