How to append several dictionaries while looping through pagination (API) - python 3

问题

I would like to loop through several pages of data which I am accessing through an API, and concatenate the pages to the same dictionary as the loop progresses.

So far I managed to loop through all the pages and could print out as the loop goes on, so when the loop is over, all the pages are printed out.

But my dictionary contains only the last page!

page = 5
i = 0
for i in range(0, page):
    url =f'http://hotell.difi.no/api/json/npd/wellbore/withcoordinates?page={i+1}'

    dataset_all = requests.get(url)
    json_dataset_all = dataset_all.json()
    json_dataset_all.update()
    print(json_dataset_all)
    i = i+1

I am not succeeding in going through the loop and updating my dictionary page1 with page2, page3, page4 and page5.

The 'print' shows that the loop through the pages works, but it's not storing the pages in the dictionary as it progresses. It only contains the latest page.

Thanks in advance for help.

回答1:

The pages have an identical "key" called "entries" and it's value is a list of dictionaries. When you update your dictionary, it will therefore just overwrite its value (the list of dictionaries).

If you want to have one single list with all the dictionaries, you could do something like this. You can then also easily put it in a Pandas DataFrame.

import requests
import pandas as pd

page = 5
i = 0
all_data_list = []
for i in range(0, page):
    url =f'http://hotell.difi.no/api/json/npd/wellbore/with-coordinates?page={i+1}'

    dataset_all = requests.get(url)
    dataset_all_json = dataset_all.json()
    number_of_entries = len(dataset_all_json['entries'])

    for x in range(0, number_of_entries):
        all_data_list.append(dataset_all_json['entries'][x])

    i = i+1

# Put all the results in a Pandas DataFrame
df = pd.DataFrame(all_data_list)
print(df)

(by the way, your URL is missing the '-' in "withcoordinates")

来源：https://stackoverflow.com/questions/52768509/how-to-append-several-dictionaries-while-looping-through-pagination-api-pyth

标签

python

dictionary

for-loop

pagination

concatenation