How to fetch data over multiple pages?

问题

My project is based on React, redux, redux-saga, es6 and I try to fetch data from this API:

http://api.dhsprogram.com/rest/dhs/data/BD,2000,2004,2007?&returnFields=CharacteristicLabel,Indicator,IndicatorId,Value&f=json

As you can see, this specific API call shows data with a limit of 100 data per page spread on 40 pages.

According to this answer: http://userforum.dhsprogram.com/index.php?t=msg&th=2086&goto=9591&S=Google it says that you can extend the limit to a maximum of 3000data per page.

However, in some cases I would do an API call that exceeds that limit which means I would not receive all my data doing it like this:

export function fetchMetaData(countryCode: string, surveyYears: string) {
return (fetch('http://api.dhsprogram.com/rest/dhs/data/' + countryCode + ',' + surveyYears + '?returnFields=CharacteristicLabel,Indicator,IndicatorId,Value&f=json')
    .then(response => response.json())
    .then(json => json.Data.map(survey => survey)))
}

So my question is; what is the best way to get all data from this API given that I know the total pages of data. The answer in the forum link suggest to loop through the API. However, I can't find the right syntax usage to do this.

My idea would be doing one api call to get the total number of pages. Then store this in a state using redux+redux-saga. Then do a new request sending the total pages as parameter and fetch this total number of pages times. And by doing this I can't figure out the syntax to store the data for each iteration.

回答1:

A possible solution -the idea is to get the number of pages first, then make the appropriate number of API calls, pushing the promise from each call into an array. We then wait for all the promises to resolve, and do something with the returned data.

function fetchMetaData() {
    let pagesRequired = 0;

    fetch('apiUrlToGetPageNumber')
    .then(resp = > {
        const apiPromises = [];
        pagesRequired = resp.data.pagesRequired;

        for (let i=pagesRequired; i>0;i--) {
            apiPromises.push(fetch('apiUrlToSpecificPage?page = ' + i));
        }

        Promise.all(apiPromises)
        .then(responses => {
            const processedResponses = [];
            responses.map(response => {
                processedResponses.push(response);
            }

            // do something with processedResponses here
        });
    }
}

回答2:

Here is another possible solution using async/await. The beauty of this is that the total_pages count is dynamic, so that if it increases while you're processing your request, it'll make sure you get it all.

async function fetchMetaData() {
  let allData = [];
  let morePagesAvailable = true;
  let currentPage = 0;

  while(morePagesAvailable) {
    currentPage++;
    const response = await fetch(`http://api.dhsprogram.com/rest/dhs/data?page=${currentPage}`)
    let { data, total_pages } = await response.json();
    data.forEach(e => allData.unshift(e));
    morePagesAvailable = currentPage < total_pages;
  }

  return allData;
}

来源：https://stackoverflow.com/questions/40677764/how-to-fetch-data-over-multiple-pages

标签

javascript

reactjs

redux

fetch

redux-saga