问题
I am trying to fetch the data from salesforce using the simple_salesforce library in python.
I am able to get the correct count of records while running the count query.
But while I am trying to put that results (in the form of list) into s3 as a JSON object, not as many reocrds are getting persisted as I captured from Salesforce.
Here is the piece of code:
result = sf.query("SELECT ID FROM Opportunity")['records']
object.put(Body=(bytes(json.dumps(result, indent=2).encode('UTF-8'))))
Is the problem on the Salesforce side or am I running into an issue using AWS's SDK to put the objects into S3?
回答1:
Salesforce API returns stuff in chunks, default is 2000 records at a time. If it'd return to you 1M records it could kill your memory usage. Retrieve a chunk, process it (save to file?), request next chunk.
It's straight on the project's homepage:
If, due to an especially large result, Salesforce adds a nextRecordsUrl to your query result, such as "nextRecordsUrl" : "/services/data/v26.0/query/01gD0000002HU6KIAW-2000", you can pull the additional results with either the ID or the full URL (if using the full URL, you must pass ‘True’ as your second argument)
sf.query_more("01gD0000002HU6KIAW-2000")
sf.query_more("/services/data/v26.0/query/01gD0000002HU6KIAW-2000", True)
As a convenience, to retrieve all of the results in a single local method call use
sf.query_all("SELECT Id, Email FROM Contact WHERE LastName = 'Jones'")
来源：https://stackoverflow.com/questions/57590301/unable-to-fetch-complete-records-from-salesforce-using-python