Getting more than 100 search results with PRAW?

问题

I'm using the following code to obtain reddit search results with PRAW 4.4.0:

params = {'sort':'new', 'time_filter':'year'}
return reddit.subreddit(subreddit).search('', **params)

I'd like to scrape an indefinite amount of posts from the subreddit, for a period of up to a year. Reddit's search functionality (and correspondingly, their API) achieves this with the 'after' parameter. However, the above search function doesn't accept 'after' as a parameter. Is there a way to use PRAW's .search() to obtain more than 100 search results?

回答1:

Yes, by sending parameter limit=None will increase that to 1000, but will not guarantee any timeframe and no way to grab more that 1000. However you can use cloudsearch syntax. It is described in detail in reddit wiki https://www.reddit.com/wiki/search#wiki_cloudsearch_syntax and is pretty powerful search enhancer.

To support it with some code, example usage like this case can be achieved in this way:

import datetime
params = {'sort':'new', 'limit':None, 'syntax':'cloudsearch'}
time_now = datetime.datetime.now()
return reddit.subreddit(subreddit).search('timestamp:{0}..{1}'.format(
    int((time_now - datetime.timedelta(days=365)).timestamp()),
    int(time_now.timestamp())),
    **params)

This has limit of 1000 results per query, but due to specified timeframe you can query multiple times for different timeframes. I.e. grab 1000 submissions, get utc_time from oldest one and send that time as first parameter for timestamp, which will give you results starting at the point in time that your last query stopped.

回答2:

The functionality that you want is available via the submissions function of a subreddit. You need only provide it with a start and end timestamp:

http://praw.readthedocs.io/en/latest/code_overview/models/subreddit.html#praw.models.Subreddit.submissions

来源：https://stackoverflow.com/questions/42658676/getting-more-than-100-search-results-with-praw

标签

python

praw