How can I see all notes of a Tumblr post from Python?

Question

Say I look at the following Tumblr post: http://ronbarak.tumblr.com/post/40692813…
It (currently) has 292 notes.

I\'d like to get all the above n

Answer 1

Unfortunately looks like the Tumblr API has some limitations (lacks of meta information about Reblogs, notes limited by 50), so you can't get all the notes.

It is also forbidden to do page scraping according to the Terms of Service.

"You may not do any of the following while accessing or using the Services: (...) scrape the Services, and particularly scrape Content (as defined below) from the Services, without Tumblr's express prior written consent;"

Source:

https://groups.google.com/forum/?fromgroups=#!topic/tumblr-api/ktfMIdJCOmc

Answer 2

how to load all notes on tumblr? also covers the topic, but unor's response (above) does it very well.

Answer 3

Like Fabio implies, it is better to use the API.

If for whatever reasons you cannot, then the tools you will use will depend on what you want to do with the data in the posts.

• for a data dump: urllib will return a string of the page you want
• looking for a specific section in the html: lxml is pretty good
• looking for something in unruly html: definitely beautifulsoup
• looking for a specific item in a section: beautifulsoup, lxml, text parsing is what you need.
• need to put the data in a database/file: use scrapy

Tumblr url scheme is simple: url/scheme/1, url/scheme/2, url/scheme/3, etc... until you get to the end of the posts and the servers just does not return any data anymore.

So if you are going to brute force your way to scraping, you can easily tell your script to dump all the data on your hard drive until, say the contents tag, is empty.

One last word of advice, please remember to put a small sleep(1000) in your script, because you could put some stress on Tumblr servers.

Answer 4

Without JS you get separate pages that only contain the notes. For the mentioned blog post the first page would be:

http://ronbarak.tumblr.com/notes/40692813320/4Y70Zzacy

Following pages are linked at the bottom, e.g.:

• http://ronbarak.tumblr.com/notes/40692813320/4Y70Zzacy?from_c=1358403506
• http://ronbarak.tumblr.com/notes/40692813320/4Y70Zzacy?from_c=1358383221
• http://ronbarak.tumblr.com/notes/40692813320/4Y70Zzacy?from_c=1358377013
• …

(See my answer on how to find the next URL in a’s onclick attribute.)

Now you could use various tools to download/parse the data.

The following wget command should download all notes pages for that post:

wget --recursive --domains=ronbarak.tumblr.com --include-directories=notes http://ronbarak.tumblr.com/notes/40692813320/4Y70Zzacy