log into a website to grab the data using RCurl

Question

I wanted to login to the website using RCurl and grab the data from the web (The data cannot be seen without logging in.)

I wanted to export this (for example) "http://www.appannie.com/app/ios/instagram/ranking/history/chart_data/?s=2010-10-06&e=2012-06-04&c=143441&f=ranks&d=iphone" into R after I log in using RCurl. The issue is I cannot log in using RCurl. I haven't tried this before so mostly I referred to http://www.omegahat.org/RCurl/philosophy.html.

So here's what I tried. (here, 'me@gmail.com' is my user ID and '9999' is my Password - i just made it up.)

library(RJSONIO)
library(rjson)
library(RCurl)
appannie <- getURL("http://www.appannie.com/app/ios/instagram/ranking/history/chart_data/.json?s=2010-10-06&e=2012-06-04&c=143441&f=ranks&d=iphone, userpwd = me@gmail.com:9999", verbose = TRUE)

But this gave me the message below :

About to connect() to www.appannie.com port 80 (#0)
*   Trying 69.167.138.64... * connected
* Connected to www.appannie.com (69.167.138.64) port 80 (#0)
> GET /app/ios/instagram/ranking/history/chart_data/?s=2010-10-06&e=2012-06-04&c=143441&f=ranks&d=iphone HTTP/1.1
Host: www.appannie.com
Accept: */*

< HTTP/1.1 403 FORBIDDEN
< Server: nginx/1.1.19
< Date: Fri, 01 Mar 2013 23:41:32 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Keep-Alive: timeout=10
< Vary: Accept-Encoding
< Vary: Cookie,Accept-Encoding
< 
* Connection #0 to host www.appannie.com left intact

So, I went back and read this http://www.omegahat.org/RCurl/philosophy.html again and didn't know what to do, so I tried this after I saw the similar question from stackoverflow.

getURL("http://www.appannie.com/app/ios/instagram/ranking/history/chart_data/?s=2010-10-06&e=2012-06-04&c=143441&f=ranks&d=iphone",.opts=list(userpwd=me@gmail.com:9999"))

But this gives me below output.

[1] ""

Can anyone give me a hint? (After a bunch of different trial, the website starts to send me warnings =(

Answer 1

This is some sort of authentication issue not anything you did wrong with RCurl most likely.

You got through to the server but either your login was incorrect, it wasn't valid or the data is not available via the API.

http://en.wikipedia.org/wiki/HTTP_403