Post request using cookies with cURL, RCurl and httr

Question

In Windows cURL I can post a web request similar to this:

curl  --dump-header cook.txt ^
  --data \"RURL=http=//www.example.com/r&user=bob&password=hell         


        

Answer 1

Here is a way to create a post request, keep and reuse the resulting cookies with RCurl, for example to get web pages when authentication is required :

library(RCurl)
curl <- getCurlHandle()
curlSetOpt(cookiejar="/tmp/cookies.txt", curl=curl)
postForm("http://example.com/login", login="mylogin", passwd="mypasswd", curl=curl)
getURL("http://example.com/anotherpage", curl=curl)

Answer 2

httr automatically preserves cookies across calls to the same site, as illustrated by these two calls to http://httpbin.org

GET("http://httpbin.org/cookies/set?a=1")
# Response [http://httpbin.org/cookies]
#   Status: 200
#   Content-type: application/json
# {
#    "cookies": {
#     "a": "1"
#   }
# } 

GET("http://httpbin.org/cookies")
# Response [http://httpbin.org/cookies]
#   Status: 200
#   Content-type: application/json
# {
#   "cookies": {
#     "a": "1"
#   }
# } 

Perhaps the problem is that you're sending your data as application/x-www-form-urlencoded, but the default in httr is multipart/form-data, so use multipart = FALSE in your POST call.

Answer 3

Based on Juba suggestion, here is a working RCurl template.

The code emulates a browser behaviour, as it:

1. retrieves cookies on a login screen and
2. reuses them on the following page requests containing the actual data.

### RCurl login and browse private pages ###

library("RCurl")

loginurl ="http=//www.*****"
mainurl  ="http=//www.*****"
agent    ="Mozilla/5.0"

#User account data and other login pars
pars=list(
     RURL="http=//www.*****",
     Username="*****",
     Password="*****"
)

#RCurl pars     
curl = getCurlHandle()
curlSetOpt(cookiejar="cookiesk.txt",  useragent = agent, followlocation = TRUE, curl=curl)
#or simply
#curlSetOpt(cookiejar="", useragent = agent, followlocation = TRUE, curl=curl)

#post login form
web=postForm(loginurl, .params = pars, curl=curl)

#go to main url with real data
web=getURL(mainurl, curl=curl)

#parse/print content of web
#..... etc. etc.


#This has the side effect of saving cookie data to the cookiejar file 
rm(curl)
gc()