RCurl getURLContent detect content type through final redirect

问题

This is a followup question to RCurl getURL with loop - link to a PDF kills looping :

I have the following getURL command:

require(RCurl)
#set a bunch of options for curl
options(RCurlOptions = list(cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl")))
agent="Firefox/23.0" 
curl = getCurlHandle()
curlSetOpt(
  cookiejar = 'cookies.txt' ,
  useragent = agent,
  followlocation = TRUE ,
  autoreferer = TRUE ,
  httpauth = 1L, # "basic" http authorization version -- this seems to make a difference for India servers
  curl = curl
)


x = getURLContent('http://timesofindia.indiatimes.com//articleshow/2933019.cms')
class(x)
#[1] "character"
attr(x, "Content-Type")
#"text/plain"

In a browser, the link above ends up redirecting to:

x = getURLContent('http://timesofindia.indiatimes.com/photo.cms?msid=2933009')
class(x)
#[1] "raw"
attr(x, "Content-Type")
#"application/pdf"

Assuming I know only the first link, how can I detect that the final location of the redirect (or redirects) is of a certain type (in this case PDF)?

Thanks!!

回答1:

Maybe there's a better solution, but one way could be this:

# ...
h <- basicTextGatherer()
x = getBinaryURL('http://timesofindia.indiatimes.com//articleshow/2933019.cms', 
                 headerfunction = h$update, curl = curl)
r <- gregexpr("Content-Type:.*?\n", h$value())
tail(regmatches(h$value(), r)[[1]], 1)
# [1] "Content-Type: application/pdf\r\n"

回答2:

I have run into a similar issue trying to run getURLContent using digest authentication to get at binary data (using a non-standard mime type). I am running RCurl v1.95-4.1 on R 2.15.3.

If I run getURLContent without the binary=TRUE flag, it won't autoswitch to binary=TRUE because of the mime header for this data type, so it attempts to perform a rawToChar() and throws a 'embedded NULL in string' error. However the authentication does work.

If I add a binary=TRUE flag to the getURLContent call, it seems to cause issues with the authentication step, since I then get an 'Error: Unauthorized' response.

What finally worked was to replace getURLContent() with getBinaryURL() [as in the example above], which allowed the userpwd="u:p" authorization to work and delivered the binary data to my assigned object.

I think that the author of RCurl has made improvements to getURLContent's handling of binary data for v1.97, based on what I see in GitHub, so this may become a thing of the past...except for those of us still running older R setups.

来源：https://stackoverflow.com/questions/25474682/rcurl-geturlcontent-detect-content-type-through-final-redirect

标签

rcurl

html-content-extraction

geturl