rcurl

SourceForge Research Data Archive (SRDA) is one of the data sources for my dissertation research. I'm having difficulty in debugging the following issue related to SRDA data collection. Data collection from SRDA requires authentication and then submitting Web form with an SQL query. Upon successful processing of the query, the system generates a text file with query results . While testing my R code for SRDA data collection, I've changed the SQL request to make sure that the results file is being regenerated. However, I've discovered that the file contents stays the same (corresponds to

I'm a starter in web scraping and I'm not yet familiarized with the nomenclature for the problems I'm trying to solve. Nevertheless, I've searched exhaustively for this specific problem and was unsuccessful in finding a solution. If it is already somewhere else, I apologize in advance and thank your suggestions. Getting to it. I'm trying to build a script with R that will: 1. Search for specific keywords in a newspaper website; 2. Give me the headlines, dates and contents for the number of results/pages that I desire. I already know how to post the form for the search and scrape the results

I am trying to download Google Trends data in csv format. For basic queries I have been successful (following a blog post by Christoph Riedl). Problem : By default trends are returned starting from January 2004. I would prefer it to return trends starting from January 2011. However when I add a date parameter to the url request it is completely ignored. I'm not sure how to overcome this. The following is code will reproduce the issue. # Just copy/paste this stuff - these are helper functions require(RCurl) # This gets the GALX cookie which we need to pass back with the login form getGALX <-

Using RCurl's getForm function, which is the only nice way of passing in GET-parameters, I need to alter some http headers. In getURI, you just pass httpheader = c(Whatever='whatever',...) and it'll work. Unfortunately, that argument seems to be ignored by getForm . How do I set the http headers in a getForm request? Welcome to the confusing world of RCurl ! You've discovered that its syntax makes no sense, which is not your fault. In getForm you pass headers as the second argument (the ... ). See the usage section of ? getForm : getForm(uri, ..., .params = character(), .opts = list(), curl =

I use http://www.regexper.com to view a picto representation regular expressions a lot. I would like a way to ideally: send a regular expression to the site open the site with that expression displayed For example let's use the regex: "\\s*foo[A-Z]\\d{2,3}" . I'd go tot he site and paste \s*foo[A-Z]\d{2,3} (note the removal of the double slashes). And it returns: I'd like to do this process from within R. Creating a wrapper function like view_regex("\\s*foo[A-Z]\\d{2,3}") and the page ( http://www.regexper.com/#%5Cs*foo%5BA-Z%5D%5Cd%7B2%2C3%7D ) with the visual diagram would be opened with the