Downloading multiple files in R with variable length, nested URLs

Question

New member here. Trying to download a large number of files from a website in R (but open to suggestions as well, such as wget.)

From this post, I understand I must

Answer 1

If all your agency codes are the same within each state code you could use the below to create your vector of urls to loop through. (You will also need a vector of destinations the same size).

#Getting all combinations
States <- c("AA","BB")
Agency <- c("ABCDEFG","HIJKLMN")
AllCombinations <- expand.grid(States, Agency)
AllCombinationsVec <- paste0("http://website.gov/" ,AllCombinations$Var1, "_",AllCombinations$Var2,".zip" )

You can then try looping through each file something like this:

#loop method

for(i in seq(AllCombinationsVec)){
  download.file(AllCombinationsVec[i], destinations[i], mode="wb")}

This is also another way of looping through items apply functions will apply a function to every item in a list or vector.

#lapply method

mapply(function(x, y) download.file(x,y, mode="wb"),x = AllCombinationsVec, y = destinations)