Downloading file from the right Apache Mirror with wget

Question

To use a specific example, I want to download the binary release of Hadoop 2.7.2. The web site points to http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.2/hadoop-2

Answer 1

The source code of closer.lua actually states that the action and filename query parameters may be used to produce a redirect to the requested file on the automatically chosen mirror instead of the usual HTML mirror selection page.

So you can download the files directly via this URL: https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz:

GET /dyn/mirrors/mirrors.cgi?action=download&filename=hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: www.apache.org



HTTP/1.1 302 Found
Cache-Control: max-age=3600
Connection: Keep-Alive
Content-Length: 0
Date: Mon, 13 Mar 2017 18:08:00 GMT
Expires: Mon, 13 Mar 2017 19:08:00 GMT
Keep-Alive: timeout=30, max=100
Location: http://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
Server: Apache/2.4.7 (Ubuntu)