问题
I use wget to access a list of links from a text file. A link example would be:
http://localhost:8888/data/test.php?value=ABC123456789
The PHP file returns a table with information from which the response is to be appended to another text file. As to the error, it is obvious that currently it cannot handle the amount of URLs because it exceeds the character limit. If I use 2 URLs only, it works perfectly fine.
The text file contains a total of 10 000 URLs. The command I am using is:
wget -i /Applications/MAMP/htdocs/data/URLs.txt -O - >> /Applications/MAMP/htdocs/data/append.txt
According to my research, a quick way to "fix" this is to change the LimitRequestLine or adding it if it does not exist. Since I use MAMP (for MacOS) what I did was:
Open /Applications/MAMP/conf/apache/httpd.conf
And insert under AccessFileName .htaccess:
LimitRequestLine 1000000000
LimitRequestFieldSize 1000000000
But I still get the same error. I don't know why this happens.
May it be easier to use cURL? If yes, what would be a similar command?
回答1:
your 414: Request-URI Too Large error has nothing to do with the amount of urls, and no, using curl wouldn't help.
the problem is that some (or 1?) of your urls is simply too long for the target server, causing the error.
you can probably identify the url causing the error by doing
cat URLs.txt | awk '{print length, $0}' | sort -nr | head -1
(thanks to https://stackoverflow.com/a/1655488/1067003 for that command)
another possible cause is that you're not properly line-terminating the urls in URLs.txt , and some of the urls (or all of them?) gets concatenated. for the record, the terminating character is "\n", aka hex code 0A - not the \r\n that most windows-editors use, i'm not sure how wget would handle such malformed line terminators (per its definition)
note that if you are downloading loads of .HTML files (or any other compressible files), curl would be much faster than wget, as curl supports compressed transfers with the --compressed argument (utilizing gzip and deflate as of speaking), while wget doesn't support compression at all - and HTML compresses very very well (easily 5-6 times smaller than the uncompressed version with gzip)
来源：https://stackoverflow.com/questions/45012726/wget-error-414-request-uri-too-large