How to properly handle a gzipped page when using curl?

Question

I wrote a bash script that gets output from a website using curl and does a bunch of string manipulation on the html output. The problem is when I run it against a site that

Answer 1

curl will automatically decompress the response if you set the --compressed flag:

curl --compressed "http://example.com"

--compressed (HTTP) Request a compressed response using one of the algorithms libcurl supports, and save the uncompressed document. If this option is used and the server sends an unsupported encoding, curl will report an error.

gzip is most likely supported, but you can check this by running curl -V and looking for libz somewhere in the "Features" line:

$ curl -V
...
Protocols: ...
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz 

---

Note that it's really the website in question that is at fault here. If curl did not pass an Accept-Encoding: gzip request header, the server should not have sent a compressed response.