Ubuntu 12.04
nginx 1.2.4
avconv -version
avconv version 0.8.10-4:0.8.10-0ubuntu0.12.04.1, Copyright (c) 2000-2013 the Libav developers
built on
The answer detailing reducing the hls_playlist to 4s and the hls_fragment to 1s is good. However, it is important to note that if the source video has a key frame interval greater than the hls_fragment length then nginx will usually have to create longer fragments and the latency will increase.
The lowest latency we have achived was with the hls_fragment at 1s and the source video key frame interval also at 1s. With these settings we achieved latency of less than 7 seconds.