There appear to be two dominating conventions (assuming you also abide by some reasonable standard for numbering the releases themselves):
The advantages of v1.2.3 are that the Git documentation (and also the Mercurial documentation) uses that format in its examples, and that several "authorities" such as the Linux kernel and Git itself use it. (The mentioned Semantic Versioning used to use it but doesn't any more.)
The advantages of 1.2.3 are that gitweb or GitHub can automatically offer a tarball or zip download of the form packagename-$tag.tar.gz (and I think it's quite established that a tarball should not be named package-v1.2.3.tar.gz). Alternatively, you can use git describe directly to generate tarball version numbers. For lightweight projects without a formal release process, these possibilities can be quite convenient. It should also be noted that Semantic Versioning is by no means the only or a universally accepted standard for version numbering. And notable projects such as GNOME as well as countless other projects do use the 1.2.3 tag naming.
I think it's probably too late to consolidate these positions. As always, be consistent and make sense.
Update: As mentioned in this comment, GitHub now offers a tarball name with the 'v' stripped off of the tag.
