you need to set nullglob before your loop
shopt -s nullglob
nullglob:
If set, bash allows patterns which match no files
(see Pathname Expansion above) to
expand to a null string, rather than
themselves.
When you're done and want to reset it to the original behavior, use:
shopt -u nullglob
As pointed out by Dennis Williamson in one of the comments.