The difference is, strings between double quotes (") are parsed for variable and escape sequence substitution. Strings in single quotes (') aren't.
So, using double quotes (") you can do:
$count = 3;
echo "The count is:\t$count";
which will produce
The count is:<tab>3
The same in single quotes returns the literal string.
Also, the characters that need to be escaped. If you have a string like:
'John said, "Hello"'
you would probably use single quotes, to avoid having to escape the quotes in the string and vice-versa.