Having a symbol to denote variables makes string interpolation simple and clear. Shell, Perl and PHP grew out of the need for quick and easy string manipulation, including interpolation, so I imagine using a variable prefix seemed like a good idea.
I.e. in PHP:
$var = 'val';
$strVar = "The var is $var";
Compare to typical string formatting:
var = 'val'
strVal = 'The var is %s' %(var)