Python - 171
drhirsch's C ported to python.
S=input();R=range(-S+1,S)
for w in R:
p="";v=-w
for u in R:d=v=u)^d];x=4*y*y-x-y+1+2*d*(x-y);p+=" *"[(x>1)*all(x%f for f in range(2,x))]
print p
echo 20 |python ulam.py
* * * * * *
* * * * * *
* * * * *
* * * * * *
* * * *
* * * * * *
* * * * * *
* * * * * * *
* * * * * * *
* * * * *
* * * * * * * *
* * * * * * *
* * * * *
* * * * * *
* * * * * * * * *
* * *
* * * * * * * * * *
* * * * * * * * * *
* * * *
* * ** * * * * * *
* * * *
* *
* * * * * * * * * * *
* * * * * * * *
* *
* * * * * * * *
* * * * * * *
* * * * *
* * * * * *
* * * * * * * * *
* * * *
* * * * * * * *
* * * * * *
* * * * * *
* * * * * *
* * * *
* * * * * *
* *
* * * * * *