I wrote a function which works as expected but i don\'t understand why the output is like that.
datatype prop = Atom of string | Not of
This is just an output restriction (yes, it's confusing) - by default the depth of value printouts in the top-level (interactive shell) is limited to a fairly small number (i.e. 5). The skipped parts are printed with #.
You can override this depth - at least, in SML-NJ - with printDepth variable:
Control.Print.printDepth := 1024;
P.S. By the way, you don't need a separate do_xor and local function here - just
fun xor(alpha, beta) = Or(...);
will do.