ls just lists what arguments it is presented with. *.txt gets expanded to a.txt c.txt before ls sees it, try echo *.txt.
To do what you ask with sed you can delete the pattern from its input, for example:
ls | sed '/\.txt$/d'
Would delete all lines ending with .txt.
With bash and zsh you can have the shell do the inverted expansion, with bash it would be:
ls !(*.txt)
zsh:
ls *~*.txt
Note that both shells need the extended glob option to be enabled, shopt -s extglob with bash and setopt extendedglob with zsh.
