You can use text that is only accessible to screen readers by placing it in a span which you hide in an accessible way. Place the x in the CSS which can't be read by screen readers, thus won't confuse, but is visible on the page, and also accessible by keyboard users.