I am formatting numbers to string using the following format string \"# #.##\", at some point I need to turn back these number strings like (1 234 567) into something like 1234
Maybe you could to use a regex to replace those empty chars:
Regex.Replace(input, @"\p{Z}", "");
This will remove "any kind of whitespace or invisible separator".