Replacing characters in C# (ascii)

Question

I got a file with characters like these: à, è, ì, ò, ù - À. What i need to do is replace those characters with normal characters eg: à = a, è = e and so on..... This is my c

Answer 1

Sounds like what you want to do is convert Extended ASCII (eight-bit) to ASCII (seven-bit) - so searching for that might help.

I've seen libraries to handle this in other languages but have never had to do it in C#, this looks like it might be somewhat enlightening though:

Convert two ascii characters to their 'corresponding' one character extended ascii representation

Answer 2

Others have commented on using a Unicode lookup table to remove Diacritics. I did a quick Google search and found this example. Code shamelessly copied, (re-formatted), and posted below:

using System;
using System.Text;
using System.Globalization;

public static class Remove
{
    public static string RemoveDiacritics(string stIn)
    {
        string stFormD = stIn.Normalize(NormalizationForm.FormD);
        StringBuilder sb = new StringBuilder();

        for(int ich = 0; ich < stFormD.Length; ich++) {
            UnicodeCategory uc = CharUnicodeInfo.GetUnicodeCategory(stFormD[ich]);
            if(uc != UnicodeCategory.NonSpacingMark) {
                sb.Append(stFormD[ich]);
            }
        }

        return(sb.ToString().Normalize(NormalizationForm.FormC));
    }
}

So, your code could clean the input by calling:

line = Remove.RemoveDiacritics(line);

Answer 3

Don't know if it is useful but in an internal tool to write message on a led screen we have the following replacements (i'm sure that there are more intelligent ways to make this work for the unicode tables, but this one is enough for this small internal tool) :

        strMessage = Regex.Replace(strMessage, "[éèëêð]", "e");
        strMessage = Regex.Replace(strMessage, "[ÉÈËÊ]", "E");
        strMessage = Regex.Replace(strMessage, "[àâä]", "a");
        strMessage = Regex.Replace(strMessage, "[ÀÁÂÃÄÅ]", "A");
        strMessage = Regex.Replace(strMessage, "[àáâãäå]", "a");
        strMessage = Regex.Replace(strMessage, "[ÙÚÛÜ]", "U");
        strMessage = Regex.Replace(strMessage, "[ùúûüµ]", "u");
        strMessage = Regex.Replace(strMessage, "[òóôõöø]", "o");
        strMessage = Regex.Replace(strMessage, "[ÒÓÔÕÖØ]", "O");
        strMessage = Regex.Replace(strMessage, "[ìíîï]", "i");
        strMessage = Regex.Replace(strMessage, "[ÌÍÎÏ]", "I");
        strMessage = Regex.Replace(strMessage, "[š]", "s");
        strMessage = Regex.Replace(strMessage, "[Š]", "S");
        strMessage = Regex.Replace(strMessage, "[ñ]", "n");
        strMessage = Regex.Replace(strMessage, "[Ñ]", "N");
        strMessage = Regex.Replace(strMessage, "[ç]", "c");
        strMessage = Regex.Replace(strMessage, "[Ç]", "C");
        strMessage = Regex.Replace(strMessage, "[ÿ]", "y");
        strMessage = Regex.Replace(strMessage, "[Ÿ]", "Y");
        strMessage = Regex.Replace(strMessage, "[ž]", "z");
        strMessage = Regex.Replace(strMessage, "[Ž]", "Z");
        strMessage = Regex.Replace(strMessage, "[Ð]", "D");
        strMessage = Regex.Replace(strMessage, "[œ]", "oe");
        strMessage = Regex.Replace(strMessage, "[Œ]", "Oe");
        strMessage = Regex.Replace(strMessage, "[«»\u201C\u201D\u201E\u201F\u2033\u2036]", "\"");
        strMessage = Regex.Replace(strMessage, "[\u2026]", "...");

One thing to note is that if in most language the text is still understandable after such a treatment it's not always the case and will often force the reader to refer to the context of the sentence to be able to understand it. Not something you want if you have the choice.

---

Note that the correct solution would be to use the unicode tables, replacing characters with integrated diacritics with their "combined diacritical mark(s)"+character form and then removing the diacritics...

Answer 4

Why are you making things complicated?

line = line.Replace('à', 'a');

Update:

The docs for File.ReadAllText say:

This method attempts to automatically detect the encoding of a file based on the presence of byte order marks. Encoding formats UTF-8 and UTF-32 (both big-endian and little-endian) can be detected.

Use the ReadAllText(String, Encoding) method overload when reading files that might contain imported text, because unrecognized characters may not be read correctly.

What encoding is C:/Joiner.csv in? Maybe you should use the other overload for File.ReadAllText where you specify the input encoding yourself?

Answer 5

Doing it the easy way. The code below will replace all special characters to ASCII characters in just 2 lines of code. It gives you the same result as Julien Roncaglia's solution.

byte[] bytes = System.Text.Encoding.GetEncoding("Cyrillic").GetBytes(inputText);
string outputText = System.Text.Encoding.ASCII.GetString(bytes);

Answer 6

Use this:

     if (line.Contains(“OldChar”))
     {
        line = line.Replace(“OldChar”, “NewChar”);
     }