Dealing with commas in a CSV file

Question

I am looking for suggestions on how to handle a csv file that is being created, then uploaded by our customers, and that may have a comma in a value, like a company name.

Answer 1

Add a reference to the Microsoft.VisualBasic (yes, it says VisualBasic but it works in C# just as well - remember that at the end it is all just IL).

Use the Microsoft.VisualBasic.FileIO.TextFieldParser class to parse CSV file Here is the sample code:

 Dim parser As TextFieldParser = New TextFieldParser("C:\mar0112.csv")
 parser.TextFieldType = FieldType.Delimited
 parser.SetDelimiters(",")      

   While Not parser.EndOfData         
      'Processing row             
      Dim fields() As String = parser.ReadFields         
      For Each field As String In fields             
         'TODO: Process field                   

      Next      
      parser.Close()
   End While 

Answer 2

If you're interested in a more educational exercise on how to parse files in general (using CSV as an example), you may check out this article by Julian Bucknall. I like the article because it breaks things down into much smaller problems that are much less insurmountable. You first create a grammar, and once you have a good grammar, it's a relatively easy and methodical process to convert the grammar into code.

The article uses C# and has a link at the bottom to download the code.

Answer 3

I think the easiest solution to this problem is to have the customer to open the csv in excel, and then ctrl + r to replace all comma with whatever identifier you want. This is very easy for the customer and require only one change in your code to read the delimiter of your choice.

Answer 4

As this is about general practices let's start from rules of the thumb:

1. Don't use CSV, use XML with a library to read & write the xml file instead.

2. If you must use CSV. Do it properly and use a free library to parse and store the CSV files.

To justify 1), most CSV parsers aren't encoding aware so if you aren't dealing with US-ASCII you are asking for troubles. For example excel 2002 is storing the CSV in local encoding without any note about the encoding. The CSV standard isn't widely adopted :(. On the other hand xml standard is well adopted and it handles encodings pretty well.

To justify 2), There is tons of csv parsers around for almost all language so there is no need to reinvent the wheel even if the solutions looks pretty simple.

To name few:

• for python use build in csv module

• for perl check CPAN and Text::CSV

• for php use build in fgetcsv/fputcsv functions

• for java check SuperCVS library

Really there is no need to implement this by hand if you aren't going to parse it on embedded device.

Answer 5

As mentioned in my comment to harpo's answer, his solution is good and works in most cases, however in some scenarios when commas as directly adjacent to each other it fails to split on the commas.

This is because of the Regex string behaving unexpectedly as a vertabim string. In order to get this behave correct, all " characters in the regex string need to be escaped manually without using the vertabim escape.

Ie. The regex should be this using manual escapes:

",(?=(?:[^\"\"]*\"\"[^\"\"]*\"\")*(?![^\"\"]*\"\"))"

which translates into ",(?=(?:[^""]*""[^""]*"")*(?![^""]*""))"

When using a vertabim string @",(?=(?:[^""]*""[^""]*"")*(?![^""]*""))" it behaves as the following as you can see if you debug the regex:

",(?=(?:[^"]*"[^"]*")*(?![^"]*"))"

So in summary, I recommend harpo's solution, but watch out for this little gotcha!

I've included into the CsvReader a little optional failsafe to notify you if this error occurs (if you have a pre-known number of columns):

if (_expectedDataLength > 0 && values.Length != _expectedDataLength) 
throw new DataLengthException(string.Format("Expected {0} columns when splitting csv, got {1}", _expectedDataLength, values.Length));

This can be injected via the constructor:

public CsvReader(string fileName, int expectedDataLength = 0) : this(new FileStream(fileName, FileMode.Open, FileAccess.Read))
{
    _expectedDataLength = expectedDataLength;
}

Answer 6

First, let's ask ourselves, "Why do we feel the need to handle commas differently for CSV files?"

For me, the answer is, "Because when I export data into a CSV file, the commas in a field disappear and my field gets separated into multiple fields where the commas appear in the original data." (That it because the comma is the CSV field separator character.)

Depending on your situation, semi colons may also be used as CSV field separators.

Given my requirements, I can use a character, e.g., single low-9 quotation mark, that looks like a comma.

So, here's how you can do it in Go:

// Replace special CSV characters with single low-9 quotation mark
func Scrub(a interface{}) string {
    s := fmt.Sprint(a)
    s = strings.Replace(s, ",", "‚", -1)
    s = strings.Replace(s, ";", "‚", -1)
    return s
}

The second comma looking character in the Replace function is decimal 8218.

Be aware that if you have clients that may have ascii-only text readers that this decima 8218 character will not look like a comma. If this is your case, then I'd recommend surrounding the field with the comma (or semicolon) with double quotes per RFC 4128: https://tools.ietf.org/html/rfc4180