ETL & Parsing CSV files in Cloud Dataflow

Question

I\'m new to cloud dataflow and Java so I\'m hoping this is the right question to ask.

I have a csv file with n number of columns and rows that could be a string, intege

Answer 1

line.split(",");

String.split doesn't make sense if the line data like this:

a,b,c,"we,have a string contains comma",d,e

A property way to deal with the csv data is to import a csv library:

        
            com.opencsv
            opencsv
            3.7
        

and use codes below inside ParDo:

public void processElement(ProcessContext c) throws IOException {
    String line = c.element();
    CSVParser csvParser = new CSVParser();
    String[] parts = csvParser.parseLine(line);
}