How to merge CSV files in Java

Question

My first csv file looks like this with header included(header is included only at the top not after every entry):

NAME,SURNAME,AGE
Fred,Krueger,Unknown
....          


        

Answer 1

I'd create a model for the 'bigger' format (a simple class with four fields and a collection for instances of this class) and implemented two parsers, one for the first, one for the second model. Create records for all rows of both csv files and implement a writer to output the csv in the correct format. IN brief:

 public void convert(File output, File...input) {

   List records = new ArrayList();
   for (File file:input) {
     if (input.isThreeColumnFormat()) {
        records.addAll(ThreeColumnFormatParser.parse(file));
     } else {
        records.addAll(FourColumnFormatParser.parse(file));
     }
   }
   CsvWriter.write(output, records);
 }

---

From your comment I see, that you a lot of different csv formats with some common columns.

You could define the model for any row in the various csv files like this:

public class Record {
  Object id; // some sort of unique identifier
  Map values; // all key/values of a single row
  public Record(Object id) {this.id=id;}
  public void put(String key, String value){
    values.put(key, value);
  }
  public void get(String key) {
    values.get(key);
  }
}

For parsing any file you would first read the header and add the column headers to a global keystore (will be needed later on for outputting), then create records for all rows, like:

//...
List records = new ArrayList()

for (File file:getAllFiles()) {
  List keys = getColumnsHeaders(file);
  KeyStore.addAll(keys);  // the store is a Set
  for (String line:file.getLines()) {
    String[] values = line.split(DELIMITER);
    Record record = new Record(file.getName()+i);  // as an example for id
    for (int i = 0; i < values.length; i++) {
      record.put(keys.get(i), values[i]);
    }
    records.add(record);
  }
}
// ...

Now the keystore has all used column header names and we can iterate over the collection of all records, get all values for all keys (and get null if the file for this record didn't use the key), assemble the csv lines and write everything to a new file.