Merging CSV files with different headers using CSVhelper C#

问题

When trying to merge multiple .csv files from a directory into one .csv file using CSVhelper. In the directory there are 50 .csv files, among these 50 files there are two sets of file structures, one with 7 columns and one with 6. Every file has the exact same first 5 headers however depending on the file the last two columns will change.

Example of CSVfile format 1:

Example of CSVfile format 2:

Every file in the directory will hold either of these structures with different data in the columns. The output of the new file will have data from all the columns bar Action, Code and Error Message. If i use files only with the structure of example 1 the file comes together perfectly. However, if i include files with both structures and try and use 'ErrorIPAddress' from example 2 in my new file i get the following error:

An unhandled exception of type 'CsvHelper.TypeConversion.CsvTypeConverterException' occurred in CsvHelper.dll

On this line: `IEnumerable dataRecord = reader.GetRecords().ToList();

My question is: How to use columns from one file thats not in the other? I have tried mapping it with the following:Map(m => m.ErrorIPAddress).Index(5); and i believe this is the line causing me the issue as if i comment it out the error doesn't persist however obviously i won't get the data i need into the new .csv. If i try and map by name with: Map( m => m.ErrorIPAddress ).Name( "ErrorIPAddress" ); I get the error message that ErrorIPAddress is not in the .csv file which it won't be as not all files have that column.

Output .csv format:

The final column will be generated by the ErrorIPAddress column in format 2.

回答1:

I'm assuming you are using a single class definition with all the fields that looks something like this:

public class StudentWebAccess
{
    public int StudentID { get; set; }
    public string Gender { get; set; }
    public int Grade { get; set; }        
    public int IPAddress { get; set; } // Also ErrorIPAddress?
    public DateTime DateTime { get; set; }
    public string Action { get; set; }
    public string Code { get; set; } // Also ErrorMessage?
}

So in order to read file format 2 you are using CsvClassMap but aren't matching the properties and field names correctly. It should look something like this:

public class CsvFile2Map : CsvClassMap<StudentWebAccess>
{
    public CsvFile2Map()
    {            
        Map(m => m.IPAddress).Name("ErrorIPAddress");
        Map(m => m.Code).Name("ErrorMessage");
    }
}

If your class file uses ErrorIPAddress instead of IPAddress you have to reverse the mapping.

Map(m => m.ErrorIPAddress).Name("IPAddress");

回答2:

You do not need an external library. Use the code below

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Globalization;
using System.Data;
using System.Data.OleDb;
using System.IO;


namespace ConsoleApplication1
{
    class Program
    {
        const string FOLDER = @"c:\temp\test";
        static void Main(string[] args)
        {
            CSVReader reader = new CSVReader();

            //table containing merged csv files
            DataTable dt = new DataTable();

            //get csv files one at a time
            foreach (string file in Directory.GetFiles(FOLDER, "*.csv"))
            {
                //read csv file into a new dataset
                DataSet ds = reader.ReadCSVFile(file, true);
                //datatable containing new csv file
                DataTable dt1 = ds.Tables[0];

                //add new columns to datatable dt if doesn't exist
                foreach(DataColumn col in dt1.Columns.Cast<DataColumn>())
                {
                    //test if column exists and add if it doesn't
                    if (!dt.Columns.Contains(col.ColumnName))
                    {
                        dt.Columns.Add(col.ColumnName, typeof(string));
                    }
                }

                //array of column names in new table
                string[] columnNames = dt1.Columns.Cast<DataColumn>().Select(x => x.ColumnName).ToArray();

                //copy row from dt1 into dt
                foreach(DataRow row in dt1.AsEnumerable())
                {
                    //add new row to table dt
                    DataRow newRow = dt.Rows.Add();

                    //add data from dt1 into dt
                    for(int i = 0; i < columnNames.Count(); i++)
                    {
                        newRow[columnNames[i]] = row[columnNames[i]];
                    }
                }
            }

        }
    }
    public class CSVReader
    {

        public DataSet ReadCSVFile(string fullPath, bool headerRow)
        {

            string path = fullPath.Substring(0, fullPath.LastIndexOf("\\") + 1);
            string filename = fullPath.Substring(fullPath.LastIndexOf("\\") + 1);
            DataSet ds = new DataSet();

            try
            {

                //read csv file using OLEDB Net Library
                if (File.Exists(fullPath))
                {
                    string ConStr = string.Format("Provider=Microsoft.Jet.OLEDB.4.0;Data Source={0}" + ";Extended Properties=\"Text;HDR={1};FMT=Delimited\\\"", path, headerRow ? "Yes" : "No");
                    string SQL = string.Format("SELECT * FROM {0}", filename);
                    OleDbDataAdapter adapter = new OleDbDataAdapter(SQL, ConStr);
                    adapter.Fill(ds, "TextFile");
                    ds.Tables[0].TableName = "Table1";
                }

                //replace spaces in column names with underscore
                foreach (DataColumn col in ds.Tables["Table1"].Columns)
                {
                    col.ColumnName = col.ColumnName.Replace(" ", "_");
                }
            }

            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
            }
            return ds;
        }
    }
}

来源：https://stackoverflow.com/questions/43831267/merging-csv-files-with-different-headers-using-csvhelper-c-sharp

标签

csv

csvhelper