OutOfMemoryException while trying to read big Excel file into DataTable

问题

I'm using SSIS package to clean and load data from .Xlsx file to SQL Server table. I have also to highlight cells containing wrong data in .Xlsx file, for this I have to get back column and row indexes based on column name and row id(witch I have in my data spreadsheet). For that I compare each column name from my first spreadsheet (Error_Sheet) with rows of a column that I added in a second spreadsheet and do the same for rows, and if I have the same value of cells I get back the column and row indexes of my data spreadsheet and highlight the cell based on that column and row index. The script worked fine, but after trying to run it from a server I got an Memory exception and also on my workstation where it was working fine before.

I've tried to reduce the range that I'm taking data from : AC1:AC10000 to AC1:AC100, it worked only after the first time compilation, but it keeps throwing exception again.

string strSQLErrorColumns = "Select * From [" + Error_Sheet + "AC1:AC100]";
OleDbConnection cn = new OleDbConnection(strCn);

OleDbDataAdapter objAdapterErrorColumns = new OleDbDataAdapter(strSQLErrorColumns, cn);
System.Data.DataSet dsErrorColumns = new DataSet();
objAdapterErrorColumns.Fill(dsErrorColumns, Error_Sheet);
System.Data.DataTable dtErrorColumns = dsErrorColumns.Tables[Error_Sheet];
dsErrorColumns.Dispose();
objAdapterErrorColumns.Dispose();

foreach (DataColumn ColumnData in dtDataColumns.Columns){
    ColumnDataCellsValue = dtDataColumns.Columns[iCntD].ColumnName.ToString();
    iCntE = 0;

    foreach (DataRow ColumnError in dtErrorColumns.Rows){
        ColumnErrorCellsValue = dtErrorColumns.Rows[iCntE].ItemArray[0].ToString();

        if (ColumnDataCellsValue.Equals(ColumnErrorCellsValue)){

            ColumnIndex = ColumnData.Table.Columns[ColumnDataCellsValue].Ordinal;
            iCntE = iCntE + 1;
            break;
            }
        }

        iCntD = iCntD + 1;
    }

ColumnIndexHCell = ColumnIndex + 1;          
RowIndexHCell = RowIndex + 2;

Range rng = xlSheets.Cells[RowIndexHCell, ColumnIndexHCell] as Excel.Range;
rng.Interior.Color = System.Drawing.ColorTranslator.ToOle(System.Drawing.Color.Yellow);

There is any other way to load data in DataTable to get column and row index without using a lot of memory or by using Excel.Range.Cell instead of dataset and DataTable to get Cell value, column and row index from xlsx file please ?

I didn't show the whole code because it's long. Please keep me informed if more information needed.

回答1:

When trying to read data from an Excel with huge number of Rows, it is better to read data by chunk (in OleDbDataAdapter you can use paging option to achieve that).

int result = 1;
int intPagingIndex = 0;
int intPagingInterval = 1000;

while (result > 0){

    result = daGetDataFromSheet.Fill(dsErrorColumns,intPagingIndex, intPagingInterval , Error_Sheet);
    System.Data.DataTable dtErrorColumns = dsErrorColumns.Tables[Error_Sheet];

    //Implement your logic here

    intPagingIndex += intPagingInterval ;

}

This will prevent an OutOfMemory Exception. And no more need to specify a range like AC1:AC10000

References

Paging Through a Query Result
Fill(DataSet, Int32, Int32, String)

来源：https://stackoverflow.com/questions/54713652/outofmemoryexception-while-trying-to-read-big-excel-file-into-datatable

标签

excel

ssis

etl

script-task