SQL command terminates before reaching end of large csv file

问题

I have a large csv file with lots of data that I need to be able to analysis (~6M rows). I want to connect to the file and run SQL command against it to return only the data I'm interested in analysing. The VBA I'm writing is in Excel 2010.

Everything works fine when the number of rows in the csv file is < 4432669. When the csv file has more rows than this, the command seem to terminate at that point in the file and just returns what ever it has found up to that point. No Error is thrown (CN.Errors), I first though it might be that the command timedout but when I increase this it made no difference. I also checked with different csv files just incase that row contained corrupted data, but no luck. Recordset maxrecords is set to 0 (No limit).

I've tried using Microsoft.Jet.OLEDB.4.0; and driver={Microsoft Text Driver (*.txt; *.csv)}; in the connectionstring, both behave the same as described above.

Here is test code I'm using,

Dim CN As New ADODB.Connection
Dim RS As New ADODB.Recordset
Dim Err As ADODB.Error

providerstr = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\cygwin\home\MarkM\csvimport\filtertest4\;" & _
"Extended Properties=" & Chr(34) & "text;HDR=Yes;FMT=Delimited" & Chr(34) & ";"
CN.ConnectionString = providerstr
CN.Mode = adModeRead
CN.CommandTimeout = 900
CN.Open

RS.Open "SELECT exCode FROM 5M_MBP1R04.csv", CN, adOpenStatic, adLockReadOnly
RS.MoveLast
MsgBox "Number of rows = " & RS.RecordCount

    For Each Err In CN.Errors
        strError = "Error #" & Err.Number & vbCr & _
            "   " & Err.Description & vbCr & _
            "   (Source: " & Err.Source & ")" & vbCr & _
            "   (SQL State: " & Err.SqlState & ")" & vbCr & _
            "   (NativeError: " & Err.NativeError & ")" & vbCr
        If Err.HelpFile = "" Then
            strError = strError & "   No Help file available"
        Else
            strError = strError & _
               "   (HelpFile: " & Err.HelpFile & ")" & vbCr & _
               "   (HelpContext: " & Err.HelpContext & ")" & _
               vbCr & vbCr
        End If

        Debug.Print strError
    Next

Really appreciate any help as I'm completely stuck now.

BR's Mark.

回答1:

Perhaps you are exceeding a memory constraint due to the CursorType. Try changing it to adOpenForwardOnly

Here is the MSDN page describing Cursor Types. https://msdn.microsoft.com/en-us/library/windows/desktop/ms681771(v=vs.85).aspx

来源：https://stackoverflow.com/questions/26441875/sql-command-terminates-before-reaching-end-of-large-csv-file