Improve large data import performance into SQLite with C#

Question

I am using C# to import a CSV with 6-8million rows.

My table looks like this:

CREATE TABLE [Data] ([ID] VARCHAR(100)  NULL,[Raw] VARCHAR(200)  NULL)
CREA         


        

Answer 1

One thing you might try is to create the index after the data has been inserted - typically it's much faster for databases to build indexes in a single operation than to update it after each insert (or transaction).

I can't say that it'll definitely work with SQLite, but since it only needs two lines to move it's worth trying.

I'm also wondering if a 6 million row transaction might be going too far - could you change the code to try different transaction sizes? Say 100, 1000, 10000, 100000? Is there a "sweet spot"?