How do I insert 800000 records into an MS Access table?

Question

I need to insert 800000 records into an MS Access table. I am using Delphi 2007 and the TAdoXxxx components. The table contains some integer fields, one float f

Answer 1

Get SQL Server Express (free) and connect to it from Access an external table. SQL express is much faster than MS Access.

Answer 2

I would prefill the database, and hand them the file itself, rather than filling an existing (but empty) database.

If the data you have to fill changes, then keep an ODBC access database (MDB file) synchronized on the server using a bit of code to see changes in the main database and copy them to the access database.

When the user requests a new database zip up the MDB, transfer it to them, and open it.

Alternately, you may be able to find code that opens and inserts data into databases directly.

Alternately, alternately, you may be able to find another format (other than csv) which access can import that is faster.

-Adam

Answer 3

Your text based solution seems the fastest, but you can get it quicker if you could get an preallocated MS Access in a size near the end one. You can do that by filling an typical user database, closing the application (so the buffers are flushed) and doing a manual deletion of all records of that big table - but not shrinking/compacting it.

So, use that file to start the real filling - Access will not request any (or very few) additional disk space. Don't remeber if MS Access have a way to automate this, but it can help much...

Answer 4

Also check to see how long it takes to copy the file. That will be the lower bound of how fast you can write data. In db's like SQL, it usually takes a bulk load utility to get close to that speed. As far as I know, MS never created a tool to write directly to MS Access tables the way bcp does. Specialized ETL tools will also optimize some of the steps surrounding the insert, such as the way SSIS does transformations in memory, DTS likewise has some optimizations.

Answer 5

What about skipping the text file and using ODBC or OLEDB to import directly from the source table? That would mean altering your FROM clause to use the source table name and an appropriate connect string as the IN '' part of the FROM clause.

EDIT: Actually I see you say the original format is xBase, so it should be possible to use the xBase ISAM that is part of Jet instead of needing ODBC or OLEDB. That would look something like this:

INSERT INTO table (...) 
SELECT * 
FROM tablename IN 'c:\somedir\'[dBase 5.0;HDR=NO;IMEX=2;];

You might have to tweak that -- I just grabbed the connect string for a linked table pointing at a DBF file, so the parameters might be slightly different.

Answer 6

You won't be importing 800,000 records in less than a minute, as someone mentioned; that's really fast already.

You can skip the annoying translate-to-text-file step however if you use the right method (DAO recordsets) for doing the inserts. See a previous question I asked and had answered on StackOverflow: MS Access: Why is ADODB.Recordset.BatchUpdate so much slower than Application.ImportXML?

Don't use INSERT INTO even with DAO; it's slow. Don't use ADO either; it's slow. But DAO + Delphi + Recordsets + instantiating the DbEngine COM object directly (instead of via the Access.Application object) will give you lots of speed.