Combine inserts into one transaction Python SQLite3

问题

I am trying to input 1000's of rows on SQLite3 with insert however the time it takes to insert is way too long. I've heard speed is greatly increased if the inserts are combined into one transactions. However, i cannot seem to get SQlite3 to skip checking that the file is written on the hard disk.

this is a sample:

if repeat != 'y':
    c.execute('INSERT INTO Hand (number, word) VALUES (null, ?)', [wordin[wordnum]])
    print wordin[wordnum]

data.commit()

This is what i have at the begining.

data = connect('databasenew')
data.isolation_level = None
c = data.cursor()  
c.execute('begin')

However, it does not seem to make a difference. A way to increase the insert speed would be much appreciated.

回答1:

According to Sqlite documentation, BEGIN transaction should be ended with COMMIT

Transactions can be started manually using the BEGIN command. Such transactions usually persist until the next COMMIT or ROLLBACK command. But a transaction will also ROLLBACK if the database is closed or if an error occurs and the ROLLBACK conflict resolution algorithm is specified. See the documentation on the ON CONFLICT clause for additional information about the ROLLBACK conflict resolution algorithm.

So, your code should be like this:

data = connect('databasenew')
data.isolation_level = None
c = data.cursor()  
c.execute('begin')

if repeat != 'y':
    c.execute('INSERT INTO Hand (number, word) VALUES (null,?)', [wordin[wordnum]])
    print wordin[wordnum]

    data.commit()

    c.execute('commit')

回答2:

https://stackoverflow.com/a/3689929/1147726 answers the question. execute('begin') does not have any effect. Apparently, a connection.commit() is sufficient.

回答3:

(In case someone is still looking for an answer to this)

You should use executemany if you are just doing 1000's of inserts successively.

Look at What is the optimized way to insert large number of records (more than 40,000) in sqlite3

I just struggled with a LOT (order millions) of execute's that were taking about 30 minutes to complete - Switched to executemany and I now have it down to about 10 minutes.

回答4:

You can use executemany, see this SO question: python sqlite question - Insert method

来源：https://stackoverflow.com/questions/5055314/combine-inserts-into-one-transaction-python-sqlite3

标签

python

transactions

insert

sqlite