SQLite/C# Connection Pooling and Prepared Statement Confusion

Question

I have been spending some time reading different best practices for databases and for SQLite specifically. While reading I found I was doing many things I shouldn\'t be doin

Answer 1

It helps to remember that both connection pooling and prepared (compiled) statements are just tools that have their limits and no approach can be equally suitable to all possible situations. With this in mind, let's remember when one might want to use connection pooling and prepared statements.

Possible Reasons to Use Connection Pooling

Connection pooling is useful when connections are expensive, for example:

• It takes significant time to establish a connection (network connections to a SQL Server or Oracle DB) and it is beneficial to "cache" open connections in an attempt to improve system performance.
• Connections are limited and shared within an application (connections from a web application serving multiple concurrent requests) or between applications so they have to be released as soon as possible to let the other clients continue.

Possible Reasons to Use Prepared Statements

Prepared statements are simply meant to improve performance of re-usable queries by cutting down the parsing time.

SQLite: What's the Best Choice?

The answer depends on your application requirements. Personally, I'm not sure if SQLite connection pooling is necessarily a good choice. If your application is single-threaded, it might be best to use a single permanent connection to the SQLite DB, which could be much faster than pooling and would allow you to use prepared statements too. This is different from SQL Server where connection pooling is a very reasonable default.

If performance matters, you should definitely profile the application to see if the SQLite connection pooling is beneficial for your scenario.

Specific Questions

Most of the answers are related to the current System.Data.SQLite provider source.

If I am opening and closing my connection (which may or may not mean the connection is being closed due to the thread pool) then how much use am I really getting from a prepared statement?

Generally, you should treat a connection coming out of the pool as new, i.e. you should not expect to get any benefit from statements prepared previously. The statement will be "re-prepared" unless you keep both the command and connection.

However I don't believe I would see a benefit from saving a single object in a transaction because once I close the connection the prepared statement that was generated from the first object is now lost. Is this a true statement?

This is a true statement.

If I create a SQLiteCommand that represents a query that I will be executing often do I need to keep that SQLiteCommand in memory for the prepared statement to stay active?

Yes, you need to keep it. SQLiteCommand holds a reference to the prepared statement.

If I create a new SQLiteCommand with the same SQLite statement is it recognized that the new SQLiteCommand is the same as the previous and thus has a prepared statement that can be used?

I don't think it's supported.

If I keep a SQLiteCommand in memory and change it's parameters and connection as I open and close the connection for different transactions am I essentially keeping a prepared statement alive between different connections?

If you change the SQLiteCommand's connection, the statement will be "re-prepared".

Answer 2

I didn't catch exactly what is the core problem, but if the problem is how to insert bulk insert statements in one transaction in so little time.

Here is a helper class I found earlier that could help you:

SQLiteBulkInsertHelper.cs

You can use it like this:

SQLiteBulkInsertHelper ContactBlk = new SQLiteBulkInsertHelper("<SQLiteConnection>","<Table Name>");
ContactBlk.AllowBulkInsert = true;
ContactBlk.AddParameter("<Column Name>", /*Column Data Type*/System.Data.DbType.Int64);
ContactBlk.AddParameter("<Column Name>", /*Column Data Type*/System.Data.DbType.String);
ContactBlk.Insert(new object[] {<First Column Value>,<Second Column Value>});
ContactBlk.Flush();

Give it a try if you see it as a solution to your problem.