Add new column without table lock?

Question

In my project having 23 million records and around 6 fields has been indexed of that table.

Earlier I tested to add delta column for Thinking Sphinx search but it turns in holding the whole database lock for an hour. Afterwards when the file is added and I try to rebuild indexes this is the query that holds the database lock for around 4 hours:

"update user_messages set delta = false where delta = true"

Well for making the server up I created a new database from db dump and promote it as database so server can be turned live.

Now what I am looking is that adding delta column in my table with out table lock is it possible? And once the column delta is added then why is the above query executed when I run the index rebuild command and why does it block the server for so long?

PS.: I am on Heroku and using Postgres with ika db model.

Answer 1

Updating in PostgreSQL means writing a new version of the row. Your question does not provide all the information, but that probably means writing millions of new rows.

If your data model and available disk space allow for it, CREATE a new table in the background and then, in one transaction: DROP the old table, RENAME the new one.
Details and caveats in this related later answer on dba.SE.

Also, if a major proportion of the table is affected, remove all indexes before doing the mass UPDATE, because all indexes have to be updated too. It is faster to delete them and recreate them after the mass UPDATE is done.

And finally, while creating the new table in the background: Apply all changes at once, or you create multiple updated versions of the affected rows.

Disclaimer: Adding a new column without DEFAULT (= DEFAULT NULL) will not normally create a new row and is very cheap per se. Adding values to it creates new rows.
Details in the maunal.

If you cannot remove the original table because of constraints, another fast way is to build a temporary table, TRUNCATE the original one and mass INSERT the new rows - sorted, if that helps performance. All in one transaction. Something like this:

BEGIN

SET temp_buffers = 1000MB;  -- or whatever you can spare temporarily

CREATE TEMP TABLE tmp AS
SELECT * FROM tbl LIMIT 0; -- copy layout of table

ALTER TABLE tmp ADD column delta boolean; -- NOT DEFAULT

INSERT INTO tmp (col1, col2, ... , delta)
SELECT col1, col2, ... , FALSE
FROM   tbl;                -- copy existing rows plus new value
-- ORDER BY ???

-- DROP all indexes here

TRUNCATE tbl;              -- empty table - truncate is super fast

ALTER TABLE tbl ADD column delta boolean DEFAULT FALSE; -- NOT NULL?

INSERT INTO tbl
SELECT * FROM tmp;         -- insert back surviving rows.

-- recreate all indexes here

COMMIT;

Answer 2

You could add another table with the one column, there won't be any such long locks. Of course there should be another column, a foreign key to the first column.

For the indexes, you could use "CREATE INDEX CONCURRENTLY", it doesn't use too heavy locks on this table http://www.postgresql.org/docs/9.1/static/sql-createindex.html.