why writes in a table prevent vacuums in another?

Question

Having READ COMMITTED isolation level, idle transactions that have performed a write operation will prevent vacuum to cleanup dead rows for the tables that transaction wrote in.

That is clear for tables that were written by transactions that are still in progress. Here you can find a good explanation.

But it is not clear to me why this limitation affects also to any other tables.

For example: transaction T is started and it updates table B, vacuum is executed for table A while T is in "idle in transaction" state. In this scenario, why dead rows in A cannot be removed?

Here what I did:

# show default_transaction_isolation;
 default_transaction_isolation 
-------------------------------
 read committed
(1 row)
# create table a (v int);
CREATE TABLE
# create table b (v int);
CREATE TABLE

# insert into a values (generate_series(1,1000));
INSERT 0 1000

At this point I do an update to generate new 1000 dead rows

# update a set v = v + 1;
UPDATE 1000

Vacuuming will remove them as expected:

# vacuum verbose a;
INFO:  vacuuming "public.a"
INFO:  "a": removed 1000 row versions in 5 pages
INFO:  "a": found 1000 removable, 1000 nonremovable row versions in 9 out of 9 pages
DETAIL:  0 dead row versions cannot be removed yet.
There were 0 unused item pointers.
0 pages are entirely empty.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
VACUUM

I now start transaction T writing in table b:

# begin;
BEGIN
# insert into b values (generate_series(1,1000));
INSERT 0 1000

I generate more dead rows again in a different transaction T1 that started after T:

# begin;
# update a set v = v + 1;
# commit;

In a different transaction:

# vacuum verbose a;
INFO:  vacuuming "public.a"
INFO:  "a": found 0 removable, 2000 nonremovable row versions in 9 out of 9 pages
DETAIL:  1000 dead row versions cannot be removed yet.
There were 34 unused item pointers.
0 pages are entirely empty.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
VACUUM

This is the relevant part: DETAIL: 1000 dead row versions cannot be removed yet.

If I commit transaction T and execute again vacuum I get dead rows removed as expected:

# vacuum verbose a;
INFO:  vacuuming "public.a"
INFO:  "a": removed 1000 row versions in 5 pages
INFO:  "a": found 1000 removable, 1000 nonremovable row versions in 9 out of 9 pages
DETAIL:  0 dead row versions cannot be removed yet.
There were 34 unused item pointers.
0 pages are entirely empty.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
VACUUM

Answer 1

cant reproduce:

first session script:

-bash-4.2$ cat prim.sql
create table a (v int);
create table b (v int);
insert into a values (generate_series(1,1000));
update a set v = v + 1;
vacuum verbose a;
begin;
  insert into b values (generate_series(1,1000));
  select pg_sleep(9);
  select e'I\'m still open transaction'::text prim;

second session and check state:

-bash-4.2$ cat 1.sh
(sleep 3; psql t -c "vacuum verbose a;") &
(sleep 5; psql t -c "select state,query from pg_stat_activity where state != 'idle' and pid <> pg_backend_pid()") &
psql t -f prim.sql

and run:

-bash-4.2$ bash 1.sh
CREATE TABLE
CREATE TABLE
INSERT 0 1000
UPDATE 1000
psql:prim.sql:5: INFO:  vacuuming "public.a"
psql:prim.sql:5: INFO:  "a": removed 1000 row versions in 5 pages
psql:prim.sql:5: INFO:  "a": found 1000 removable, 1000 nonremovable row versions in 9 out of 9 pages
DETAIL:  0 dead row versions cannot be removed yet.
There were 0 unused item pointers.
0 pages are entirely empty.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
VACUUM
BEGIN
INSERT 0 1000
INFO:  vacuuming "public.a"
INFO:  "a": found 0 removable, 1000 nonremovable row versions in 9 out of 9 pages
DETAIL:  0 dead row versions cannot be removed yet.
There were 1000 unused item pointers.
0 pages are entirely empty.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
VACUUM
 state  |        query
--------+---------------------
 active | select pg_sleep(9);
(1 row)

 pg_sleep
----------

(1 row)

            prim
----------------------------
 I'm still open transaction
(1 row)

As you can see the first session was active before, while and after the vacuum in different session took place.

the version I tried at is:

t=# select version();
                                                   version
--------------------------------------------------------------------------------------------------------------
 PostgreSQL 9.3.14 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit
(1 row)

Answer 2

It is important to generate the dead rows again in a transaction started AFTER the transaction that remains open.

I have been able to reproduce the problem with the following versions:

• PostgreSQL 9.3.19 on x86_64-unknown-linux-gnu, compiled by gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4, 64-bit

• PostgreSQL 9.5.9 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4, 64-bit'

Answer 3

Following this question up via Twitter.

Current (at least up to PostgreSQL 9.6) behavior is:

Live transactions performing a write operation in any table will prevent vacuuming dead rows generated by commited transactions that started after first live transaction in any other table.

Even this limitation is not required from the conceptual point of view, it is how current algorithm is implemented for performance on checking dead rows reasons.