PostgreSQL: How to select and update from a table as if it were a queue

问题

I'd like several processes to pull from a table as if it were a queue. The requirements:

Order of the items is important, and must be specified in the query
Each process (of which there are an unknown amount) can get chunks of N items at a time
Each item must only be processed once
I don't want to lock the table (for performance reasons)

I have a working solution, but would like some other opinions.

First attempt:

UPDATE foo
SET should_select=0
FROM
        (SELECT
                *
        FROM foo
        WHERE
                should_select=1
        ORDER BY id ASC
        LIMIT $N FOR UPDATE) as to_select
WHERE
    foo.id = to_select.id
RETURNING to_select.*;

This works, but does not return the $N results in order, as "RETURNING" does not guarantee order. I could sort on client side, but that will require loading all of the results into memory, and if $N is large this is not efficient.

Second attempt:

I can use an advisory lock so only 1 process can grab from the queue at a time, and then do two queries:

SELECT pg_advisory_lock( 123456 );

SELECT
        *
FROM foo
WHERE
        should_select=1
ORDER BY id ASC
LIMIT $N;

UPDATE foo
SET should_select=0
WHERE id IN (<list of $N ids...>);

SELECT pg_advisory_unlock( 123456 );

However, I'm not a fan of the IN clause.. this seems very bad for performance.

I'm using version 9.3.

Thoughts?

回答1:

I looked around a bit. There is some talk about controlling the returning order here : returning order. This was an older question (pre 9.1 maybe), but there was a reference to the CTE. That's the ticket, right? Here is my favorite page about stuff you should be using in postgres (including CTEs): The best Postgres Features .... So, using a with clause I could coerce the order of a returning * like:

WITH FOOUP as (
UPDATE foo
SET should_select=0
FROM
    (SELECT
            *
    FROM foo
    WHERE
            should_select=1
    ORDER BY id ASC
    LIMIT $N FOR UPDATE) as to_select
WHERE
    foo.id = to_select.id
RETURNING to_select.*)
select * from FOOUP order by id asc;

That looks like it should be super efficient, and you should get the tuples back in the correct order. Maybe you could leave off the FOR UPDATE?

-g

回答2:

DELETE FROM queue
WHERE itemid = (
  SELECT itemid
  FROM queue
  ORDER BY itemid
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING *;

Source:

1) https://blog.2ndquadrant.com/what-is-select-skip-locked-for-in-postgresql-9-5/

来源：https://stackoverflow.com/questions/26637198/postgresql-how-to-select-and-update-from-a-table-as-if-it-were-a-queue

标签

postgresql

locking

sql-update

queue