Queue in php and postgres

Question

I would like to find out a good way to go about implementing a jobs queue using postgres and PDO (php).

Basically I have an even

Answer 1

As written, another worker trying to claim the job would block at query 1. It can see the old version of the row, but cannot update it--it would block.

So don't do it in a single transaction. Claim and commit; do the work; then resolve and commit. Any workers coming along will see that the row is already claimed. Also, you can see that it is claimed, which will help you in debugging and monitoring.

When you claim the row you should mark with something distinctive (a pid, if there only one worker machine, or a hostname and pid, if there are several) rather than simply with 'ongoing'. That way if a worker dies you can manually clean up after it.

Answer 2

You cannot see changes made in a transaction, from outside of that transaction by definition.

Transactions are a fundamental concept of all database systems. The essential point of a transaction is that it bundles multiple steps into a single, all-or-nothing operation. The intermediate states between the steps are not visible to other concurrent transactions, and if some failure occurs that prevents the transaction from completing, then none of the steps affect the database at all.

For concurrency issues, I would recommend using the serializable transaction isolation level and / or row-level locking.

Answer 3

As it is presented, this is not possible. PostgreSQL doesn't have dirty reads, and QUERY1 is pointless since its effect will be overrided by QUERY2 before ever being visible.

But even if it was committed and visible immediately (if committed independantly), this wouldn't be satisfying anyway. In a high concurrency environment, the time between the SELECT of a row in the queue and its UPDATE with the ongoing state is enough for another worker to SELECT it too and create the confusion you want to avoid.

I think a close alternative to your design that should work can be achieved by replacing your QUERY1 with an advisory lock on the queue ID.

Pseudo-code:

BEGIN;
SELECT pg_try_advisory_xact_lock(3) INTO result;
IF result=true THEN
  -- grabbed the exclusive right to process this entry
  -- recheck the status now that the lock is taken
  SELECT status INTO var_status FROM events WHERE id=3;
  IF var_status='needs-to-be-done' THEN
     -- do the work...
     -- work is done
     UPDATE events SET status = 'resolved' WHERE id = 3;
  END IF;
ELSE
 -- nothing to do, another worker is on it
END IF;
COMMIT;

This kind of lock is automatically released at the end of the transaction. Contrary to the SELECT followed by UPDATE, the lock is guaranteed to be granted or denied atomically.