PHP How do I implement queue processing in php

Question

I want the data sent by my clients (via post) to be placed in a queue and a php script on my server first checks if the queue is empty. If the queue is not empty, then the s

Answer 1

Problem with cronjob approach is, cronjob could at the most set to 1 minute interval, so there is a delay of 1 minute in the job execution, if that's acceptable then that's fine otherwise should use queues with polling script.

Answer 2

This is something you could easily do with enqueue library. First, you can choose from a variety of transports, such as AMQP, STOMP, Redis, Amazon SQS, Filesystem and so on.

Secondly, That's super easy to use. Let's start from installation:

You have to install the enqueue/simple-client library and one of the transports. Assuming you choose the filesystem one, install enqueue/fs library. To summarize:

composer require enqueue/simple-client enqueue/fs 

Now let's see how you can send messages from your POST script:

<?php
// producer.php

use Enqueue\SimpleClient\SimpleClient;

include __DIR__.'/vendor/autoload.php';

$client = new SimpleClient('file://'); // the queue will store messages in tmp folder

$client->sendEvent('a_topic', 'aMessageData');

The consumption script:

<?php
// consumer.php

use Enqueue\SimpleClient\SimpleClient;
use Enqueue\Psr\PsrProcessor;
use Enqueue\Psr\PsrMessage;

include __DIR__.'/vendor/autoload.php';

$client = new SimpleClient('file://');

$client->bind('a_topic', 'a_processor_name', function(PsrMessage $psrMessage) {
   // processing logic here

   return PsrProcessor::ACK;
});

// this call is optional but it worth to mention it.
// it configures a broker, for example it can create queues and excanges on RabbitMQ side. 
$client->setupBroker();

$client->consume();

Run as many consumer.php processes as you by using supervisord or other process managers, on local computer you can run it without any extra libs or packages.

That's a basic example and enqueue has a lot of other features that might come in handy. If you are interested, check the enqueue documentation out.

Answer 3

You could use something like Zero MQ

See Example by Rasmus Lerdorf.

You could also consider using Gearman to distribute the load.

Answer 4

Take a look at this.

It uses memcached for persistence.

Answer 5

Since relational DB's (Ex: MySQL) are so flexible, and well understood by web developers, they are used for many type of job queues. Many PHP applications use this solution as a fallback when object caching is unconfigured. It's the method of last resort because it's a very expensive way to implement a queue.

If you must use MySQL as your queue, one of the Percona engineers wrote this blog entry on managing the potential pain points.

If you want the most scalable implementation, I would highly recommend ZeroMQ. However it's not a default, or particularly common, PHP extension. So for a project where you would not be controlling the web server stack: Use APC Objects, Memcache or Memcached, and then fallback to a MySQL cache table.

Answer 6

Here is another great tutorial for this:

http://squirrelshaterobots.com/programming/php/building-a-queue-server-in-php-part-1-understanding-the-project/

The other solution is using Gearman which they seem to have incorporated into PHP (it wasn't the last time I played with it): http://php.net/manual/en/book.gearman.php