gearman | 易学教程

A queuing system which supports job batching (e.g. several jobs for 1 worker at once)

阅读更多关于 A queuing system which supports job batching (e.g. several jobs for 1 worker at once)

问题 I'm looking for a queuing system that could support the following scenario: A client adds a job - to check how many Facebook likes a particular url (URL1) has; A client adds another job - to check the same information for URL2; [....] A worker picks up anything from 1 to 50 jobs (urls) from the queue (e.g., if there's only 5 - it picks up 5, if there's 60 - picks up 50, leaving others for another worker), and issues a request against Facebook API (which allows multiple urls per request). If

wdlinux centos 7 gearman安装

阅读更多关于 wdlinux centos 7 gearman安装

一、GearMan环境安装依赖环境：yum install -y boost-devel gperf libevent-devel libuuid-devel 如果boost版本太低，则要自己编译安装 //注意：wdlinux centos 5.11 验证的版本 //ICU4C ：icu4c-56_1-src.zip //Boost：boost_1_50_0.tar.gz //安装ICU4C wget http://downloads.sourceforge.net/project/icu/ICU4C/4.0/icu4c-4_0-src.tgz?use_mirror=cdnetworks-kr-2 tar zxvf icu4c-4_0-src.tgz cd icu/source ./configure –prefix=/usr make make install ldconfig //安装Boost wget http://sourceforge.net/projects/boost/files/boost/1.43.0/boost_1_43_0.tar.gz/download tar zxvf boost_1_43_0.tar.gz cd boost_1_43_0 rm -rf /usr/include/boost/ rm -rf /usr/lib/libboost* .

Scheduling with gearman vs. cron?

阅读更多关于 Scheduling with gearman vs. cron?

问题 I have noticed a lot of people discussing Gearman and it's scheduling features making it enable to distribute work onto other servers. However, I have not yet seen a comparison to native cronjobs. What are the differences between cron and Gearman? 回答1: If you're doing pure scheduling, using Gearman is unnecessary. The main differences between Gearman and cron are that: cron jobs are only triggered only based on time, while Gearman functions are triggered by calls by other applications.

Python Gearman get data from background task

阅读更多关于 Python Gearman get data from background task

问题 I'm try use gearman with backgroud tasks and get data progress from worker. In documentation I'm see methods: send_job_data and send_job_status, but with background first method not work (I'm not see data in job.data_updates), but status changes in job.status. I'm use this code for test workers: from gearman import GearmanWorker import time worker = GearmanWorker(['192.168.1.79:4730']) def long_task(work, job): work.send_job_data(job, 'long task') work.send_job_status(job, 0, 3) time.sleep(60

Gearman使用Redis 做数据持久化

阅读更多关于 Gearman使用Redis 做数据持久化

2014年的时候，老大说数据持久化，当时听的一愣一愣的，不知道啥是持久化；讲了一堆Gearman使用 Redis 和MySQL的东西，表示听不懂；然后他自己在那搞；左后搞出来了，还在launchpad.net 报告了一个bug，当时确实是第一个报告那种使用情况下bug的第一人。队列，很多流量大的网站都会使用。对于数据，是否需要考虑数据的可靠性，对于一般不重要的数据丢失也没关系；但是对于订单和钱后关系的这类数据是需要高可靠性的。当时老大设计的系统几乎就是准备跑在缓存上的一个平台系统（当然，不使用缓存一样运行），还有个队列Gearman。内存，数据都在内存上，丢失咋办；先写数据库（持久化）在改内存，这样就没啥意思了。所有的写操作都是先到缓存，判断缓存操作是否成功，失败写日志记录；然后使用队列Gearman写到数据库，那如果缓存中的数据加入到了队列的Job, 还没来得及Worker 消费数据，服务器出现故障，内存数据全部丢失，与之相关操作全部出错。解决方案：使用Redis 做数据的持久化，即使memcached 数据丢失，在Redis 上还有持久化的数据，当服务器可用后， Gearman 再从Redis 消费数据，持久化到数据库MySQL 毕业后，自己动手配置过该环境，当时虽然配置出来了，但是没有做记录，当再次遇到这个问题时，发现已经有人做过了 http://www

Installing Gearman PHP extension on Debian 6

阅读更多关于 Installing Gearman PHP extension on Debian 6

问题 I'm desperately trying to get the Gearman PHP extension working on a Debian 6. I have all the binaries and sources root@debian:/tmp/pear/install# aptitude search gearman i gearman - A distributed job queue i A gearman-job-server - Job server for the Gearman i gearman-tools - Tools for the Gearman i libgearman-dev - Development files for the i libgearman4 - Library providing Gearman ...but when there are some bugs in the PHP extension confing which don't let me install it # /usr/local/pear/bin

Gearman , php extension problem : Class 'GearmanWorker' not found in .. using terminal but works on browser

阅读更多关于 Gearman , php extension problem : Class 'GearmanWorker' not found in .. using terminal but works on browser

问题 I've installed gearman on ubuntu 10.04 recently and installed it's pecl extension. Now , when I run a php file in the browser that contains : $client = new GearmanWorker(); die(var_Dump($client)); I get object(GearmanWorker)#1 (0) { } but when running the a real worker file in terminal (by root) , I get this: sudo php worker.php PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/imagick.ini on line 1 in Unknown on line 0 PHP Deprecated: Comments starting with '#

pcntl_signal callbacks are not executed

阅读更多关于 pcntl_signal callbacks are not executed

I am using php 5.4. I am using gearman and supervisor to control gearman workers. I want to catch the TERM signal produced by supervisor in PHP. I have the following worker code: pcntl_signal(SIGTERM, function($sig){ error_log("interrupted", false); }); while(true){ pcntl_signal_dispatch(); $gw->work(); } } My supervisor conf file is as follows: [program:public_pdf_convert] process_name = pdf_convert_%(process_num)02d command = /usr/local/bin/php worker_pdf_convert.php numprocs = 1 directory = /projects/mv2/deliver/batch/batch/workers/ By default, supervisor use TERM . When I stop supervisor,

Gearman PHP Extension: Dead Job Server = Slow Response from all Workers

阅读更多关于 Gearman PHP Extension: Dead Job Server = Slow Response from all Workers

I started with this question: Gearman: 3 seconds between client request and worker receive. Is this normal? Environment: Ubuntu 12.04 Desktop PHP 5.3.10 Gearman (libgearman 1.1.5 with PHP Extension 1.1.1) Multiple servers on LAN I couldn't get a worker response time of less than 3 sec and I couldn't figure out why. I narrowed it down to a wrapper class I'd built. I then narrowed it further to a specific method within the class. Long story short the real problem seems to lie in the addServer method of the GearmanWorker in the PHP extension. My wrapper class was attempting to connect to 3

Simple scalable work/message queue with delay

阅读更多关于 Simple scalable work/message queue with delay

问题 I need to set up a job/message queue with the option to set a delay for the task so that it's not picked up immediately by a free worker, but after a certain time (can vary from task to task). I looked into a couple of linux queue solutions (rabbitmq, gearman, memcacheq), but none of them seem to offer this feature out of the box. Any ideas on how I could achieve this? Thanks! 回答1: I've used BeanstalkD to great effect, using the delay option on inserting a new job to wait several seconds till