pika

pyspider + RabbitMQ 使用记

我的梦境 提交于 2019-12-06 18:16:25
首先我们需要安装 RabbitMQ,然后通过服务启动它,默认为注册到本机的5672端口。我们的爬虫和数据库写入脚本都需要连接到 RabbitMQ,一边往队列中写入数据,另一边从队列中取出数据,然后插入到数据。 Python 中使用 RabbitMQ 可以通过调用 pika 这个库,安装过程见 官方文档 ,对于 RabbitMQ 本身也有 中文教程 。 本项目用到的模型是一对一的,用 pika 写很容易,代码如下: import pika # 导入库 connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost')) # 设置一个新连接,连接到本地的 RabbitMQ 服务端。 channel = connection.channel() # 注册到 books 队列 channel.queue_declare(queue='books') channel.basic_publish(exchange='', routing_key='books', body='Whats up') # 发送消息 body connection.close() # 在 basic_publish 这个函数中,我们设置 exchange 为空,而 routing-key 为 books,此时 basic

rabbitmq multiple consumers on a queue- only one get the message

醉酒当歌 提交于 2019-12-06 00:02:07
问题 I implemented multiple consumers, who are fetching messages from a single queue, I am doing this using something similar to this example, except that I'm doing basic.get on an infinite loop for polling. Any idea how do I prevent racing between all consumers, in that only one consumer will get the message and the other will continue to do polling until another message comes? I trying to follow a logic in which as soon as I get the message I ack it for the message to be removed, but its seems

Python Pika - Consumer into Thread

牧云@^-^@ 提交于 2019-12-05 20:47:50
I'm working on a Python app with a background thread for consuming message from a RabbitMQ Queue (topic scenario). I start the thread on on_click event of a Button. Here is my code, please take attention on "#self.receive_command()". def on_click_start_call(self,widget): t_msg = threading.Thread(target=self.receive_command) t_msg.start() t_msg.join(0) #self.receive_command() def receive_command(self): syslog.syslog("ENTERED") connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost')) syslog.syslog("1") channel = connection.channel() syslog.syslog("2") channel.exchange

python 使用 RabbitMQ

喜夏-厌秋 提交于 2019-12-05 17:45:22
一、RabbitMQ消息队列介绍 RabbitMQ是在两个独立得python程序,或其他语言交互时使用。 RabbitMQ:erlang语言 开发的。 Python中连接RabbitMQ的模块:pika 、Celery(分布式任务队列) 、haigha 可以维护很多得队列 RabbitMQ 教程官网: http://www.rabbitmq.com/getstarted.html 几个概念说明: Broker:简单来说就是消息队列服务器实体。 Exchange:消息交换机,他制定消息按什么规则,路由到哪个队列。 Queue:消息队列载体,每个消息都会被投入一个或多个队列。 Binding:绑定,他的作用就是把exchange和queue按照路由规则绑定起来。 Routing Key:路由关键字,exchange根据这个关键字进行消息投递。 vhost:虚拟主机,一个broker里可以设多个vhost,用作不同用户得权限分离。 producer:消息生产者,就是投递消息得程序。 consumer:消息消费者,就是接受消息得程序。 channel:消息通道,在客户端得每个连接里。可以建立多个channel,每个channel代表一个会话任务。 二、RabbitMQ基本示范 1.Rabbitmq安装 ubuntu系统 install rabbitmq-server # 直接搞定 ---

Set Timeout for Pika ioloop async (RabbitMQ)

戏子无情 提交于 2019-12-05 13:59:07
I need to be able to gracefully stop a consumer (worker) who works in a Pika ioloop. The worker should stop after 60 seconds. Currently processed messages should be finished. I tried to put a connection.close() inside the callback function but that only stopped the current thread and not the complete ioloop. And it gave a terrible error output. Please see line 16 and following in my code: I used the (basic example about Pika ioloop http://pika.github.com/connecting.html#cps-example : from pika.adapters import SelectConnection channel = None def on_connected(connection): connection.channel(on

Getting “pika.exceptions.ConnectionClosed” error while using rabbitmq in python

一世执手 提交于 2019-12-05 11:59:10
I am using "hello world" tutorial in : http://www.rabbitmq.com/tutorials/tutorial-two-python.html . worker.py looks like this import pika import time connection = pika.BlockingConnection(pika.ConnectionParameters( host='localhost')) channel = connection.channel() channel.queue_declare(queue='task_queue', durable=True) print ' [*] Waiting for messages. To exit press CTRL+C' def callback(ch, method, properties, body): print " [x] Received %r" % (body,) time.sleep( body.count('.') ) print " [x] Done" ch.basic_ack(delivery_tag = method.delivery_tag) channel.basic_qos(prefetch_count=1) channel

RabbitMQ pika.exceptions.ConnectionClosed

拟墨画扇 提交于 2019-12-04 22:28:32
问题 I tried to send message and receive message using RabbitMQ. I dont have computer science background, the terms I used could not be very accurate. I try to copy the tutorial file: When submitting my html form, my python script (cgi) the message is submitting to the queue connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost')) channel = connection.channel() channel.queue_declare(queue='task_queue', durable=True) message = PN channel.basic_publish(exchange='', routing

Message sequence acting unexpectedly in RabbitMQ

廉价感情. 提交于 2019-12-04 20:40:46
问题 I'd like to implement a RabbitMQ topology similar to Option 3 here, except for some differences: The new topology should handle a few 1000 messages per day. And it shall have two exchanges: one to deal with the main queues (about 30), the other to deal with retry and error queues (about 60). I've been following this tutorial and the usual RMQ tutorials, plus many SO posts. The RMQ server is fired up in a Docker container. The problem I am facing is that the not all messages are being picked

Which form of connection to use with pika

混江龙づ霸主 提交于 2019-12-04 08:10:06
问题 I've been trying to figure out which form of connection i should use when using pika, I've got two alternatives as far as I understand. Either the BlockingConnection or the SelectConnection , however I'm not really sure about the differences between these two (i.e. what is the BlockingConnection blocking? and more) The documentation for pika says that SelectConnection is the preferred way to connect to rabbit since it provides "multiple event notification methods including select, epoll,

python rabbitmq的库,rabbitpy代替pika

耗尽温柔 提交于 2019-12-03 14:59:17
之前看网上都是清一色pika包的例子,就用的pika包,最大问题是非多线程安全,改为使用rabbitpy。大幅改善了pika多线程需要加锁,和外网推送延迟又不能开多线程导致推送慢的问题。 rabbitpy有个适配器,可以把rabbitpy包的channel适配成与pika包的channel的相同公有方法,减少了难度。 高层次封装,使用参数来控制使用什么包来操作rabbitmq。 # -*- coding: utf-8 -*- # @Author : ydf from collections import Callable import time from threading import Lock import rabbitpy from pika import BasicProperties # noinspection PyUnresolvedReferences from rabbitpy.message import Properties import pika from pika.adapters.blocking_connection import BlockingChannel from app.utils_ydf import LogManager from app.utils_ydf.mixins import LoggerMixin from app.utils