gevent

Celery Gevent Pool - ConcurrentObjectUseError

▼魔方 西西 提交于 2021-02-18 07:06:09
问题 I have a celery worker that is using the gevent pool which does HTTP requests and adds another celery task with page source. I'm using Django, RabbitMQ as a broker, Redis as a celery result backend, Celery 4.1.0. The task has ignore_result=True but I'm getting this error pretty often ConcurrentObjectUseError: This socket is already used by another greenlet: <bound method Waiter.switch of <gevent.hub.Waiter...> I see it is related to the Redis connection. I can't figure out how to solve this.

Celery Gevent Pool - ConcurrentObjectUseError

我是研究僧i 提交于 2021-02-18 07:04:51
问题 I have a celery worker that is using the gevent pool which does HTTP requests and adds another celery task with page source. I'm using Django, RabbitMQ as a broker, Redis as a celery result backend, Celery 4.1.0. The task has ignore_result=True but I'm getting this error pretty often ConcurrentObjectUseError: This socket is already used by another greenlet: <bound method Waiter.switch of <gevent.hub.Waiter...> I see it is related to the Redis connection. I can't figure out how to solve this.

Celery Gevent Pool - ConcurrentObjectUseError

女生的网名这么多〃 提交于 2021-02-18 07:04:28
问题 I have a celery worker that is using the gevent pool which does HTTP requests and adds another celery task with page source. I'm using Django, RabbitMQ as a broker, Redis as a celery result backend, Celery 4.1.0. The task has ignore_result=True but I'm getting this error pretty often ConcurrentObjectUseError: This socket is already used by another greenlet: <bound method Waiter.switch of <gevent.hub.Waiter...> I see it is related to the Redis connection. I can't figure out how to solve this.

Python 中多进程、多线程、协程

有些话、适合烂在心里 提交于 2021-02-15 06:29:23
进程:   一个运行的程序(代码)就是一个进程,没有运行的代码叫程序,进程是系统资源分配的最小单位,进程拥有自己独立的内存空间,所以进程间数据不共享、开销大。 线程:   调度执行的最小单位,也叫执行路径,不能独立存在,依赖进程存在一个进程至少有一个线程,叫主线程,而多个线程共享内存(数据共享,共享全局变量),从而极大的提高了程序的运行效率。 协程:   是一种用户态的轻量级线程,协程的调度完全由用户控制,协程拥有自己的寄存器上下文和栈,协程调度切换时。将寄存器上下文和栈保存到其他地方,在切回来的时候,恢复先前保存的寄存器和上下文和栈,直接操作栈基本没有内核切换的开销,可以不加锁的访问全局变量,所以上下文的切换非常快。 什么时线程竞争?   线程时非独立的,同一个进程里线程是数据共享的,当各个线程方位数据资源时会出现竞争状态即:       数据几乎同步会被多个线程占用,造成数据混乱,即所谓的线程不安全,那怎么解决多线程竞争问题? -- 锁  锁的好处:     确保了某段关键代码(共享数据资源)只能由一个线程从头到尾完整的执行能解决多线竞争下的原子操作问题。  锁的坏处:        阻止了多线程并发执行,包含锁的某段代码实际上只能以单线程模式执行,效率大大的降低了,锁的致命问题:死锁。 多进程:   在Linux系统下,使用 os.fork(), 调用一次,返回两次

Python 从业十年是种什么体验?

主宰稳场 提交于 2021-02-14 14:27:08
点击上方“ 咸鱼学Python ”,选择“ 加为星标 ” 第一时间关注Python技术干货! 作者: laisky(基于 CC BY 4.0 授权许可) 原题: Python之路(内容略有调整) 来源: https://laisky.com/p/python-road 一、概述 本文起源于我在 Twitter 上发布的关于 Python 经历的一系列话题。 出于某些原因,想记录一下我过去数年使用 Python 的经验和一些感悟。毕竟算是一门把我带入互联网行业的语言,而我近期已经几乎不再写 Py 代码, 做一个记录,也许会对他人起到些微的帮助,也算是纪念与感恩了。 二、摘录 推文地址:https://twitter.com/ppcelery/status/1159620182089728000 最早接触 py 是 2010 年左右,那之前主要是使用 c、fortran 和 matlab 做数值运算。当时在做一些文件文本处理时觉得很麻烦,后来看到 NASA 说要用 py 取代 matlab,就去接触了 py。 python 那极为简洁与优美的语法给了当时的我极大的震撼,时至今日,写 py 代码对我而言依然是一种带有艺术意味的享受。 首先开宗明义的说一句:python 并不慢,至少不够慢。拿一个 web 后端来说,一台垃圾 4 核虚机,跑 4 个同步阻塞的 django,假设

Python 从业十年是种什么体验?

a 夏天 提交于 2021-02-13 17:51:45
点击“ Python编程与实战 ”,选择“置顶公众号” 第一时间获取 Python 技术干货! 作者: laisky(基于 CC BY 4.0 授权许可) 原题: Python之路(内容略有调整) 来源: https://laisky.com/p/python-road 一、概述 本文起源于我在 Twitter 上发布的关于 Python 经历的一系列话题。 出于某些原因,想记录一下我过去数年使用 Python 的经验和一些感悟。毕竟算是一门把我带入互联网行业的语言,而我近期已经几乎不再写 Py 代码, 做一个记录,也许会对他人起到些微的帮助,也算是纪念与感恩了。 二、摘录 推文地址:https://twitter.com/ppcelery/status/1159620182089728000 最早接触 py 是 2010 年左右,那之前主要是使用 c、fortran 和 matlab 做数值运算。当时在做一些文件文本处理时觉得很麻烦,后来看到 NASA 说要用 py 取代 matlab,就去接触了 py。 python 那极为简洁与优美的语法给了当时的我极大的震撼,时至今日,写 py 代码对我而言依然是一种带有艺术意味的享受。 首先开宗明义的说一句:python 并不慢,至少不够慢。拿一个 web 后端来说,一台垃圾 4 核虚机,跑 4 个同步阻塞的 django,假设

Wait until one of several greenlets finished

霸气de小男生 提交于 2021-02-08 08:30:32
问题 I have two functions that receives data from two different connections, and i should close both connections after getting result from one of them. def first(): gevent.sleep(randint(1, 100)) # i don't know how much time it will work return 'foo' def second(): gevent.sleep(randint(1, 100)) # i don't know how much time it will work return 'bar' Then i spawn each function: lst = [gevent.spawn(first), gevent.spawn(second)] gevent.joinall blocks current greenlet until both two greenlets from lst

Can python abstract base classes inherit from C extensions?

拥有回忆 提交于 2021-02-06 16:28:49
问题 It seems as if that when I have an abstract base class that inherits from gevent.Greenlet (which inherits from the C extension module greenlet: https://github.com/python-greenlet/greenlet) then classes that implement it do not raise any of the abc errors about unimplemented methods. class ActorBase(gevent.Greenlet): __metaclass__ = abc.ABCMeta @abc.abstractmethod def foo(self): print "foo" class ActorBaseTest(ActorBase): def bar(self): print "bar" abt = ActorBaseTest() # no errors! If I

Can python abstract base classes inherit from C extensions?

旧巷老猫 提交于 2021-02-06 16:27:51
问题 It seems as if that when I have an abstract base class that inherits from gevent.Greenlet (which inherits from the C extension module greenlet: https://github.com/python-greenlet/greenlet) then classes that implement it do not raise any of the abc errors about unimplemented methods. class ActorBase(gevent.Greenlet): __metaclass__ = abc.ABCMeta @abc.abstractmethod def foo(self): print "foo" class ActorBaseTest(ActorBase): def bar(self): print "bar" abt = ActorBaseTest() # no errors! If I

Can python abstract base classes inherit from C extensions?

你离开我真会死。 提交于 2021-02-06 16:26:53
问题 It seems as if that when I have an abstract base class that inherits from gevent.Greenlet (which inherits from the C extension module greenlet: https://github.com/python-greenlet/greenlet) then classes that implement it do not raise any of the abc errors about unimplemented methods. class ActorBase(gevent.Greenlet): __metaclass__ = abc.ABCMeta @abc.abstractmethod def foo(self): print "foo" class ActorBaseTest(ActorBase): def bar(self): print "bar" abt = ActorBaseTest() # no errors! If I