gil | 易学教程

When are Python threads fast?

阅读更多关于 When are Python threads fast?

问题 We're all aware of the horrors of the GIL, and I've seen a lot of discussion about the right time to use the multiprocessing module, but I still don't feel that I have a good intuition about when threading in Python (focusing mainly on CPython) is the right answer. What are instances in which the GIL is not a significant bottleneck? What are the types of use cases where threading is the most appropriate answer? 回答1: Threading really only makes sense if you have a lot of blocking I/O going on.

Embedding python in multithreaded C application

阅读更多关于 Embedding python in multithreaded C application

问题 I'm embedding the python interpreter in a multithreaded C application and I'm a little confused as to what APIs I should use to ensure thread safety. From what I gathered, when embedding python it is up to the embedder to take care of the GIL lock before calling any other Python C API call. This is done with these functions: gstate = PyGILState_Ensure(); // do some python api calls, run python scripts PyGILState_Release(gstate); But this alone doesn't seem to be enough. I still got random

numpy and Global Interpreter Lock

阅读更多关于 numpy and Global Interpreter Lock

问题 I am about to write some computationally-intensive Python code that'll almost certainly spend most of its time inside numpy 's linear algebra functions. The problem at hand is embarrassingly parallel. Long story short, the easiest way for me to take advantage of that would be by using multiple threads. The main barrier is almost certainly going to be the Global Interpreter Lock (GIL). To help design this, it would be useful to have a mental model for which numpy operations can be expected to

Multiprocessing useless with urllib2?

阅读更多关于 Multiprocessing useless with urllib2?

I recently tried to speed up a little tool (which uses urllib2 to send a request to the (unofficial)twitter-button-count-url (> 2000 urls) and parses it´s results) with the multiprocessing module (and it´s worker pools). I read several discussion here about multithreading (which slowed the whole thing down compared to a standard, non-threaded version) and multiprocessing, but i could´t find an answer to a (probably very simple) question: Can you speed up url-calls with multiprocessing or ain´t the bottleneck something like the network-adapter? I don´t see which part of, for example, the

Parallel file matching, Python

阅读更多关于 Parallel file matching, Python

问题 I am trying to improve on a script which scans files for malicious code. We have a list of regex patterns in a file, one pattern on each line. These regex are for grep as our current implementation is basically a bash script find\grep combo. The bash script takes 358 seconds on my benchmark directory. I was able to write a python script that did this in 72 seconds but want to improve more. First I will post the base-code and then tweaks I have tried: import os, sys, Queue, threading, re

Python Global Interpreter Lock (GIL) workaround on multi-core systems using taskset on Linux?

阅读更多关于 Python Global Interpreter Lock (GIL) workaround on multi-core systems using taskset on Linux?

So I just finished watching this talk on the Python Global Interpreter Lock (GIL) http://blip.tv/file/2232410 . The gist of it is that the GIL is a pretty good design for single core systems (Python essentially leaves the thread handling/scheduling up to the operating system). But that this can seriously backfire on multi-core systems and you end up with IO intensive threads being heavily blocked by CPU intensive threads, the expense of context switching, the ctrl-C problem[*] and so on. So since the GIL limits us to basically executing a Python program on one CPU my thought is why not accept

Green-threads and thread in Python

阅读更多关于 Green-threads and thread in Python

问题 As Wikipedia states: Green threads emulate multi-threaded environments without relying on any native OS capabilities, and they are managed in user space instead of kernel space, enabling them to work in environments that do not have native thread support. Python's threads are implemented as pthreads (kernel threads) , and because of the global interpreter lock (GIL), a Python process only runs one thread at a time. [ QUESTION ] But in the case of Green-threads (or so-called greenlet or

day33_8_14 并发编程三线程的GIL

阅读更多关于 day33_8_14 并发编程三线程的GIL

一。GIL 　　什么是GIL？　　GIL是一个全局排他锁，简单来说就是为了防止多线程并行操作的锁。这里有官方解释： In CPython, the global interpreter lock, or GIL, is a mutex that prevents multiple native threads from executing Python bytecodes at once. This lock is necessary mainly because CPython’s memory management is not thread-safe. (However, since the GIL exists, other features have grown to depend on the guarantees that it enforces.) 在cpython中，全局解释器锁（gil）是一个互斥体，它可以防止多个本机线程同时执行python字节码。这个锁是必要的，主要是因为cpython 的内存管理不是线程安全的。（然而，由于gil存在，其他特性也越来越依赖于它所执行的保证。）　　GIL的存在是因为CPython解释器的内存管理不是线程安全的　　在一个进程中有多个线程，多个线程之间如果可以并行会是什么样子？以垃圾回收机制为例。　

GIL全局解释锁.死锁与递归锁

阅读更多关于 GIL全局解释锁.死锁与递归锁

一.GIL全局解释器 In CPython, the global interpreter lock, or GIL, is a mutex that prevents multiple native threads from executing Python bytecodes at once. This lock is necessary mainly because CPython’s memory management is not thread-safe. (However, since the GIL exists, other features have grown to depend on the guarantees that it enforces.) View Code Python解释器有很多种，最常见的Python解释器就是Cpython解释器 GIL(global interpreter lock) :本质是一把互斥锁，把并发变成串行牺牲效率来保证数据的安全性在Cpython解释器中，同一个进程开启多线程，同一时刻只能有一个线程执行，无法利用多核优势。（只能并发，无法并行） GIL 存在的原因：是因为CPython解释器的内存管理线程是不安全的 GIL 作用：保证python解释器同一时间只能执行一个线程 GIL 不是python的特点

Day32 并发编程(三) GIL锁和其他知识

阅读更多关于 Day32 并发编程(三) GIL锁和其他知识

1.全局解释器锁GIL 什么是GIL 　　GIL是全局解释器锁他规定了每个线程在被CPU执行前都要获得一个GIL,并且同一时刻只有一个线程被执行为什么要有GIL 　　因为CPython解释器的内存管理不是线程安全的,所以在CPython中增加的一个GIL锁线程释放GIL锁的情况　　在I/O操作等会引起阻塞状态的操作时,会暂时释放GIL,其他线程可以获取GIL继续执行 python的多线程和多进程的使用场景　　1.当CPU处于多任务计算密集型的情况下　　　　单核:两者差不多,但是多线程更加省资源　　　　多核:由于多线程不能利用多核的优势,所以使用多进程更有优势　　2.当CPU处于多任务I/O密集型的情况下　　　　单核:和上述一样,多线程更省资源　　　　多核:由于I/O密集型的任务耗费的时间大部分在I/O操作上,并且cpu在阻塞状态时可以暂时释放GIL,所以还是多线程更省资源(开启进程是耗费资源,当需要进行的任务过多,多进程耗费的时间会大大超过多线程) 1 # 计算密集型 2 from multiprocessing import Process 3 import os,time 4 def work(): 5 res=0 6 for i in range(100000000): 7 res*=i 8 9 10 if __name__ == '__main__':