Speed up millions of regex replacements in Python 3

后端未结

关注

 9  1195

醉酒成梦 2020-11-22 05:44

I\'m using Python 3.5.2

I have two lists

a list of about 750,000 \"sentences\" (long strings)
a list of about 20,000 \"words\" that I would l

9条回答

不思量自难忘° (楼主)

2020-11-22 06:23

Practical approach

A solution described below uses a lot of memory to store all the text at the same string and to reduce complexity level. If RAM is an issue think twice before use it.

With join/split tricks you can avoid loops at all which should speed up the algorithm.
Concatenate a sentences with a special delimeter which is not contained by the sentences:

merged_sentences = ' * '.join(sentences)

Compile a single regex for all the words you need to rid from the sentences using | "or" regex statement:

regex = re.compile(r'\b({})\b'.format('|'.join(words)), re.I) # re.I is a case insensitive flag

Subscript the words with the compiled regex and split it by the special delimiter character back to separated sentences:

clean_sentences = re.sub(regex, "", merged_sentences).split(' * ')

Performance

"".join complexity is O(n). This is pretty intuitive but anyway there is a shortened quotation from a source:

for (i = 0; i < seqlen; i++) {
    [...]
    sz += PyUnicode_GET_LENGTH(item);

Therefore with join/split you have O(words) + 2*O(sentences) which is still linear complexity vs 2*O(N²) with the initial approach.

b.t.w. don't use multithreading. GIL will block each operation because your task is strictly CPU bound so GIL have no chance to be released but each thread will send ticks concurrently which cause extra effort and even lead operation to infinity.

0 讨论(0)

查看其它9个回答

Speed up millions of regex replacements in Python 3

Practical approach

Performance