Fastest Text search method in a large text file

前端 未结 4 1427
南旧
南旧 2021-01-19 04:35

I am doing a text search in a rather big txt file (100k lines, 7mo) Text is not that big but I need a lot of searches. I want to look for a target string and return the line

4条回答
  •  挽巷
    挽巷 (楼主)
    2021-01-19 05:08

    1. Load the whole text in RAM at once. Don't read line by line.
    2. Search for the pattern in the blob. If you find it, use text.count('\n',0,pos) to get the line number.
    3. If you don't need the line number, look for the previous and next EOL to cut the line out of the text.

    The loop in Python is slow. String searching is very fast. If you need to look for several strings, use regular expressions.

    If that's not fast enough, use an external program like grep.

提交回复
热议问题