What is the best way to find string in txt file by using python?

Question

there must be various ways to find string in txt file by using python, but what is the best way? ( for speed, for resources .. )

My first idea was as below.

<         


        

Answer 1

If you have to use text files, you could try reading the entire file into memory instead of searching it line by line to speed things up. (If you read all of the file into memory, you don't need the loop any more)

Instead of writing a python script to do the search, you could try using grep or find.

You should consider putting your data into a database and querying it to find matches. This approach should be a lot more resource efficient and should be faster since databases can make use of indexes and they don't necessarily have to read the entire dataset into memory to find matches. If your application is simple enough, you might be able to use sqlite.

Answer 2

If your task is "I have a static text file and there are dynamic queries asking whether that text file contains a particular IP address" then just read the file into memory once, and then handle the queries as they come in.

with open('/home/socfw/src/edl/outbound_monthly.txt') as ipaddresses:
    ip = set(line.strip() for line in ipaddresses)

while True:  # notice how a boolean is the idiomatic way to express an endless loop
    queryip = somehow receive a query from a client()
    if queryip in ip:
        tell client yes()
    else:
        tell client no()

The pseudocode in the while loop might be replaced with a Flask route or something if your clients are web browsers or consumers of a web API; but this general pattern will apply to pretty much any kind of server.

There isn't any obvious way to make the reading of the text into memory more efficient - good for you if you manage to achieve 100% CPU because typically this sort of task is I/O bound, not CPU bound.

If the text file is not static, perhaps you can reread it into memory periodically, or just import it to a database whenever it is updated and have the clients query that instead.

Answer 3

You could try using a for loop this way:

for line in file:
    if inputIP in line:
        print(ok)
        break