What is the best way to find string in txt file by using python?

问题

there must be various ways to find string in txt file by using python, but what is the best way? ( for speed, for resources .. )

My first idea was as below.

file = open('/home/socfw/src/edl/outbound_monthly.txt')

inputIP = '127.0.0.1'

while (1):
    line = file.readline()
    if inputIP in line:
        print("ok")
        break

But, it's too slow to use web service properly (it is actually backend logic of my web service) txt file looks like as below

test.txt ( IPV4 addresses are in here, and they counts almost 60k)

x.x.x.x
x.x.x.x
.
.
.
.

My source code causes 100 percent CPU for several mins, so I want to find another way. Is there any good solution for me? thanks in advance.

Thank you for answering me. I changed my sources as below.

with open('/home/socfw/src/edl/outbound_monthly.txt') as outMonIPs:
    ip = set(line.strip() for line in outMonIPs)

inputIP = '111.90.150.249'
#while True:
if inputIP in ip:
    print("ok")
#        break
else:
    print("no")
#        break

I have one more question, shoud I use loop for this work? I think loop is no more need when I save whole file in memory.

回答1:

If you have to use text files, you could try reading the entire file into memory instead of searching it line by line to speed things up. (If you read all of the file into memory, you don't need the loop any more)

Instead of writing a python script to do the search, you could try using grep or find.

You should consider putting your data into a database and querying it to find matches. This approach should be a lot more resource efficient and should be faster since databases can make use of indexes and they don't necessarily have to read the entire dataset into memory to find matches. If your application is simple enough, you might be able to use sqlite.

回答2:

If your task is "I have a static text file and there are dynamic queries asking whether that text file contains a particular IP address" then just read the file into memory once, and then handle the queries as they come in.

with open('/home/socfw/src/edl/outbound_monthly.txt') as ipaddresses:
    ip = set(line.strip() for line in ipaddresses)

while True:  # notice how a boolean is the idiomatic way to express an endless loop
    queryip = somehow receive a query from a client()
    if queryip in ip:
        tell client yes()
    else:
        tell client no()

The pseudocode in the while loop might be replaced with a Flask route or something if your clients are web browsers or consumers of a web API; but this general pattern will apply to pretty much any kind of server.

There isn't any obvious way to make the reading of the text into memory more efficient - good for you if you manage to achieve 100% CPU because typically this sort of task is I/O bound, not CPU bound.

If the text file is not static, perhaps you can reread it into memory periodically, or just import it to a database whenever it is updated and have the clients query that instead.

回答3:

You could try using a for loop this way:

for line in file:
    if inputIP in line:
        print(ok)
        break

来源：https://stackoverflow.com/questions/59712678/what-is-the-best-way-to-find-string-in-txt-file-by-using-python

标签

python

while-loop

ipv4