Efficient way of reading large txt file in python

Question

I\'m trying to open a txt file with 4605227 rows (305 MB)

The way I have done this before is:

data = np.loadtxt(\'file.txt\', delimiter=\'\\t\', dtype=st         


        

Answer 1

You read it directly in as a Pandas DataFrame. eg

import pandas as pd
pd.read_csv(path)

If you want to read faster, you can use modin:

import modin.pandas as pd
pd.read_csv(path)

https://github.com/modin-project/modin