Extracting columns containing a certain name

Question

I\'m trying to use it to manipulate data in large txt-files.

I have a txt-file with more than 2000 columns, and about a third of these have a title which contains th

Answer 1

This could be done for instance with Pandas,

import pandas as pd

df = pd.read_csv('path_to_file.txt', sep='\s+')
print(df.columns)  # check that the  columns are parsed correctly 
selected_columns = [col for col in df.columns if "net" in col]
df_filtered = df[selected_columns]
df_filtered.to_csv('new_file.txt')

Of course, since we don't have the structure of your text file, you would have to adapt the arguments of read_csv to make this work in your case (see the the corresponding documentation).

This will load all the file in memory and then filter out the unnecessary columns. If your file is so large that it cannot be loaded in RAM at once, there is a way to load only specific columns with the usecols argument.