Google Datalab) How to read multiple csv files existing in Google Cloud Storage by pandas read_csv() to?

Question

I found the solution for reading a "single" csv file in a Datalab : How can i load my csv from google dataLab to a pandas data frame?

But I wonder how I could read "multiple" csv files in Datalab.

What I tried is like this :

variable_list = ['IBM', 'SPY']
for variable in variable_list:
  file_path = "gs://chois-trader-bucket/data/" + variable + ".csv"
  %%storage read --object file_path --variable variable

But this one failed because python variable is not compatible with magic command.

How can I deal with multiple csv files effectively?

Answer 1

You can use variables enclosed in braces.

e.g) %storage read --object {file_path} --variable variable

Answer 2

Alternatively, you can do a union of all the csv files in a folder using a one liner in bash as such (assuming no headers in the csv files) :

cat *.csv > unioned_file_name.csv

Then import that one file into pandas.