The fastest way to parse dates in Python when reading .csv file?

Question

I have a .csv file that has 2 separate columns for \'Date\' and \' Time\'. I read the file like this:

data1 = pd.read_csv(\'filename.cs         


        

Answer 1

Here, In your case 'Time' is in AM/PM format which take more time to parse.

You can add format to increase speed of to_datetime() method.

data0=pd.read_csv('filename.csv')

# %Y - year including the century
# %m - month (01 to 12)
# %d - day of the month (01 to 31)
data0['Date']=pd.to_datetime(data0['Date'], format="%Y/%m/%d")

# %I - hour, using a -hour clock (01 to 12)
# %M - minute
# %p - either am or pm according to the given time value
# data0['Time']=pd.to_datetime(data0['Time'], format="%I:%M %p") -> around 1 sec
data0['Time']=pd.datetools.to_time(data0['Time'], format="%I:%M %p")

For more methods info : Pandas Tools

For more format options check - datetime format directives.

For 500K rows it improved speed from around 60 seconds -> 0.01 seconds in my system.

You can also use :

# Combine date & time directly from string format
pd.Timestamp(data0['Date'][0] + " " + data0['Time'][0])