Python pandas - pd.melt a dataframe with datetime index results in NaN

Question

I have the following dataframe (sim_2005):

Date         ELEM1 ELEM2 ... ELEM1133
2005-01-01   0.021 2.455 ... 345.2
2005-01-02   0.321 2.331 ... 355.1
...            


        

Answer 1

Assuming Date is the index to your DataFrame, you can get a date column in your melted DataFrame as follows:

sim_2005_melted['Date'] = pd.concat([sim_2005.reset_index().Date 
                                     for _ in range(sim_2005.shape[1])], 
                                    ignore_index=True).values

Answer 2

Here is one way to use .stack() to solve your question.

import pandas as pd
import numpy as np

# try to simulate your data
columns = ['ELEM' + str(x) for x in np.arange(1, 1134, 1)]
sim_2005 = pd.DataFrame(np.random.randn(365, 1133), index=pd.date_range('2005-01-01', periods=365, freq='D'), columns=columns)

processed_sim_2005 = sim_2005.stack().reset_index()
processed_sim_2005.columns = ['Date', 'ELEM', 'Q_sim']

Out[82]: 
             Date      ELEM   Q_sim
0      2005-01-01     ELEM1  0.6221
1      2005-01-01     ELEM2  0.1862
2      2005-01-01     ELEM3 -1.0736
3      2005-01-01     ELEM4 -0.9756
4      2005-01-01     ELEM5  0.8397
...           ...       ...     ...
413540 2005-12-31  ELEM1129  0.0345
413541 2005-12-31  ELEM1130  0.5522
413542 2005-12-31  ELEM1131 -0.6900
413543 2005-12-31  ELEM1132 -0.2269
413544 2005-12-31  ELEM1133  0.1243

[413545 rows x 3 columns]

Answer 3

A possibly simpler solution still using .melt() is to pull your date index out into a column with .reset_index() first:

sim_2005_melted = pd.melt(sim_2005.reset_index(), id_vars=sim_2005.index.name, value_vars=list(sim_2005.columns.values), var_name='ELEM', value_name='Q_sim')

You get the same result with .stack() but this way is a bit more flexible if you want all the extra melty goodness.