How can I use melt() to reshape a pandas DataFrame to a list, creating an index from a crosstab column and creating a new variable in its place?

Question

I have a matrix of data 29523 rows x 503 cols of which 3 cols are indices (below is a subset for example).

IDX1|  IDX2  | IDX3 | 1983 Q4   |  X  | Y |  Z  |1         


        

Answer 1

Given

In [189]: df
Out[189]: 
  IDX1 IDX2 IDX3  1983 Q4  X  Y    Z  1984 Q1 X.1 Y.1  Z.1
0    A   A1    Q       10  A  F  NaN      110   A   F  NaN
1    A   A2    Q       20  B  C   40      120   B   C  240
2    A   A3    Q       30  A  F  NaN      130   A   F  NaN
3    A   A4    Q       40  B  C   80      140   B   C  280
4    A   A5    Q       50  A  F  NaN      150   A   F  NaN
5    A   A6    Q       60  B  F  120      160   B   F  320

Let us first set ['IDX1', 'IDX2', 'IDX3'] as the index.

df = df.set_index(['IDX1', 'IDX2', 'IDX3'])

The other columns have a periodic quality to them; we want to handle every 4 columns as a group. This idea of "handling as a group" leads naturally to assigning a new index level to the column index; some value which is the same for every 4 columns. This would be ideal:

               1983 Q4            1984 Q1           
                     W  X  Y    Z       W  X  Y    Z
IDX1 IDX2 IDX3                                      
A    A1   Q         10  A  F  NaN     110  A  F  NaN
     A2   Q         20  B  C  240     120  B  C  240
     A3   Q         30  A  F  NaN     130  A  F  NaN
     A4   Q         40  B  C  280     140  B  C  280
     A5   Q         50  A  F  NaN     150  A  F  NaN
     A6   Q         60  B  F  320     160  B  F  320

We can achieve this by building a MultiIndex and assigning it to df.columns:

columns = [col for col in df.columns if col[0] not in set(list('XYZ'))]
df.columns = pd.MultiIndex.from_product([columns, list('WXYZ')])

Now the desired long-format DataFrame can be obtained by calling df.stack to move the column levels into the row index:

df.columns.names = ['IDX4', 'ValueType']
series = df.stack(['IDX4', 'ValueType'], dropna=False)

---

Note also that when mangle_dupe_cols=False, the duplicate columns, X, Y, Z, get overwritten. So you lose data with mangle_dupe_cols=False. For example, when you use mangle_dupe_cols=False the last row's Z value gets assigns to every Z column regardless of the period.

So we must use mangle_dupe_cols=True, (or just leave it out since that is the default) and adjust the code accordingly. That, fortunately, is not hard to do since we are reassigning df.columns to a custom-build MultiIndex anyway.

---

Putting it all together:

import numpy as np
import pandas as pd
df = pd.read_table('data', sep=r'\s*[|]\s*')
df = df.set_index(['IDX1', 'IDX2', 'IDX3'])
columns = [col for col in df.columns if col[0] not in set(list('XYZ'))]
df.columns = pd.MultiIndex.from_product([columns, list('WXYZ')])
df.columns.names = ['IDX4', 'ValueType']
series = df.stack(['IDX4', 'ValueType'], dropna=False)
print(series.head())

yields

IDX1  IDX2  IDX3  IDX4     ValueType
A     A1    Q     1983 Q4  W             10
                           X              A
                           Y              F
                           Z            NaN
                  1984 Q1  W            110
dtype: object

Note that since we've removed all the column levels, the result is a Series. If you want a DataFrame with 6 columns, then we should follow it up with:

series.name = 'Value'
df = series.reset_index()
print(df.head())

which yields

  IDX1 IDX2 IDX3     IDX4 ValueType Value
0    A   A1    Q  1983 Q4         W    10
1    A   A1    Q  1983 Q4         X     A
2    A   A1    Q  1983 Q4         Y     F
3    A   A1    Q  1983 Q4         Z   NaN
4    A   A1    Q  1984 Q1         W   110
...