Renaming columns in a Pandas dataframe with duplicate column names?

Question

I have a df X with columns with duplicate names:

In [77]: X_R
Out[77]: 
      dollars  dollars
   0   0.7085   0.5000

I want to rename it

Answer 1

MaxU's answer helped me with this same problem. In this answer, I add in a way to find those duplicated column headers.

First, we make a dictionary of the duplicated column names with values corresponding to the desired new column names. For this, the defaultdict subclass is required.

import pandas as pd
from collections import defaultdict

renamer = defaultdict()

We iterate over the duplicated column names to create a dictionary with keys being the duplicated column name and values being a list of new column names. I have chosen this list to be original name_0, original name_1, and so on.

for column_name in df.columns[df.columns.duplicated(keep=False)].tolist():
    if column_name not in renamer:
        renamer[column_name] = [column_name+'_0']
    else:
        renamer[column_name].append(column_name +'_'+str(len(renamer[column_name])))

print(renamer)
defaultdict(None, {'b': ['b_0', 'b_1', 'b_2', 'b_3'], 'c': ['c_0', 'c_1']})

Original dataframe:

print(df)
        a   b   b   b   b   c   c   d
Item 0  2   1   0   2   8   3   9   5
Item 1  3   2   7   3   5   4   6   2
Item 2  4   3   8   1   5   7   4   4
Item 3  5   5   3   6   0   5   2   5

Rename the duplicated columns by assigning the new names from our renamer defaultdict, leaving the unduplicated columns alone

df.rename(
    columns=lambda column_name: renamer[column_name].pop(0)
    if column_name in renamer 
    else column_name
)
        a   b_0 b_1 b_2 b_3 c_0 c_1 d
Item 0  2   1   0   2   8   3   9   5
Item 1  3   2   7   3   5   4   6   2
Item 2  4   3   8   1   5   7   4   4
Item 3  5   5   3   6   0   5   2   5

(As a sidenote, a couple of people have questioned why duplicated column names existed in the first place. For myself, I encountered duplicated column names when importing with the xlwings package (to deal with password-protected Excel files). You could also inadvertently create duplicate column names by using pd.concat.