multi-column factorize in pandas

Question

The pandas factorize function assigns each unique value in a series to a sequential, 0-based index, and calculates which index each series entry belongs to.

I'd like to accomplish the equivalent of pandas.factorize on multiple columns:

import pandas as pd
df = pd.DataFrame({'x': [1, 1, 2, 2, 1, 1], 'y':[1, 2, 2, 2, 2, 1]})
pd.factorize(df)[0] # would like [0, 1, 2, 2, 1, 0]

That is, I want to determine each unique tuple of values in several columns of a data frame, assign a sequential index to each, and compute which index each row in the data frame belongs to.

Factorize only works on single columns. Is there a multi-column equivalent function in pandas?

Answer 1

You need to create a ndarray of tuple first, pandas.lib.fast_zip can do this very fast in cython loop.

import pandas as pd
df = pd.DataFrame({'x': [1, 1, 2, 2, 1, 1], 'y':[1, 2, 2, 2, 2, 1]})
print pd.factorize(pd.lib.fast_zip([df.x, df.y]))[0]

the output is:

[0 1 2 2 1 0]

Answer 2

I am not sure if this is an efficient solution. There might be better solutions for this.

arr=[] #this will hold the unique items of the dataframe
for i in df.index:
   if list(df.iloc[i]) not in arr:
      arr.append(list(df.iloc[i]))

so printing the arr would give you

>>>print arr
[[1,1],[1,2],[2,2]]

to hold the indices, i would declare an ind array

ind=[]
for i in df.index:
   ind.append(arr.index(list(df.iloc[i])))

printing ind would give

 >>>print ind
 [0,1,2,2,1,0]

Answer 3

You can use drop_duplicates to drop those duplicated rows

In [23]: df.drop_duplicates()
Out[23]: 
      x  y
   0  1  1
   1  1  2
   2  2  2

EDIT

To achieve your goal, you can join your original df to the drop_duplicated one:

In [46]: df.join(df.drop_duplicates().reset_index().set_index(['x', 'y']), on=['x', 'y'])
Out[46]: 
   x  y  index
0  1  1      0
1  1  2      1
2  2  2      2
3  2  2      2
4  1  2      1
5  1  1      0

Answer 4

df = pd.DataFrame({'x': [1, 1, 2, 2, 1, 1], 'y':[1, 2, 2, 2, 2, 1]})
tuples = df[['x', 'y']].apply(tuple, axis=1)
df['newID'] = pd.factorize( tuples )[0]