Dealing with sparse categories in Pandas - replace everything not in top categories with “Other”

问题

I often come across the following common problem when cleaning the data there are some more common categories (let's say top 10 movie genres) and lots and lots of others which are sparse. Usual practice here would be to combine sparse genres into "Other" for example.

Easily done when there are not many sparse categories:

# Join bungalows as they are sparse classes into 1
df.property_type.replace(['Terraced bungalow','Detached bungalow', 'Semi-detached bungalow'], 'Bungalow', inplace=True)

but if for example I have a movies dataset with majority of the movies produced by let's say 8 big studios and I would like to combine everything else under "other" studio, it makes sense to get top 8 studios:

top_8_list = []
top_8 = df.studio.value_counts().head(8)
for key, value in top_8.iteritems():
    top_8_list.append(key)

top_8_list
top_8_list
['Universal Pictures',
 'Warner Bros.',
 'Paramount Pictures',
 'Twentieth Century Fox Film Corporation',
 'New Line Cinema',
 'Columbia Pictures Corporation',
 'Touchstone Pictures',
 'Columbia Pictures']

and then do something like

replace studio where studio is not in the top 8 list with "other"

so the question, if anyone knows any elegant solution in pandas for this? This is very common data cleaning task

回答1:

You could convert the column to type Categorical which has added memory benefits:

top_cats = df.studio.value_counts().head(8).index.tolist() + ['other']
df['studio'] = pd.Categorical(df['studio'], categories=top_cats).fillna('other')

回答2:

You can use pd.DataFrame.loc with Boolean indexing:

df.loc[~df['studio'].isin(top_8_list), 'studio'] = 'Other'

Note there's no need to construct your list of top 8 studios via a manual for loop:

top_8_list = df['studio'].value_counts().index[:8]

来源：https://stackoverflow.com/questions/52663432/dealing-with-sparse-categories-in-pandas-replace-everything-not-in-top-categor