I have fed the following CSV file into iPython Notebook:
public = pd.read_csv(\"categories.csv\")
public
I\'ve also imported pandas as pd, nump
You can use the pandas.DataFrame.apply
method along with a lambda
expression to solve this. In your example you could use
df[['parks', 'playgrounds', 'sports']].apply(lambda x: x.astype('category'))
I don't know of a way to execute this inplace, so typically I'll end up with something like this:
df[df.select_dtypes(['object']).columns] = df.select_dtypes(['object']).apply(lambda x: x.astype('category'))
Obviously you can replace .select_dtypes
with explicit column names if you don't want to select all of a certain datatype (although in your example it seems like you wanted all object
types).
In my case, I had big Dataframe with many objects that I would like to convert it to category.
Therefore, what I did is I selected the object columns and filled anything that is NA to missing and then saved it in the original Dataframe as in
# Convert Object Columns to Categories
obj_df =df.select_dtypes(include=['object']).copy()
obj_df=obj_df.fillna('Missing')
for col in obj_df:
obj_df[col] = obj_df[col].astype('category')
df[obj_df.columns]=obj_df[obj_df.columns]
df.head()
I hope this might be a helpful resource for later reference
As of pandas 0.19.0, What's New describes that read_csv
supports parsing Categorical
columns directly.
This answer applies only if you're starting from read_csv
otherwise, I think unutbu's answer is still best.
Example on 10,000 records:
import pandas as pd
import numpy as np
# Generate random data, four category-like columns, two int columns
N=10000
categories = pd.DataFrame({
'parks' : np.random.choice(['strongly agree','agree', 'disagree'], size=N),
'playgrounds' : np.random.choice(['strongly agree','agree', 'disagree'], size=N),
'sports' : np.random.choice(['important', 'very important', 'not important'], size=N),
'roading' : np.random.choice(['important', 'very important', 'not important'], size=N),
'resident' : np.random.choice([1, 2, 3], size=N),
'children' : np.random.choice([0, 1, 2, 3], size=N)
})
categories.to_csv('categories_large.csv', index=False)
pd.read_csv('categories_large.csv').dtypes # inspect default dtypes
children int64
parks object
playgrounds object
resident int64
roading object
sports object
dtype: object
For mixed dtypes
parsing as Categorical
can be implemented by passing a dictionary dtype={'colname' : 'category', ...}
in read_csv
.
pd.read_csv('categories_large.csv', dtype={'parks': 'category',
'playgrounds': 'category',
'sports': 'category',
'roading': 'category'}).dtypes
children int64
parks category
playgrounds category
resident int64
roading category
sports category
dtype: object
A slight speed-up (local jupyter notebook), as mentioned in the release notes.
# unutbu's answer
%%timeit
public = pd.read_csv('categories_large.csv')
for col in ['parks', 'playgrounds', 'sports', 'roading']:
public[col] = public[col].astype('category')
10 loops, best of 3: 20.1 ms per loop
# parsed during read_csv
%%timeit
category_cols = {item: 'category' for item in ['parks', 'playgrounds', 'sports', 'roading']}
public = pd.read_csv('categories_large.csv', dtype=category_cols)
100 loops, best of 3: 14.3 ms per loop
I found that using a for loop works well.
for col in ['col_variable_name_1', 'col_variable_name_2', ect..]:
dataframe_name[col] = dataframe_name[col].astype(float)
Sometimes, you just have to use a for-loop:
for col in ['parks', 'playgrounds', 'sports', 'roading']:
public[col] = public[col].astype('category')
To make things easier. No apply. No map. No loop.
cols=data.select_dtypes(exclude='int').columns.to_list()
data[cols]=data[cols].astype('category')