Convert Pandas dataframe to Dask dataframe

。_饼干妹妹 提交于 2020-02-26 07:13:54

问题


Suppose I have pandas dataframe as:

df=pd.DataFrame({'a':[1,2,3],'b':[4,5,6]})

When I convert it into dask dataframe what should name and divisions parameter consist of:

from dask import dataframe as dd 
sd=dd.DataFrame(df.to_dict(),divisions=1,meta=pd.DataFrame(columns=df.columns,index=df.index))

TypeError: init() missing 1 required positional argument: 'name'

Edit : Suppose I create a pandas dataframe like:

pd.DataFrame({'a':[1,2,3],'b':[4,5,6]})

Similarly how to create dask dataframe as it needs three additional arguments as name,divisions and meta.

sd=dd.Dataframe({'a':[1,2,3],'b':[4,5,6]},name=,meta=,divisions=)

Thank you for your reply.


回答1:


I think you can use dask.dataframe.from_pandas:

from dask import dataframe as dd 
sd = dd.from_pandas(df, npartitions=3)
print (sd)
dd.DataFrame<from_pa..., npartitions=2, divisions=(0, 1, 2)>

EDIT:

I find solution:

import pandas as pd
import dask.dataframe as dd
from dask.dataframe.utils import make_meta

df=pd.DataFrame({'a':[1,2,3],'b':[4,5,6]})

dsk = {('x', 0): df}

meta = make_meta({'a': 'i8', 'b': 'i8'}, index=pd.Index([], 'i8'))
d = dd.DataFrame(dsk, name='x', meta=meta, divisions=[0, 1, 2])
print (d)
dd.DataFrame<x, npartitions=2, divisions=(0, 1, 2)>


来源:https://stackoverflow.com/questions/39721800/convert-pandas-dataframe-to-dask-dataframe

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!