How to extract non NA values in a list or dict from a pandas dataframe

问题

I have a df like this,

df,

    AAA BBB CCC
0   4   10  100
1   5   20  50
2   6   30  -30
3   7   40  -50

df_mask = pd.DataFrame({'AAA' : [True] * 4, 'BBB' : [False] * 4,'CCC' : [True,False] * 2}) and df.where(df_mask) is

    AAA BBB CCC
0   4   NaN 100.0
1   5   NaN NaN
2   6   NaN -30.0
3   7   NaN NaN

I am trying to extract the non null values like this.

I tried, df[df.where(df_mask).notnull()].to_dict() but it gives all the values

My expected output is,

{'AAA': {0: 4, 1: 5, 2: 6, 3: 7},
 'CCC': {0: 100.0, 2: -30.0}}

回答1:

Let's use agg here:

v = df.where(df_mask).agg(lambda x: x.dropna().to_dict())

On older versions, apply does the same thing (albeit a bit slower).

v = df.where(df_mask).apply(lambda x: x.dropna().to_dict())

And now, filter out rows with empty dictionaries for the final step:

res = v[v.str.len() > 0].to_dict()

print(res)
{'AAA': {0: 4.0, 1: 5.0, 2: 6.0, 3: 7.0}, 'CCC': {0: 100.0, 2: -30.0}}

Another apply-free option is a dict-comprehension:

v = df.where(df_mask)  
res = {k : v[k].dropna().to_dict() for k in df} 

print(res)
{'AAA': {0: 4, 1: 5, 2: 6, 3: 7}, 'BBB': {}, 'CCC': {0: 100.0, 2: -30.0}}

Note that this (slightly) simpler solution retains keys with empty values.

回答2:

You can iterate df's columns and apply dropna Serieswise

{col: df[col].dropna().values for col in df}

Which yields

{'AAA': array([4, 5, 6, 7]),
 'BBB': array([], dtype=float64),
 'CCC': array([ 100.,  -30.])}

You can filter out empty arrays such as 'BBB' with

{key: val for key, val in ddict.items() if val}

来源：https://stackoverflow.com/questions/50903622/how-to-extract-non-na-values-in-a-list-or-dict-from-a-pandas-dataframe

标签

python

pandas

dataframe

data-analysis