How to drop NaN elements in a groupby on a pandas dataframe?

Question

Suppose I have this dataframe :

my_df = pd.DataFrame({\'A\':[np.nan,np.nan,\'gate\',\'ball\'],\'B\':[\'car\',np.nan,np.nan,np.nan],\'C\':[np.nan,\'edge\',np.         


        

Answer 1

Use custom function with dropna, but for empty values add NaNs:

print (my_df.groupby("D").agg(lambda x: np.nan if x.isnull().all() else x.dropna()))
        A    B     C
D                   
id1  gate  car  edge
id2  ball  NaN   NaN

Similar solution with custom function:

def f(x):
    y = x.dropna()
    return np.nan if y.empty else y

print (my_df.groupby("D").agg(f))
        A    B     C
D                   
id1  gate  car  edge
id2  ball  NaN   NaN

Answer 2

Your approach is much better I guess but add a replace at the end

my_df.fillna("").groupby("D").max().replace('',np.nan) 

You can also do :

def get_notnull(x):
    if x.notnull().any():
        return  x[x.notnull()]
    else:
        return np.nan

my_df.groupby('D').agg(get_notnull)

Output :

     A    B     C
D                   
id1  gate  car  edge
id2  ball  NaN   NaN