Computing the first non-missing value from each column in a DataFrame [duplicate]

Answer 1

The built in function DataFrame.groupby().column.first() returns the first non null value in the column, while last() returns the last.

http://pandas.pydata.org/pandas-docs/stable/generated/pandas.core.groupby.GroupBy.first.html

If you don't wish to get the first value for each group, you can add a dummy column of 1s. Then get the first non null value using the groupby & first functions.

from Pandas import DataFrame

df = DataFrame({'a':[None,1,None],'b':[None,2,None]})
df['dummy'] = 1
df.groupby('dummy').first()
df.groupby('dummy').last()

Answer 2

You should use the apply function which applies a function on either each column (default) or each row efficiently:

>>> first_valid_indices = df.apply(lambda series: series.first_valid_index())
>>> first_valid_indices
1125400   2013-05-22 00:00:00
5430095   2013-05-28 00:00:00
1095751   2013-05-22 00:00:00

first_valid_indiceswill then be a series containing the first_valid_index for each column.

You could also define the lambda function as a normal function outside:

def first_valid_index(series):
    return series.first_valid_index()

and then call apply like this:

df.apply(first_valid_index)

Answer 3

By compute I assume you mean access?

The simplest way to do this is with the pd.Series.first_valid_index() method probably inside a dict comprehension:

values = {col : DF.loc[DF[col].first_valid_index(), col] for col in DF.columns}
values

Just to be clear, each column in a pandas DataFrame is a Series. So the above is the same as doing:

values = {}
for column in DF.columns:
    First_Non_Null_Index = DF[column].first_valid_index()
    values[column] = DF.loc[First_Non_Null_Index, column]

So the operation in my one line solution is on a per column basis. I.e. it is not going to create the type of error you seem to be suggesting in the edit you made to the question. Let me know if it does not work as expected.