问题
I want to backfill each column based on time (1 day ,2 day) with different label. here is the code:
from datetime import datetime, timedelta
import pandas as pd
import numpy as np
import random
np.random.seed(11)
date_today = datetime.now()
ndays = 15
df = pd.DataFrame({'date': [date_today + timedelta(days=x) for x in range(ndays)],
'test': pd.Series(np.random.randn(ndays)), 'test2':pd.Series(np.random.randn(ndays))})
df = df.set_index('date')
df = df.mask(np.random.random(df.shape) < .7)
print(df) # this will be the dataset that I generate for this question
# my orginal data set have labels that is why I convert it to str
df['test']=df['test'].astype(str)
df['test2']=df['test2'].astype(str)
df.replace('nan', np.nan, inplace = True)
for I in df.dropna().index.values:
end=I
start=end-np.timedelta64(24,'h')
start2=end-np.timedelta64(48,'h')
df[(df.index >= start) & (df.index <= end)]=df[(df.index >= start) & (df.index <= end)].bfill()
my initial dataset will look like this:
test test2
date
2018-03-07 11:28:23.028856 NaN NaN
2018-03-08 11:28:23.028856 NaN NaN
2018-03-09 11:28:23.028856 -0.484565 1.574634
2018-03-10 11:28:23.028856 -2.653319 NaN
2018-03-11 11:28:23.028856 NaN NaN
2018-03-12 11:28:23.028856 NaN NaN
2018-03-13 11:28:23.028856 -0.536629 NaN
2018-03-14 11:28:23.028856 NaN 0.725752
2018-03-15 11:28:23.028856 NaN 1.549072
2018-03-16 11:28:23.028856 -1.065603 0.630080
2018-03-17 11:28:23.028856 NaN NaN
2018-03-18 11:28:23.028856 -0.475733 0.732271
2018-03-19 11:28:23.028856 NaN -0.642575
2018-03-20 11:28:23.028856 NaN -0.178093
2018-03-21 11:28:23.028856 NaN -0.573955
What I want to get is like this: I tried different things but I couldn't find a way to this with bfill, bfill don't get any parameters for value and fillna only get method or value.
test test2
date
2018-03-07 11:28:23.028856 -0.484565_2D 1.574634_2D
2018-03-08 11:28:23.028856 -0.484565_D 1.574634_D
2018-03-09 11:28:23.028856 -0.484565 1.574634
2018-03-10 11:28:23.028856 -2.653319 NaN
2018-03-11 11:28:23.028856 -0.536629_2D NaN
2018-03-12 11:28:23.028856 -0.536629_D 0.725752_2D
2018-03-13 11:28:23.028856 -0.536629 0.725752_D
2018-03-14 11:28:23.028856 -1.065603_2D 0.725752
2018-03-15 11:28:23.028856 -1.065603_D 1.549072
2018-03-16 11:28:23.028856 -1.065603 0.630080
2018-03-17 11:28:23.028856 -0.475733_D 0.732271_D
2018-03-18 11:28:23.028856 -0.475733 0.732271
2018-03-19 11:28:23.028856 NaN -0.642575
2018-03-20 11:28:23.028856 NaN -0.178093
2018-03-21 11:28:23.028856 NaN -0.573955
Update: The timestamp of my original dataset is non uniform so this code create similar time stamps:
date_today = datetime.now()
ndays = 15
df = pd.DataFrame({'date': [date_today + timedelta(days=(abs(np.random.randn(1))*2)[0]*x) for x in range(ndays)],
'test': pd.Series(np.random.randn(ndays)), 'test2':pd.Series(np.random.randn(ndays))})
df1=pd.DataFrame({'date': [date_today + timedelta(hours=x) for x in range(ndays)],
'test': pd.Series(np.random.randn(ndays)), 'test2':pd.Series(np.random.randn(ndays))})
df2=pd.DataFrame({'date': [date_today + timedelta(days=x)-timedelta(seconds=100*x) for x in range(ndays)],
'test': pd.Series(np.random.randn(ndays)), 'test2':pd.Series(np.random.randn(ndays))})
df=df.append(df1)
df=df.append(df2)
df = df.set_index('date')
df = df.mask(np.random.random(df.shape) < .7)
print(df) # this will be the dataset that I generate for this question
# my orginal data set have labels that is why I convert it to str
df['test']=df['test'].astype(str)
df['test2']=df['test2'].astype(str)
df.replace('nan', np.nan, inplace = True)
I really appreciate if anyone can help me with this
Thanks in advance.
回答1:
create a filled dataframe using fillna with method backfill & limit 2
filled = df.fillna(method='bfill', limit=2)
# filled outputs:
test test2
date
2018-03-07 16:12:25.944362 -0.484565132221 1.5746340731
2018-03-08 16:12:25.944362 -0.484565132221 1.5746340731
2018-03-09 16:12:25.944362 -0.484565132221 1.5746340731
2018-03-10 16:12:25.944362 -2.65331855926 NaN
2018-03-11 16:12:25.944362 -0.536629362235 NaN
2018-03-12 16:12:25.944362 -0.536629362235 0.725752224799
2018-03-13 16:12:25.944362 -0.536629362235 0.725752224799
2018-03-14 16:12:25.944362 -1.06560298045 0.725752224799
2018-03-15 16:12:25.944362 -1.06560298045 1.54907163337
2018-03-16 16:12:25.944362 -1.06560298045 0.630079822493
2018-03-17 16:12:25.944362 -0.475733492683 0.732271353885
2018-03-18 16:12:25.944362 -0.475733492683 0.732271353885
2018-03-19 16:12:25.944362 NaN -0.642575392433
2018-03-20 16:12:25.944362 NaN -0.178093175312
2018-03-21 16:12:25.944362 NaN -0.57395455941
create a bool data frame to indicate whether a cell was filled
is_filled = df.isnull() & filled.notnull()
# is_filled outputs:
test test2
date
2018-03-07 16:12:25.944362 True True
2018-03-08 16:12:25.944362 True True
2018-03-09 16:12:25.944362 False False
2018-03-10 16:12:25.944362 False False
2018-03-11 16:12:25.944362 True False
2018-03-12 16:12:25.944362 True True
2018-03-13 16:12:25.944362 False True
2018-03-14 16:12:25.944362 True False
2018-03-15 16:12:25.944362 True False
2018-03-16 16:12:25.944362 False False
2018-03-17 16:12:25.944362 True True
2018-03-18 16:12:25.944362 False False
2018-03-19 16:12:25.944362 False False
2018-03-20 16:12:25.944362 False False
2018-03-21 16:12:25.944362 False False
create masks to indicate filled values that need suffix _1D or _2D
one_d = (is_filled & ~is_filled.shift(-1).fillna(False)).applymap(lambda x: '_1D' if x else '')
two_d = (is_filled & is_filled.shift(-1).fillna(False)).applymap(lambda x: '_2D' if x else '')
suffix = pd.concat([one_d, two_d]).groupby('date').agg('max')
# suffix outputs:
test test2
date
2018-03-07 16:12:25.944362 _2D _2D
2018-03-08 16:12:25.944362 _1D _1D
2018-03-09 16:12:25.944362
2018-03-10 16:12:25.944362
2018-03-11 16:12:25.944362 _2D
2018-03-12 16:12:25.944362 _1D _2D
2018-03-13 16:12:25.944362 _1D
2018-03-14 16:12:25.944362 _2D
2018-03-15 16:12:25.944362 _1D
2018-03-16 16:12:25.944362
2018-03-17 16:12:25.944362 _1D _1D
2018-03-18 16:12:25.944362
2018-03-19 16:12:25.944362
2018-03-20 16:12:25.944362
2018-03-21 16:12:25.944362
join the suffix dataframe to the filled dataframe convert the float to strings & append the appropriate suffixes
final = filled.join(suffix, rsuffix='_x')
final.apply(lambda x: '{}{}'.format(x.test, x.test_x), axis=1)
# outputs:
date
2018-03-07 16:12:25.944362 -0.484565132221_2D
2018-03-08 16:12:25.944362 -0.484565132221_1D
2018-03-09 16:12:25.944362 -0.484565132221
2018-03-10 16:12:25.944362 -2.65331855926
2018-03-11 16:12:25.944362 -0.536629362235_2D
2018-03-12 16:12:25.944362 -0.536629362235_1D
2018-03-13 16:12:25.944362 -0.536629362235
2018-03-14 16:12:25.944362 -1.06560298045_2D
2018-03-15 16:12:25.944362 -1.06560298045_1D
2018-03-16 16:12:25.944362 -1.06560298045
2018-03-17 16:12:25.944362 -0.475733492683_1D
2018-03-18 16:12:25.944362 -0.475733492683
2018-03-19 16:12:25.944362 nan
2018-03-20 16:12:25.944362 nan
2018-03-21 16:12:25.944362 nan
similarly you can generate the filled & suffixed series for test2. However, I would recommend that you keep test & test2 as numeric types and store the fill & lag information in separate columns (here the column suffix stores that info in the dataframe final).
来源：https://stackoverflow.com/questions/49158268/back-fill-missing-data-with-a-label-for-a-window-of-a-time