f1_score metric in lightgbm

Question

I want to train a lgb model with custom metric : f1_score with weighted average.

I went through the advanced examples of lightgbm over here

Answer 1

Regarding Toby's answers:

def lgb_f1_score(y_hat, data):
    y_true = data.get_label()
    y_hat = np.round(y_hat) # scikits f1 doesn't like probabilities
    return 'f1', f1_score(y_true, y_hat), True

I suggest change the y_hat part to this:

y_hat = np.where(y_hat < 0.5, 0, 1)  

Reason: I used the y_hat = np.round(y_hat) and fonud out that during training the lightgbm model will sometimes(very unlikely but still a change) regard our y prediction to multiclass instead of binary.

My speculation: Sometimes the y prediction will be small or higher enough to be round to negative value or 2?I'm not sure,but when i changed the code using np.where, the bug is gone.

Cost me a morning to figure this bug,although I'm not really sure if the np.where solution is good.

Answer 2

The docs are a bit confusing. When describing the signature of the function that you pass to feval, they call its parameters preds and train_data, which is a bit misleading.

But the following seems to work:

from sklearn.metrics import f1_score

def lgb_f1_score(y_hat, data):
    y_true = data.get_label()
    y_hat = np.round(y_hat) # scikits f1 doesn't like probabilities
    return 'f1', f1_score(y_true, y_hat), True

evals_result = {}

clf = lgb.train(param, train_data, valid_sets=[val_data, train_data], valid_names=['val', 'train'], feval=lgb_f1_score, evals_result=evals_result)

lgb.plot_metric(evals_result, metric='f1')

To use more than one custom metric, define one overall custom metrics function just like above, in which you calculate all metrics and return a list of tuples.

Edit: Fixed code, of course with F1 bigger is better should be set to True.