Feature selection for multilabel classification (scikit-learn)

Question

I'm trying to do a feature selection by chi-square method in scikit-learn (sklearn.feature_selection.SelectKBest). When I'm trying to apply this to a multilabel problem, I get this warning:

UserWarning: Duplicate scores. Result may depend on feature ordering.There are probably duplicate features, or you used a classification score for a regression task. warn("Duplicate scores. Result may depend on feature ordering."

Why is it appearning and how to properly apply feature selection is this case?

Answer 1

The code warns you that arbitrary tie-breaking may need to be performed because some features have exactly the same score.

That said, feature selection does not actually work for multilabel out of the box; the best you can currently do is tie feature selection and a classifier together in a pipeline, then feed that to a multilabel meta-estimator. Example (untested):

clf = Pipeline([('chi2', SelectKBest(chi2, k=1000)),
                ('svm', LinearSVC())])
multi_clf = OneVsRestClassifier(clf)

(This warning is, I believe, issued even when the tied features aren't actually the k'th and (k+1)'th, I think. It can usually be ignored safely.)

Answer 2

I know the topic is a bit old but the following is working for me:

clf = Pipeline([('chi2', SelectKBest(chi2, k=1000)),
            ('lasso', OneVsRestClassifier(LogisticRegression()))])