Python scikit-learn KMeans is being killed (9) while computing silhouette score

Question

I'm currently working on an image dataset (250 000 images, so just as much as features vectors, everyone of them composed of 132 features) and trying to use the KMeans function provided by sklearn.

I run it on Mac OS X 10.10, Python 2.7 and sklearn 0.15.2, and after a while I only obtain a:

Killed: 9

Error when running these command lines:

nb_cls = int(raw_input("Number of clusters chosen :"))
clusterer = sklearn.cluster.KMeans(n_clusters=nb_cls)
clusters_labels = clusterer.fit_predict(X)
silhouette = sklearn.metrics.silhouette_score(X, clusters_labels)
print "n clusters =", nb_cls, "/ silhouette_score =", silhouette

Please note that whitout the calculation of the silhouette score, the code isn't killed

For smaller datasets (± 2 500 images) the same algorithm is efficient and there is no such Python error.

How could I avoid this Killed 9 error? Is this calculation too ambitious for my laptop?

Answer 1

It means your script was killed by the OS. In most cases it's because it was using too much memory. It seems likely in your case as your code works fine when you use only 2 500 images.

If it is a memory problem, you will have to either get more RAM (not possible on a mac ?), use another computer with more RAM or reduce the size of the dataset.