hierarchical-clustering

I am working on a clustering problem and it has to be scalable for a lot of data. I would like to try hierarchical clustering in Spark and compare my results with other methods. I have done some research on the web about using hierarchical clustering with Spark but haven't found any promising information. If anyone has some insight about it, I would be very grateful. Thank you. Gabe Church The Bisecting Kmeans Approach Seems to do a decent job, and runs quite fast in terms of performance. Here is a sample code I wrote for utilizing the Bisecting-Kmeans algorithm in Spark (scala) to get cluster

I have lot of data points which are clustered in the following way using Scipy Hierarchical Clustering. Let's say I want to prune the dendogram at level '1500'? How to do that? (I've tried using 'p' parameter and that is not what I'm expecting) Z = dendrogram(linkage_matrix, truncate_mode='lastp', color_threshold=1, labels=df.session.tolist(), distance_sort='ascending') plt.title("Hierachical Clustering") plt.show() As specified in the scipy documentation , if a cluster node is under color_threshold , then all of its descendants will be the same color (not blue). The links connecting nodes

I am trying to figure out how the output of scipy.cluster.hierarchy.dendrogram works... I thought I knew how it worked and I was able to use the output to reconstruct the dendrogram but it seems as if I am not understanding it anymore or there is a bug in Python 3 's version of this module. This answer, how do I get the subtrees of dendrogram made by scipy.cluster.hierarchy , implies that the dendrogram output dictionary gives dict_keys(['icoord', 'ivl', 'color_list', 'leaves', 'dcoord']) w/ all of the same size so you can zip them and plt.plot them to reconstruct the dendrogram. Seems simple

I usually use scipy.cluster.hierarchical linkage and fcluster functions to get cluster labels. However, the sklearn.cluster.AgglomerativeClustering has the ability to also consider structural information using a connectivity matrix , for example using a knn_graph input, which makes it interesting for my current application. However, I usually assign labels in fcluster by either a 'distance' or 'inconsistent' criterion, and AFAIK the AgglomerativeClustering function in sklearn only has the option to define the number of desired clusters (so criterion='maxclust' in the scipy library). I am

I want to cluster 2d points (latitude/longitude) on a map. The number of points is 400K so the input matrix would be 400k x 2. When I run scikit-learn's Agglomerative Clustering I run out of memory and my memory is about 500GB. class sklearn.cluster.AgglomerativeClustering(n_clusters=2, affinity='euclidean', memory=Memory(cachedir=None), connectivity=None, n_components=None, compute_full_tree='auto', linkage='ward', pooling_func=<function mean at 0x2b8085912398>)[source] I also tried the memory=Memory(cachedir) option with no success. Does anybody have a suggestion (another library or change