Finding Connected Components using Hadoop/MapReduce

后端 未结 4 1236
误落风尘
误落风尘 2021-02-06 06:11

I need to find connected components for a huge dataset. (Graph being Undirected)

One obvious choice is MapReduce. But i\'m a newbie to MapReduce and am quiet short of ti

4条回答
  •  滥情空心
    2021-02-06 06:41

    I don't really know if an API is available which has methods to find strongly connected components. But, I implemented the BFS algorithm to find distance from source node to all other nodes in the graph (the graph was a directed graph as big as 65 million nodes).

    The idea was to explore the neighbors (distance of 1) for each node in one iteration and feeding the output of reduce back to map, until the distances converge. The map emits the shortest distances possible from each node, and reduce updated the node with the shortest distance from the list.

    I would suggest to check this out. Also, this could help. These two links would give you the basic idea about graph algorithms in map reduce paradigm (if you are already not familiar). Essentially, you need to twist the algorithm to use DFS instead of BFS.

提交回复
热议问题