What two different keys go to the same reducer by the default hash partitioner in Hadoop?

谁说我不能喝 提交于 2020-01-03 04:51:08

问题


As we know that Hadoop guarantees that the same keys which come from different mappers will be sent to the same reducer.

But if two different keys have the same hash value, they definitely will go to the same reducer, so will them be sent to the same reduce method orderly? Which part is responsible for this logic?

Thanks a lot!


回答1:


Q1:so will them be sent to the same reduce method orderly

Ans : yes


Q2:Which part is responsible for this logic

Ans : shuffle sort


Example :

key  value
 1       2
 1       2
 2       5
 3       19
 6       20

Lets say number of reducer is 5 so now .

Reduce 0 will get key NO key-value pairs
Reduce 1 will get key 1,6 in same order 
Reduce 2 will get key 2
Reduce 3 will get key 3
Reduce 4 will get key NO key-value pairs


来源:https://stackoverflow.com/questions/20757318/what-two-different-keys-go-to-the-same-reducer-by-the-default-hash-partitioner-i

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!