Which HBase connector for Spark 2.0 should I use?

ぐ巨炮叔叔 提交于 2019-12-28 13:51:51

问题


Our stack is composed of Google Data Proc (Spark 2.0) and Google BigTable (HBase 1.2.0) and I am looking for a connector working with these versions.

The Spark 2.0 and the new DataSet API support is not clear to me for the connectors I have found:

  • spark-hbase : https://github.com/apache/hbase/tree/master/hbase-spark
  • spark-hbase-connector : https://github.com/nerdammer/spark-hbase-connector
  • hortonworks-spark/shc : https://github.com/hortonworks-spark/shc

The project is written in Scala 2.11 with SBT.

Thanks for your help


回答1:


Update: SHC now seems to work with Spark 2 and the Table API. See https://github.com/GoogleCloudPlatform/cloud-bigtable-examples/tree/master/scala/bigtable-shc

Original answer:

I don't believe any of these (or any other existing connector) will do all that you would like today.

  • spark-hbase will probably the right solution when it is release (HBase 1.4?), but currently only builds at head and is still working on Spark 2 support.
  • spark-hbase-connector only seems to support RDD APIs, but since they are more stable, might be somewhat helpful.
  • hortonworks-spark/shc probably won't work because I believe it only supports Spark 1 and uses the older HTable APIs which do not work with BigTable.

I would recommend just using HBase MapReduce APIs with RDD methods like newAPIHadoopRDD (or possibly the spark-hbase-connector?). Then manually convert RDDs into DataSets. This approach is a lot easier in Scala or Java than Python.

This is an area that the HBase community is working to improve and Google Cloud Dataproc will incorporate those improvements as they happen.




回答2:


In addition to the above answer, using newAPIHadoopRDD means that, you get all the data from HBase and from then on, its all core spark. You would not get any HBase specific API like Filters etc. And the current spark-hbase, only snapshots are available.



来源:https://stackoverflow.com/questions/40908891/which-hbase-connector-for-spark-2-0-should-i-use

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!