Avoid starting HiveThriftServer2 with created context programmatically

二次信任 提交于 2020-01-14 07:17:06

问题


We are trying to use ThriftServer to query data from spark temp tables, in spark 2.0.0.

First, we have created sparkSession with enabled Hive Support. Currently, we start ThriftServer with sqlContext like this:

HiveThriftServer2.startWithContext(spark.sqlContext());

We have spark stream with registered temp table "spark_temp_table":

StreamingQuery streamingQuery = streamedData.writeStream()
                                             .format("memory")
                                             .queryName("spark_temp_table")
                                             .start();

With beeline we are able to see temp tables (running SHOW TABLES);

When we want to run second job (with second sparkSession) with this approach we have to start second ThriftServer with different port.

I have two questions here:

  1. Is there any way to have one ThriftServer on one port with access to all temp tables in a different sparkSessions?

  2. HiveThriftServer2.startWithContext(spark.sqlContext()); is annotated with @DeveloperApi. Is there any way to start thrift server with context not in the code programatically?
    I saw there is configuration --conf spark.sql.hive.thriftServer.singleSession=true passed to ThriftServer on startup (sbin/start-thriftserver.sh) but I don't understand how to define this for a job. I tried to set this configuration property in sparkSession builder , but beeline didn't display temp tables.


回答1:


Is there any way to have one ThriftServer on one port with access to all temp tables in a different sparkSessions?

No. ThriftServer uses specific session and temporary tables can be accessed only within this session. This is why:

beeline didn't display temp tables.

when you start independent server with sbin/start-thriftserver.sh.

spark.sql.hive.thriftServer.singleSession doesn't mean you get a single session for multiple servers. It uses the same session for all connections to a single Thrift server. Possible use case:

  • you start thrift server.
  • client1 connects to this server and creates temp table foo.
  • client2 connects to this server and reads foo


来源:https://stackoverflow.com/questions/39719043/avoid-starting-hivethriftserver2-with-created-context-programmatically

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!