Spark - How can get the Logical / Physical Query execution using - Thirft - Hive Interactor

Question

Spark - How can get the Logical / Physical Query execution using the following

1. Via Thrift
2. Via SparkInteractor

Answer 1

If you are using Spark 1, You can get the explain on a query this way:

sqlContext.sql("your SQL query").explain(true)

If you are using Spark 2, it's the same:

spark.sql("your SQL query").explain(true)

The same logic is available on a dataframe:

yourDF.explain(true) or yourDF.someOperation.explain(true)

Where someOperation could be maybe a select on a join or something else.

Answer 2

You can use explain statement with query as below in beeline via thrift.

EXPLAIN EXTENDED select * from sr23 join sr12 [<join condidtion>]

What do you mean be spark interceptor.? is it spark-sql shell.? if it is, then you can use above query.

If you meant spark-shell, then you need to call explain() function on dataframes.

eg:

val df1 = sqlContext.sql(" < your sql query > ");

df1.explain;

this will give both physical and logical plans. You can also see them from spark web UI in SQL tab.