apache-spark-2.3

问题 I cant save a table to hive database anymore using metastore. I see the tables in spark using spark.sql but I cant see the same tables in hive database. I tried this but it doesnt store the table to hive. How can I configure the hive metastore? The spark version is 2.3.1. If you want more details please comment. %spark import org.apache.spark.sql.SparkSession val spark = (SparkSession .builder .appName("interfacing spark sql to hive metastore without configuration file") .config("hive

问题 While working on improving code performance as I had many jobs fail (aborted), I thought about using persist() function on Spark Dataframe whenever I need to use that same dataframe on many other operations. When doing it and following the jobs, stages in the Spark application UI, I felt like it's not really always optimal to do so, it depends on the number of partitions and the data size. I wasn't sure until I got the job aborted because of a fail in the persist stage. I'm questioning if the