问题
It is understood , while migrating/load from oracle db to hdfs/parquet , it is preferred to use SQOOP rather than SPARK with JDBC driver.
Spark suppose to be 100x faster when processing right ? Then what is wrong with Spark ? Why people prefer SQOOP while loading data from oracle db tables ?
Please suggest me what should i need to do make Spark faster when loading data from oracle.
回答1:
Spark is fast when it knows how to parallelize queries. If you're just executing single query, then Spark doesn't know what to do. You can improve speed by using parameters lowerBound, upperBound, numPartitions when reading data with spark.read.jdbc, but it depends really on the design of your tables.
You can find more documentation here.
回答2:
The major point is already covered in Alex's answer.
I just wanted to add an example,
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[4]").appName("Test-JDBC").getOrCreate()
ds = spark.read.jdbc("jdbc:mysql://localhost:3306/stackexchange", "(select min(id), max(id) from post_history) as ph",
properties={"user": "devender", "password": "*****", "driver": "com.mysql.jdbc.Driver"})
r = ds.head()
minId = r[0]
maxId = r[1]
ds = spark.read.jdbc("jdbc:mysql://localhost:3306/stackexchange", "(select * from post_history) as ph",
properties={"user": "devender", "password": "*****", "driver": "com.mysql.jdbc.Driver"},
numPartitions=4, column="id", lowerBound=minId, upperBound=maxId)
count = ds.count()
print(count)
For more details, https://gist.github.com/devender-yadav/5c4328918602b7910ba883e18b68fd87
Note: Sqoop automatically executes boundary query to fetch MIN, MAX value for split by column (that query can also be overridden)
来源：https://stackoverflow.com/questions/56288481/why-spark-is-slower-when-compared-to-sqoop-when-it-comes-to-jdbc