spark-jdbc

问题 I want to write around 10 GB of data everyday to Azure SQL server DB using PySpark.Currently using JDBC driver which takes hours making insert statements one by one. I am planning to use azure-sqldb-spark connector which claims to turbo boost the write using bulk insert. I went through the official doc: https://github.com/Azure/azure-sqldb-spark. The library is written in scala and basically requires the use of 2 scala classes : import com.microsoft.azure.sqldb.spark.config.Config import com

问题 I am new to Spark and we are currently using the spark-java to create orc files from Oracle database. I was able to configure the connection with sqlContext.read().jdbc(url,table,props) However, I couldn't find any way in the properties to specify the trustStore or trustStoreType. Can someone help me about how to specify these properties? I already tried populating the properties as props.put("trustStore", "<PATH_TO_SSO>"); props.put("trustStoreType", "sso"); But it didn't work for me Update1

问题 I'm perplexed between the behaviour of numPartitions parameter in the following methods: DataFrameReader.jdbc Dataset.repartition The official docs of DataFrameReader.jdbc say following regarding numPartitions parameter numPartitions : the number of partitions. This, along with lowerBound (inclusive), upperBound (exclusive), form partition strides for generated WHERE clause expressions used to split the column columnName evenly. And official docs of Dataset.repartition say Returns a new

问题 I am using a query to fetch data from MYSQL as follows: var df = spark.read.format("jdbc") .option("url", "jdbc:mysql://10.0.0.192:3306/retail_db") .option("driver" ,"com.mysql.jdbc.Driver") .option("user", "retail_dba") .option("password", "cloudera") .option("dbtable", "orders") .option("partitionColumn", "order_id") .option("lowerBound", "1") .option("upperBound", "68883") .option("numPartitions", "4") .load() Question is, can I use a pseudo column (like ROWNUM in Oracle or RRN(employeeno)