Pyspark to Netezza write issue : failed to create external table for bulk load

问题

While writing from Pyspark to Netezza Im constantly getting following error (Issue persists when the size of the dataframe increases, no issues in appending for small dataframes with 10 to 60 rows) :

org.netezza.error.NzSQLException: failed to create external table for bulk load
    at org.netezza.sql.NzPreparedStatament.executeBatch(NzPreparedStatament.java:1140)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:667)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:834)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:834)
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I have all permissions to create external tables and normal tables in Netezza. Im able to create external tables in Netezza using the same user directly in Netezza.

Following is my code :

failed_df \
    .write \
    .format(("jdbc")) \
    .options(url='jdbc:netezza://###/',
             driver='org.netezza.Driver',
             dbtable='###',
             user='abc',
             password='***') \
    .mode('append') \
    .save()

来源：https://stackoverflow.com/questions/61590729/pyspark-to-netezza-write-issue-failed-to-create-external-table-for-bulk-load

标签

apache-spark

jdbc

pyspark

netezza