Writing to a PostgreSQL [CIDR] column with Spark JDBC

不羁岁月 提交于 2021-02-11 14:34:14

问题


I'm trying to write a Spark 2.4.4 dataframe to PostgreSQL via JDBC. I'm using Scala.

batchDF.
    write.
    format("jdbc").
    option("url", "jdbc:postgresql://...").
    option("driver", "org.postgresql.Driver").
    option("dbtable", "traffic_info").
    option("user", "xxxx").
    option("password", "xxxx").
    mode(SaveMode.Append).
    save()

One of the fields (remote_prefix) is of CIDR type in my table but is StringType in my dataframe, so I cannot write it as-is:

ERROR: column "remote_prefix" is of type cidr but expression is of type character varying

I tried to cast it explicitely using:

option("customSchema", "remote_prefix CIDR")

or

withColumn("remote_prefix", functions.expr("CAST(remote_prefix AS CIDR)"))

But in both cases Spark returned an error:

20/09/17 13:52:00 ERROR MicroBatchExecution: Query [id = xxx, runId = xxx] terminated with error
org.apache.spark.sql.catalyst.parser.ParseException: 
DataType cidr is not supported.(line 1, pos 14)

== SQL ==
remote_prefix CIDR
--------------^^^

How can I get around this?

来源:https://stackoverflow.com/questions/63939841/writing-to-a-postgresql-cidr-column-with-spark-jdbc

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!