问题
I'm trying to write a Spark 2.4.4 dataframe to PostgreSQL via JDBC. I'm using Scala.
batchDF.
write.
format("jdbc").
option("url", "jdbc:postgresql://...").
option("driver", "org.postgresql.Driver").
option("dbtable", "traffic_info").
option("user", "xxxx").
option("password", "xxxx").
mode(SaveMode.Append).
save()
One of the fields (remote_prefix
) is of CIDR
type in my table but is StringType
in my dataframe, so I cannot write it as-is:
ERROR: column "remote_prefix" is of type cidr but expression is of type character varying
I tried to cast it explicitely using:
option("customSchema", "remote_prefix CIDR")
or
withColumn("remote_prefix", functions.expr("CAST(remote_prefix AS CIDR)"))
But in both cases Spark returned an error:
20/09/17 13:52:00 ERROR MicroBatchExecution: Query [id = xxx, runId = xxx] terminated with error
org.apache.spark.sql.catalyst.parser.ParseException:
DataType cidr is not supported.(line 1, pos 14)
== SQL ==
remote_prefix CIDR
--------------^^^
How can I get around this?
来源:https://stackoverflow.com/questions/63939841/writing-to-a-postgresql-cidr-column-with-spark-jdbc