Databricks - Creating permanent User Defined Functions (UDFs)

Question

I am able to create a UDF function and register to spark using spark.UDF method. However, this is per session only. How to register python UDF functions automatically when

Answer 1

This is not possible; this is not like UDFs in Hive.

Code the UDF as part of the package / program you submit or in the jar included in the Spark App, if using spark-submit.

However,

spark.udf.register.udf("...

is required to be done as well. This applies to Databrick notebooks, etc. The UDFs need to be re-registered per Spark Context/Session.

Answer 2

acutally you can create a permanent function but not from a notebook you need to create it from a JAR file

https://docs.databricks.com/spark/latest/spark-sql/language-manual/create-function.html

CREATE [TEMPORARY] FUNCTION [db_name.]function_name AS class_name [USING resource, ...]

resource: : (JAR|FILE|ARCHIVE) file_uri