Spark History Server Authentication in Kerberized Cluster

Question

I set-up a HDP cluster which contains amongst others Spark. I also enabled Kerberos for the cluster, so that all services and users have to authenticate via their principals

Answer 1

I have found a solution to this in IBM's documentation

You re-use Hadoop's jetty authentication filter for Kerberos/SPNEGO org.apache.hadoop.security.authentication.server.AuthenticationFilter

You can do this by setting in Spark's default.conf spark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter and spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params= type=kerberos,kerberos.principal=${spnego_principal_name},kerberos.keytab=${spnego_keytab_path} Be careful with those replacement variables, they didn't work for me when setting these values in Ambari. Also consider adding cookie.domain and signature.secret.file similar to the other Hadoop SPNEGO-configurations.

Obviously this only works when the Spark History Server runs with the Hadoop-classes in its classpath -- so it's not an out-of-the-box solution for a SMACK-stack for example.