How to use Delta Lake with spark-shell?

Question

I\'m trying to write as Spark DF as a DeltaTable. It\'s working fine in my IDE Intelliji , But with the same dependencies and versions it\'s not working in

Answer 1

Spark itself has a dependency on Jackson, and the version you're instructing spark-shell to use is incompatible. Per https://github.com/apache/spark/blob/v2.4.0/pom.xml, 2.4.0 uses Jackson 2.6.7. Is there a particular reason that you need Jackson 2.10 in this case?