问题
I am running Pycharm on my MacBook.
Client settings: Python Interpreter -> Python 3.7 (dtabricks-connect-6.4)
Cluster settings: Databricks Runtime Version -> 6.4 (includes Apache Spark 2.4.5, Scala 2.11)
It worked well for months but suddenly, without any updates made, I cant run my python script from Pycharm against databricks cluster anymore. The Error is ...
Caused by: `java.lang.IllegalArgumentException: The cluster is running server version `dbr-6.4` but this client only supports Set(dbr-5.5)....`
I restarted Pycharm, I witched back and for the interpreter, I restarted the cluster and I even restarted my MacBook but it didnt helped. The Error message is simply false, because both, cluster and client are using the SAME version. I can see that when I execute my python script that the cluster is being started but fails at the end.
pyenv activate databricks-connect-6-4
pip freeze
Cython==0.29.21
databricks-connect==6.4.0
numpy==1.19.2
pandas==1.0.1
py4j==0.10.7
pyarrow==0.13.0
pycountry==20.7.3
python-dateutil==2.8.1
pytz==2020.1
six==1.15.0
回答1:
It looks like that it was caused by some internal changes on the server side, and this prevents databricks-connect from working. You can always disable this check by setting environment variable DEBUG_IGNORE_VERSION_MISMATCH to value 1 (export DEBUG_IGNORE_VERSION_MISMATCH=1 in the console before executing databricks-connect test) - you can also set this environment variable in the PyCharm.
Update: it should be fixed by Databricks Connect 6.4.2 that was just released.
回答2:
Thanks for raising this. The Databricks-Connect team has acknowledged this issue and we are working on a patch to address this issue. Will keep you posted. In the meantime you can use the DEBUG_IGNORE_VERSION_MISMATCH as Alex pointed out.
Update: A compatible db-connect client has been released to fix this problem: version 6.4.2 (https://pypi.org/project/databricks-connect/6.4.2/, install with: pip install databricks-connect==6.4.2).
来源：https://stackoverflow.com/questions/64695287/databricks-connect-6-4-not-able-to-communicate-with-server-anymore