问题
I can see logs using %sh command on databricks driver node. How can I copy them on my windows machine for analysis?
%sh cd eventlogs/4246832951093966440 gunzip eventlog-2019-07-22--14-00.gz ls -l head -1 eventlog-2019-07-22--14-00 Version":"2.4.0","Timestamp":1563801898572,"Rollover Number":0,"SparkContext Id":4246832951093966440}
Thanks
回答1:
There are different ways to copy driver logs to your local machine.
Option1: Cluster Driver Logs:
Go to Azure Databricks Workspace => Select the cluster => Click on Driver Logs => To download to local machine.
The direct print and log statements from your notebooks and libraries goes to the driver logs. The logs have three outputs:
Standard output
Standard error
Log4j logs
The log files are rotated periodically. Older log files appear at the top of the page, listed with timestamp information. You can download any of the logs for troubleshooting.
Option2: Cluster Log Delivery:
When you create a cluster, you can specify a location to deliver Spark driver and worker logs. Logs are delivered every five minutes to your chosen destination. When a cluster is terminated, Databricks guarantees to deliver all logs generated up until the cluster was terminated.
The destination of the logs depends on the cluster ID. If the specified destination is dbfs:/cluster-log-delivery, cluster logs for 0630-191345-leap375 are delivered to dbfs:/cluster-log-delivery/0630-191345-leap375.
To configure the log delivery location:
On the cluster configuration page, click the Advanced Options toggle.
At the bottom of the page, click the Logging tab.
Select a destination type.
Enter the cluster log path.
To Download the Cluster Logs to Local Machine:
Install the Databricks CLI, configure it with your Databricks credentials, and use the CLI's dbfs cp command. For example: dbfs cp dbfs:/FileStore/azure.txt ./azure.txt.
If you want to download an entire folder of files, you can use dbfs cp -r <DBFS Path> <LocalPath>.
- Open cmd prompt.
- Install Python: https://www.python.org/downloads/
- Install Databricks:
pip install databricks-cli Copy the host and Generated access token.
host: https://centralus.azuredatabricks.net/?
token: 46546fgdsgfdhhh65fdh546sdsdrfdfghf12
Databricks configure: Run the
datbricks configure
Databricks Host (should begin with https://): https://centralus.azuredatabricks.net/
Username: username@microsoft.com
Password: paste Access token
Repeat for confirmation: paste Access token
- Now Run the below cmdlet to copy logs to local machine
dbfs cp -r dbfs:/cluster-logs/0731-081420-tees851/driver C:\Users\Azure\Desktop\Logs
Hope this helps.
来源：https://stackoverflow.com/questions/57148116/databricks-how-can-i-copy-driver-logs-to-my-machine