azure-databricks

问题 H|*|D|*|PA|*|BJ|*|S|*|2019.05.27 08:54:24|##| H|*|AP_ATTR_ID|*|AP_ID|*|OPER_ID|*|ATTR_ID|*|ATTR_GROUP|*|LST_UPD_USR|*|LST_UPD_TSTMP|##| 779045|*|Sar|*|SUPERVISOR HIERARCHY|*|Supervisor|*|2|*|128|*|2019.05.14 16:48:16|##| 779048|*|KK|*|SUPERVISOR HIERARCHY|*|Supervisor|*|2|*|116|*|2019.05.14 16:59:02|##| 779054|*|Nisha - A|*|EXACT|*|CustomColumnRow120|*|2|*|1165|*|2019.05.15 12:11:48|##| T|*||*|2019.05.27 08:54:28|##| file name is PA.dat. I need to skip first line and also last line of the

问题 I am able to execute a simple SQL statement using PySpark in Azure Databricks but I want to execute a stored procedure instead. Below is the PySpark code I tried. #initialize pyspark import findspark findspark.init('C:\Spark\spark-2.4.5-bin-hadoop2.7') #import required modules from pyspark import SparkConf, SparkContext from pyspark.sql import SparkSession from pyspark.sql import * import pandas as pd #Create spark configuration object conf = SparkConf() conf.setMaster("local").setAppName("My

问题 I am able to execute a simple SQL statement using PySpark in Azure Databricks but I want to execute a stored procedure instead. Below is the PySpark code I tried. #initialize pyspark import findspark findspark.init('C:\Spark\spark-2.4.5-bin-hadoop2.7') #import required modules from pyspark import SparkConf, SparkContext from pyspark.sql import SparkSession from pyspark.sql import * import pandas as pd #Create spark configuration object conf = SparkConf() conf.setMaster("local").setAppName("My

问题 I am able to execute a simple SQL statement using PySpark in Azure Databricks but I want to execute a stored procedure instead. Below is the PySpark code I tried. #initialize pyspark import findspark findspark.init('C:\Spark\spark-2.4.5-bin-hadoop2.7') #import required modules from pyspark import SparkConf, SparkContext from pyspark.sql import SparkSession from pyspark.sql import * import pandas as pd #Create spark configuration object conf = SparkConf() conf.setMaster("local").setAppName("My

问题 I am able to execute a simple SQL statement using PySpark in Azure Databricks but I want to execute a stored procedure instead. Below is the PySpark code I tried. #initialize pyspark import findspark findspark.init('C:\Spark\spark-2.4.5-bin-hadoop2.7') #import required modules from pyspark import SparkConf, SparkContext from pyspark.sql import SparkSession from pyspark.sql import * import pandas as pd #Create spark configuration object conf = SparkConf() conf.setMaster("local").setAppName("My

问题 I am trying to read partitioned Avro data which is partitioned based on Year, Month and Day and that seems to be significantly slower than pointing it directly to the path. In the Physical plan I can see that the partition filters are getting passed on, so it is not scanning the entire set of directories but still it is significantly slower. E.g. reading the partitioned data like this profitLossPath="abfss://raw@"+datalakename+".dfs.core.windows.net/datawarehouse/CommercialDM.ProfitLoss/"

问题 I am trying to read partitioned Avro data which is partitioned based on Year, Month and Day and that seems to be significantly slower than pointing it directly to the path. In the Physical plan I can see that the partition filters are getting passed on, so it is not scanning the entire set of directories but still it is significantly slower. E.g. reading the partitioned data like this profitLossPath="abfss://raw@"+datalakename+".dfs.core.windows.net/datawarehouse/CommercialDM.ProfitLoss/"

问题 I have multiple data sources I want to add a validation in azure data factory before loading into tables it should check for file size so that it is not empty. So if the file size is more than 10 kb or if it is not empty loading should start and if it is empty then loading should not start. I checked validation activity in Azure Data Factory but it is not showing size for multiple files in a folder. Any suggestions appreciated basically if I can add any python notebook for this validation

问题 Help :) I need to connect from my Azure databricks cluster to a SQL Azure instance using my Azure AD credentials. I have tested and I can connect to the target database using SSMS (SQL Server Management Studio) through my Azure AD credentials so that works fine. Firewall connectivity is fine. I have been able to temporarily test with a SQL username and password and this works fine, but that is about to be taken away from me. However, connecting through databricks I get: om.microsoft.sqlserver

问题 Recently, Databricks launched Databricks Connect that allows you to write jobs using Spark native APIs and have them execute remotely on an Azure Databricks cluster instead of in the local Spark session. It works fine except when I try to access files in Azure Data Lake Storage Gen2. When I execute this: spark.read.json("abfss://...").count() I get this error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs