databricks

问题 I have hundreds of zipped files sitting in different folders, which I can access using MS Storage Explorer. I just setup a SQL Server DB in Azure. Now I am trying to figure out how I can pull data from each file in each folder, unzip it, parse it, and load it into tables. The data is coming in daily, so the folders are named '1', '2', '3', etc. '31', for the days of the month. Also, I have monthly folders '1' through '12', for the 12 months of the year. Finally, I have folders named '2017',

问题 I have hundreds of zipped files sitting in different folders, which I can access using MS Storage Explorer. I just setup a SQL Server DB in Azure. Now I am trying to figure out how I can pull data from each file in each folder, unzip it, parse it, and load it into tables. The data is coming in daily, so the folders are named '1', '2', '3', etc. '31', for the days of the month. Also, I have monthly folders '1' through '12', for the 12 months of the year. Finally, I have folders named '2017',

问题 Trying to implement SCD Type 2 logic in Spark 2.4.4. I've two Data Frames; one containing 'Existing Data' and the other containing 'New Incoming Data'. Input and expected output are given below. What needs to happen is: All incoming rows should get appended to the existing data. Only following 3 rows which were previously 'active' should become inactive with appropriate 'endDate' populated as follows: pk=1, amount = 20 => Row should become 'inactive' & 'endDate' is the 'startDate' of

问题 I am trying to store some strings of a dataframe in cassandra table. I tried with cassandra table columns defining as float/double/decimal. But every type only storing 2 precision , i.e. 8.00005 as stored as 8.00 69.345 as 69.34 , what is wrong with cassandra table? Why it is not holding all precision digits. How fix this issue ? Let me know if needed any more information of the problem. 回答1: This issue seeme to be with the precision settings for cqlsh. The cassandra is storing the values

问题 I am reading a parquet file in Azure databricks: Using SparkR > read.parquet() Using Sparklyr > spark_read_parquet() Both the dataframes are different, Is there any way to convert SparkR dataframe into the sparklyr dataframe and vice-versa ? 回答1: sparklyr creates tbl_spark. This is essentially just a lazy query written in Spark SQL. SparkR creates a SparkDataFrame which is more of a collection of data that is organized using a plan. In the same way you can't use a tbl as a normal data.frame

问题 Question related to this problem was not answered by anyone I tried implementing error logging using python in azure data bricks. If i try the below code in python(pycharm) it is working as expected. But when i try the same code in azure databricks(python) it is not creating a file and not writing any contents into the file. I tried creating a file in azure data lake gen2. i have given the path with mount point of data lake store gen2. Can you please help why the python code is not working as

问题 I have a datafame and would like to add columns to it, based on values from a list. The list of my values will vary from 3-50 values. I'm new to pySpark and I'm trying to append these values as new columns (empty) to my df. I've seen recommended code of how to add [one column][1] to a dataframe but not multiple from a list. mylist = ['ConformedLeaseRecoveryTypeId', 'ConformedLeaseStatusId', 'ConformedLeaseTypeId', 'ConformedLeaseRecoveryTypeName', 'ConformedLeaseStatusName',

问题 Help :) I need to connect from my Azure databricks cluster to a SQL Azure instance using my Azure AD credentials. I have tested and I can connect to the target database using SSMS (SQL Server Management Studio) through my Azure AD credentials so that works fine. Firewall connectivity is fine. I have been able to temporarily test with a SQL username and password and this works fine, but that is about to be taken away from me. However, connecting through databricks I get: om.microsoft.sqlserver

问题 Recently, Databricks launched Databricks Connect that allows you to write jobs using Spark native APIs and have them execute remotely on an Azure Databricks cluster instead of in the local Spark session. It works fine except when I try to access files in Azure Data Lake Storage Gen2. When I execute this: spark.read.json("abfss://...").count() I get this error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs

问题 I'm working on two pyspark dataframes and doing a left-anti join on them to track everyday changes and then send an email. The first time I tried: diff = Table_a.join( Table_b, [Table_a.col1== Table_b.col1, Table_a.col2== Table_b.col2], how='left_anti' ) Expected output is a pyspark dataframe with some or no data. This diff dataframe gets it's schema from Table_a. The first time I ran it, showed no data as expected with the schema representation. The next time onwards just throws