pyspark-dataframes

问题 I have a PySpark dataframe that keeps track of changes that occur in a product's price and status over months. This means that a new row is created only when a change occurred (in either status or price) compared to the previous month, like in the dummy data below ---------------------------------------- |product_id| status | price| month | ---------------------------------------- |1 | available | 5 | 2019-10| ---------------------------------------- |1 | available | 8 | 2020-08| ------------

问题 I am new to spark ,I want to do a broadcast join and before that i am trying to get the size of my data frame that i want to broadcast.. Is there anyway to find the size of a data frame . I am using Python as my programming language for spark Any help much appreciated 回答1: If you are looking for size in bytes as well as size in row count follow this- Alternative-1 // ### Alternative -1 /** * file content * spark-test-data.json * -------------------- * {"id":1,"name":"abc1"} * {"id":2,"name":

问题 I am new to spark ,I want to do a broadcast join and before that i am trying to get the size of my data frame that i want to broadcast.. Is there anyway to find the size of a data frame . I am using Python as my programming language for spark Any help much appreciated 回答1: If you are looking for size in bytes as well as size in row count follow this- Alternative-1 // ### Alternative -1 /** * file content * spark-test-data.json * -------------------- * {"id":1,"name":"abc1"} * {"id":2,"name":

问题 I try to load the following data.json file in a spark dataframe: {"positionmessage":{"callsign": "PPH1", "name": 0.0, "mmsi": 100}} {"positionmessage":{"callsign": "PPH2", "name": 0.0, "mmsi": 200}} {"positionmessage":{"callsign": "PPH3", "name": 0.0, "mmsi": 300}} by the following code: from pyspark.sql import SparkSession from pyspark.sql.types import ArrayType, StructField, StructType, StringType, IntegerType appName = "PySpark Example - JSON file to Spark Data Frame" master = "local" #

问题 I have the following sample Spark dataframe import pandas as pd import pyspark import pyspark.sql.functions as fn from pyspark.sql.window import Window raw_df = pd.DataFrame([ (1115, dt.datetime(2019,8,5,18,20), dt.datetime(2019,8,5,18,40)), (484, dt.datetime(2019,8,5,18,30), dt.datetime(2019,8,9,18,40)), (484, dt.datetime(2019,8,4,18,30), dt.datetime(2019,8,6,18,40)), (484, dt.datetime(2019,8,2,18,30), dt.datetime(2019,8,3,18,40)), (484, dt.datetime(2019,8,7,18,50), dt.datetime(2019,8,9,18

问题 I have recently started working with pySpark so don't know about many details regarding this. I am trying to create a BinaryType column in a data frame? But struggling to do it... for example, let's take a simple df df.show(2) +---+----------+ | col1|col2| +---+----------+ | "1"| null| | "2"| "20"| +---+----------+ Now I want to have a third column "col3" with BinaryType like | col1|col2| col3| +---+----------+ | "1"| null|[1 null] | "2"| "20"|[ 2 20] +---+----------+ How should i do it? 回答1:

问题 I want to start value of indexes in data frame with certain value instead of default value zero, if there is any parameter we can use for zipWithIndex() in pyspark. 回答1: the following solution will help to start zipwithIndex with default value. df = df_child.rdd.zipWithIndex().map(lambda x: (x[0], x[1] + index)).toDF() where index is default number you want to start with zipWithIndex. 来源： https://stackoverflow.com/questions/60124599/start-index-with-certain-value-zipwithindex-in-pyspark

问题 I have a dataframe like this sample df = spark.createDataFrame( [(2, "A" , "A2" , 2500), (2, "A" , "A11" , 3500), (2, "A" , "A12" , 5500), (4, "B" , "B25" , 7600), (4, "B", "B26" ,5600), (5, "C" , "c25" ,2658), (5, "C" , "c27" , 1100), (5, "C" , "c28" , 1200)], ['parent', 'group' , "brand" , "usage"]) output : +------+-----+-----+-----+ |parent|group|brand|usage| +------+-----+-----+-----+ | 2| A| A2| 2500| | 2| A| A11| 3500| | 4| B| B25| 7600| | 4| B| B26| 5600| | 5| C| c25| 2658| | 5| C|

问题 I have the following sample Spark dataframe import pandas as pd import pyspark import pyspark.sql.functions as fn from pyspark.sql.window import Window raw_df = pd.DataFrame([ (1115, dt.datetime(2019,8,5,18,20), dt.datetime(2019,8,5,18,40)), (484, dt.datetime(2019,8,5,18,30), dt.datetime(2019,8,9,18,40)), (484, dt.datetime(2019,8,4,18,30), dt.datetime(2019,8,6,18,40)), (484, dt.datetime(2019,8,2,18,30), dt.datetime(2019,8,3,18,40)), (484, dt.datetime(2019,8,7,18,50), dt.datetime(2019,8,9,18

问题 Apologies for the lengthy post for a seemingly simple curiosity, but I wanted to give full context... In Databricks, I am creating a "row" of data based on a specific schema definition, and then inserting that row into an empty dataframe (also based on the same specific schema). The schema definition looks like this: myschema_xb = StructType( [ StructField("_xmlns", StringType(), True), StructField("_Version", DoubleType(), True), StructField("MyIds", ArrayType( StructType( [ StructField("_ID