问题
I have 2 dataframes with same schema, i need to compare the rows of dataframes and keep a count of rows with at-least one column with value 1 in both the dataframes
Right now i am making a list of the rows and then comparing the 2 lists to find even if one value is equal in both the list and equal to 1
rowOgList = []
for row in cat_og_df.rdd.toLocalIterator():
rowOgDict = {}
for cat in categories:
rowOgDict[cat] = row[cat]
rowOgList.append(rowOgDict)
#print(rowOgList[0])
rowPredList = []
for row in prob_df.rdd.toLocalIterator():
rowPredDict = {}
for cat in categories:
rowPredDict[cat] = row[cat]
rowPredList.append(rowPredDict)
But here the function rdd.tolocalIterator gives me a heap space error when i try it on a huge dataset. for example: this is the 1st dataframe
+-------+-------+-------+-------+
|column1|column2|column3|column4|
+-------+-------+-------+-------+
| 0| 0| 0| 0|
| 0| 0| 0| 0|
| 0| 0| 0| 0|
| 1| 0| 0| 0|
| 0| 0| 0| 0|
| 0| 0| 0| 0|
| 0| 1| 0| 0|
| 0| 0| 0| 0|
| 0| 0| 1| 0|
| 0| 0| 0| 0|
| 0| 0| 0| 1|
| 0| 0| 0| 0|
| 0| 0| 0| 0|
| 0| 0| 0| 0|
| 0| 0| 0| 0|
| 0| 0| 0| 0|
| 1| 0| 0| 0|
| 0| 0| 1| 0|
| 0| 0| 0| 0|
| 0| 0| 0| 0|
+-------+-------+-------+-------+
this is the 2nd dataframe
+-------+-------+-------+-------+
|column1|column2|column3|column4|
+-------+-------+-------+-------+
| 1| 0| 1| 0|
| 1| 0| 1| 0|
| 0| 0| 1| 1|
| 0| 0| 1| 1|
| 1| 0| 1| 0|
| 1| 0| 1| 0|
| 1| 0| 1| 0|
| 1| 0| 1| 0|
| 0| 0| 1| 1|
| 1| 0| 1| 0|
| 0| 0| 1| 1|
| 1| 0| 1| 0|
| 1| 0| 1| 0|
| 1| 0| 1| 0|
| 1| 0| 1| 0|
| 1| 0| 1| 0|
| 1| 0| 1| 0|
| 1| 0| 1| 0|
| 1| 0| 1| 0|
| 1| 0| 1| 0|
+-------+-------+-------+-------+
here rows 9,11,17,18 have at least one column with same value and that value as 1 so here the count = 4
Can this be done in any optimized way, Thanks.
回答1:
Note : As mentioned by pault, this will work better if you have unique row indices that connect both dataframes. Otherwise, the row orders may not be guaranteed in some Spark operations.
(1) Setup the environment and some sample data.
import numpy as np
from pyspark.ml.feature import VectorAssembler
from pyspark.sql import functions as F
df1 = spark.createDataFrame([
(0, 0, 1),
(1, 0, 0),
(0, 0, 1)
], ["column1", "column2", "column3"])
df2 = spark.createDataFrame([
(0, 0, 0),
(1, 0, 1),
(0, 0, 1)
], ["column1", "column2", "column3"])
(2) Collect all columns into a Spark vector.
assembler = VectorAssembler(
inputCols=["column1", "column2", "column3"],
outputCol="merged_col")
df1_merged = assembler.transform(df1)
df2_merged = assembler.transform(df2)
df1_merged.show()
+-------+-------+-------+-------------+
|column1|column2|column3| merged_col|
+-------+-------+-------+-------------+
| 0| 0| 1|[0.0,0.0,1.0]|
| 1| 0| 0|[1.0,0.0,0.0]|
| 0| 0| 1|[0.0,0.0,1.0]|
+-------+-------+-------+-------------+
(3) Get the row and column index of non-zero elements. Using numpy.nonzero() on RDD of Spark Vector.
def get_nonzero_index(args):
(row, index) = args
np_arr = np.array(row.merged_col)
return (index, np_arr.nonzero()[0].tolist())
df1_ind_rdd = df1_merged.rdd.zipWithIndex().map(get_nonzero_index)
df2_ind_rdd = df2_merged.rdd.zipWithIndex().map(get_nonzero_index)
df1_ind_rdd.collect()
[(0, [2]), (1, [0]), (2, [2])]
df2_ind_rdd.collect()
[(0, []), (1, [0, 2]), (2, [2])]
(4) You can then do your comparison on these 2 Python lists easily.
Note that this method will not be efficient (due to collect) if the number of rows you have is very large. In that case, you will want to do all processing in Spark by doing a join on the 2 dataframes.
(5) To do the matching purely in Spark, you can try the methods below that rely on a join on row index.
df1_index = spark.createDataFrame(df1_ind_rdd, ["row_index_1", "column_index_1"])
df2_index = spark.createDataFrame(df2_ind_rdd, ["row_index_2", "column_index_2"])
df_joined = df1_index.join(df2_index, df1_index.row_index_1 == df2_index.row_index_2)
Then expand the list so that we get an element on each row.
df_exploded = df_joined.withColumn("column_index_exp_1", F.explode(df_joined.column_index_1))\
.withColumn("column_index_exp_2", F.explode(df_joined.column_index_2))
Check for match between the two columns and finally convert into integer for summing.
df_match_bool = df_exploded.withColumn("match_bool", df_exploded.column_index_exp_1 == df_exploded.column_index_exp_2)
df_match_int = df_match_bool.withColumn("match_integer", df_match_bool.match_bool.cast("long"))
df_match_bool.show()
+-----------+--------------+-----------+--------------+------------------+------------------+----------+
|row_index_1|column_index_1|row_index_2|column_index_2|column_index_exp_1|column_index_exp_2|match_bool|
+-----------+--------------+-----------+--------------+------------------+------------------+----------+
| 1| [0]| 1| [0, 2]| 0| 0| true|
| 1| [0]| 1| [0, 2]| 0| 2| false|
| 2| [2]| 2| [2]| 2| 2| true|
+-----------+--------------+-----------+--------------+------------------+------------------+----------+
df_match_int.groupBy().sum("match_integer").collect()[0][0]
2
回答2:
For Spark 2.4 and for smallish number of columns and with a degree of performance penalty as whole array processed, but in parallel. Num cols is 5, as an example. Dynamic schema columns definitions. Tidy up of declaration required here This approach is for any value, need to restrict to 1's only. Filter added. Certain approaches do not appear to work in lower versions of Spark. Tested this.
from pyspark.sql.functions import udf, col, split, arrays_zip, expr, lit
from pyspark.sql import functions as F
from pyspark.sql.types import *
from pyspark.sql import Row
df1 = spark.createDataFrame([
(1, 1, 0, 0, 0),
(1, 0, 0, 0, 1),
(0, 0, 0, 0, 0) ], ["column1", "column2", "column3", "column4", "column5"])
df2 = spark.createDataFrame([
(1, 1, 1, 1, 1),
(0, 1, 1, 1, 1),
(0, 0, 0, 0, 0) ], ["column1", "column2", "column3", "column4", "column5"])
schema1 = StructType(df1.schema.fields[:] + [StructField("index1", LongType(), True)])
schema2 = StructType(df2.schema.fields[:] + [StructField("index2", LongType(), True)])
allCols = [x for x in df1.columns] # at this stage common to both DFs - df1 & df2
rdd1 = df1.rdd.zipWithIndex()
rdd2 = df2.rdd.zipWithIndex()
# All narrow transformations, so zipWithIndex should be fine
rddA = rdd1.map(lambda row: tuple(row[0].asDict()[c] for c in schema1.fieldNames()[:-1]) + (row[1],))
dfA = spark.createDataFrame(rddA, schema1)
rddB = rdd2.map(lambda row: tuple(row[0].asDict()[c] for c in schema2.fieldNames()[:-1]) + (row[1],))
dfB = spark.createDataFrame(rddB, schema2)
dfA = dfA.withColumn("merged_col1", F.concat_ws(',', *allCols))
dfB = dfB.withColumn("merged_col2", F.concat_ws(',', *allCols))
dfC = dfA.join(dfB, dfA.index1 == dfB.index2).select("index1", "merged_col1", "merged_col2")
dfD = dfC.select(col("index1"), split(col("merged_col1"), ",\s*").cast("array<int>").alias("ev1"), split(col("merged_col2"), ",\s*").cast("array<int>").alias("ev2"))
dfE = dfD.withColumn("matches", expr("filter(sequence(0,size(ev1)-1), (i -> ev1[i] == 1 and ev1[i] == ev2[i]))"))
dfF = dfE.withColumn("matchesSize", F.size(F.col("matches")))
dfF.filter(F.col("matchesSize") > 0).show()
returns in this simulated case:
+------+---------------+---------------+-------+-----------+
|index1| ev1| ev2|matches|matchesSize|
+------+---------------+---------------+-------+-----------+
| 0|[1, 1, 0, 0, 0]|[1, 1, 1, 1, 1]| [0, 1]| 2|
| 1|[1, 0, 0, 0, 1]|[0, 1, 1, 1, 1]| [4]| 1|
+------+---------------+---------------+-------+-----------+
You need to manipulate the data further in terms of count, etc. and what you want to display. There is enough data to do that.
I am not a pyspark expert by any means, but interesting question. And this does not have explode etc. Simpler than other answer I feel with good parallelism possible, can optimize further, but we leave that to you in terms of partitioning. Down-side is all values processed when in fact when should stop on 1st find. Would need a UDF for that I suspect.
来源：https://stackoverflow.com/questions/57849724/compare-rows-of-two-dataframes-to-find-the-matching-column-count-of-1s