Spark Equivalent of IF Then ELSE

Question

I have seen this question earlier here and I have took lessons from that. However I am not sure why I am getting an error when I feel it should work.

I want to crea

Answer 1

There are different ways you can achieve if-then-else.

1. Using when function in DataFrame API. You can specify the list of conditions in when and also can specify otherwise what value you need. You can use this expression in nested form as well.

2. expr function. Using "expr" function you can pass SQL expression in expr. PFB example. Here we are creating new column "quarter" based on month column.

cond = """case when month > 9 then 'Q4'
            else case when month > 6 then 'Q3'
                else case when month > 3 then 'Q2'
                    else case when month > 0 then 'Q1'
                        end
                    end
                end
            end as quarter"""

newdf = df.withColumn("quarter", expr(cond))

3. selectExpr function. We can also use the variant of select function which can take SQL expression. PFB example.

    cond = """case when month > 9 then 'Q4'
                else case when month > 6 then 'Q3'
                    else case when month > 3 then 'Q2'
                        else case when month > 0 then 'Q1'
                            end
                        end
                    end
                end as quarter"""

    newdf = df.selectExpr("*", cond)