I have a input dataframe: df_input (updated df_input)
|comment|inp_col|inp_val| |11 |a |a1 | |12 |a |a2 |
You can simply use regex_replace like this:
regex_replace
df.withColumn("new_col", regex_replace(col("inp_val"), "&", ""))