Pyspark - Check column string against list of words and remove

Question

PySpark Version 2.4.3 AWS Glue

I have a list of words(patterns):

to_remove = [\'_test1"\',\'_test2\',\'_test3\']

and a DF like so