How to split Vector into columns - using PySpark

Question

Context: I have a DataFrame with 2 columns: word and vector. Where the column type of \"vector\" is VectorUDT.

An Example:

Answer 1

For anyone trying to split the rawPrediction or probability columns generated after training a PySpark ML model into Pandas columns, you can split like this:

your_pandas_df['probability'].apply(lambda x: pd.Series(x.toArray()))