I have this dataframe path_df:
path_df.show() +---------------+-------------+----+ |FromComponentID|ToComponentID|Cost| +---------------+-------------+----+
Easiest way I know is the below (but has Pandas dependency):
path_df.toPandas().set_index('FromComponentID').T.to_dict('list')