My main goal is to color elements differently in my plot. To do so, I manually added an extra column with the desired color for each category:
manually
category
ggplot(mtcars) + geom_point(aes(wt, mpg, size = disp, color = color)) + scale_color_identity(guide = 'legend')