Can this be done faster with numpy?

Question

There\'s a color image, a numpy array of shape (h,w,3) with N=h*w pixels; there\'s an array labels of shape