Restrict SwiftUI ColorPicker to only colors that work in light/dark mode

Question

Is there a way to restrict the ColorPicker to only show colors that work in light/dark mode?

Right now I only know of the default