I have some objects in the scene, some may occlude others. When I click the mouse or drag-select to get a selection rectangle, I want to select/pick only the objects that I can
I decided to implement the selection using the back buffer. Here's my attempt to answer my questions:
If GL_SELECT is deprecated in recent OpenGL, what alternatives are developers supposed to be using?
I think it's best to not employ OpenGL to do this task but to use spatial acceleration structures as user chamber85 suggested in comments to the original question.
I've only ever read about occlusion queries being employed to speed up rendering. Can they be used for selection/picking, and are there drawbacks?
I'm sure they could but one would need to know all the objects they want to query for occlusion before the draw. Using back buffer and colour selection, one can just see what is under the cursor or within a rectangular region and filter from there.
The existing application has a handful of glColorXXX calls. If I went the back buffer route, and used glColorMask(FALSE,FALSE,FALSE,FALSE), will this effectively turn the glColorXXX calls into calls that have no effect, thereby letting me control the colour in a single place when rendering in select mode?
The answer is no. Calling glColorMask() with all GL_FALSE parameters will not mean that glColor3ub() calls (for example) will not be honoured. It simply specifies a filter/mask for colours just before they are written to the colour buffer. The original thought was to set the colour to the object id, then call glColorMask() to ignore all subsequent glColorXXX() calls. This strategy is doomed as the colour representing the object id would also be masked out.
4) Which route is best/canonical? I would say the back buffer colour selection is generally best as it doesn't require setting up the occlusion queries before/during the draw.