I have a complex 3D scene that I need to display HTML elements on top of, based on a 3D coordinate. (I'm simply overlaying a div tag on top and positioning it with CSS.) However, I also need to partially hide it (e.g., making it transparent) when the 3D coordinate is obscured by a model (or phrased in another way, when it's not visible in the camera). These models may be have many hundreds of thousands of faces, and I need a way to find out if it's obscured that's fast enough to be run many times per second.
Currently, I am using Three.js's built-in raytracer, with the following code:
// pos = vector with (normalized) x, y coordinates on canvas
// dir = vector from camera to target point
const raycaster = new THREE.Raycaster();
const d = dir.length(); // distance to point
let intersects = false;
raycaster.setFromCamera(pos, camera);
const intersections = raycaster.intersectObject(modelObject, true);
if (intersections.length > 0 && intersections[0].distance < d)
intersects = true;
// if ray intersects at a point closer than d, then the target point is obscured
// otherwise it is visible
However, this is very slow (frame rate drops from 50 fps to 8 fps) on these complex models. I've been looking for better ways to do this, but so far I haven't found any that work well in this case.
Are there any better, more effective ways of finding out if a point is visible or obscured by models in the scene?
I am not aware of any really quick way, but you do have a few options. I don't know enough about three.js to tell you how to do it with that library, but speaking about WebGL in general...
If you can use WebGL 2.0, you can use occlusion queries. This boils down to
var query = gl.createQuery();
gl.beginQuery(gl.ANY_SAMPLES_PASSED, query);
// ... draw a small quad at the specified 3d position ...
gl.endQuery(gl.ANY_SAMPLES_PASSED);
// some time later, typically a few frames later (after a fraction of a second)
if (gl.getQueryParameter(query, gl.QUERY_RESULT_AVAILABLE))
{
gl.getQueryParameter(query, gl.QUERY_RESULT);
}
Note though, that the result of the query is only available a few frames later.
If WebGl 2.0 is not an option, then you should probably draw the scene to a framebuffer, where you attach your own texture to use in place of the normal z-buffer. There is an extension to use proper depth textures (more details here), but where that is not possible you could always fall back to drawing your scene with a fragment shader that outputs the depth of each pixel.
You can then use gl.ReadPixels() on the depth texture. Again, be aware of the latency for the GPU->CPU transfer, that's always going to be significant.
Having said all that, depending on what your DOM objects look like, it could be far easier and quicker to render your DOM objects into a texture and draw that texture using a quad as part of your 3d scene.
I will assume content of desired html tags are third party data such as image or iframe, and cannot be used with webgl, so it must be html tag and cannot be sprite.
There is cookbook for GPU computation. Repeat each time scene changes. Sorry I can't do it for Three.js (don't know the engine).
Stage 1, Construct image with tags visibility
Create array (and index) buffer containing size of the html tags, tag IDs (int numbers from 0) and tag positions.
Create renderbuffer and new WebGL program, that will be used to render into it. Shaders of this program will render simplified scene including "shadows of tags". Now the simplified algoritm for fragment shader is following: for any object render white color. Except for tag, render color based on tag ID.
If your current program has fog, transparent objects, height maps or some procedural logic, it might be also contained in shaders (depends if it can cover tag or not).
Result could look like this (but it is not important):
If color is different then white, there is tag. (Assuming I have 3 tags only, then my colors are #000000, #010000, #020000, which all look like black, but is not. )
Stage 2, Collect transparency data about tags in image
We need another WebGL program and renderbuffer. We will render points into renderbuffer, each point is one pixel large and is next to each other. Points represent tags. So we will need array buffer with tag positions (and tag IDs, but this can be deduced in shader). We also bind texture from previous stage.
Now the code of vertex shader will do following, based on tag ID attribute, it will set position of the point. Then it calculate transparency with texture lookup, pseudocode:
attribute vec3 tagPosition;
attribute float tagId;
float calculateTransparency(vec2 tagSize, vec2 tagPosition) {
float transparency = 0;
for(0-tagSize.x+tagPosition.x) {
for(0-tagSize.y+tagPosition.y) {
if(textureLookup == tagId) transparency++; // notice that texture lookup is used only for area where tag could be
}
}
return transparency/totalSize;
}
vec2 tagSize2d = calculateSize(tagPosition);
float transparency = calculateTransparency(tagSize2d, tagPosition.xy);
Point position and transparency will enter as varying to FS. FS will render some color based on transparency (for example white for full visible, black for invisible and shades of grey for partialy visible).
Result of this stage is image, where each pixel present one tag and color of the pixel is tag transparency. Depending on how many tags you have, some pixels might mean nothing and has clearColor value. Pixel position coresponds to tag ID.
Stage 3, read values with javascript
To read data back use readPixels (or might use texImage2D?). Simple way to do it.
Then you use forloop based on tag IDs and write data from typed array to your javascript state machine for example. Now you have transparency values in javascript and you can change CSS values.
Ideas
In stage 1, reducing size of renderbuffer will cause significant performance boost (it lowers also texture lookups in stage 2) with almost zero cost.
If you use readPixels directly after stage 1 and try to read data from screen with javascript, even if you use renderbuffer only 320*200px large, js have to do as many iterations as the resolution is. So in case scene will change every moment, then just empty forloop:
var time = Date.now();
for(var i=0;i<320*200*60;i++) { // 64000*60 times per second
}
console.log(Date.now() - time);
tooks ~4100ms on my machine. But with stage 2, you must do only as many iterations as you have tags in visible area. (For 50 tags it might be 3000*60).
The biggest problem I see is complexity of the implementation.
The bottleneck of this technique is readpixels and texture lookups. You might consider to not call stage 3 at FPS speed, but at slower predefined speed instead.
Assuming your div positioning is synchronized with the underlying 3D scene, you should be able to query just one pixel with readPixels "underneath" your div.
Again assuming you are in control of the geometry, wouldn't it be a feasible hack to just add an "out of context" color ( or alpha value ) in the texture where the div will cover and test against it?
In case of no textures, modify the geometry to "surround" a single vertex within the boundaries of the overlying div and give it an equivalent "out of context" color or alpha value, to supply to the fragment shader.
Here in this answer you find a nice example using the THREE.Frustum to detect whether objects are visible:
var frustum = new THREE.Frustum();
var cameraViewProjectionMatrix = new THREE.Matrix4();
camera.updateMatrixWorld();
camera.matrixWorldInverse.getInverse( camera.matrixWorld );
cameraViewProjectionMatrix.multiplyMatrices( camera.projectionMatrix, camera.matrixWorldInverse );
frustum.setFromMatrix( cameraViewProjectionMatrix );
visible = frustum.intersectsObject( object );
Not sure if this will give you the performance you are after though. Maybe you can test to see how well it works and leave a comment with your findings for others who end up here looking for a similar solution.
来源：https://stackoverflow.com/questions/36745711/how-to-quickly-find-if-a-point-is-obscured-in-a-complex-scene