How do I find my mouse point in a scene using SceneKit?

Question

I have set up a scene in SceneKit and have issued a hit-test to select an item. However, I want to be able to move that item along a plane in my scene. I continue to receive mouse drag events, but don't know how to transform those 2D coordinates into 3D coordinate in the scene.

My case is very simple. The camera is located at 0, 0, 50 and pointed at 0, 0, 0. I just want to drag my object along the z-plane with a z-value of 0.

The hit-test works like a charm, but how do I translate the mouse point from a drag event into a new position in the scene for the 3D object I am dragging?

Answer 1

You don't need to use invisible geometry — Scene Kit can do all the coordinate conversions you need without having to hit test invisible objects. Basically you need to do the same thing you would in a 2D drawing app for moving an object: find the offset between the mouseDown: location and the object position, then for each mouseMoved:, add that offset to the new mouse location to set the object's new position.

Here's an approach you could use...

1. Hit-test the initial click location as you're already doing. This gets you an SCNHitTestResult object identifying the node you want to move, right?

2. Check the worldCoordinates property of that hit test result. If the node you want to move is a child of the scene's rootNode, these is the vector you want for finding the offset. (Otherwise you'll need to convert it to the coordinate system of the parent of the node you want to move — see convertPosition:toNode: or convertPosition:fromNode:.)

3. You're going to need a reference depth for this point so you can compare mouseMoved: locations to it. Use projectPoint: to convert the vector you got in step 2 (a point in the 3D scene) back to screen space — this gets you a 3D vector whose x- and y-coordinates are a screen-space point and whose z-coordinate tells you the depth of that point relative to the clipping planes (0.0 is on the near plane, 1.0 is on the far plane). Hold onto this z-coordinate for use during mouseMoved:.

4. Subtract the position of the node you want to move from the mouse location vector you got in step 2. This gets you the offset of the mouse click from the object's position. Hold onto this vector — you'll need it until dragging ends.

5. On mouseMoved:, construct a new 3D vector from the screen coordinates of the new mouse location and the depth value you got in step 3. Then, convert this vector into scene coordinates using unprojectPoint: — this is the mouse location in your scene's 3D space (equivalent to the one you got from the hit test, but without needing to "hit" scene geometry).

6. Add the offset you got in step 3 to the new location you got in step 5 - this is the new position to move the node to. (Note: for live dragging to look right, you should make sure this position change isn't animated. By default the duration of the current SCNTransaction is zero, so you don't need to worry about this unless you've changed it already.)

(This is sort of off the top of my head, so you should probably double-check the relevant docs and headers. And you might be able to simplify this a bit with some math.)

Answer 2

As an experiment I implemented Mr Bishop's helpful answer. The drag doesn't quite work (the object - a chess piece - jumps off screen) because of differences in the coordinate magnitudes between the mouse click and the 3-D world. I've inserted log outputs here and there among the code.

I asked on the Apple forums if anyone knew the secret sauce to homogenize the coordinates but didn't get a decisive answer. One thing, I had made some experimental changes to Mr Bishop's method and the forum members advised me to return to his technique.

Despite my code's failings, I thought someone might find it a useful starting point. I suspect there are only one or two small problems with the code.

Note that the log of the world transform matrix of the object (chess piece) is not part of the process but one Apple forum member advised me that the matrix often offers a useful 'sanity check' - which indeed it did.

- (NSPoint)
viewPointForEvent: (NSEvent *) event_
{
    NSPoint   windowPoint    = [event_ locationInWindow];
    NSPoint   viewPoint        = [self.view convertPoint: windowPoint
                                             fromView: nil];
    return viewPoint;
}

- (SCNHitTestResult *)
hitTestResultForEvent: (NSEvent *) event_
{
    NSPoint      viewPoint        = [self viewPointForEvent: event_];
    CGPoint      cgPoint        = CGPointMake (viewPoint.x, viewPoint.y);
    NSArray * points        = [(SCNView *) self.view hitTest: cgPoint
                                                     options: @{}];
    return points.firstObject;
}

- (void)
mouseDown: (NSEvent *) theEvent
{
    SCNHitTestResult * result = [self hitTestResultForEvent: theEvent];

    SCNVector3 clickWorldCoordinates = result.worldCoordinates;
    log output: clickWorldCoordinates x 208.124578, y -12827.223365, z 3163.659073
    SCNVector3 screenCoordinates = [(SCNView *) self.view projectPoint: clickWorldCoordinates];
    log output: screenCoordinates x 245.128906, y 149.335938, z 0.985565
    // save the z coordinate for use in mouseDragged
    mouseDownClickOnObjectZCoordinate = screenCoordinates.z;

    selectedPiece = result.node;  // save selected piece for use in mouseDragged

    SCNVector3    piecePosition = selectedPiece.position;
    log output: piecePosition x -18.200000, y 6.483060, z 2.350000

    offsetOfMouseClickFromPiece.x = clickWorldCoordinates.x - piecePosition.x;
    offsetOfMouseClickFromPiece.y = clickWorldCoordinates.y - piecePosition.y;
    offsetOfMouseClickFromPiece.z = clickWorldCoordinates.z - piecePosition.z;
    log output: offsetOfMouseClickFromPiece x 226.324578, y -12833.706425, z 3161.309073  
}

- (void)
mouseDragged: (NSEvent *) theEvent;
{
    NSPoint   viewClickPoint        = [self viewPointForEvent: theEvent];

    SCNVector3 clickCoordinates;
    clickCoordinates.x = viewClickPoint.x;
    clickCoordinates.y = viewClickPoint.y;
    clickCoordinates.z = mouseDownClickOnObjectZCoordinate;
    log output:  clickCoordinates x 246.128906, y 0.000000, z 0.985565

    log output:  pieceWorldTransform: 
      m11 = 242.15889219510001, m12 = -0.000045609300002524833, m13 = -0.00000721691076126, m14 = 0, 
      m21 = 0.0000072168760805499971, m22 = -0.000039452697396149999, m23 = 242.15890446329999, m24 = 0, 
      m31 = -0.000045609300002524833, m32 = -242.15889219510001, m33 = -0.000039452676995750002, m34 = 0, 
      m41 = -4268.2349924762348, m42 = -12724.050221935429, m43 = 4852.6652710104272, m44 = 1)

    SCNVector3 newPiecePosition;
    newPiecePosition.x = offsetOfMouseClickFromPiece.x + clickCoordinates.x;
    newPiecePosition.y = offsetOfMouseClickFromPiece.y + clickCoordinates.y;
    newPiecePosition.z = offsetOfMouseClickFromPiece.z + clickCoordinates.z;
    log output: newPiecePosition x 472.453484, y -12833.706425, z 3162.294639

    selectedPiece.position = newPiecePosition;
}

Answer 3

I used the code written by Steve and with little modification it worked for me.

On mouseDown I save clickWorldCoordinates on a property called startClickWorldCoordinates.

On mouseDragged I calculate the selectedPiece position in this way:

SCNVector3 worldClickCoordinate = [(SCNView *) self.view unprojectPoint:clickCoordinates.x];

newPiecePosition.x = selectedPiece.position.x + worldClickCoordinate.x - startClickWorldCoordinates.x;
newPiecePosition.y = selectedPiece.position.y + worldClickCoordinate.y - startClickWorldCoordinates.y;
newPiecePosition.z = selectedPiece.position.z + worldClickCoordinate.z - startClickWorldCoordinates.z;

selectedPiece.position = newPiecePosition;

startClickWorldCoordinates = worldClickCoordinate;