Synchronization between command buffers in Vulkan

Question

There are several ways to handle synchronization in Vulkan. This is how I understand it:

• Fences are GPU to CPU syncs.
• Semaphores are GPU to GPU syncs, they are used to sync queue submissions (on the same or different queues).
• Events are more general, reset and checked on both CPU and GPU.
• Barriers are used for synchronization inside a command buffer.

In my case I have two command buffers. And I want the second command buffer to execute after the first one.

submitInfo.pCommandBuffers = &firstCommandBuffer;
vkQueueSubmit(queue, 1, &submitInfo, VK_NULL_HANDLE);

// wait for first command buffer to finish
submitInfo.pCommandBuffers = &secondCommandBuffer;
vkQueueSubmit(queue, 1, &submitInfo, VK_NULL_HANDLE);

What kind of synchronization is best for this? If I use vkQueueWaitIdle(queue)), is that the same thing as using a fence or should I use event or semaphores for this?

If I send multiple commandbuffer to the queue at the same time:

std::vector<VkCommandBuffer> submitCmdBuffers = {
        firstCommandBuffer,
        secondCommandBuffer
    };
    submitInfo.commandBufferCount = submitCmdBuffers.size();
    submitInfo.pCommandBuffers = submitCmdBuffers.data();

Is there still a way to synchronize between the first and the second one?

Answer 1

The first command buffer is rendering object with the depth test turned on. The second command buffer is rendering outlines of meshes with the depth test turned off. Because it has to be on top of the other objects.

For this case, what you need rather depends on what those command buffers are.

If those are secondary command buffers executed within the same render pass instance, then you don't need any synchornization. Not unless you are manually reading from the depth texture in the secondary command buffer. Why?

Because section 2.2.1's API Ordering protects you. Depth testing and depth writing within a render-pass instance will always proceed in API order. So later commands, whether in the same CB or a different one, will be ordered with regard to depth testing/writing.

However, if you need to read that depth buffer or your command buffers are in different render pass instances, then you need explicit synchronization via an event.

In this case, the stage mask for the vkCmdSetEvent command should be the stage that writes the depth value. This could be EARLY_FRAGMENT_TESTS_BIT or LATE_FRAGMENT_TESTS_BIT. To be safe, use both. However, since you're probably updating the same color buffer, you also need the COLOR_ATTACHMENT_OUTPUT_BIT stage. Insert this command at the end of the first command buffer (or after all the depth writing is done).

For the vkCmdWaitEvent, you want to wait on the pipeline stages that need it. In your case, this is again the fragment tests and color attachment. But if a shader stage is going to read the depth, you also need that stage in the wait command.

Since memory is involved, your vkCmdWaitEvent will also need to use a memory dependency on the depth and color buffers.

Really though, all of this complexity is why you should try to put these command buffers in the same render pass instance if at all possible. The only reason you would be unable to do so is if you needed to read from the depth buffer in a shader.

Answer 2

For your scenario you should use events. These should be the most light-weight synchronization objects to sync execution of two command buffers in a given order, even if you submit them at once. But note that events do not work across different queues. If you only use one, use events and try to keep src and dst pipeline stage masks as narrow as possible.

Semaphores are another way of synchronizing command buffer execution, but these only work on the queue submission, so they're more heavy-weight than events.