Is there a simpler way to achieve this style of user messaging?

Question

I have created a messaging system for users, it allows them to send a message to another user. If it is the first time they have spoken then a new conversation is initiated,

Answer 1

Extending the answer suggested by Watcher.

You should consider dropping the "conversation" concept to simplify further.

+----+---------+------+------------------+--------+----------+
| id | message | read | time             | toUser | fromUser |
+----+---------+------+------------------+--------+----------+
| 1  |  test 1 |  0   | (some timestamp) |  3     |   4      |
| 2  |  test 2 |  0   | (some timestamp) |  4     |   3      |
+----+---------+------+------------------+--------+----------+

List of all conversations for user 123:

SELECT * FROM (
    SELECT id, message, toUser, fromUser   
    FROM userMessages 
    WHERE toUser = 123 OR fromUser = 123 
    ORDER BY id DESC
) AS internalTable 
GROUP BY toUser, fromUser 

List entire conversation between user 123 and user 456:

SELECT * 
FROM userMessages
WHERE (toUser = 123 OR fromUser = 123) 
AND (toUser = 456 OR fromUser = 456)
ORDER BY time DESC