Use OpenSSL with a custom channel

Question

I developed (in CPP) a unique protocol over HTTP and I use it to communicate with my server. Now, I want to establish SSL connection over my proprietary protocol to transfer the

Answer 1

Second option - a protocol that has its own messages and uses HTTP to pass them between the client and the server.

If you're using HTTP to pass your own messages, using OpenSSL for SSL/TLS would imply that you'd need to write your own HTTP library library too.

Instead, use an HTTP library that supports HTTPS (most do), via OpenSSL or not. Exchanging your custom messages on top of HTTPS should be fairly transparent and similar to using plain HTTP. You'd just need to configure HTTPS normally.