Use OpenSSL with a custom channel

Question

I developed (in CPP) a unique protocol over HTTP and I use it to communicate with my server. Now, I want to establish SSL connection over my proprietary protocol to transfer the

Answer 1

Use BIO pairs. You can find an example in the ssltest.c program, search the source for bio_pair. The basic idea is that you treat the OpenSSL engine as a black box.

There are four things your code has to do:

1. When you receive encrypted data over the connection to the other side, you have to write it to the SSL engine's encrypted BIO.

2. When the SSL engine wants to send encrypted data to the other side, you have to read it from the SSL engine's encrypted BIO and transport it to the other side.

3. When you have plaintext you want to encrypt and send, you have to write it to the SSL engine's plaintext BIO.

4. When the SSL engine has plaintext it has decrypted for you, you have to read it from the SSL engine's plaintext BIO.

OpenSSL acts purely as an engine following the SSL protocol and moving data between the two BIOs. It does all the protocol negotiation and operations for you, so long as you keep all four of these data streams moving.

One caution I can give you is this -- do not assume any special relationship between these things. For example, you might have some plaintext you want to encrypt and send, and when you write it to the SSL engine's plaintext BIO, the SSL engine might not be able to make forward progress until it receives some encrypted data from the other side. Treat the SSL engine as a black box and do all these four things whenever possible . Do not try to "look through" the SSL engine and, for example, expect that because you handed the SSL engine some encrypted data it will have necessarily plaintext for you. It might, but it might also need to send encrypted data to the other side.

One other caution: the SSL engine has only one state. It does not have a read state and a write state. (Search this thread for "the nightmare scenario" if you want the ugly details.) This is most likely to bite you if you use an SSL connection with multiple threads and expect it to behave just like a TCP connection (where the read and write sides are independent except in the case of a fatal error or connection close).

Answer 2

Second option - a protocol that has its own messages and uses HTTP to pass them between the client and the server.

If you're using HTTP to pass your own messages, using OpenSSL for SSL/TLS would imply that you'd need to write your own HTTP library library too.

Instead, use an HTTP library that supports HTTPS (most do), via OpenSSL or not. Exchanging your custom messages on top of HTTPS should be fairly transparent and similar to using plain HTTP. You'd just need to configure HTTPS normally.