How to correctly send binary data over HTTPS POST?

Question

I send binary data from client (Debian 6.0.3) to server (Windows Server 2003). To bypass most firewalls I use HTTPS POST. Client and server are implemented usin

Answer 1

I was misled from the start with this problem. Narrowed it down to surprising details:

Sending over SSL socket fails if I use Boost.Asio multi-buffers in Boost v.1.48 (the most recent one at this moment). Example:

// data to send, protocol is [packet size: 4 bytes][packet: packet_size bytes]
std::vector<char> packet = ...;
uint32_t packet_size = packet.size();
// prepare buffers
boost::array<boost::asio::const_buffer, 2> bufs = {{boost::asio::buffer(&packet_size, sizeof(packet_size)), boost::asio::buffer(packet)}};
// send multi buffers by single call
boost::asio::async_write(socket, bufs, ...);

Sending separately packet_size and packet in this example works around the problem. I'm far from calling any suspicious behavior as a bug, especially if it's related with Boost libraries. But this one really looks like a bug. Tried on Boost v.1.47 - works fine. Tried with usual TCP socket (not SSL one) - works fine. The same on both Linux and Windows.

I'm going to find any reports about this problem in Asio mailing list and will report it if nothing found.

Answer 2

(EDIT: I had originally deleted this because I had realised it wasn't using HTTP really. Following a comment where you think you might have a MITM proxy and should use proper HTTP, I'm undeleting/editing.)

POST / HTTP/1.1
User-Agent: my custom client v.1

[binary data]

Whether it's binary data or not, in plain HTTP or with SSL/TLS, you'll need a Content-Length header or to use chunked transfer encoding. This this section of the HTTP spec. A Content-Type header would be useful too.

Chunked transfer encoding is for when you don't necessarily know the length of the stream. (You always need some form of delimiters when sending data, if only to detect reliably when it ends.)

This being said, you should be able to find out whether you're behind a MITM proxy that looks into the application layer on top of SSL/TLS if you get a certificate that's not your servers. If you do still get a successful handshake with your won server cert, there isn't such a proxy. Even an HTTP proxy would use CONNECT and relay everything, without altering the SSL/TLS connection (and thus without altering your original pseudo-HTTP on top).

Answer 3

If you don't have to operate in front of web server, you don't have to use HTTPS protocol. From the firewall point of view HTTPS looks like yet another SSL connection and it has no idea what going through. So if the only thing you need is just to pass the data - not to actual web server, use just SSL connection over 443 port.

So just troubleshoot your SSL connection the problem has nothing to do with HTTP.

---

If you want to use HTTP web server and not custom client:

Two points:

1. You need to specify Content-Length.
2. If you are using HTTP/1.1 you need to specify Host header.

The simplest would be

POST /url HTTP/1.0
User-Agent: my custom client v.1
Content-Type: application/octet-stream
Content-Length: NNN

Actual Content

Or for HTTP/1.1

POST /url HTTP/1.1
Host: www.example.com
User-Agent: my custom client v.1
Content-Type: application/octet-stream
Content-Length: NNN

Actual Content

Note: you can't send infinite data. HTTP protocol requires fixed content-lenght and most of the time web servers would load the data first before passing it to the backend.

So you will have to transfer data by chunks.