Sending structured data over a network [closed]

问题

I'm a begginer in network programming, so, sorry if my questions may appear a little obvious.

I'm trying to send some data from Qt application to a Python server which will process them and send back some answer.

the methods that allows me to send data in the QTcpSocket class are:

// ...
write(const QByteArray &)
write(const char *)
// ...

my application will manage: authentification, sending and receiving some complexe data like struct, and files.

I've many questions about this situation:

Are the methods mentioned above sufficient to send complexe data, and how ?
how to deal with the data types in the server side (with Python) ?
do you think I should use an other protocole like HTTP (with the QNetworkAccessManager class) ?

回答1:

Trying to answer your questions:

Are the methods mentioned above sufficient to send complexe data, and how ?

Well, yes, sending a raw byte array is the lowest level format. However, you need a something that can get you uniquely from your complex data to the byte array and from the byte array back to your complex data.

This process is called in different ways, encoding, serializing, marshalling.... But in general it just means creating a system for encoding complex structures into a sequence of bytes or characters

There are many you can choose from: ASN.1, JSON, XML, Google's protocol buffers or MIME....

You can even design your own (e.g. a simple schema is using TLV: (Tag-Length-Value), where Tag is an identifier of the Type and Value can be a basic type [and you have to define a representation for each type that you consider basic] or again one or more TLV), Length indicates how many bytes/characters are used to encode the Value.

What to choose depends a lot of where you encode (language/platform) and where you decode (language/platform) and your requirements for speed, bandwidth usage, transport, whether messages should be inspected... etc.

If you're dealing with heterogenous architectures you might need to think about endianness.

Finally, you should distinguish between the format (i.e. how the complex structure is expressed as a sequence of bytes in the line) and the library used for encoding (or the library used for decoding). Sometimes they will be linked, sometimes for the same format you will have a choice of libraries to use.

how to deal with the data types in the server side (with Python) ?

So, here you have a requirement... if you're going for a externally provided format, you must make sure it has a python library able to decode it.

if you're going for a home-grown solution, one of things you should define is the expression of your complex C++ structures as Python structures.

An additional possibility is to do everything in C++, and for the python server side use one of the systems for creating python extensions in C++ (e.g. boost-python or swig....)

do you think I should use an other protocol like HTTP (with the QNetworkAccessManager class) ?

It depends on what you try to do. There are many HTTP libraries widely available that you can use on different languages and different architectures.

You still need to solve the problem of deciding the formatting of your information (although HTTP have some defined practices).

In addition HTTP is clearly biased towards the communication of a client with a server, with the action always initiated by the client.

Things get complex (or less widely supported) when is the server the one that needs to initiate the communication or the one that needs to send spontaneous information.

回答2:

I do not think it is the language data structure type that should be differentiated, but it is more about the data you send over. Note that, different languages may have different language structures and so on. That is just really low-level details. What is more important is what you send.

You could look into the following example how the serialization/deserialization works with json format in QtCore. Json is also supported in python quite well by the json module, so you would have no issue on the server side to deserialize it:

JSON Save Game Example

This is basically the important part that would give you some hint on the client side. Do not get lost at saving into a file. It is basically writing the raw bytes to the file, which you would replace by sending over the network:

void Game::write(QJsonObject &json) const
{
    QJsonObject playerObject;
    mPlayer.write(playerObject);
    json["player"] = playerObject;

    QJsonArray levelArray;
    foreach (const Level level, mLevels) {
        QJsonObject levelObject;
        level.write(levelObject);
        levelArray.append(levelObject);
    }
    json["levels"] = levelArray;
}

... and then you would do something like this on the server side, again instead of reading from file, you would read from the network, but that is not a biggie as both are IO.

import json
json_data=open(file_directory).read()

data = json.loads(json_data)
pprint(data)

You could use raw protocol to design your own, or just use an extending. I would suggest to go with something standard, like http (tcp/udp). Then, you would only need to define the json format for your own data, and not deal with all the rest, like one-way or two-way communication, transaction identifier against reply attack, timestamp, data size and so on.

This would allow you to truly concentrate on the important stuff for you. Once, you have your own json format defined, you could look into the QtNetwork module to send post, get, put and delete requests as you wish.

You would probably work closely with the QNetworkManager, QNetworkReply classes, and so on. Here you can find a simple client implementation in Qt with QtCore's json for a simple pastebin functionality:

#include <QSslError>
#include <QNetworkAccessManager>
#include <QNetworkRequest>
#include <QNetworkReply>
#include <QTcpSocket>

#include <QJsonDocument>
#include <QJsonObject>
#include <QJsonParseError>
#include <QFile>
#include <QScopedPointer>
#include <QTextStream>
#include <QStringList>
#include <QCoreApplication>

#include <QDebug>

int main(int argc, char **argv)
{
    QCoreApplication application{argc, argv};
    application.setOrganizationName(R"("CutePaste")");
    application.setApplicationName(R"("CutePaste Desktop Console Frontend")");

    QTextStream standardOutputStream{stdout};
    QFile dataFile;
    QString firstArgument{QCoreApplication::arguments().size() < 2 ? QString() : QCoreApplication::arguments().at(1)};
    if (!firstArgument.isEmpty()) {
        dataFile.setFileName(firstArgument);
        dataFile.open(QIODevice::ReadOnly);
    } else {
        dataFile.open(stdin, QIODevice::ReadOnly);
    }

    QByteArray pasteTextByteArray{dataFile.readAll()};

    QJsonObject requestJsonObject;
    requestJsonObject.insert(QStringLiteral("data"), QString::fromUtf8(pasteTextByteArray));
    requestJsonObject.insert(QStringLiteral("language"), QStringLiteral("text"));

    QJsonDocument requestJsonDocument{requestJsonObject};

    QString baseUrlString{QStringLiteral(R"("http://pastebin.kde.org")")};

    QNetworkRequest networkRequest;
    networkRequest.setAttribute(QNetworkRequest::DoNotBufferUploadDataAttribute, true);
    networkRequest.setHeader(QNetworkRequest::ContentTypeHeader, R"("application/json")");
    networkRequest.setUrl(QUrl(baseUrlString + R"("/api/json/create")"));

    QNetworkAccessManager networkAccessManager;
    QScopedPointer<QNetworkReply> networkReplyScopedPointer(networkAccessManager.post(networkRequest, requestJsonDocument.toJson()));
    QObject::connect(networkReplyScopedPointer.data(), &QNetworkReply::finished, [&] {

        QJsonParseError jsonParseError;
        QByteArray replyJsonByteArray{networkReplyScopedPointer->readAll()};
        QJsonDocument replyJsonDocument{QJsonDocument::fromJson(replyJsonByteArray, &jsonParseError)};
        if (jsonParseError.error != QJsonParseError::NoError) {
            qDebug() << R"("The json network reply is not valid json:")" << jsonParseError.errorString();
            QCoreApplication::quit();
        }

        if (!replyJsonDocument.isObject()) {
            qDebug() << R"("The json network reply is not an object")";
            QCoreApplication::quit();
        }

        QJsonObject replyJsonObject{replyJsonDocument.object()};
        QJsonValue resultValue{replyJsonObject.value(QStringLiteral("result"))};

        if (!resultValue.isObject()) {
            qDebug() << R"("The json network reply does not contain an object for the "result" key")";
            QCoreApplication::quit();
        }

        QJsonValue identifierValue{resultValue.toObject().value(QStringLiteral("id"))};

        if (!identifierValue.isString()) {
            qDebug() << R"("The json network reply does not contain a string for the "id" key")";
            QCoreApplication::quit();
        }

        endl(standardOutputStream << baseUrlString << '/' << identifierValue.toString());

        QCoreApplication::quit();
    });

    QObject::connect(networkReplyScopedPointer.data(), static_cast<void (QNetworkReply::*)(QNetworkReply::NetworkError)>(&QNetworkReply::error), [&](QNetworkReply::NetworkError networkReplyError) {
        if (networkReplyError != QNetworkReply::NoError)
            endl(standardOutputStream << networkReplyScopedPointer->errorString());
    });

    QObject::connect(networkReplyScopedPointer.data(), &QNetworkReply::sslErrors, [&](QList<QSslError> networkReplySslErrors) {
        if (!networkReplySslErrors.isEmpty()) {
            for (const auto &networkReplySslError : networkReplySslErrors)
                endl(standardOutputStream << networkReplySslError.errorString());
        }
    });

    int returnValue{application.exec()};

    dataFile.close();
    if (dataFile.error() != QFileDevice::NoError)
        endl(standardOutputStream << dataFile.errorString());

    return returnValue;
}

The JSON is defined in here:

http://sayakb.github.io/sticky-notes/pages/api/

For sure, it is not the only way of doing it, e.g. if you need efficiency, you may well look into a binary format like capnproto.

来源：https://stackoverflow.com/questions/23578456/sending-structured-data-over-a-network

标签

python

c++

qtcore

qtnetwork