Server thread inside a qt class (need mutex?)

Question

I made this server class that starts a thread when new connection comes in. It works ok with some cases, but it\'s not very stable. I am trying to solve where it breaks. My

Answer 1

The code you present is exemplary of cargo cult coding: you do various unnecessary things, apparently in hopes of fixing the problem.

The Likely Crasher ...

There are tons of problems with the code, but I think the cause of the crash is this: tcpSocket.write(block) does not send out a zero-terminated string down the wire. The block is zero-terminated, but the assignment to a byte array does not add this zero termination to the size() of QByteArray. The following code prints 1, even though there is a zero terminating byte internally in the contents of the byte array.

QByteArray arr = "a";
qDebug("len=%d", arr.size());

The receiving code expects the zero termination, but never receives it. You then proceed to assign a non-zero-terminated buffer to std::string:

string instring;
instring = inbuffer;
instring.resize(joj);

The subsequent resize is cargo cult: you're trying to fix the problem after std::string & std::string::operator=(const char*) has already read past your buffer, in all likelihood.

Do not take this to mean that fixing just that is the right way to proceed. Not at all. The right way to proceed is to delete the code you wrote and do it right, without a ton of unnecessary incantations that don't help.

... and All The Other Problems

You've fallen into the trap of believing in magic, perpetuated endlessly in various forums.

The threads are not magical objects that you can just apply to any problem out there in hopes that they help. I don't know what makes people think that threads are magical, but the rule of thumb is: If someone tells you "oh, you should try threads", they are most likely wrong. If they tell that in relation to networking, they are pretty much never right, they are unhelpful, and they don't understand your problem at all (neither do you, it seems). More often than not, threads will not help unless you clearly understand your problem. Qt's networking system is asynchronous: it doesn't block the execution of your code, if you don't use the waitxxxx() functions. You shouldn't use them, by the way, so all is good here. No need for a bazillion threads.

So, it is completely unnecessary to start a new thread per each incoming connection. It will decrease the performance of your server -- especially if the server does simple processing because you add the overhead of context switching and thread creation/dismantling to each connection. You want less than 2 threads per each core in your system, so using QThread::idealThreadCount() for the number of threads in the pool would be a good starting point.

You are also depriving yourself of the benefit of threading since you use the networking thread only to receive the data, and you then send out a fromThreadString(string) signal. I presume that signal is sent to your application's main thread. Now that's just silly, because receiving a bunch of bytes from a network socket is downright trivial. Your threads don't do any work, all the work they do is wasted on their creation and removal.

The code below is a simple example of how one might correctly use the Qt APIs to implement a client-server system that distributes work across the physical cores in a round-robin fashion. It should perform quite well. The Fortune client example included in Qt is very unfortunate indeed, because it's precisely the wrong way to go about things.

What one will notice is:

1. It's not entirely trivial. Qt could be more helpful, but isn't.

2. Both the clients and the senders are moved into threads from a thread pool.

3. Disconnected clients are not deleted, but merely returned to a list of clients kept by the tread pool. They are reused when a client is called for.

4. QThread is not derived from. QTcpServer is only derived to access the socket handle.

5. No functions whose name begins with wait() are used. Everything is handled asynchronously.

6. The ThreadPool keeps a looked-up QMetaMethod for the newConnection(int) slot of the Client. This is faster than using QMetaObject::invokeMethod() as it has to look things up every time.

7. A timer running in the main thread sets off a signal-slot chain by deleting the first sender. Each senders' deletion triggers the deletion of the next one. Eventually, the last sender sets off the quit() slot in the thread pool. The latter emits the finished() signal when all threads are indeed finished.

#include 
#include 
#include 
#include 
#include 
#include 
#include 

// Processes data on a socket connection
class Client : public QObject
{
    Q_OBJECT
public:
    Client(QObject* parent = 0) : QObject(parent), socket(new QTcpSocket(this))
    {
        connect(socket, SIGNAL(readyRead()), SLOT(newData()));
        connect(socket, SIGNAL(stateChanged(QAbstractSocket::SocketState)),
                SLOT(newState(QAbstractSocket::SocketState)));
        qDebug("Client()");
    }
    ~Client() { qDebug("~Client()"); }
signals:
    void done();
public slots:
    void newConnection(int descriptor) {
        socket->setSocketDescriptor(descriptor);
    }
private slots:
    void newData() {
        QByteArray data = socket->readAll();
        if (0) qDebug("got %d bytes", data.size());
        if (0) qDebug("got a string %s", data.constData());
        // here we can process the data
    }
    void newState(QAbstractSocket::SocketState state) {
        qDebug("client new state %d", state);
        if (state == QAbstractSocket::UnconnectedState) { emit done(); }
    }
protected:
    QTcpSocket* socket;
    int descriptor;
};

// Connects to a client and sends data to it
class Sender : public QObject
{
    Q_OBJECT
public:
    Sender(const QString & address, quint16 port, QObject * parent = 0) :
        QObject(parent), socket(new QTcpSocket(this)),
        bytesInFlight(0), maxBytesInFlight(65536*8)
    {
        connect(socket, SIGNAL(stateChanged(QAbstractSocket::SocketState)),
                SLOT(newState(QAbstractSocket::SocketState)));
        connect(socket, SIGNAL(bytesWritten(qint64)), SLOT(sentData(qint64)));
        socket->connectToHost(address, port);
        qDebug("Sender()");
    }
    ~Sender() { qDebug("~Sender()"); }
protected:
    // sends enough data to keep a maximum number of bytes in flight
    void sendData() {
        qint64 n = maxBytesInFlight - bytesInFlight;
        if (n <= 0) return;
        bytesInFlight += n;
        socket->write(QByteArray(n, 44)); // 44 is the answer, after all
    }
protected slots:
    void sentData(qint64 n) {
        bytesInFlight -= n;
        Q_ASSERT(bytesInFlight >= 0);
        sendData();
    }
    void newState(QAbstractSocket::SocketState state) {
        qDebug("sender new state %d", state);
        if (state == QAbstractSocket::ConnectedState) sendData();
    }
protected:
    QTcpSocket* socket;
    qint64 bytesInFlight;
    qint64 maxBytesInFlight;
};

// Keeps track of threads and client objects
class ThreadPool : public QTcpServer
{
    Q_OBJECT
public:
    ThreadPool(QObject* parent = 0) : QTcpServer(parent), nextThread(0) {
        for (int i=0; i < QThread::idealThreadCount(); ++i) {
            QThread * thread = new QThread(this);
            connect(thread, SIGNAL(finished()), SLOT(threadDone()));
            thread->start();
            threads << thread;
        }
        const QMetaObject & mo = Client::staticMetaObject;
        int idx = mo.indexOfMethod("newConnection(int)");
        Q_ASSERT(idx>=0);
        method = mo.method(idx);
    }
    void poolObject(QObject* obj) const {
        if (nextThread >= threads.count()) nextThread = 0;
        QThread* thread = threads.at(nextThread);
        obj->moveToThread(thread);
    }
protected:
    void incomingConnection(int descriptor) {
        Client * client;
        if (threads.isEmpty()) return;
        if (! clients.isEmpty()) {
            client = clients.dequeue();
        } else {
            client = new Client();
            connect(client, SIGNAL(done()), SLOT(clientDone()));
        }
        poolObject(client);
        method.invoke(client, Q_ARG(int, descriptor));
    }
signals:
    void finished();
public slots:
    void quit() {
        foreach (QThread * thread, threads) { thread->quit(); }
    }
private slots:
    void clientDone() {
        clients.removeAll(qobject_cast(sender()));
    }
    void threadDone() {
        QThread * thread = qobject_cast(sender());
        if (threads.removeAll(thread)) delete thread;
        if (threads.isEmpty()) emit finished();
    }
private:
    QList threads;
    QQueue clients;
    QMetaMethod method;
    mutable int nextThread;
};

int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);
    ThreadPool server;
    if (!server.listen(QHostAddress::Any, 1101)) qCritical("cannot establish a listening server");
    const int senderCount = 10;
    Sender *prevSender = 0, *firstSender = 0;
    for (int i = 0; i < senderCount; ++ i) {
        Sender * sender = new Sender("localhost", server.serverPort());
        server.poolObject(sender);
        if (!firstSender) firstSender = sender;
        if (prevSender) sender->connect(prevSender, SIGNAL(destroyed()), SLOT(deleteLater()));
        prevSender = sender;
    }
    QTimer::singleShot(3000, firstSender, SLOT(deleteLater())); // run for 3s
    server.connect(prevSender, SIGNAL(destroyed()), SLOT(quit()));
    qApp->connect(&server, SIGNAL(finished()), SLOT(quit()));
    // Deletion chain: timeout deletes first sender, then subsequent senders are deleted,
    // finally the last sender tells the thread pool to quit. Finally, the thread pool
    // quits the application.
    return a.exec();
}

#include "main.moc"

Given your explanation, you game engine starts up and creates a connection to some port on localhost. Your Qt program is supposed to accept that connection on port 1101, receive some strings, process them, then send them back.

The code is modified to accept the connection on a fixed port number. All of the data processing, including sending the response back, has to be done from the newData() slot. You can also pass that data off to a different thread, if your computations are very complex. By complex I mean tens of thousands of operations like additions and multiplications, or thousands of trig operations.

The Sender class is there just as an example. Your game engine does the sending, of course, so you don't need the Sender class.