RPC can't decode arguments for TCP transport

Question

I\'m working on a multithreaded RPC server based on the example from this page: http://bderzhavets.blogspot.ca/2005/11/multithreaded-rpc-server-in-white-box.html

Un

Answer 1

You've got a few issues.

From the xen page, when it does the pthread_create in square_prog_2, it first calls pthread_attr_setdetachstate, but it needs to do pthread_attr_init before that. Also, attr appears to be static/global--put it in the function's stack frame.

square_prog_2 gets two args: rqstp and transp. These get saved into a malloc'ed struct data_str [so each thread has their own copy]. But, I wonder what the rqstp and transp values are (e.g. printf("%p")). They need to different or each thread will collide with each other when trying to use them [thus needing pthread_mutex_lock]. The malloc doesn't clone rqstp/transp so if they are the same, that's the issue because you may have two threads trying to riff on the same buffers simultaneously.

There is a return code of 11. Barring some special code, that looks suspiciously like SIGSEGV on a thread. This would be completely accounted for by the rqstp/transp overlap.

You may need to rearchitect this as I suspect XDR is not thread safe--nor should it need to be. Also, I don't think svc_* is thread safe/aware.

Start single threaded. As a test, have square_prog_2 call serv_request directly (e.g. do not do pthread_*). I bet that works in all modes.

If so, hold onto your hat--the example code using threads is broken--full of race conditions and will segfault, etc. If you're not hung up on using threads (no need for such a light duty task as x * x), you can just enjoy as is.

Otherwise, the solution is a bit more sophisticated. The main thread must do all the access to the socket and all XDR parsing/encoding. It can't use svc_run--you have to roll your own. The child can only do the actual work (e.g. x * x) and may not touch the socket/req/transp, etc.

Main thread:

while (1) {
    if (svc_getreq_poll()) {
        // parse XDR
        // create data/return struct for child thread
        // create thread
        // add struct to list of "in-flight" requests
    }

    forall struct in inflight {
        if (reqdone) {
            // take result from struct
            // encode into XDR
            // do send_reply
            // remove struct from list
        }
    }
}

For the child struct it would look like:

struct child_struct {
    int num;
    int num_squared;
};

And the child's thread function becomes a one liner:ptr->num_squared = ptr->num * ptr->num

UPDATE: Multithread RPC servers appear to not be supported under Linux or FreeBSD

Here's a document: https://www.redhat.com/archives/redhat-list/2004-June/msg00439.html This has a cleaner example to start from.

From that: Remember -A option of rpcgen is not supported under Linux. Library calls providing by SunOS RPC to build Multithreaded RPC Server are unavailable under Linux as well

Here's the Linux rpcgen man page: http://linux.die.net/man/1/rpcgen No mention of -M. IMO, this means the rpcgen program has the option and does generate the stubs, but the underlying support is not there, so they left it out of the doc.

Here's the FreeBSD man page [and the reason why there's no support]: http://www.freebsd.org/cgi/man.cgi?query=rpcgen&sektion=1&manpath=FreeBSD+5.0-RELEASE See the doc for -M within this:

M -- Generate multithread-safe stubs for passing arguments and results between rpcgen generated code and user written code. This option is useful for users who want to use threads in their code. However, the rpc_svc_calls(3) functions are not yet MT-safe, which means that rpcgen generated server-side code will not be MT-safe.

An alternate way:

Why bother with RPC/XDR at all? The overhead is huge for the large arrays you intend to use. Most of the standard uses are for things like yellow pages, that don't have much data.

Most systems are little endian these days. Just blast the native buffer to a socket that you open directly. On the server, have a daemon do a listen, then fork a child and have the child do the accept, read in the data, do the calculations, and send back the reply. At worst, the child will need to do an endian swap but that's easily done in a tight loop using bswap_32.

A simple little control struct at the beginning of each message in either direction that prefixes the data payload:

struct msgcontrol {
    int what_i_am;
    int operation_to_perform;
    int payload_length;
    int payload[0];
};

A special note: I've done this commercially before (e.g. MPI and roll my own) and you may have to issue setsockopt calls to increase the size of the kernel socket buffer to something large enough to sustain a barrage of data

Actually, now that I think of it, if you don't want to roll your own, MPI may be of interest. However, having used it, I'm not a true fan. It had unexpected problems and we had to remove it in favor of controlling our sockets directly.