I want to create a Web app which would allow the user to upload some C code, and see the results of its execution (the code would be compiled on the server). The users are u
The few details you provide imply that you have administrative control over the server itself, so my suggestion makes this assumption.
I'd tackle this as a batch system. The web server accepts an upload of the source file, a process polls the submission directory, processes the file, and then submits the result to another directory which the web application polls until it finds the result and displays it.
The fun part is how to safely handle the execution.
My OS of choice is FreeBSD, so I'd set up a pre-configured jail (not to be confused with a vanilla chroot jail) that would compile, run, and save the output. Then, for each source file submission, launch a pristine copy of the jail for each execution, with a copy of the source file inside.
Provided that the jail's /dev is pruned down to almost nothing, system resource limits are set safely, and that the traffic can't route out of the jail (bound to unroutable address or simply firewalled), I would personally be comfortable running this on a server under my care.
Since you use Linux, I'd investigate User Mode Linux or Linux-VServer, which are very similar in concept to FreeBSD jails (I've never used them myself, but have read about them). There are several other such systems listed here.
This method is much more secure than a vanilla chroot jail, and it is much more light-weight than using full virtualization such as qemu/kvm or VMware.
I'm not a programmer, so I don't what kind of AJAX-y thing you could use to poll for the results, but I'm sure it could be done. As an admin, I would find this a fun project to partake in. Have fun. :)
ptrace-based confinement for untrusted programs can be used like the one described in http://www.cs.vu.nl/~rutger/publications/jailer.pdf, http://www.cs.vu.nl/~guido/mansion/publications/ps/secrypt07.pdf.
They have a change-root-ing policy rule, CHRDIR, whose effect is similar to chroot. (Section "The jailing policy")
However, they might have not published their source code (partially based on a modified strace http://www.liacs.nl/~wichert/strace/ -- Section "Implementation")...
See also other available ptrace-based approaches to chroot-in-userpace: https://unix.stackexchange.com/a/72697/4319
See this page on Google Chrome's sandboxing methods for Linux. As you can see, there are plenty of methods, but none of them are great for a distributable application like Chrome because some distros might not include them. This is not really a problem for a web application though, because you can control what is installed on your server.
Personally, my favorite is Seccomp, because it has a very low overhead compared to other tools like ptrace (switch address spaces on every syscall!) or KVM (big memory hungry virtual machine), and it is incredibly simple compared to tools like SELinux (and therefore more likely to be secure).
Spawning a new VM under KVM or qemu to compile and run the code looks like the way to go. Running the code under jail/LXC can compromise the machine if it exploits the unsecured parts of the OS like networking code. Advantage of running under a VM are obvious. One can only hack the VM but not the machine itself. But the side effect is you need lots of resources (CPU and Memory) to spawn a VM for each request.
Although it is still in development, and not yet considered secure, you might check out the technology behind Google Native Client. It is designed to allow untrusted native code to be run in a web browser, but could probably be adapted for use on a web server. You might use something like this on top of other techniques such as a virtual machine, for additional security.
Along with the other sugestions you might find this useful.
http://www.eelis.net/geordi/
This is from http://codepad.org/about, codepad.org's about page.