I am considering helping an open-source gaming project with creating their setup file. The game is currently weights around 300 MB and is packaged in a zip file, playable st
Not in the light category... but may help you chose between the light ones .. InstallShield Multiplatform (or its more recent cousins) will give you the benefit of an entire language (Java) to configure and deploy your applicaiton
Benefits
Drawbacks
I haven't tried some of the above suggestions but after struggling with arcane install issues for many years this one finally let me forget I had an installer... Which is the most important feature by far.
For Windows use InnoSetup (in conjunction with Inno Setup Form Designer, that will make Form creation easier)
BitRock InstallBuilder is a multiplatform installation tool that supports all your requirements and is already used by multiple gaming/3D companies like GarageGames, Newtek or DAZ3D. We are not open source but we provide free licenses to projects like yours, just drop us an email support/at/bitrock.com
For Windows, I've always used the Nullsoft Installer (http://nsis.sourceforge.net/Main_Page). I created my first setup.exe within about 5 minutes using one of the templates they have, and from there, it was quite easy to add installation types, extra pages (license, etc), and other "features".
On Linux, YoFrankie has been using Zero Install. That's LGPL and also supports checking for updates.
You can either let users install over the web, or run 0export to create a self-extracting archive. 300 MB is quite big... would it be useful to split it into several packages (so that e.g. updates to the core engine don't require users to re-download all the level data)? Zero Install can handle dependencies like that.
Some other options can be found in the Management & Distribution section on gnomefiles.
For OSX, this looks nice for making DMG's:
http://www.araelium.com/dmgcanvas/
Catch is that it costs $15 or so.