I\'m looking to setup video uploads for users on a site and want to have them viewed through a Flash player. The site is already partially built (by someone else) and I\'m w
ffmpeg is the tool for you. It's a major opensource video encoding library that a lot of other tools are based on. It's a bit tricky to use directly, but I think there are a few wrappers around.
There is this highly underrated post with 3 great links for open source solutions that are like youtube and fits perfectly your question (and at least another one).
I actually found that using other services to encode the file is the easiest option and also the fastest to setup. I ended up using Hey!Watch for this project. I haven't worried about a streaming server/service yet because for the most part the videos are short and people won't be skipping around within the video.
Just FYI: If you have HQ videos you need not to encode them into FLV. Flash Player 10 has builtin H264 support, this is what HQ does in popular video sites.
Use the YouTube API and let them do the heavy lifting for you.