Are there any open source, high level tools that would facilitate and simplify development of experimental network protocols (TCP/UDP) using a GUI?
Basically, something
Give up the GUI for a text editor and enter the world of protocol specification languages. Most of these tools take a description of some protocol, try to prove it's not broken, and generate an implementation and tests. Here's a few to get you started, but there's many more:
If you are going to aim for an eventually python target anyway, who not roll your own test rig (in python) from the start?
State machines aren't that hard to build in code, and their a heck of a lot easier to read unambiguously and verify than pictures in a GUI, especially when they get complex.
It sounds to me you'd do better with an embedded DSL than a GUI based designer.
Edit: If, as you said in the comments, you just threw python out as an example, I'd suggest you go with timday's answer. If you're not actually worried about embedding in python, go with a pre-existing DSL as he suggests.