Are there any open source, high level tools that would facilitate and simplify development of experimental network protocols (TCP/UDP) using a GUI?
Basically, something
If you are going to aim for an eventually python target anyway, who not roll your own test rig (in python) from the start?
State machines aren't that hard to build in code, and their a heck of a lot easier to read unambiguously and verify than pictures in a GUI, especially when they get complex.
It sounds to me you'd do better with an embedded DSL than a GUI based designer.
Edit: If, as you said in the comments, you just threw python out as an example, I'd suggest you go with timday's answer. If you're not actually worried about embedding in python, go with a pre-existing DSL as he suggests.