Efficient Python IPC [closed]

Answer 1

I've used the Redis engine for this. Extremely simple, and lightweight.

Server side does:

import redis
r = redis.Redis() # Init
r.subscribe(['mychannel']) # Subscribe to "channel"
for x in r.listen():
  print "I got message",x

Client side does:

import redis
r = redis.Redis() # Init
r.publish('mychannel',mymessage)

"messages" are strings (of any size). If you need to pass complex data structures, I like to use json.loads and json.dumps to convert between python dicts/arrays and strings - "pickle" is perhaps the better way to do this for python-to-python communication, though JSON means "the other side" can be written in anything.

Now there are a billion other things Redis is good for - and they all inherently are just as simple.

Answer 2

The solution is dbus

It is a mature solution and availiable for a lot of languages (C, Python, ..., just google for dbus + your favorite language), though not as fast as shared memory, but still fast enough for pretty much everything not requiring (hard) realtime properties.

Answer 3

You are asking for a lot of things from the framework; network enabled, multi-platform, multi-language, high performance (which ideally should be further specified - what does it mean, bandwidth? latency? what is "good enough"; are we talking kB/s, MB/s, GB/s? 1 ms or 1000 ms round-trip?) Plus there are a lot of things not mentioned which can easily come into play, e.g. do you need authentication or encryption? Some frameworks give you such functionality, others rely on implementing that part of the puzzle yourself.

There probably exists no silver bullet product which is going to give you an ideal solution which optimizes all those requirements at the same time. As for the 'additional' component of your question - yes, if you restrict language requirements to python only, or further distinguish between key vs. nice-to-have requirements, there would be more solutions available.

One technology you might want to have a look at is Versile Python (full disclosure: I am one of the developers). It is multi-platform and supports python v2.6+/v3, and java SE6+. Regarding performance, it depends on what are your requirements. If you have any questions about the technology, just ask on the forum.

Answer 4

The three you mentioned seem a good fit and will uphold your requirements. I think you should go on with what you feel most comfortable\familiar with.

From my personal experience, I do believe ZeroMQ is the best combination between efficiency, ease of use and inter-operability. I had an easy time integrating zmq 2.2 with Python 2.7, so that would be my personal favorite. However as I said I'm quite sure you can't go wrong with all 3 frameworks.

Half related: Requirements tend to change with time, you may decide to switch framework later on, therefore encapsulating the dependency on the framework would be a good design pattern to use. (e.g. having a single conduit module that interacts with the framework and have its API use your internal datastructures and domain language)

Answer 5

I'll take a different tack here and say why not use the de facto RPC language of the Internet? I.e. HTTP REST APIs?

With Python Requests on the client side and Flask on the server side, you get these kinds of benefits:

• Existing HTTP REST tools like Postman can access and test your server.
• Those same tools can document your API.
• If you also use JSON, then you get a lot of tooling that works with that too.
• You get proven security practices and solutions (session based security and SSL).
• It's a familiar pattern for a lot of different developers.