Concurrent XMLRPC-C calls with global ServerProxy object not working as expected

老子叫甜甜 提交于 2019-12-13 02:36:24

问题


I am encountering a strange behaviour while sending XMLRPC requests to a server (XMLRPC-C). I intend to send several requests simultaneously with a pool of processes, as shown below. Each time I make a request I create a new ServerProxy object. I know that this leads to poor performance, instead I should be creating only one object for each process and then reusing those objects for subsequent calls. But the issue I am having is more clearly visible (and easily explained) this way.

The parameter MAX_CONN of the XMLRPC-C library is at its default of 16.

So, lets supose that I have a text with 50 sentences and that I will be launching one XMLRPC request for each sentence (I want to translate them). With the code below, everything works as expected, the first 20 requests are launched, only 16 are executed simultaneously (MAX_CONN=16), and when those 16 requests finish, the remaining 34 requests are executed (when its turn comes).

However, the problem comes if I make the "server" variable global (by adding "global server" at the beginning of the "rpc_call" function). In that case, what happens is:

  • 20 requests launched
  • 16 get to the server and finish
  • the 30 requests not launched in the first batch of 20 enter the server and finish

Note that there are 4 requests "lost", those launched in the first batch of 20 that don't get executed inmediately. The client does not finish as it remains waiting for those responses. However, if the client is killed (Ctrl-C), then those 4 sentences finally get executed, but then there is no client to return them to.

Obviously I don't need to make that variable global in this example. But to do what I say in the first paragraph (reusing the ServerProxy objects) I do need it (this comes frome here: Python Multiprocessing. How to enqueue XMLRPC ServerProxy objects), and I would like to understand why this is happening so I can sort some things out.

aux.py

def translate(sentence, server):
    param= {'text': '', 'align': False}
    param['text']=sentence
    txt=server.translate(param)
    return txt

main.py

def rpc_call(sentence):
    server = xmlrpclib.ServerProxy('http://myserver:6060/RPC2',)
    result= aux.translate(sentence, server)

if __name__ == '__main__':
    text=list() 
    [...]

    processes=20
    pool = Pool (processes)

    for sentence in text:
        pool.apply_async(rpc_call, args=(sentence,))

    pool.close()
    pool.join()

来源:https://stackoverflow.com/questions/22154884/concurrent-xmlrpc-c-calls-with-global-serverproxy-object-not-working-as-expected

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!