improving speed of Python module import

前端 未结 4 1548
[愿得一人]
[愿得一人] 2020-12-07 16:12

The question of how to speed up importing of Python modules has been asked previously (Speeding up the python "import" loader and Python -- Speed Up Imports?) but

相关标签:
4条回答
  • 2020-12-07 16:53

    1.35 seconds isn't long, but I suppose if you're used to half that for a "quick check" then perhaps it seems so.

    Andrea suggests a simple client/server setup, but it seems to me that you could just as easily call a very slight modification of your script and keep it's console window open while you work:

    • Call the script, which does the imports then waits for input
    • Minimize the console window, switch to your work, whatever: *Do work*
    • Select the console again
    • Provide the script with some sort of input
    • Receive the results with no import overhead
    • Switch away from the script again while it happily awaits input

    I assume your script is identical every time, ie you don't need to give it image stack location or any particular commands each time (but these are easy to do as well!).

    Example RAAC's_Script.py:

    import numpy as np
    import matplotlib.pyplot as plt
    import matplotlib.animation as animation
    import scipy.ndimage
    import scipy.signal
    import sys
    import os
    
    print('********* RAAC\'s Script Now Running *********')
    
    while True: # Loops forever
        # Display a message and wait for user to enter text followed by enter key.
        # In this case, we're not expecting any text at all and if there is any it's ignored
        input('Press Enter to test image stack...')
    
        '''
        *
        *
        **RAAC's Code Goes Here** (Make sure it's indented/inside the while loop!)
        *
        *
        '''
    

    To end the script, close the console window or press ctrl+c.

    I've made this as simple as possible, but it would require very little extra to handle things like quitting nicely, doing slightly different things based on input, etc.

    0 讨论(0)
  • 2020-12-07 16:58

    you could build a simple server/client, the server running continuously making and updating the plot, and the client just communicating the next file to process.

    I wrote a simple server/client example based on the basic example from the socket module docs: http://docs.python.org/2/library/socket.html#example

    here is server.py:

    # expensive imports
    import numpy as np
    import matplotlib.pyplot as plt
    import matplotlib.animation as animation
    import scipy.ndimage
    import scipy.signal
    import sys
    import os
    
    # Echo server program
    import socket
    
    HOST = ''                 # Symbolic name meaning all available interfaces
    PORT = 50007              # Arbitrary non-privileged port
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind((HOST, PORT))
    s.listen(1)
    while 1:
        conn, addr = s.accept()
        print 'Connected by', addr
        data = conn.recv(1024)
        if not data: break
        conn.sendall("PLOTTING:" + data)
        # update plot
        conn.close()
    

    and client.py:

    # Echo client program
    import socket
    import sys
    
    HOST = ''    # The remote host
    PORT = 50007              # The same port as used by the server
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect((HOST, PORT))
    s.sendall(sys.argv[1])
    data = s.recv(1024)
    s.close()
    print 'Received', repr(data)
    

    you just run the server:

    python server.py
    

    which does the imports, then the client just sends via the socket the filename of the new file to plot:

    python client.py mytextfile.txt
    

    then the server updates the plot.

    On my machine running your imports take 0.6 seconds, while running client.py 0.03 seconds.

    0 讨论(0)
  • 2020-12-07 17:01

    Not an actual answer to the question, but a hint on how to profile the import speed with Python 3.7 and tuna (a small project of mine):

    python3.7 -X importtime -c "import scipy" 2> scipy.log
    tuna scipy.log
    

    0 讨论(0)
  • 2020-12-07 17:03

    You can import your modules manually instead, using imp. See documentation here.

    For example, import numpy as np could probably be written as

    import imp
    np = imp.load_module("numpy",None,"/usr/lib/python2.7/dist-packages/numpy",('','',5))
    

    This will spare python from browsing your entire sys.path to find the desired packages.

    See also:

    Manually importing gtk fails: module not found

    0 讨论(0)
自定义标题
段落格式
字体
字号
代码语言
提交回复
热议问题