Import python module over the internet/multiple protocols or dynamically create module

Question

Is it possible to import a Python module from over the internet using the http(s), ftp, smb or any other pro

Answer 1

Another version,

I like this answer. when applied it, i simplified it a bit - similar to the look and feel of javascript includes over HTTP.

This is the result:

import os
import imp
import requests

def import_cdn(uri, name=None):
    if not name:
        name = os.path.basename(uri).lower().rstrip('.py')

    r = requests.get(uri)
    r.raise_for_status()

    codeobj = compile(r.content, uri, 'exec')
    module = imp.new_module(name)
    exec (codeobj, module.__dict__)
    return module

Usage:

redisdl = import_cdn("https://raw.githubusercontent.com/p/redis-dump-load/master/redisdl.py")

# Regular usage of the dynamic included library
json_text = redisdl.dumps(host='127.0.0.1')

• Tip - place the import_cdn function in a common library, this way you could re-use this small function
• Bear in mind It will fail when no connectivity to that file over http

Answer 2

This seems to be a use case for a self-written import hook. Look up in PEP 302 how exactly they work.

Essentially, you'll have to provide a finder object which, in turn, provides a loader object. I don't understand the process at the very first glance (otherwise I'd be more explicit), but the PEP contains all needed details for implementing the stuff.

Answer 3

In principle, yes, but all of the tools built-in which kinda support this go through the filesystem.

To do this, you're going to have to load the source from wherever, compile it with compile, and exec it with the __dict__ of a new module. See below.

I have left the actually grabbing text from the internet, and parsing uris etc as an exercise for the reader (for beginners: I suggest using requests)

In pep 302 terms, this would be the implementation behind a loader.load_module function (the parameters are different). See that document for details on how to integrate this with the import statement.

import imp
modulesource = 'a=1;b=2' #load from internet or wherever
def makemodule(modulesource,sourcestr='http://some/url/or/whatever',modname=None):
    #if loading from the internet, you'd probably want to parse the uri, 
    # and use the last part as the modulename. It'll come up in tracebacks
    # and the like.
    if not modname: modname = 'newmodulename'
    #must be exec mode
    # every module needs a source to be identified, can be any value
    # but if loading from the internet, you'd use the URI
    codeobj = compile(modulesource, sourcestr, 'exec')
    newmodule = imp.new_module(modname)
    exec(codeobj,newmodule.__dict__)
    return newmodule
newmodule = makemodule(modulesource)
print(newmodule.a)

At this point newmodule is already a module object in scope, so you don't need to import it or anything.

modulesource = '''
a = 'foo'
def myfun(astr):
    return a + astr
'''
newmod = makemodule(modulesource)
print(newmod.myfun('bat'))

Ideone here: http://ideone.com/dXGziO

Tested with python 2, should work with python 3 (textually compatible print used;function-like exec syntax used).

Answer 4

As glglgl's has it this import hook has been implemented for Python2 and Python3 in a module called httpimport. It uses a custom finder/loader object to locate resources using HTTP/S.

Additionally, the import_cdn function in Jossef Harush's answer is almost identically implemented in httpimport's github_repo, and bitbucket_repo functions.

@Marcin's answer contains a good portion of the code of the httpimport's loader class.