Running interactive command line code from Jupyter notebook

Question

There is an interesting option in Ipython Jupyter Notebook to execute command line statements directly from the notebook. For example:

! mkdir ...
! python f         


        

Answer 1

Assuming you are asking about interactivity, there is something you can try.

If you ever wondered how Jupyter knows when the output of a cell ends: well, it apparently does not know, it just dumps any captured output into the most recently active cell:

import threading,time
a=5
threading.Thread(target=lambda:[print(a),time.sleep(20),print(a)]).start()

(deliberately shorter-than-nice example, as this is just side-info. While the 20-second wait is running you have time to activate another cell, perhaps via issuing an a=6)

This can be used to get the output of some console code to the screen, while controlling it from the main thread:

import sys,threading,subprocess

proc=subprocess.Popen('/bin/sh',stdout=subprocess.PIPE,stdin=subprocess.PIPE,stderr=subprocess.STDOUT)
pout=proc.stdout
pin=proc.stdin

def outLoop():
    running=True
    while(running):
        line=pout.readline().decode(sys.stdout.encoding)
        print(line,end='')
        running='\n' in line
    print('Finished')

threading.Thread(target=outLoop).start()

Then you can isssue commands, like

pin.write(b'ls -l\n')
pin.flush()

and

pin.write(b'exit\n')
pin.flush()

Even b'ls\nexit\n' works, that is why outLoop is so long (a simple while(proc.poll() is None)-print(...) loop would finish sooner than it has grabbed all output.

Then the whole thing can be automated as:

while(proc.poll() is None):
    inp=bytearray(input('something: ')+'\n',sys.stdin.encoding)
    if(proc.poll() is None):
        pin.write(inp)
        pin.flush()

This works well on https://try.jupyter.org/, but obviously I did not want to try installing conda packages there, so I do not know what happens when conda asks a question.

A lucky thing is that the input field stays at the bottom of the cell (tested with ls;sleep 10;ls). An unlucky thing is that the input field needs an extra entry at the end to disappear (and that is already the 'nice' way, when it was a simple while(...)-write(bytearray(input()))-flush() 3-liner, it was exiting with an exception.

If someone wants to try this on Windows, it works with 'cmd', but I suggest using a hardcoded 'windows-1252' instead of sys.stdin/out.encoding: they say UTF-8, but a simple dir command already produces output which is neither UTF-8 nor ASCII (the non-breakable space between the 3-digit groups in sizes is a 0xA0 character). Or just remove the decode part (and use running=0xA in line)

Answer 2

the !commandsyntax is an alternative syntax of the %system magic, which documentation can be found here.

As you guessed, it's invoking os.system and as far as os.system works there is no simple way to know whether the process you will be running will need input from the user. Thus when using the notebook or any multi-process frontend you have no way to dynamically provide input to your the program you are running. (unlike a call to input in Python we can intercept).

As you explicitly show interest in installing packages from the notebook, I suggest reading the following from Jake Van Der Plas which is a summary of a recent discussion on the subject, and explain some of the complications of doing so. You can of course go with --yes option of conda, but it does not guarantee that installing with conda will always work.

Note also that !command is an IPython feature, not a Jupyter one.

Answer 3

Good option for most of the commands I've encountered is using non-interactive args. E.g. in the above case:

conda install package -y

If you absolutely need to feed the prompts, you can use printf hack, e.g.:

printf 'y\n' | conda install package

This supports multiple inputs, you separate them by '\n'

Answer 4

I'm posting this as an answer. It's not a good answer, but the way I'd handle the problem is to write a bash script to be run in the background. I've looked into the '!' operator and it doesn't seem to have a lot of documentation. I can't even find it in the Jupyter source. This article:

[Safari book about Jupyter Predecessor and component IPython][1]https://www.safaribooksonline.com/blog/2014/02/12/using-shell-commands-effectively-ipython/

suggests that this is simply the way things were before and probably forever shall be. Unless you want to hack into the Magic Commands part of the Jupyter Notebook and fix it yourself.

That said, given that with a little bash programming (It's simple and focused) you can do what you're trying to do, you might consider that route. Especially if you need results enough to put reputation on it.

If you want to look into running bash scripts with against an expected response this answer is what you're looking for: Have bash script answer interactive prompts