Redirect print command in python script through tqdm.write()

Question

I'm using tqdm in Python to display console-progressbars in our scripts. However, I have to call functions which print messages to the console as well and which I can't change. In general, writing to the console while displaying progress bars in the console messes up the display like so:

from time import sleep
from tqdm import tqdm

def blabla():
  print "Foo blabla"

for k in tqdm(range(3)):
  blabla()
  sleep(.5)

This creates the output:

0%|                                           | 0/3 [00:00<?, ?it/s]Foo
blabla
33%|###########6                       | 1/3 [00:00<00:01,  2.00it/s]Foo
blabla
67%|#######################3           | 2/3 [00:01<00:00,  2.00it/s]Foo
blabla
100%|###################################| 3/3 [00:01<00:00,  2.00it/s]

According to the documentation of tqdm the method tqdm.write() provides a means to write messages to the console without breaking the displayed progressbars. Thus, the right output is provided by this snippet:

from time import sleep
from tqdm import tqdm

def blabla():
  tqdm.write("Foo blabla")

for k in tqdm(range(3)):
  blabla()
  sleep(.5)

And looks like this:

Foo blabla
Foo blabla
Foo blabla
100%|###################################| 3/3 [00:01<00:00,  1.99it/s]

On the other hand, there is this solution which permits to silence those functions by quite elegantly redirecting sys.stdout into the void. This works perfectly well for silencing the functions.

Since I want to display the messages from these functions nonetheless without breaking the progress bars, I tried to merge both solutions into one by redirecting sys.stdout to tqdm.write() and, in turn, letting tqdm.write() write to the old sys.stdout. This results in the snippet:

from time import sleep

import contextlib
import sys

from tqdm import tqdm

class DummyFile(object):
  file = None
  def __init__(self, file):
    self.file = file

  def write(self, x):
    tqdm.write(x, file=self.file)

@contextlib.contextmanager
def nostdout():
    save_stdout = sys.stdout
    sys.stdout = DummyFile(save_stdout)
    yield
    sys.stdout = save_stdout

def blabla():
  print "Foo blabla"

for k in tqdm(range(3)):
  with nostdout():
    blabla()
    sleep(.5)

However, this actually creates an even more messed up output as before:

0%|                                           | 0/3 [00:00<?, ?it/s]Foo
blabla


33%|###########6                       | 1/3 [00:00<00:01,  2.00it/s]Foo
blabla


67%|#######################3           | 2/3 [00:01<00:00,  2.00it/s]Foo
blabla


100%|###################################| 3/3 [00:01<00:00,  2.00it/s]

FYI: calling tqdm.write(..., end="") inside DummyFile.write() creates the same result as the first output which is still messed up.

I can't understand why this wouldn't work, since tqdm.write() is supposed to manage clearing the progress bar before writing the message and then rewriting the progress bar.

What am I missing?

Answer 1

Redirecting sys.stdout is always tricky, and it becomes a nightmare when two applications are twiddling with it at the same time.

Here the trick is that tqdm by default prints to sys.stderr, not sys.stdout. Normally, tqdm has an anti-mixup strategy for these two special channels, but since you are redirecting sys.stdout, tqdm gets confused because the file handle changes.

Thus, you just need to explicitly specify file=sys.stdout to tqdm and it will work:

from time import sleep

import contextlib
import sys

from tqdm import tqdm

class DummyFile(object):
  file = None
  def __init__(self, file):
    self.file = file

  def write(self, x):
    # Avoid print() second call (useless \n)
    if len(x.rstrip()) > 0:
        tqdm.write(x, file=self.file)

@contextlib.contextmanager
def nostdout():
    save_stdout = sys.stdout
    sys.stdout = DummyFile(sys.stdout)
    yield
    sys.stdout = save_stdout

def blabla():
  print("Foo blabla")

# tqdm call to sys.stdout must be done BEFORE stdout redirection
# and you need to specify sys.stdout, not sys.stderr (default)
for _ in tqdm(range(3), file=sys.stdout):
    with nostdout():
        blabla()
        sleep(.5)

print('Done!')

I also added a few more tricks to make the output nicer (eg, no useless \n when using print() without end='').

/EDIT: in fact it seems you can do the stdout redirection after starting tqdm, you just need to specify dynamic_ncols=True in tqdm.

Answer 2

It might be the bad way, but I change built-in print function.

import inspect
import tqdm
# store builtin print
old_print = print
def new_print(*args, **kwargs):
    # if tqdm.tqdm.write raises error, use builtin print
    try:
        tqdm.tqdm.write(*args, **kwargs)
    except:
        old_print(*args, ** kwargs)
# globaly replace print with new_print
inspect.builtins.print = new_print

Answer 3

By mixing, user493630 and gaborous answers, I created this context manager which avoid having to use the file=sys.stdout parameter of tqdm.

import inspect
import contextlib
import tqdm

@contextlib.contextmanager
def redirect_to_tqdm():
    # Store builtin print
    old_print = print
    def new_print(*args, **kwargs):
        # If tqdm.tqdm.write raises error, use builtin print
        try:
            tqdm.tqdm.write(*args, **kwargs)
        except:
            old_print(*args, ** kwargs)

    try:
        # Globaly replace print with new_print
        inspect.builtins.print = new_print
        yield
    finally:
        inspect.builtins.print = old_print

To use it, simply:

for i in tqdm.tqdm(range(100)):
    with redirect_to_tqdm():
        time.sleep(.1)
        print(i)

To simplify even further, it is possible to wrap the code in a new function:

def tqdm_redirect(*args, **kwargs):
    with redirect_to_tqdm():
        for x in tqdm.tqdm(*args, **kwargs):
            yield x

for i in tqdm_redirect(range(20)):
    time.sleep(.1)
    print(i)

Answer 4

OP's solution is almost correct. The test in tqdm library that messes up your output is this one (https://github.com/tqdm/tqdm/blob/master/tqdm/_tqdm.py#L546-L549):

if hasattr(inst, "start_t") and (inst.fp == fp or all(
           f in (sys.stdout, sys.stderr) for f in (fp, inst. 
    inst.clear(nolock=True)
    inst_cleared.append(inst)

tqdm.write is testing the file you provided to see if there is risk of collision between the text to print and potential tqdm bars. In your case stdout and stderr get mixed in the terminal so there is a collision. To counter this, when the test passes, tqdm clears the bars, prints the text and draws the bars back after.

Here, the test fp == sys.stdout fails because sys.stdout became DummyFile, and fp is the real sys.stdout, so the cleaning behavior is not enabled. A simple equality operator in DummyFile fixes everything.

class DummyFile(object):
    def __init__(self, file):
        self.file = file

    def write(self, x):
        tqdm.write(x, end="", file=self.file)

    def __eq__(self, other):
        return other is self.file

Also, since print passes a newline to sys.stdout (or not depending on the user choice), you don't want tqdm to add another one on its own, so it's better to set the option end='' than performing strip on the content.

Advantages of this solution

With gaborous's answer, tqdm(..., file=sys.stdout) pollutes your output stream with pieces of bar. By keeping file=sys.stdout (default), you keep your streams separated.
With Conchylicultor and user493630's answers, you only patch print. However other systems such as logging directly stream to sys.stdout, so they don't go through tqdm.write.