Keras verbose training progress bar writing a new line on each batch issue

Question

running a Dense feed-forward neural net in Keras. there are class_weights for two outputs, and sample_weights for a third output. fore some reason it prints the progress ver

Answer 1

Import the library:

from tqdm.keras import TqdmCallback

Specify using tqdm library:

model.fit(xs, ys, epochs=10000, verbose=0, callbacks=[TqdmCallback(verbose=1)])

Result:

100%|██████████████| 10000/10000 [01:34<00:00, 105.57epoch/s, loss=1.56e+3]
 29%|███▌          |  2891/10000 [00:28<01:05, 108.02epoch/s, loss=1.57e+3]

For my case, verbose=1 is what really works for me. My answer is based on casper.dcl's answer.

Answer 2

It was mentioned before, but I will rewrite it to be more visible for future users.

You have too narrow terminal to print all these values - just set width argument of Progbar constructor to smaller number or remove/rename some of the provided values.

Answer 3

This seems to be a consistent problem with Keras. I tried to find the lines

sys.stdout.write('\b' * prev_total_width)

sys.stdout.write('\r')

at the Keras/utils/generic_utils.py file and they are (as of the current version) at 258 and 259 accordingly. I commented like 258 but this seems not to resolve the issue. I did manage to make the progress bar work by commenting the line:

line 303: sys.stdout.write(info)

It seems as if the info makes the bar too long for the terminal and so it breaks to a new line.

So I finally resolved the issue. It seems like it was rather simple at the end....

Just make the terminal wider...

Note: Tested on Linux Ubuntu 16.04 | Keras Version 2.0.5

Answer 4

I had a similar issue, but have not had the time to investigate it further. The problem seems to be related to the class Progbar in generic_utils.py of keras, see link, and perhaps Python >= 3.3.

The following lines are found in the update function of the class:

Line 107: sys.stdout.write('\b' * prev_total_width)
Line 108: sys.stdout.write('\r')

I simply removed line 107 as a quick fix, so instead of backspacing the previous line then performing a shift to the beginning of the line, I only perform the shift. I guess there is some better ways than altering the source code though.

Answer 5

I've added built-in support for keras in tqdm so you could use it instead (pip install "tqdm>=4.41.0"):

from tqdm.keras import TqdmCallback
...
model.fit(..., verbose=0, callbacks=[TqdmCallback(verbose=2)])

This turns off keras' progress (verbose=0), and uses tqdm instead. For the callback, verbose=2 means separate progressbars for epochs and batches. 1 means clear batch bars when done. 0 means only show epochs (never show batch bars).

If there are problems with it please open an issue at https://github.com/tqdm/tqdm/issues

Answer 6

The solutions of @user11353683 and @Panos can be completed in order to not modify the sources (which can create future problems): just install ipykernel and import it in your code:

pip install ipykernel Then import ipykernel

In fact, in the Keras generic_utils.py file, the probematic line was:

            if self._dynamic_display:
                sys.stdout.write('\b' * prev_total_width)
                sys.stdout.write('\r')
            else:
                sys.stdout.write('\n')

And the value self._dynamic_display was initiated such as:

        self._dynamic_display = ((hasattr(sys.stdout, 'isatty') and
                                  sys.stdout.isatty()) or
                                 'ipykernel' in sys.

So, loading ipykerneladded it to sys.modules and fixed the problem for me.