Beautifulsoup, maximum recursion depth reached

Question

This is a beautifulsoup procedure that grabs content within all

html tags. After grabbing content from some web pages, I get an error th

Answer 1

You probably hit a string. Iterating over a string yields 1-length strings. Iterating over that 1-length string yields a 1-length string. Iterating over THAT 1-length string...

Answer 2

Your printText() calls itself recursively if it encounters anything other than a NavigableString. This includes subclasses of NavigableString, such as Comment. Calling printText() on a Comment iterates over the text of the comment, and causes the infinite recursion you see.

I recommend using isinstance() in your if statement instead of comparing class objects:

if isinstance(tag, basestring):

I diagnosed this problem by inserting a print statement before the recursion:

print "recursing on", tag, type(tag)
printText(tag)

Answer 3

I had the same problem. If you have nested tags with a depth of about 480 levels, and you want to convert this tag to string/unicode, you get the RuntimeError maximum recursion depth reached. Every level needs two nested method calls and soon you hit the default of 1000 nested python calls. You can raise this level, or you can use this helper. It extracts all text from the html and displays it in a pre-environment:

def beautiful_soup_tag_to_unicode(tag):
    try:
        return unicode(tag)
    except RuntimeError as e:
        if not str(e).startswith('maximum recursion'):
            raise
        # If you have more than 480 level of nested tags you can hit the maximum recursion level
        out=[]
        for mystring in tag.findAll(text=True):
            mystring=mystring.strip()
            if not mystring:
                continue
            out.append(mystring)
        return u'<pre>%s</pre>' % '\n'.join(out)