“Failed to decode response from marionette” message in Python/Firefox headless scraping script

前端 未结 7 2043
既然无缘
既然无缘 2020-12-03 03:29

Good Day, I\'ve done a number of searches on here and google and yet to find a solution that address this problem.

The scenario is:

I have a Python script (

相关标签:
7条回答
  • 2020-12-03 04:02

    The likely real issue behind this is that the DOM has not loaded yet and you are triggering searches on the next page. That's why the sleep(3) is working in most cases. The proper fix is to use a wait class.

    This is an example test case using a wait function for Nextcloud. It's from my docker-selenium-firefox-python image: https://hub.docker.com/r/nowsci/selenium

    Notice how the wait class is called surrounding any click or get calls. Basically, what this does is takes advantage of the fact that selenium changes the ID for the HTML tag on page load. The wait function checks if the new ID is different than the old, and when it is, the DOM has loaded.

    import time
    from selenium.webdriver import Firefox
    from selenium.webdriver.firefox.options import Options
    from selenium.webdriver.common.keys import Keys
    
    class wait(object):
    
        def __init__(self, browser):
            self.browser = browser
    
        def __enter__(self):
            self.old_page = self.browser.find_element_by_tag_name('html')
    
        def page_has_loaded(self):
            new_page = self.browser.find_element_by_tag_name('html')
            return new_page.id != self.old_page.id
    
        def __exit__(self, *_):
            start_time = time.time()
            while time.time() < start_time + 5:
                if self.page_has_loaded():
                    return True
                else:
                    time.sleep(0.1)
            raise Exception('Timeout waiting for page load.')
    
    def test():
        try:
            opts = Options()
            opts.headless = True
            assert opts.headless  # Operating in headless mode
            browser = Firefox(options=opts)
        except Exception as e:
            print("  -=- FAIL -=-: Browser setup - ", e)
            return
    
        # Test title
        try:
            with wait(browser):
                browser.get('https://nextcloud.mydomain.com/index.php/login')
            assert 'Nextcloud' in browser.title
        except Exception as e:
            print("  -=- FAIL -=-: Initial load - ", e)
            return
        else:
            print("  Success: Initial load")
    
        try:
            # Enter user
            elem = browser.find_element_by_id('user')
            elem.send_keys("MYUSER")
    
            # Enter password
            elem = browser.find_element_by_id('password')
            elem.send_keys("MYPASSWORD")
    
            # Submit form
            elem = browser.find_element_by_id('submit')
            with wait(browser):
                elem.click()
    
            # Check title for success
            assert 'Files' in browser.title
        except Exception as e:
            print("  -=- FAIL -=-: Login - ", e)
            return
        else:
            print("  Success: Login")
    
        print("  Finished.")
    
    print("Testing nextcloud...")
    test()
    

    Combine this with the answer from @myol if you are using Docker.

    0 讨论(0)
提交回复
热议问题