“Failed to decode response from marionette” message in Python/Firefox headless scraping script

Question

Good Day, I\'ve done a number of searches on here and google and yet to find a solution that address this problem.

The scenario is:

I have a Python script (

Answer 1

The problem is that you do not close the driver. I made the same mistake. Whit htop in Linux, I have noticed that I occupy all the 26 GB of my pc whit firefox unclosed process.

Answer 2

This error message...

Message: failed to decode response from marionette

...implies that the communication between GeckoDriver and Marionette was interrupted/broken.

---

Some of the reasons and solution for this issue are as follows:

• In the discussion Crash during command execution results in "Internal Server Error: Failed to decode response from marionette" @whimboo mentions, while executing your tests Selenium may force a crash of the parent process of Firefox with an error as:

  DEBUG   <- 500 Internal Server Error {"value":{"error":"unknown error","message":"Failed to decode response from marionette","stacktrace":...}...}
  

  • Analysis: The current message is somewhat misleading and Geckdriver needs to handle this situation and reporting that the application has unexpectedly quit, in a better way. This issue is still open.

• In the discussion Failed to decode response from marionette with Firefox >= 65 @rafagonc mentioned, this issue can occur when using GeckoDriver / FirefoxDriver or ChromeDriver in docker environment, due to presence of Zombie process that hangs even after invoking driver.quit(). At times, when you open many browsing instances one after another, your system may run out of memory or out of PIDs. See: Selenium using too much RAM with Firefox

  • As a solution @andreastt mentions, the following configuration should resolve the out of memory issue with Docker:

    --memory 1024mb --shm-size 2g
    

Steps: Configure SHM size in the docker container

• Similarly, while executing your test in your localhost, it is advisable to keep the following (minimum) configuration:

  --memory 1024mb
  

---

Additional Considerations

This issue can also occur due to incompatibility between the version of the binaries you are using.

Solution:

• Upgrade JDK to recent levels JDK 8u341.
• Upgrade Selenium to current levels Version 3.141.59.
• Upgrade GeckoDriver to GeckoDriver v0.26.0 level.
• Upgrade Firefox version to Firefox v72.0 levels.
• Execute your Test as a non-root user.

---

GeckoDriver, Selenium and Firefox Browser compatibility chart

---

tl; dr

[e10s] Crash in libyuv::ARGBSetRow_X86

---

Reference

You can find a relevant detailed discussion in:

• Browsing context has been discarded using GeckoDriver Firefox through Selenium

Answer 3

Try this, for Ubuntu 16.04:

1. Install firefox

sudo apt update
sudo apt install firefox

2. Check that firefox is well installed

which firefox

Will return /usr/bin/firefox

3. Go to the geckodriver releases page. Find the latest version of the driver for your platform and download it. For example:

wget https://github.com/mozilla/geckodriver/releases/download/v0.24.0/geckodriver-v0.24.0-linux64.tar.gz

4. Extract the file with:

tar -xvzf geckodriver*

5. Make it executable:

chmod +x geckodriver

6. Move it to $PATH and give root access

sudo mv geckodriver /usr/bin/
cd /usr/bin
sudo chown root:root geckodriver

7. Install selenium

pip3 install selenium

8. Add firefox and geckodriver to $PATH

sudo vim ~/.bashrc

Add the two lines:

export PATH=$PATH:"/usr/bin/firefox"
export PATH=$PATH:"/usr/bin/geckodriver"

9. Reboot your instance

sudo reboot

Answer 4

I hope this saves some other poor soul the hours I just spent on this.

Download an old version of firefox (specifically, v66 for me), and point selenium there:

firefox_binary='/home/user/Downloads/old_firefox/firefox/firefox'

Answer 5

For anyone else experiencing this issue when running selenium webdriver in a Docker container, increasing the container size to 2gb fixes this issue.

I guess this affects physical machines too if the OP fixed their issue by upgrading their server RAM to 2Gb, but could be coincidence.

Answer 6

I still don't know why this is happening but I may have found a work around. I read in some documentation there may be a race condition (on what, I am not sure since there shouldn't be two items competing for the same resources).

I changed the scraping code to do this:

import time

browserObj.get(url)

time.sleep(3)

soup = BeautifulSoup(browserObj.page_source, 'lxml')

No specific reason why I chose 3 seconds but since adding this delay I have not had the Message: failed to decode response from marionette error from any of my list of URLs to scrape.

---

Update: October, 2018

This continues to be an issue over six months later. Firefox, Geckodriver, Selenium and PyVirtualDisplay have all been updated to their latest versions. This error kept reoccurring spontaneously without pattern: sometimes working and sometimes not.

What fixed this issue is increasing RAM on my server from 1 GB to 2 GB. Since the increase there have been no failures of this sort.