How to fix python-selenium error “connection refused” when initializing a selenium webdriver?

Question

I am running very complex python-selenium tests on non-public webpages. In most cases these tests run fine, but sometimes one of these tests fail during the initialization o

Answer 1

This error message...

{'status': 500, 'value': '{"value":{"error":"unknown error","message":"connection refused","stacktrace":"stack backtra...s::imp::thread::{{impl}}::new::thread_start\n at /checkout/src/libstd/sys/unix/thread.rs:84"}}'}

...implies that the GeckoDriver was unable to initiate/spawn a new WebBrowsing Session i.e. Firefox Browser session.

In a comment within the discussion DELETE '/session/{session id}' no longer working @andreastt mentions that:

geckodriver is implicitly ending the (previous) session when the last window closes. If driver.quit() is called as the subsequent command it will fail because the session has already been implicitly deleted.

In these cases GeckoDriver should detect that the session has been closed implicitly after driver.close() or ignore the response from driver.quit() in case the session has already been closed.

In such cases the following trace logs are generated:

1505753594121   webdriver::server   DEBUG   Last window was closed, deleting session
1505753594121   webdriver::server   DEBUG   Deleting session
1505753594121   geckodriver::marionette DEBUG   Stopping browser process
1505753594364   webdriver::server   DEBUG   <- 200 OK {"value": []}
1505753594523   webdriver::server   DEBUG   -> DELETE /session/a8312282-af00-4931-94d4-0d401abf01c9 
1505753594524   webdriver::server   DEBUG   <- 500 Internal Server Error {"value":{"error":"session not created","message":"Tried to run command without establishing a connection","stacktrace":"stack backtrace:\n   0:           0x4f388c - backtrace::backtrace::trace::h736111741fa0878e\n   1:           0x4f38c2 - backtrace::capture::Backtrace::new::h63b8a5c0787510c9\n   2:           0x442c61 - webdriver::error::WebDriverError::new::hc4fe6a1ced4e57dd\n   3:           0x42a926 - <webdriver::server::Dispatcher<T, U>>::run::hba9181b5aacf8f04\n   4:           0x402c59 - std::sys_common::backtrace::__rust_begin_short_backtrace::h19de262639927233\n   5:           0x40c065 - std::panicking::try::do_call::h6c1659fc4d01af51\n   6:           0x5e38ec - panic_unwind::__rust_maybe_catch_panic\n                        at /checkout/src/libpanic_unwind/lib.rs:98\n   7:           0x420d32 - <F as alloc::boxed::FnBox<A>>::call_box::h953e5f59694972c5\n   8:           0x5dc00b - alloc::boxed::{{impl}}::call_once<(),()>\n                        at /checkout/src/liballoc/boxed.rs:661\n                         - std::sys_common::thread::start_thread\n                        at /checkout/src/libstd/sys_common/thread.rs:21\n                         - std::sys::imp::thread::{{impl}}::new::thread_start\n                        at /checkout/src/libstd/sys/unix/thread.rs:84"}}
1505753594533   webdriver::server   DEBUG   -> DELETE /session/a8312282-af00-4931-94d4-0d401abf01c9 
1505753594542   webdriver::server   DEBUG   <- 500 Internal Server Error {"value":{"error":"session not created","message":"Tried to run command without establishing a connection","stacktrace":"stack backtrace:\n   0:           0x4f388c - backtrace::backtrace::trace::h736111741fa0878e\n   1:           0x4f38c2 - backtrace::capture::Backtrace::new::h63b8a5c0787510c9\n   2:           0x442c61 - webdriver::error::WebDriverError::new::hc4fe6a1ced4e57dd\n   3:           0x42a926 - <webdriver::server::Dispatcher<T, U>>::run::hba9181b5aacf8f04\n   4:           0x402c59 - std::sys_common::backtrace::__rust_begin_short_backtrace::h19de262639927233\n   5:           0x40c065 - std::panicking::try::do_call::h6c1659fc4d01af51\n   6:           0x5e38ec - panic_unwind::__rust_maybe_catch_panic\n                        at /checkout/src/libpanic_unwind/lib.rs:98\n   7:           0x420d32 - <F as alloc::boxed::FnBox<A>>::call_box::h953e5f59694972c5\n   8:           0x5dc00b - alloc::boxed::{{impl}}::call_once<(),()>\n                        at /checkout/src/liballoc/boxed.rs:661\n                         - std::sys_common::thread::start_thread\n                        at /checkout/src/libstd/sys_common/thread.rs:21\n                         - std::sys::imp::thread::{{impl}}::new::thread_start\n                        at /checkout/src/libstd/sys/unix/thread.rs:84"}}
1505753594549   webdriver::server   DEBUG   -> GET /shutdown 
1505753594551   webdriver::server DEBUG <- 404 Not Found {"value":{"error":"unknown command","message":"GET /shutdown did not match a known command","stacktrace":"stack backtrace:\n 0: 0x4f388c - backtrace::backtrace::trace::h736111741fa0878e\n 1: 0x4f38c2 - backtrace::capture::Backtrace::new::h63b8a5c0787510c9\n 2: 0x442d88 - webdriver::error::WebDriverError::new::hea6d4dbf778b2b24\n 3: 0x43c65f - <webdriver::server::HttpHandler<U> as hyper::server::Handler>::handle::hd03629bd67672697\n 4: 0x403a04 - std::sys_common::backtrace::__rust_begin_short_backtrace::h32e6ff325c0d7f46\n 5: 0x40c036 - std::panicking::try::do_call::h5f902dc1eea01ffe\n 6: 0x5e38ec - panic_unwind::__rust_maybe_catch_panic\n at /checkout/src/libpanic_unwind/lib.rs:98\n 7: 0x4209a2 - <F as alloc::boxed::FnBox<A>>::call_box::h032bafb4b576d1cd\n 8: 0x5dc00b - alloc::boxed::{{impl}}::call_once<(),()>\n 

Though the error codes for the error you are seeing is 'status': 500 and the error sample I have provided is 404 Not Found, apparently looks different, the core reason is similar as:

"message":"connection refused" 

due to:

imp::thread::{{impl}}::new::thread_start

from:

/checkout/src/libstd/sys/unix/thread.rs:84

From another perspective, while you use GeckoDriver, Selenium and Firefox ensure that the binaries are compatible as follows:

---

Analysis

There had been significant changes in the geckodriver binary since the availability of geckodriver 0.19.1. A couple of the changes are:

• GeckoDriver v0.22.0 (2018-09-15):
  • The HTTP status codes used for [script timeout] and [timeout] errors has changed from Request Timeout (408) to Internal Server Error (500) in order to not break HTTP/1.1 Keep-Alive support, as HTTP clients interpret the old status code to mean they should duplicate the request.
  • The HTTP/1.1 Keep-Alive timeout for persistent connections has been increased to 90 seconds.
  • An [invalid session ID] error is now returned when there is no active session.
  • The handshake when geckodriver connects to Marionette has been hardened by killing the Firefox process if it fails.
  • The handshake read timeout has been reduced to 10 seconds instead of waiting forever.
• GeckoDriver v0.21.0 (2018-06-15):
  • WebDriver commands that do not have a return value now correctly return {value: null} instead of an empty dictionary.
  • Force use of IPv4 network stack.

  • On certain system configurations, where localhost resolves to an IPv6 address, geckodriver would attempt to connect to Firefox on the wrong IP stack, causing the connection attempt to time out after 60 seconds. We now ensure that geckodriver uses IPv4 consistently to both connect to Firefox and for allocating a free port.

• GeckoDriver v0.20.1 (2018-04-06):
  • Avoid attempting to kill Firefox process that has stopped.

  • With the change to allow Firefox enough time to shut down in 0.20.0, geckodriver started unconditionally killing the process to reap its exit status. This caused geckodriver to inaccurately report a successful Firefox shutdown as a failure.

• GeckoDriver v0.20.0 (2018-03-08):
  • Backtraces from geckodriver no longer substitute for missing Marionette stacktraces.
  • The Firefox process is now given ample time to shut down, allowing enough time for the Firefox shutdown hang monitor to kick in.

  • Firefox has an integrated background monitor that observes long-running threads during shutdown. These threads will be killed after 63 seconds in the event of a hang. To allow Firefox to shut down these threads on its own, geckodriver has to wait that time and some additional seconds.

---

Solution

• Upgrade Selenium to current levels Version 3.14.0.
• Upgrade GeckoDriver to GeckoDriver v0.22.0 level.
• Upgrade Firefox version to Firefox v62.0.2 levels.
• If your base Web Client version is too old, then uninstall it through Revo Uninstaller and install a recent GA and released version of Web Client.
• Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.
• Execute your Test as a non-root user.

---

Update

As per your question update of-coarse you can induce a loop for multiple trials to initialize the selenium webdriver instance as follows:

• Ensure that there is no dangling instances of geckodriver by invoking taskkill command (WindowsOS specific) as follows:

  os.system("taskkill /f /im geckodriver.exe /T") 
  

• Ensure that there is no dangling instances of geckodriver by invoking kill() command (Cross platform) as follows:

  from selenium import webdriver
  import psutil
  from selenium.common.exceptions import WebDriverException
  
  for counter in range(5):
      try:
          webdriver = webdriver.Firefox(executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe') 
          print("WebDriver and WebBrowser initialized ...")
          break
      except WebDriverException:
          #Cross platform
          PROCNAME = "geckodriver"
          for proc in psutil.process_iter():
              # check whether the process name matches
              if proc.name() == PROCNAME:
                  proc.kill()        
          print("Retrying ...")
  print("Out of loop ...")