Python, Selenium and Chromedriver - endless loop using find_element_by_id causes CPU problem

Question

Good day to all! I\'ve been experiencing this problem for a week now but I don\'t think I can solve it and I also do not see any solution based on articles online. Hopefully

Answer 1

It would be tough to guess the exact reason of 100% CPU Usage without any visibility to your code blocks specifically the WebDriver configuration. So the answer will be pretty much based on generic guidelines as follows:

• Never close the browser (by pressing the X button). Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.
  • You can find a detailed discussion in PhantomJS web driver stays in memory
• Never terminate the script (by pressing Control+C). Incase there are presence of zombie WebDriver or Web Browser instances you can programatically remove them.
  • You can find a detailed discussion in Selenium : How to stop geckodriver process impacting PC memory, without calling driver.quit()?

• A couple of useful ChromeOptions() and their usage are as follows:

  options.addArguments("start-maximized"); // open Browser in maximized mode
  options.addArguments("disable-infobars"); // disabling infobars
  options.addArguments("--disable-extensions"); // disabling extensions
  options.addArguments("--disable-gpu"); // applicable to windows os only
  options.addArguments("--disable-dev-shm-usage"); // overcome limited resource problems
  options.addArguments("--no-sandbox"); // Bypass OS security model
  

• Using hardcoded sleeps in the form of time.sleep(1) is a big No.

  • You can find a detailed discussion in How to sleep webdriver in python for milliseconds
• Incase you are using Chrome in headless mode, there had been a lot of discussion going around about the unpredictable CPU and Memory Consumption by Chrome Headless sessions.
  • You can find a detailed discussion in Limit chrome headless CPU and memory usage
• Always keep your Test Environment updated with the latest released binaries as follows:
  • Upgrade ChromeDriver to current ChromeDriver v2.44 level.
  • Keep Chrome version between Chrome v69-71 levels. (as per ChromeDriver v2.44 release notes)
  • Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
  • If your base Web Client version is too old, then uninstall it through Revo Uninstaller and install a recent GA and released version of Web Client.
  • Take a System Reboot.
  • Execute your @Test.
• From Space and Memory Management perspective:
  • (WindowsOS only) Use CCleaner tool to wipe off all the OS chores before and after the execution of your Test Suite.
  • (LinuxOS only) Free Up and Release the Unused/Cached Memory in Ubuntu/Linux Mint before and after the execution of your Test Suite.

Answer 2

Have you tried releasing memory into the loop? Maybe by picking up the values (list out of the loop?) and then resetting those variables to None you can avoid excessive memory consumption.

...
while True:

...
    close1 = close2 = close3 = close4 = close5 = close6 = None

...

You can also try forcing the garbage collector:

import gc

while True: 
...
    gc.collect()

If you think that the reason may be a script another another solution to detect the problem might be to enable Chrome to do remote debug and debug the page.

--remote-debugging-port=9222

I hope some of this helps you.