How to check for webpages' popups?

Question

Is there a possibility if I code a program in python that allows to automatically browse a given website using mechanize to detect if there are popup windows (suggesting adv

Answer 1

Unfortunately, Mechanize's browser seems to skip the pop-ups so the title, URL, and HTML are identical for both pop-ups and normal pages.

Frankly, Python is not the right tool for this job and is lagging behind in this respect IMHO. Having spent months doing web crawling, for sites that use Javascript extensively (the number of which is greatly increasing nowadays), I find that using Javascript-Based environments like PhantomJS or SlimerJS are simply better for what you're trying to do.

If you have the luxury to use Javascript-Based environments, I'd say go right ahead. However, you can still use python. PhantomJS embeds Ghost Driver. You can use Ghost.py to utilize the power of PhantomJS. Or you can use Selenium with Python as illustrated here.

Answer 2

Mechanize cannot handle javascript and popup windows:

• How do I use Mechanize to process JavaScript?
• Mechanize and Javascript

To accomplish the goal, you need to utilize a real browser, headless or not. This is where selenium would help. It has a built-in support for popup dialogs:

Selenium WebDriver has built-in support for handling popup dialog boxes. After you’ve triggerd and action that would open a popup, you can access the alert with the following:

alert = driver.switch_to_alert()

Example (using this jsfiddle):

from selenium import webdriver

url = "http://fiddle.jshell.net/ebkXh/show/"
driver = webdriver.Firefox()
driver.get(url)

button = driver.find_element_by_xpath('//button[@type="submit"]')

# dismiss
button.click()
driver.switch_to.alert.dismiss()

# accept
button.click()
driver.switch_to.alert.accept()

See also:

• Handle Popup Windows
• Click the javascript popup through webdriver