Website denies get request using Selenium

问题

I am trying to use selenium in order to automate for login into a website https://afghanbids.com/ but when I run my python program, the browser loads a page which says access denied. You may please see the screen shot below.

My code:

driver = webdriver.Chrome(executable_path="chromedriver.exe")
driver.get("https://afghanbids.com/")

It shows that:

How can I make the request so that the target website does not deny my request?

回答1:

I took your code, added a few tweaks and encountered the access_denied page. Here are the observation results:

Code Block:

from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://afghanbids.com/")
print(driver.page_source)

Browser Snapshot:

Console Output:

<html lang="en"><head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <title>AfghanBids</title>

    <!--favicon-->
    <!-- <link rel="apple-touch-icon" href="/theme/images/apple-touch-icon.png"> -->
    <link rel="shortcut icon" href="https://afghanbids.com/assets/backend/images/favicon_1.ico" type="image/x-icon">

    <script type="text/javascript">
      var base_url = "https://afghanbids.com/";
      var number = 1;
      var total_size = 0;
      var allow = false;
    </script>

    <!-- bootstrap -->
    <link href="https://afghanbids.com/assets/plugins/bootstrap-3.3.2/css/bootstrap.min.css" rel="stylesheet">

    <!-- Icons -->
    <link href="https://afghanbids.com/assets/plugins/font-awesome-4.2.0/css/font-awesome.min.css" rel="stylesheet">

    <!-- Themes styles-->
    <link href="https://afghanbids.com/assets/theme/css/theme.css" rel="stylesheet">
    <!-- Your custom css -->
    <link href="https://afghanbids.com/assets/theme/css/theme-custom.css" rel="stylesheet">
    <link href="https://afghanbids.com/assets/custom/custom.css" rel="stylesheet">


    <script type="text/javascript" src="https://afghanbids.com/assets/backend/js/jquery.min.js"></script>
    <!-- TO SET TIMEZONE OF VISITOR -->
    <script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/jstimezonedetect/1.0.4/jstz.min.js"></script>
    <script type="text/javascript">
    $(document).ready(function(){
        var tz = jstz.determine(); // Determines the time zone of the browser client
        var timezone = tz.name(); //'Asia/Kolhata' for Indian Time.
        url = base_url+"admin/user/set_timezone";
        $.post(url, {timezone: timezone}, function(data){
        // alert(data);
        });
    });
    </script>

    <!-- FOR GOOGLE RECAPTCHA -->

    <!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
    <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
    <!--[if lt IE 9]>
      <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
      <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
    <![endif]-->

      </head>
  <body>
     <div class="loading-page overlay" id="myNav">
      <!-- <a href="javascript:void(0)" class="closebtn" onclick="closeNav()">&times;</a> -->
    <div class="counter">
      <p>File Uploading</p>
      <p style="font-size:12px">Please wait ...</p>
      <h1>0%
        <!--
        h1.abs loading
        h1.abs.color loading
        -->
      </h1>
      <hr>
    </div>
      </div>

    <!-- wrapper page -->
    <div class="wrapper">
      <!-- main-header -->
      <header class="main-header">


    <!-- main navbar -->
    <nav class="navbar navbar-default main-navbar hidden-sm hidden-xs">
      <div class="container">
        <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">

          <ul class="nav navbar-nav">
        <li class=""><a class="my-nav-a" href="https://afghanbids.com/home"><strong>HOME</strong></a></li>
        <li class=""><a class="my-nav-a" href="https://afghanbids.com/buyer_front"><strong>BUYERS</strong></a></li>
        <li class=""><a class="my-nav-a" href="https://afghanbids.com/seller_front"><strong>SELLERS</strong></a></li>
        <li class=""><a class="my-nav-a" href="https://afghanbids.com/home/contact"><strong>CONTACT US</strong></a></li>
        <li class=""><a class="my-nav-a" href="https://afghanbids.com/home/about"><strong>ABOUT US</strong></a></li>
                <li class=""><a href="https://afghanbids.com/admin/user/register"><strong>REGISTER</strong></a></li>              </ul>
          <form action="https://afghanbids.com/admin/user/login" id="quick_login" hidden="" method="post"></form>
          <ul class="nav navbar-nav navbar-right">
                  <li class="link-btn"><input type="text" maxlength="64" class="form-control" required="" form="quick_login" name="email_username" placeholder="Email / Username"></li>
        <li class="link-btn"><input type="password" maxlength="32" class="form-control" required="" form="quick_login" name="password" pattern=".{6,}" placeholder="Password"></li>
        <li class="link-btn"><button class="btn btn-theme btn-pill btn-xs btn-line" form="quick_login" type="submit">Login</button></li>

                 </ul>
        </div>
      </div>
    </nav><!-- end main navbar -->

    <!-- mobile navbar -->
    <div class="container">
      <nav class="mobile-nav hidden-md hidden-lg">
        <a href="#" class="btn-nav-toogle first">
          <span class="bars"></span>
          Menu
        </a>
        <div class="mobile-nav-block">
          <h4>Navigation</h4>
          <a href="#" class="btn-nav-toogle">
        <span class="barsclose"></span>
        Close
          </a>

          <ul class="nav navbar-nav">
        <li class=""><a href="https://afghanbids.com/home"><strong>Home</strong></a></li>
        <li class=""><a href="https://afghanbids.com/buyer_front"><strong>BUYERS</strong></a></li>
        <li class=""><a href="https://afghanbids.com/seller_front"><strong>SELLERS</strong></a></li>
<li class=""><a href="https://afghanbids.com/home/contact"><strong>CONTACT US</strong></a></li>
        <li class=""><a href="https://afghanbids.com/home/about"><strong>ABOUT US</strong></a></li>

                 <li class=""><a href="https://afghanbids.com/admin/user/register"><strong>Registration</strong></a></li>
        <li><a href="https://afghanbids.com/admin/user/login_page/1"><strong>Login</strong></a></li>
                   </ul>
        </div>
      </nav>
    </div><!-- mobile navbar -->

    <!-- <div class="container"> -->
      <!-- <img class="banner-image" src="header-banner.jpg" width="100%"> -->
    <!-- </div> -->

    <!-- form search area-->
        <div class="container">
      <div class="row" style="margin-top: -30px">
        <div class="col-md-5">
          <div class="col-md-8">
        <div class="logo text-center inline">
            <!-- <img src="assets/theme/images/logo.png" alt="">  -->
            <h3 class="banner-msg"><strong></strong></h3>
            <!-- <div class="label-add">Resume</div> -->
          <!-- </a> -->
        </div>
          </div>
          <div class="col-md-4">
        <div class="logo text-center inline">
                  </div>
          </div>

        </div>
        <div class="col-md-7">

          <!-- form search  -->
          <form method="get" action="https://afghanbids.com/home/search" class="form-search-list">
        <div class="row">

          <div class="col-sm-10 col-xs-12">
            <div class="form-group" style="margin-top: 3px;">
              <label class="color-white">Search For Opportunities</label>
              <input class="form-control" maxlength="128" required="" name="keywords" placeholder="Type your keywords here...">
            </div>
          </div>
          <div class="col-sm-2 col-xs-12">
            <div class="form-group">
              <label class="hidden-xs">&nbsp;</label>
              <button class="btn btn-block btn-theme  btn-success">Search</button>
            </div>
          </div>
        </div>
        <!--<p class="text-right"><a  href="#modal-advanced" data-toggle="modal" class="link-white">Advanced Search</a></p>-->
          </form><!-- end form search  -->


        </div>
      </div>
    </div><!-- end form search area-->
          </header><!-- end main-header -->      <!-- body-content -->
      <div class="body-content clearfix">

    <div class="block-section bg-color2">
      <div class="container">
        <!-- text centered -->
        <div class="text-center">
          <h1>Sorry, page not found!</h1>

        <!--   <ul class="list-inline">
        <li> <a href="#">Home</a></li>
        <li><a href="#">Find a Job</a></li>
        <li> <a href="#">Blog</a></li>
        <li> <a href="#">Contact Us</a></li>
          </ul> -->
        </div><!-- end text centered -->

        <!-- big text error -->
        <div class="big-error">404</div><!-- end big text error -->
      </div>
    </div>

      </div><!--end body-content -->

       <!-- main-footer -->
      <footer class="main-footer">


    <div class="container">
      <div class="row">
        <div class="col-sm-6">
          <ul class="list-inline link-footer text-center-xs">
        <li><a href="https://afghanbids.com/">Home</a></li>
        <li><a href="https://afghanbids.com/home/about">About Us</a></li>
        <li><a href="https://afghanbids.com/home/contact">Contact Us</a></li>
          </ul>
        </div>
        <div class="col-sm-6 ">
          <span class="pull-right">Powered By: <a target="_blank" href="http://www.codezone.af">CodeZone</a></span>
        </div>

      </div>
    </div>
      </footer><!-- end main-footer -->

    </div><!-- end wrapper page -->




    <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
    <!-- <script src="/plugins/jquery.js"></script> -->

    <!-- jQuery Bootstrap -->
    <script src="https://afghanbids.com/assets/plugins/bootstrap-3.3.2/js/bootstrap.min.js"></script>
    <script src="https://afghanbids.com/assets/plugins/jquery.easing-1.3.pack.js"></script>
    <!-- Lightbox -->
    <!-- <script src="https://afghanbids.com/assets/plugins/magnific-popup/jquery.magnific-popup.min.js"></script> -->

    <!-- Theme JS -->
    <script src="https://afghanbids.com/assets/theme/js/theme.js"></script>


</body></html>

Analysis

Initially I suspected that Selenium driven ChromeDriver initiated google-chrome Browsing Context was getting detected.

But surprisingly, even the windows based shortcuts to access the google-chrome-devtools remains disabled through either of the following ways on the manually inititiated session:

F12
ctrl+shift+I
ctrl+shift+J
ctrl+shift+C

Additionally, If you forcefully open the google-chrome-devtools through More tools -> Developer tools, the manually inititiated session starts showing Sorry, page not found!. Hence it would be a bit difficult even to analyze the HTML.

Finally, I must admit the developers did a commendable job in restricting the automated bots.

回答2:

Looking at the webpage, it appears when you are using chrome driver it does indeed redirect to an access denied page.

I had a feeling it could be some javascript that was forcing the redirect.

According to this post: How to disable java script in Chrome Driver Selenium Python, you can disable javascript within chrome driver.

Using the code from the answers:

chrome_options = Options()
chrome_options.add_experimental_option("prefs", {'profile.managed_default_content_settings.javascript': 2})
chrome = webdriver.Chrome('chromedriver', chrome_options=chrome_options)

I was able to prevent the redirect to the access denied page. Not the ideal solution but seems to be a useful workaround

来源：https://stackoverflow.com/questions/62978920/website-denies-get-request-using-selenium

标签

python

selenium

google-chrome

selenium-webdriver

selenium-chromedriver