问题
I am trying to crawl this currency rate website: https://banking.nonghyup.com/servlet/PGEF0011I.view
(Just click the exchange rate check and you'll see the pic like below)
I want to crawl this dynamical table and I find its url via "Inspect->Network is : https://banking.nonghyup.com/servlet/PGEF0012R.frag
However, when I start to request it by selenium, it returns error due to abnormal requests, which comes up "We're sorry for causing you any inconvenience. You will not be able to use the service due to abnormal request. Please try again after confirmation." I am not sure how to proceed to the next step. Could you help? Many thanks!
driver.get('https://banking.nonghyup.com/servlet/PGEF0012R.frag')
soup = BeautifulSoup(driver.page_source, 'html.parser')
回答1:
Seems you were close. Accessing the url https://banking.nonghyup.com/servlet/PGEF0011I.view manually for the first time also shows the following error:
We're sorry for causing you any inconvenience. You will not be able to use the service due to abnormal request. Please try again after confirmation.
Guide Code : 90304, 16.49-03
Snapshot:
However, as per your mentioned steps through clicking on the element with text as Exchange Rate Check also you can crawl the currency rates using the following Locator Strategies:
Code Block:
from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC options = webdriver.ChromeOptions() options.add_argument("start-maximized") options.add_experimental_option("excludeSwitches", ["enable-automation"]) options.add_experimental_option('useAutomationExtension', False) driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe') driver.get("https://banking.nonghyup.com/servlet/PGEF0011I.view") WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "Exchange Rate Check"))).click() WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[starts-with(@summary, 'Country,Currency Namem,Cash')]//td"))) print(driver.find_element_by_xpath("//table[starts-with(@summary, 'Country,Currency Namem,Cash')]").get_attribute("outerHTML"))Console Output:
<table class="tb_col t_center" summary="Country,Currency Namem,Cash(When Buying,When Selling),Remittance(When Sending,When Receiving),Base Exchange Rate,Compared to D-1"> <caption>??????2</caption> <colgroup> <col style="width:12%;"> <col style="width:auto;"> <col style="width:13%;"> <col style="width:13%;"> <col style="width:13%;"> <col style="width:13%;"> <col style="width:13%;"> <col style="width:13%;"> </colgroup> <thead> <tr> <th rowspan="2">Country</th> <th rowspan="2">Currency Name</th> <th colspan="2">Cash</th> <th colspan="2">Remittance</th> <th rowspan="2">Base Exchange Rate</th> <th rowspan="2" style="border-right: none;">Compared to the previous day</th> </tr> <tr> <th>When Buying</th> <th>When Selling</th> <th>When Sending</th> <th>When Receiving</th> </tr> </thead> <tbody> <tr> <td>USA</td> <td>USD</td> <td class="t_right">1,188.74</td> <td class="t_right">1,147.86</td> <td class="t_right">1,179.60</td> <td class="t_right">1,157.00</td> <td class="t_right">1,168.30</td> <td style="border-right: none;"> <span class="raise"><em>??</em>5.80</span> </td> </tr> <tr> <td>Japan</td> <td>JPY(100)</td> <td class="t_right">1,090.24</td> <td class="t_right">1,052.74</td> <td class="t_right">1,081.99</td> <td class="t_right">1,060.99</td> <td class="t_right">1,071.49</td> <td style="border-right: none;"> <span class="raise"><em>??</em>7.03</span> </td> </tr> <tr> <td>EU</td> <td>EUR</td> <td class="t_right">1,312.25</td> <td class="t_right">1,260.79</td> <td class="t_right">1,299.38</td> <td class="t_right">1,273.66</td> <td class="t_right">1,286.52</td> <td style="border-right: none;"> <span class="raise"><em>??</em>4.76</span> </td> </tr> <tr> <td>China</td> <td>CNY</td> <td class="t_right">176.35</td> <td class="t_right">156.39</td> <td class="t_right">168.01</td> <td class="t_right">164.73</td> <td class="t_right">166.37</td> <td style="border-right: none;"> <span class="raise"><em>??</em>0.43</span> </td> </tr> <tr> <td>England</td> <td>GBP</td> <td class="t_right">1,529.50</td> <td class="t_right">1,469.52</td> <td class="t_right">1,514.35</td> <td class="t_right">1,484.67</td> <td class="t_right">1,499.51</td> <td style="border-right: none;"> <span class="raise"><em>??</em>9.42</span> </td> </tr> <tr> <td>Swiss</td> <td>CHF</td> <td class="t_right">1,203.57</td> <td class="t_right">1,156.39</td> <td class="t_right">1,191.66</td> <td class="t_right">1,168.30</td> <td class="t_right">1,179.98</td> <td style="border-right: none;"> <span class="raise"><em>??</em>11.06</span> </td> </tr> <tr> <td>Canada</td> <td>CAD</td> <td class="t_right">898.34</td> <td class="t_right">863.12</td> <td class="t_right">889.53</td> <td class="t_right">871.93</td> <td class="t_right">880.73</td> <td style="border-right: none;"> <span class="raise"><em>??</em>3.55</span> </td> </tr> <tr> <td>Hong Kong</td> <td>HKD</td> <td class="t_right">152.19</td> <td class="t_right">146.23</td> <td class="t_right">150.68</td> <td class="t_right">147.74</td> <td class="t_right">149.21</td> <td style="border-right: none;"> <span class="raise"><em>??</em>0.67</span> </td> </tr> <tr> <td>Sweden</td> <td>SEK</td> <td class="t_right">123.05</td> <td class="t_right">117.05</td> <td class="t_right">121.23</td> <td class="t_right">118.87</td> <td class="t_right">120.05</td> <td style="border-right: none;"> <span class="raise"><em>??</em>0.21</span> </td> </tr> <tr> <td>Australia</td> <td>AUD</td> <td class="t_right">814.61</td> <td class="t_right">782.67</td> <td class="t_right">806.54</td> <td class="t_right">790.74</td> <td class="t_right">798.64</td> <td style="border-right: none;"> <span class="raise"><em>??</em>3.32</span> </td> </tr> <tr> <td>Denmark</td> <td>DKK</td> <td class="t_right">176.47</td> <td class="t_right">167.87</td> <td class="t_right">173.87</td> <td class="t_right">170.47</td> <td class="t_right">172.17</td> <td style="border-right: none;"> <span class="raise"><em>??</em>0.64</span> </td> </tr> <tr> <td>Norway</td> <td>NOK</td> <td class="t_right">130.28</td> <td class="t_right">123.94</td> <td class="t_right">128.36</td> <td class="t_right">125.86</td> <td class="t_right">127.11</td> <td style="border-right: none;"> <span class="raise"><em>??</em>0.09</span> </td> </tr> <tr> <td>Saudi Arabia</td> <td>SAR</td> <td class="t_right">342.67</td> <td class="t_right">280.37</td> <td class="t_right">314.60</td> <td class="t_right">308.44</td> <td class="t_right">311.52</td> <td style="border-right: none;"> <span class="raise"><em>??</em>1.54</span> </td> </tr> <tr> <td>Kuwait</td> <td>KWD</td> <td class="t_right">4,230.03</td> <td class="t_right">3,460.95</td> <td class="t_right">3,883.56</td> <td class="t_right">3,807.42</td> <td class="t_right">3,845.49</td> <td style="border-right: none;"> <span class="raise"><em>??</em>19.34</span> </td> </tr> <tr> <td>Bahrain</td> <td>BHD</td> <td class="t_right">3,408.69</td> <td class="t_right">2,788.93</td> <td class="t_right">3,129.48</td> <td class="t_right">3,068.14</td> <td class="t_right">3,098.81</td> <td style="border-right: none;"> <span class="raise"><em>??</em>15.35</span> </td> </tr> <tr> <td>UAE</td> <td>AED</td> <td class="t_right">349.86</td> <td class="t_right">286.26</td> <td class="t_right">321.20</td> <td class="t_right">314.92</td> <td class="t_right">318.06</td> <td style="border-right: none;"> <span class="raise"><em>??</em>1.56</span> </td> </tr> <tr> <td>Singapore</td> <td>SGD</td> <td class="t_right">878.86</td> <td class="t_right">836.00</td> <td class="t_right">865.91</td> <td class="t_right">848.95</td> <td class="t_right">857.43</td> <td style="border-right: none;"> <span class="raise"><em>??</em>3.35</span> </td> </tr> <tr> <td>Malaysia</td> <td>MYR</td> <td class="t_right">298.22</td> <td class="t_right">256.02</td> <td class="t_right">284.12</td> <td class="t_right">278.56</td> <td class="t_right">281.34</td> <td style="border-right: none;"> <span class="raise"><em>??</em>0.58</span> </td> </tr> <tr> <td>New Zealand</td> <td>NZD</td> <td class="t_right">762.00</td> <td class="t_right">732.12</td> <td class="t_right">754.45</td> <td class="t_right">739.67</td> <td class="t_right">747.06</td> <td style="border-right: none;"> <span class="raise"><em>??</em>10.62</span> </td> </tr> <tr> <td>Thailand</td> <td>THB</td> <td class="t_right">41.13</td> <td class="t_right">36.11</td> <td class="t_right">39.00</td> <td class="t_right">38.24</td> <td class="t_right">38.62</td> <td style="border-right: none;"> <span class="raise"><em>??</em>0.31</span> </td> </tr> <tr> <td>Indonesia</td> <td>IDR(100)</td> <td class="t_right">9.11</td> <td class="t_right">7.47</td> <td class="t_right">8.37</td> <td class="t_right">8.21</td> <td class="t_right">8.29</td> <td style="border-right: none;"> <span class="raise"><em>??</em>0.03</span> </td> </tr> <tr> <td>Taiwan</td> <td>TWD</td> <td class="t_right">40.77</td> <td class="t_right">35.81</td> <td class="t_right">38.66</td> <td class="t_right">37.92</td> <td class="t_right">38.29</td> <td style="border-right: none;"> <span class="raise"><em>??</em>0.10</span> </td> </tr> <tr> <td>Philippine</td> <td>PHP</td> <td class="t_right">24.44</td> <td class="t_right">21.46</td> <td class="t_right">23.17</td> <td class="t_right">22.73</td> <td class="t_right">22.95</td> <td style="border-right: none;"> <span class="raise"><em>??</em>0.07</span> </td> </tr> <tr> <td>India</td> <td>INR</td> <td class="t_right">17.72</td> <td class="t_right">14.64</td> <td class="t_right">16.42</td> <td class="t_right">16.10</td> <td class="t_right">16.26</td> <td style="border-right: none;"> <span class="raise"><em>??</em>0.03</span> </td> </tr> <tr> <td>Russia</td> <td>RUB</td> <td class="t_right">19.40</td> <td class="t_right">15.97</td> <td class="t_right">18.31</td> <td class="t_right">17.97</td> <td class="t_right">18.14</td> <td style="border-right: none;"> <span class="decline"><em>??</em>0.01</span> </td> </tr> <tr> <td>South Africa</td> <td>ZAR</td> <td class="t_right">78.10</td> <td class="t_right">78.10</td> <td class="t_right">79.03</td> <td class="t_right">77.17</td> <td class="t_right">78.10</td> <td style="border-right: none;"> <span>0.00</span> </td> </tr> <tr> <td>Mexico</td> <td>MXN</td> <td class="t_right">60.38</td> <td class="t_right">60.38</td> <td class="t_right">60.98</td> <td class="t_right">59.78</td> <td class="t_right">60.38</td> <td style="border-right: none;"> <span class="decline"><em>??</em>0.37</span> </td> </tr> <tr> <td>Vietnam</td> <td>VND(100)</td> <td class="t_right"> - </td> <td class="t_right"> - </td> <td class="t_right"> - </td> <td class="t_right"> - </td> <td class="t_right">5.03</td> <td style="border-right: none;"> <span class="raise"><em>??</em>0.03</span> </td> </tr> </tbody> </table>
来源：https://stackoverflow.com/questions/58833294/crawl-javascript-table-but-considered-abnormal-requests