selenium

webscraping using python and selenium and tried to use multiprocessing but code not working. without it code works fine

廉价感情. 提交于 2021-02-11 16:55:15
问题 Am doing web scraping with python & selenium. I used to scrape data for one location & year at a time, by creating 1800 .py files (600 places * 3 years = 1800) and batch opening 10 at a time and waiting for it to complete. which is time-consuming so I decided to use multiprocessing. I made my code to read places data from a text file and iterate with it. the text file looks like this Aandimadam Aathur_Dindugal Aathur_Salem East Abiramam Acchirapakkam Adayar Adhiramapattinam Alandur

Iterating through each line of multiple text files, unique to each thread in Python selenium

China☆狼群 提交于 2021-02-11 16:39:28
问题 So this code below does half the job I'm looking for, but I would like to implement a second text file along with the first one that basically does the same for another variable. So the first thread will read the first line of text file 1 and also the first line of text file 2, then the second thread will read the second line of text file 1 and the second line of text file 2. Sounds confusing, buts its not. Thanks in advance!! import threading import time def test_logic(file, file2): # **

Iterating through each line of multiple text files, unique to each thread in Python selenium

房东的猫 提交于 2021-02-11 16:38:07
问题 So this code below does half the job I'm looking for, but I would like to implement a second text file along with the first one that basically does the same for another variable. So the first thread will read the first line of text file 1 and also the first line of text file 2, then the second thread will read the second line of text file 1 and the second line of text file 2. Sounds confusing, buts its not. Thanks in advance!! import threading import time def test_logic(file, file2): # **

get financial data using Python

点点圈 提交于 2021-02-11 16:31:43
问题 I have managed to write some Python code and Selenium that navigates to a webpage that contains financial data that is in some tables. I want to be able to extract the data and put it into excel. The tables seem to be html based tables code below: <tr> <td class="bc2T bc2gt">Last update</td> <td class="bc2V bc2D">03/15/2018</td><td class="bc2V bc2D">03/14/2019</td><td class="bc2V bc2D">03/12/2020</td><td class="bc2V bc2D" style="background-color:#DEFEFE;">05/22/2020</td><td class="bc2V bc2D"

Python Selenium webscraping of Tableau Public: how to assign favourites to workbook?

巧了我就是萌 提交于 2021-02-11 15:31:45
问题 I have written my first Selenium script to practise webscraping in Python. The idea is to scrape all workbooks, views and favourites from a Tableau Public profile. I managed to extract those three key variables, but I don't know how to assign favourites to their respective workbooks since not all workbooks have at least one favourite. For example "Skyler on Broadway" has no favourites, but if I were to match workbooks and favourites in a dictionary, it would pull in the next best value,

Finding an element using regex selenium java

感情迁移 提交于 2021-02-11 15:22:36
问题 I have an element whose value changes dynamically. This is the following: "Pay 13.61" I want to save the element 13.61 in a string and compare with 0. This is how the appium structure looks like: enter image description here I want to extract the value of pay. This is what I did: String amt = wait.until( ExpectedConditions.presenceOfElementLocated(By.xpath("//*[@text='Pay '+%s]"))).getText(); logger.info(amt); logger.info("Cash Pay succeeded!"); Its not able to find the element, can anyone

Python: Headless mode picks up the older version of chrome

廉价感情. 提交于 2021-02-11 15:10:05
问题 I am trying to automate sending a text message using python & selenium from https://voice.google.com/about . When I run below code, it picks up the lattest version/ instance of google chrome and works fine. But, when I run it headless mode, it uses an older version of google chrome(guessing from screenshot) and gives TimeoutException() exception. It fails while logging in. I checked an element id of sign in box in an IE emulator and it appears to be the same. url = "https://voice.google.com

Extracting table data from website using chrome

笑着哭i 提交于 2021-02-11 15:09:40
问题 i want to extract table data from website with chrome browser with selenium. i wrote below code but it's not working Sub Chartinka() Dim bot As New WebDriver, posts As WebElements, post As WebElement, i As Integer, mysheet As Worksheet, keys As Selenium.keys bot.Start "chrome", "https://chartink.com/screener/buy-15m-78" bot.Get "/" Set posts = bot.FindElementsByXPath("//*[@id='DataTables_Table_0']/tbody/tr[1]") i = 2 Set mysheet = Sheets("Sheet3") For Each post In posts ' Run time Error '438'

Selenium is not properly determining if a page has a <title> html tag

佐手、 提交于 2021-02-11 15:08:45
问题 I'm trying to get Selenium to wait until the title tag of a web page is present when loading with Python. I've tried testing this code with other types of HTML tags and only the <body> tag didn't result in an error. wait = WebDriverWait(driver, 10) driver.get(link) wait.until(EC.visibility_of_element_located((By.TAG_NAME, 'div'))) I expected the code to evaluate to completion but I got the following error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib

TestNG Selenium ChromeDriver : java.lang.NoSuchMethodError

梦想的初衷 提交于 2021-02-11 14:59:50
问题 by wanting to perform a first execution of my framework, i launch a simple testcase with Selenium - testNG on intellij IDEA and, i have an error of which I do not know the real origin. This is my POM.xml dependencies <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion>