How to scrape newspaper articles from website using selenium and beautifulsoup in python?

Question

I am trying to collect the date, title, and content from the newspaper (the new york times).

Date and title I got, but the full article I couldn\'t able to. Below i

Answer 1

It is possible. You'll have to get each link to the article and pull that. I added it below and also cleaned up the article a bit as there were extra spaces and what not in the text when you join it all together.

import time
import requests
from bs4 import BeautifulSoup
import json
import string
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

base = "https://www.nytimes.com"
browser = webdriver.Chrome('C:/chromedriver_win32/chromedriver.exe')
wait = WebDriverWait(browser, 10)
browser.get('https://www.nytimes.com/search?endDate=20190331&query=cybersecurity&sort=newest&startDate=20180401')

while True:
    try:
        time.sleep(1)
        show_more = wait.until(EC.element_to_be_clickable((By.XPATH, '//button[@type="button"][contains(.,"Show More")]')))  
        show_more.click()
    except Exception as e:
            print(e)
            break    

soup = BeautifulSoup(browser.page_source,'lxml')
search_results = soup.find('ol', {'data-testid':'search-results'})

links = search_results.find_all('a')
for link in links:
    link_url = link['href']

    title = link.find('h4').text
    date = link.find_next('time').text
    print(date + ': '+ title + '\n')

    response = requests.get(base + link_url)
    soup_link = BeautifulSoup(response.text, 'html.parser')
    scripts = soup_link.find_all('script')
    for script in scripts:
        if 'window.__preloadedData = ' in script.text:
            jsonStr = script.text
            jsonStr = jsonStr.split('window.__preloadedData = ')[-1]
            jsonStr = jsonStr.rsplit(';',1)[0]

            jsonData = json.loads(jsonStr)

            article = []
            for k, v in jsonData['initialState'].items():
                w=1
                try:
                    if v['__typename'] == 'TextInline':
                        article.append(v['text'])
                        #print (v['text'])
                except:
                    continue
            article = [ each.strip() for each in article ]
            article = ''.join([('' if c in string.punctuation else ' ')+c for c in article]).strip()
    print (article + '\n')

print("Complete")

browser.quit()

Output:

March 31: Bezos’ Security Consultant Accuses Saudis of Hacking the Amazon C.E.O.’s Phone

SEATTLE — Jeff Bezos’ security consultant accused the Saudi government of gaining unauthorized access to the Amazon chief executive’s phone, as part of an effort to harm the world’s richest man. In an opinion article in The Daily Beast on Saturday, Gavin de Becker, Mr. Bezos’ security chief, alleged the Saudis wanted to hurt Mr. Bezos because he owns The Washington Post. The Post has aggressively reported on the murder of Jamal Khashoggi, one of its columnists, who was killed last year in Turkey. United States officials have concluded Mr. Khashoggi, who was critical of Saudi leaders, was killed on the orders of the Saudi crown prince, Mohammed bin Salman. Mr. de Becker said he had turned over his findings about the Saudis and their role against Mr. Bezos to law enforcement. “Our investigators and several experts concluded with high confidence that the Saudis had access to Bezos’ phone, and gained private information,” Mr. de Becker wrote. Mr. de Becker’s claims about the Saudis are difficult to verify and raise many questions. Throughout his article, the security consultant was vague on details. He did not reveal direct evidence of his accusations and wrote that he would not speak again publicly on the issue. On Sunday, American Media Inc., the parent company of The National Enquirer, which first reported earlier this year that Mr. Bezos was having an extramarital affair, issued a statement denying “the false and unsubstantiated claims of Mr. de Becker.” Amazon did not immediately have a comment about Mr. Bezos. An official with the Saudi embassy in Washington declined to comment on Saturday. Mr. de Becker’s allegations are the latest twist in a bizarre situation that has embroiled the largest U.S. tabloid publisher and Mr. Bezos, who runs Amazon and owns The Post. The unusual set of circumstances began in January, when Mr. Bezos announced that he and his wife, MacKenzie Bezos, were divorcing. The next day, The Enquirer published an exposé revealing that Mr. Bezos was romantically involved with Lauren Sanchez, a former Los Angeles TV anchor. Then in February, Mr. Bezos published a post on Medium accusing AMI of “extortion and blackmail.” Mr. Bezos said AMI had threatened to publish graphic photographs of him, including a “below-the-belt selfie,” if he did not publicly affirm that The Enquirer’s reporting on his affair was not motivated by political concerns. He said AMI, which has had ties to the Saudis, was “apoplectic” about The Post’s reporting on the Saudis. Mr. Bezos added that he had asked Mr. de Becker, his longtime security consultant, to investigate who had leaked information and photos about him. Earlier this month, The New York Times reported that two people with direct knowledge of The Enquirer’s reporting said that everything the tabloid received on Mr. Bezos’ affair, including the “below-the-belt selfie,” came from a single source. The Wall Street Journal later reported that AMI had paid Ms. Sanchez’s brother, Michael Sanchez, $200,000 for the texts. In its statement on Sunday, AMI said directly that its source was Mr. Sanchez. “The fact of the matter is, it was Michael Sanchez who tipped the National Enquirer off to the affair on Sept. 10, 2018, and over the course of four months provided all of the materials for our investigation.” AMI added, “There was no involvement by any other third party whatsover.” Mr. de Becker on Saturday said that the effort against Mr. Bezos went beyond Mr. Sanchez and also involved the Saudis. Mr. de Becker pointed to an article published on Saturday by The New York Post in which Mr. Sanchez said The Enquirer “had seen text exchanges between the secret couple” before he was in touch with the tabloid on the matter. “Reality is complicated, and can’t always be boiled down to a simple narrative like ‘the brother did it,’” Mr. de Becker wrote. Mr. de Becker said his investigation included interviews with cybersecurity experts and “people who personally know the Saudi Crown Prince Mohammed bin Salman.” But he stopped short of saying what methods he believed the Saudis may have used to access Mr. Bezos’ personal information. He added that AMI had wanted him to make a public statement “saying that my investigation had concluded they hadn’t relied upon ‘any form of electronic eavesdropping or hacking in their news-gathering process,’” and that the tabloid’s story was not “instigated, dictated or influenced in any manner by external forces, political or otherwise.” Mr. de Becker said he told AMI in a recorded call that those claims were “not my truth.”

March 29: In Ukraine, Russia Tests a New Facebook Tactic in Election Tampering

Campaigning for Ukraine’s presidential election had just begun to heat up when the authorities announced they had thwarted a Russian plot to use Facebook to undermine the vote. Unlike the 2016 interference in the United States, which centered on fake Facebook pages created by Russians in faraway St. Petersburg, the operation in Ukraine this year had ...

...

Answer 2

You are observing only the first page of search. Where you have the list of articles. To get the content of article you have to send request to the article and fetch content from there.

Here I am fetching Title, Author, Publish Date, Content and storing them in a list. From that list we can create a DataFrame later, if required.

newyork_times_list = []
for a in search_results.find_all('a', href=True):

    newyork_times = {}
    page_url = "https://www.nytimes.com" + a['href']

    try:
        # URL
        newyork_times['URL'] =  page_url

        # Invoke URL
        page = requests.get(page_url)
        page_soup = BeautifulSoup(page.content, 'lxml')

        # Title
        newyork_times['Title'] = page_soup.find('title').text

        # Content
        page_content = ''
        page_soup_div = page_soup.find_all("div", {"class":"StoryBodyCompanionColumn"})
        for p_content in page_soup_div:
            page_content = page_content + p_content.text

        # Content
        newyork_times['Content'] =  page_content

        # Date Time
        page_soup_span = page_soup.find_all("time")
        newyork_times['Publish Date'] = page_soup_span[0].text

        # Author
        page_soup_span = page_soup.find_all("span", {"itemprop": "name"})
        newyork_times['Author'] =  page_soup_span[0].text

        newyork_times_list.append(newyork_times)

        print('Processed', page_url)
    except:
        print('ERROR!', page_url)

print('Done')

Answer 3

In order to scrape Newspaper Articles you can simply use goose library which is simple and elegant. This results you in cleaned article text and title too. For Date you can try using BeautifulSoup.

from goose import Goose
from requests import get

response = get('http://www.nytimes.com/2015/05/19/health/study-finds-dense-breast-tissue-isnt-always-a-high-cancer-risk.html?src=me&ref=general')
extractor = Goose()
article = extractor.extract(raw_html=response.content)
text = article.cleaned_text
title = article.title