(Web scraping) I've located the proper tags, now how do I extract the text?

问题

I'm creating my first web scraping application that collects the titles of games currently on the "new and trending" tab on https://store.steampowered.com/. Once I figure out how to do this, I want to repeat the process with prices, and export both to a spreadsheet in separate columns.

I've successfully found the tags that contain the text I'm trying to extract (the title), but I'm unsure how to extract the titles once I've located their containers.

from urllib.request import urlopen
from bs4 import BeautifulSoup

my_url = 'https://store.steampowered.com/'
uClient = urlopen(my_url)
page_html = uClient.read()
uClient.close()

page_soup = BeautifulSoup(page_html, "html.parser")
containers = page_soup.findAll("div",{"class":"tab_item_name"}, limit=10)

for titles in containers:
    print(titles)

What I'm trying to do is print the names of the 10 games that appear on steam's homepage in a vertical list using a for loop. What actually happens is I print out the tags that contain the titles:

<div class="tab_item_name">Destiny 2: Shadowkeep</div>
<div class="tab_item_name">Destiny 2</div>
<div class="tab_item_name">Destiny 2: Forsaken</div>
<div class="tab_item_name">Destiny 2: Shadowkeep Digital Deluxe Edition</div>
<div class="tab_item_name">NGU IDLE</div>
<div class="tab_item_name">Kaede the Eliminator / Eliminator 小枫</div>
<div class="tab_item_name">Spaceland</div>
<div class="tab_item_name">Cube World</div>
<div class="tab_item_name">Aokana - Four Rhythms Across the Blue</div>
<div class="tab_item_name">CODE VEIN</div>

回答1:

Just read the documentation:

If you only want the text part of a document or tag, you can use the get_text() method. It returns all the text in a document or beneath a tag, as a single Unicode string.

So just do:

# Should be `title` IMO, because you are currently handling a single title
for titles in containers:
    print(titles.get_text())

回答2:

use titles.text or, even titles.get_text() whichever you prefer to get the title text like below:

from urllib.request import urlopen
from bs4 import BeautifulSoup

my_url = 'https://store.steampowered.com/'
uClient = urlopen(my_url)
page_html = uClient.read()
uClient.close()

page_soup = BeautifulSoup(page_html, "html.parser")
containers = page_soup.findAll("div",{"class":"tab_item_name"}, limit=11)

for titles in containers:
    print(titles.text)

回答3:

Another very convenient way is to use lxml

import requests
import lxml.html

url = 'https://store.steampowered.com/'
# Make the request
response = requests.get(url=url, timeout=5)
# Parse tree
tree = lxml.html.fromstring(response.text)
# Select section corresponding to new games
sub_tree = tree.get_element_by_id('tab_newreleases_content')
# Extract data
games_list = [a.text_content() for a in sub_tree.find_class('tab_item_name')]

# Check
for game in games_list[:11]:
    print(game)
# Destiny 2: Shadowkeep
# Destiny 2
# Destiny 2: Forsaken
# Destiny 2: Shadowkeep Digital Deluxe Edition
# NGU IDLE
# Fernbus Simulator - MAN Lion's Intercity
# Euro Truck Simulator 2 - Pink Ribbon Charity Pack
# Spaceland
# Cube World
# CODE VEIN
# CODE VEIN

来源：https://stackoverflow.com/questions/58196341/web-scraping-ive-located-the-proper-tags-now-how-do-i-extract-the-text