python从入门到放弃篇41（selenium库，requests库，lxml库，time库，os模块，jieba库，pyecharts库）实现爬取书籍信息，并进行可视化

我们这次的案例网址是：https://book.douban.com/tag/%E7%BC%96%E7%A8%8B。
最近在研究爬虫和数据可视化的结合，所以，时间上分配的有点不太宽裕。

爬虫部分的完整代码：

from selenium import webdriver
from lxml import etree
import requests
import time
import os

options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome(options=options)

headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36 Edg/84.0.522.52'
}
url = 'https://book.douban.com/tag/%E7%BC%96%E7%A8%8B'
driver.get(url)

books = driver.find_element_by_xpath('//*[@id="content"]/div/div[1]').text
#print(books)
with open('example_book.txt','a',encoding='utf-8') as f:
    f.write(books)

运行结果，去编辑器路径下面找到文件打开：
在这里插入图片描述
接着，我们就要用jieba库（第三方库），来进行分词，分词后进行词频统计，所以有：

import jieba
txt = open("D:/pycharm_project/example_book.txt","r",encoding="utf-8").read()
words = jieba.lcut(txt,cut_all=True)
counts = {}
py_num = 0
book_num = 0
computer_num = 0
board_num = 0
for word in words:
    if len(word) == 1:
        continue
    elif word == "python编程" or word == "python":
        word1 = "python"
        py_num += 1
        counts[word1] = py_num
    elif word == "本书" or word == "书":
        word2 = "书"
        book_num += 1
        counts[word2] = book_num
    elif word == "计算机" or word == "算法":
        word3 = "计算机"
        computer_num += 1
        counts[word3] = computer_num
    elif word == "出版社" or word == "出版":
        word4 = "出版"
        board_num += 1
        counts[word4] = board_num
    else:
        continue
    
print(counts)

结果：
在这里插入图片描述
然后，我们再做修改，得到如下结果，这样我们就完成了分词和词频统计。

接下来就是数据可视化部分了。我们要用到pyecharts库（要在联网情况下操作）和OS系统模块来创建文件夹存放HTML文件，所以有：

from pyecharts.charts import Bar
import os
B = Bar()
B.add_xaxis(['出版','书','计算机'])
B.add_yaxis('词语出现次数',[42,13,19])
os.mkdir('E:/Example')
B.render('E:/Example/豆瓣图书爬取可视化.html')