Requests vs Selenium Python in Youtube

Question

When I use Selenium library to find the length of related channel in YouTube channel Page it gives me 12. But when I use Req

Answer 1

Sometimes the issue is caused by the soup object having different tags from the ones you see from dev tools, which is what is happening in your case. On analysing the soup object you'll notice the information you need is actually now in <h3 class="yt-lockup-title ">.

This code will pull the results you want:

import requests
from bs4 import BeautifulSoup

r = requests.get("https://www.youtube.com/channel/UCoykjkkJxsz7JukJR7mGrwg/about")
soup = BeautifulSoup(r.content, 'html.parser')
bb=soup.find_all('h3',class_='yt-lockup-title')
print(len(bb))

Answer 2

Every time I've run into an issue like this, it was because JS was creating the data I was after. If this is the case, you likely won't be able to use requests as it can't handle the JS.

If you navigate to that youtube page in a browser, you can see that "ytd-mini-channel-renderer" exists if you inspect it, but if you view source, you get 0 results. The code you can see from "view source" is what requests is getting.