Scraping multiple paginated links with BeautifulSoup and Requests

Question

Python Beginner here. I\'m trying to scrape all products from one category on dabs.com. I\'ve managed to scrape all products on a given page, but I\'m having trouble iterating o

Answer 1

Their next page button has a title of "Next" you could do something like:

import requests
from bs4 import BeautifulSoup as bs

url = 'www.dabs.com/category/computing/11001/'
base_url = 'http://www.dabs.com'

r = requests.get(url)

soup = bs(r.text)
elm = soup.find('a', {'title': 'Next'})

next_page_link = base_url + elm['href']

Hope that helps.