问题
I am trying to scrape Naukri job postings. Web scraping was too time-consuming, so I switched to network requests. I believe I got the request pattern for pagination by changing the URL right (not clicking the next tab).
URLs Example:
https://www.naukri.com/maintenance-jobs?xt=catsrch&qf%5B%5D=19
https://www.naukri.com/maintenance-jobs-2?xt=catsrch&qf%5B%5D=19
https://www.naukri.com/maintenance-jobs-3?xt=catsrch&qf%5B%5D=19
https://www.naukri.com/maintenance-jobs-4?xt=catsrch&qf%5B%5D=19
The below code works on the first page properly. However, it is failing to paginate and get data from the next request link. Instead, for each run, it is getting data from the first link.
Where am I going wrong? How to paginate correctly with URL requests?
import pandas as pd
import time
import re
from bs4 import BeautifulSoup
import unicodecsv
import requests
import ast
import json
import time
import logging
import datetime
headers_naukri={
"authority": 'www.naukri.com',
"user-agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 Edg/85.0.564.70',
'x-newrelic-id': 'VgUHWVFXABACXVdaBAgHUF0=',
"systemid": '109',
"content-type": "application/json",
"accept": 'application/json',
"cache-control": "no-cache",
"clientid": 'd3skt0p',
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
'referer': 'https://www.naukri.com/maintenance-jobs?xt=catsrch&qf[]=19',
"sec-fetch-site": "same-origin",
"accept-encoding": "gzip, deflate, br",
"accept-language": "en-US,en;q=0.9",
"appid": '109',
"content-type": "application/json"
}
session = requests.Session()
source_url='https://www.naukri.com/jobapi/v3/search?noOfResults=20&urlType=search_by_keyword&searchType=adv&keyword=maintenance&xt=catsrch&functionAreaId=19&seoKey=maintenance-jobs&src=jobsearchDesk&latLong='
print(source_url)
try:
req = session.get(source_url, headers = headers_naukri)
except requests.exceptions.TooManyRedirects:
time.sleep(15)
req = session.get(source_url,headers=headers_naukri)
source_name = 'naukri'
source_id = '1'
source = req.text
#print(source)
source_json = json.loads(source)
noOfJobs=source_json['noOfJobs']
print(noOfJobs)
pages = (noOfJobs // 20) + 1
print(pages)
print(source_json['jobDetails'][0]['jobId'])
a=headers_naukri['referer'].split("?")
for i in range(2,pages,1):
nexturl=source_url.replace('&src=', '-'+str(i)+'&src=')
nexturl=nexturl.replace('&xt', '&pageNo='+str(i)+'&xt')
headers_naukri['referer']= a[0]+'-'+str(i)+'?'+a[1]
print(headers_naukri['referer'])
#print(nexturl)
try:
req = session.get(source_url, headers = headers_naukri)
headers_naukri['referer']= headers_naukri['referer'][:-18]+'-'+str(i)+headers_naukri['referer'][-18:]
except requests.exceptions.TooManyRedirects:
time.sleep(15)
req = session.get(source_url,headers=headers_naukri)
source_name = 'naukri'
source_id = '1'
source = req.text
#print(source)
source_json = json.loads(source)
print(source_json['jobDetails'][1]['jobId'])
Output:
https://www.naukri.com/jobapi/v3/search?noOfResults=20&urlType=search_by_keyword&searchType=adv&keyword=maintenance&xt=catsrch&functionAreaId=19&seoKey=maintenance-jobs&src=jobsearchDesk&latLong=
143263
7164
300920001858
https://www.naukri.com/maintenance-jobs-2?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-3?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-4?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-5?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-6?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-7?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-8?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-9?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-10?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-11?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-12?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-13?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-14?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-15?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-16?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-17?xt=catsrch&qf[]=19
You can clearly see that on each request, the first job post id is the same, and it is not paginating.
回答1:
Just loop through the pages by updating the pageNo value in the payload and get whatever you need from the JSON.
Here's an example of how to get job results for the first three pages:
from urllib.parse import urlencode
import requests
url = "https://www.naukri.com/jobapi/v3/search?"
headers = {
"appid": "109",
"systemid": "109",
"clientid": "d3skt0p",
"accept": "application/json",
"accept-encoding": "gzip, deflate, br",
"user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36",
}
payload = {
"noOfResults": 20,
"urlType": "search_by_keyword",
"searchType": "adv",
"keyword": "maintenance",
"xt": "catsrch",
"functionAreaId": 19,
"seoKey": "maintenance-jobs-3",
"src": "jobsearchDesk",
}
for page_number in range(1, 4):
payload['pageNo'] = page_number
search_results = requests.get(url, params=urlencode(payload), headers=headers).json()
for job in search_results['jobDetails']:
print(job['title'])
Output:
Assistant Manager - Maintenance and Engineering
Maintenance Engineer
Electrical Maintenance Engineer - DRI Plant - Nigeria Location
Shift Engineer Mechanical Maintenance - DRI Plant - Nigeria Location
Maintenance Engineer I
Maintenance Engineer I
Senior Engineer/ Assistant Manager - Textile Maintenance
Senior Manager - Electrical - Project & Maintenance –WHRB-Nigeria
Core Maintenance Engg Utilities
State Core Maintenance Engg Utilities
State Core Maintenance Engg Fiber
State Core Maintenance Engg Utilities
State Core Maintenance Engg Utilities
Core Maintenance Engg Utilities
Electrical Maintenance Engineer
Electrical Engineer - Operations & Maintenance
Electrical Maintenance Engineer,
Job opening For Jr Maintenance Engineer For Hospital in Pune
Assistant Manager Maintenance
Mech Maintenance Head - Sr. Manager - DRI Plant - Nigeria Location
Deputy Manager Maintenance - At Plant Level Manesar Sec 3
Senior Manager - Mechanical Maintenance – WHRB (For Nigeria)
Sand Plant Mechanical Maintenance Engineer/ Sr. Engineer
Urgent Requirement For Maintenance Engineer For Rudrapur
CNC Service Engineer / Maintenance Engineer (Mechanical)
Operations & Maintenance, Mech Engr, Natural Gas Processing, Nigeria
Maintenance Engineer
CNC Maintenance Engineer
Utility Maintenance Engineer
Engineer - Maintenance
Machine Maintenance Engineer
Maintenance Engineer
Maintenance Engineer
Maintenance Engineer - Textile Industry
MACHINERY MAINTENANCE ENGINEER
Engineer - Maintenance
Assistant Engineer - Maintenance
Electrical Maintenance Technicians - Sodexo Sriperumbudur location
Electrical Maintenance Technicians - Sodexo Sriperumbudur location
Site Engineer/ Sr. Engineer- Operation & Maintenance
Senior Aircraft Maintenance Engineering
Assistant Engineer - Maintenance
Electrical Maintenance Technicians - Sodexo Sriperumbudur location
Electrical Maintenance Technicians - Sodexo Sriperumbudur location
Site Engineer/ Sr. Engineer- Operation & Maintenance
Electrical Maintenance - Telangana
Engineer/Sr. Engineer/AM in Production, Quality, Process,
Maintenance
Jr. Maintenance Engineer (Utility)
Operation & Maintenance Engineer
Electrical Maintenance Technician-sri City
Electrical Maintenance Technician
Maintenance Manager
Maintenance Manager
Sr. Exe. / Asst. Mgr. - Plant Maintenance & Engineering Deptt.- Dubai
Senior Engineering(Maintenance) Manager
Team Member Maintenance
Fleet Executive (service/repair/maintenance) Anywhere in U.P.
Executive- Maintenance(Sterile API)
Mechanical Maintenance Technicians - Sodexo Sriperumbudur location
Mechanical Maintenance Technicians - Sodexo Sriperumbudur location
来源：https://stackoverflow.com/questions/64295244/paginate-with-network-requests-scraper