问题
I need to scrape the website - https://mphc.gov.in/judgement-orders Under the section Free-Text-text, I need to Enter 'A' in the Free-Text Field. Then select the date range - Say - 19-06-2020 to 19-06-2020 , then click the Search Button.
I tried doing this using the below code :
def start_requests(self):
yield scrapy.Request(self.start_urls[0],
callback=self.parse,errback=self.errback_httpbin,dont_filter=True)
def parse(self, response):
headers={
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.9',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36',
'X-Requested-With': 'XMLHttpRequest',
'Connection': 'keep-alive'
}
data={
'btn_search':'A',
'ddl_o_j_fo':'',
'fno':'01',
'txt_order_dt':'',
'j1':'',
'j2':'',
'pj':'',
'ct':'',
'cn':'',
'cy':'',
'id':'IND',
'page':'1',
'sort':'jo',
'ad':'DESC',
'date1':'19-06-2020',
'date2':'19-06-2020',
'code':'',
'csrf_token':'1b299f78533375d78ddd36c860858d1293a0f3a1',
}
yield Request(self.start_urls[0],callback=self.start_scraping)
def start_scraping(self, response):
open_in_browser(response)
all_data = response.css('body').extract()
caseno = response.css('#dv_srh_dt a').extract()
sb=response.css('#bench+ table b:nth-child(1)').extract()
tot=response.css('.bbb b').extract()
#print(all_data)
print(tot)
print("-------------------------------------")
print(caseno)
print(sb)
But it returns the empty list as the output. No idea why it is not filling in the details and scraping the required data.
Any help would be appreciated. Thanks
回答1:
The error is in html.parser, when you use lxml you will get correct HTML:
import requests
from bs4 import BeautifulSoup
url1 = 'https://mphc.gov.in/judgement-orders'
url2 = 'https://mphc.gov.in/php/hc/judgement/search_data_get.php'
params = {'btn_search': "A", # <-- kw
'ddl_o_j_fo': "",
'fno':"01",
'txt_order_dt': "",
'j1': "",
'j2': "",
'pj': "",
'ct': "",
'cn': "",
'cy': "",
'id': "JBP",
'page': "1",
'sort': "jo",
'ad': "DESC",
'date1' : "19-06-2020", # <-- from
'date2': "19-06-2020", # <-- to
'code': "",
'csrf_token': "XXXXXXXX",
}
def prettify(txt):
return ' '.join(line.strip() for line in txt.splitlines())
with requests.session() as s:
soup = BeautifulSoup(s.get(url1, verify=False).content, 'html.parser')
params['csrf_token'] = soup.select_one('#csrf_token1')['value']
soup = BeautifulSoup(s.get(url2, params=params, verify=False).content, 'lxml') # <-- set to lxml
# print some data to screen:
for tr in soup.select('table[width="100%"] tr:has(td)'):
tds = [prettify(td.get_text(strip=True, separator='\n')) for td in tr.select('td')]
print(*tds, sep='|\t')
Prints:
1| MCRC-10826- 2020 Order| RAMANAND SINGH Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE RAJEEV KUMAR DUBEY| HON\'BLE SHRI JUSTICE RAJEEV KUMAR DUBEY| 19-06-2020| 19-06-2020
2| MCRC-16080- 2020 Order| MANJULAL BALECHA Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE B. K. SHRIVASTAVA| HON\'BLE SHRI JUSTICE B. K. SHRIVASTAVA| 19-06-2020| 19-06-2020
3| MCRC-14104- 2020 Order| YASHVANT ROY Vs THE STATE OF MADHYA PRADESH| HON'BLE SMT. JUSTICE ANJULI PALO| HON\'BLE SMT. JUSTICE ANJULI PALO| 19-06-2020| 19-06-2020
4| WP-4442- 2020 Order| BHUPAT DWIVEDI Vs THE UNION OF INDIA| HON'BLE SHRI JUSTICE SUJOY PAUL| HON\'BLE SHRI JUSTICE SUJOY PAUL| 19-06-2020| 19-06-2020
5| MCRC-13647- 2020 Order| HIMANSHU BELWANSHI Vs STATE OF M.P.| HON'BLE SHRI JUSTICE VIJAY KUMAR SHUKLA| HON\'BLE SHRI JUSTICE VIJAY KUMAR SHUKLA| 19-06-2020| 19-06-2020
6| MCRC-14878- 2020 Order| SUMIT Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VISHAL DHAGAT| HON\'BLE SHRI JUSTICE VISHAL DHAGAT| 19-06-2020| 19-06-2020
7| CRA-3251- 2020 Order| KODURAM @ KUDDHU Vs STATE OF MADHYA PRADESH| HON'BLE SMT. JUSTICE ANJULI PALO| HON\'BLE SMT. JUSTICE ANJULI PALO| 19-06-2020| 19-06-2020
8| MCRC-18688- 2020 Order| DHARAM PAL AHIRWAR Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VIJAY KUMAR SHUKLA| HON\'BLE SHRI JUSTICE VIJAY KUMAR SHUKLA| 19-06-2020| 19-06-2020
9| MCRC-12853- 2020 Order| RAJKUMAR PATEL Vs THE STATE OF MADHYA PRADESH| HON'BLE SMT. JUSTICE NANDITA DUBEY| HON\'BLE SMT. JUSTICE NANDITA DUBEY| 19-06-2020| 19-06-2020
10| MCRC-10721- 2020 Order| MRS SONALI AJMERA Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE SUNIL KUMAR AWASTHI| HON\'BLE SHRI JUSTICE SUNIL KUMAR AWASTHI| 19-06-2020| 19-06-2020
11| MCRC-16505- 2020 Order| SUNIL@BULLI Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE SUNIL KUMAR AWASTHI| HON\'BLE SHRI JUSTICE SUNIL KUMAR AWASTHI| 19-06-2020| 19-06-2020
12| CRA-3244- 2020 Order| VICKY @ VIKASH KUSHWAHA Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE SUJOY PAUL| HON\'BLE SHRI JUSTICE SUJOY PAUL| 19-06-2020| 19-06-2020
13| MCRC-1985- 2020 Order| RAJA @ GOS ALI Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VIJAY KUMAR SHUKLA| HON\'BLE SHRI JUSTICE VIJAY KUMAR SHUKLA| 19-06-2020| 19-06-2020
14| WP-7314- 2020 Order| SMT. RAVIKANTA SONI Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VISHAL DHAGAT| HON\'BLE SHRI JUSTICE VISHAL DHAGAT| 19-06-2020| 19-06-2020
15| MCRC-13818- 2020 Order| RAVI Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE JAGDISH PRASAD GUPTA| HON\'BLE SHRI JUSTICE JAGDISH PRASAD GUPTA| 19-06-2020| 19-06-2020
16| CRA-8783- 2019 Order| MAHESH @ DABBU MEENA Vs THE STATE OF MADHYA PRADESH| HON'BLE SMT. JUSTICE ANJULI PALO| HON\'BLE SMT. JUSTICE ANJULI PALO| 19-06-2020| 19-06-2020
17| CRA-2330- 2020 Final Order| ANITA SAYYAM Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VIJAY KUMAR SHUKLA| HON\'BLE SHRI JUSTICE VIJAY KUMAR SHUKLA| 19-06-2020| 19-06-2020
18| WP-8062- 2020 Order| SMT. JATINDER KAUR BHATIA Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VISHAL DHAGAT| HON\'BLE SHRI JUSTICE VISHAL DHAGAT| 19-06-2020| 19-06-2020
19| MCRC-17593- 2020 Order| MOHD. IKRAM Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE SUNIL KUMAR AWASTHI| HON\'BLE SHRI JUSTICE SUNIL KUMAR AWASTHI| 19-06-2020| 19-06-2020
20| MCRC-16284- 2020 Order| KANHAIYALAL YADAV Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE B. K. SHRIVASTAVA| HON\'BLE SHRI JUSTICE B. K. SHRIVASTAVA| 19-06-2020| 19-06-2020
21| MCRC-12054- 2020 Order| DINESH Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE SUNIL KUMAR AWASTHI| HON\'BLE SHRI JUSTICE SUNIL KUMAR AWASTHI| 19-06-2020| 19-06-2020
22| MCRC-15399- 2020 Final Order| HIMMAT SINGH Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VISHAL MISHRA| HON\'BLE SHRI JUSTICE VISHAL MISHRA| 19-06-2020| 19-06-2020
23| CRA-2001- 2020 Order| KAMAL Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE SUNIL KUMAR AWASTHI| HON\'BLE SHRI JUSTICE SUNIL KUMAR AWASTHI| 19-06-2020| 19-06-2020
24| CRA-744- 2019 Order| ASHOK SINGH Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE SUJOY PAUL and HON\'BLE SMT. JUSTICE ANJULI PALO| HON\'BLE SHRI JUSTICE SUJOY PAUL| 19-06-2020|19-06-2020
25| MCRC-15865- 2020 Order| RAJESH Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VIVEK RUSIA| HON\'BLE SHRI JUSTICE VIVEK RUSIA| 19-06-2020| 19-06-2020
26| MCRC-15862- 2020 Order| SAHIL SHARNAGT Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VIJAY KUMAR SHUKLA| HON\'BLE SHRI JUSTICE VIJAY KUMAR SHUKLA| 19-06-2020| 19-06-2020
27| MCRC-12446- 2020 Order| RAJENDRA PRAJAPATI Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE RAJEEV KUMAR DUBEY| HON\'BLE SHRI JUSTICE RAJEEV KUMAR DUBEY| 19-06-2020| 19-06-2020
28| MCRC-15152- 2020 Order| AJAY @ GOLU MEENA Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE JAGDISH PRASAD GUPTA| HON\'BLE SHRI JUSTICE JAGDISH PRASAD GUPTA| 19-06-2020| 19-06-2020
29| CRA-3255- 2020 Order| RAMKHELAWAN @ LALLU Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE B. K. SHRIVASTAVA| HON\'BLE SHRI JUSTICE B. K. SHRIVASTAVA| 19-06-2020| 19-06-2020
30| MCRC-14912- 2020 Order| KALAVATI YADAV Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VISHAL DHAGAT| HON\'BLE SHRI JUSTICE VISHAL DHAGAT| 19-06-2020| 19-06-2020
31| MCRC-12827- 2020 Order| JITENDRA Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE SUNIL KUMAR AWASTHI| HON\'BLE SHRI JUSTICE SUNIL KUMAR AWASTHI| 19-06-2020| 19-06-2020
32| MCRC-15801- 2020 Order| SURENDRA KUMAR SONDHIYA(VERMA) Vs THE STATE OF MADHYA PRADESH| HON'BLE SMT. JUSTICE ANJULI PALO| HON\'BLE SMT. JUSTICE ANJULI PALO| 19-06-2020| 19-06-2020
33| CRA-3189- 2020 Order| AKASH MISHRA Vs THE STATE OF M.P.| HON'BLE SHRI JUSTICE SUJOY PAUL| HON\'BLE SHRI JUSTICE SUJOY PAUL| 19-06-2020| 19-06-2020
34| MCRC-16090- 2020 Order| DEVENDRA KUMAR Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VISHAL DHAGAT| HON\'BLE SHRI JUSTICE VISHAL DHAGAT| 19-06-2020| 19-06-2020
35| MCRC-16282- 2020 Order| SURESH Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE SUNIL KUMAR AWASTHI| HON\'BLE SHRI JUSTICE SUNIL KUMAR AWASTHI| 19-06-2020| 19-06-2020
36| MCRC-14067- 2020 Final Order| VIKAS SAHU Vs STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE RAJEEV KUMAR DUBEY| HON\'BLE SHRI JUSTICE RAJEEV KUMAR DUBEY| 19-06-2020| 19-06-2020
37| MCRC-17254- 2020 Order| VICHHLA @ VICHLA @ KUBER SINGH Vs STATE OF M.P.| HON'BLE SHRI JUSTICE RAJEEV KUMAR DUBEY| HON\'BLE SHRI JUSTICE RAJEEV KUMAR DUBEY| 19-06-2020| 19-06-2020
38| MCRC-1405- 2020 Final Order| HARBAL Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE B. K. SHRIVASTAVA| HON\'BLE SHRI JUSTICE B. K. SHRIVASTAVA| 19-06-2020| 19-06-2020
39| MCRC-15209- 2020 Order| RANJEET MAHESHWARI Vs STATE OF M.P.| HON'BLE SHRI JUSTICE JAGDISH PRASAD GUPTA| HON\'BLE SHRI JUSTICE JAGDISH PRASAD GUPTA| 19-06-2020| 19-06-2020
40| MCRC-15916- 2020 Order| RAMESHWAR MEENA @ BANTI MEENA Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE JAGDISH PRASAD GUPTA| HON\'BLE SHRI JUSTICE JAGDISH PRASAD GUPTA| 19-06-2020|19-06-2020
41| MCRC-16885- 2020 Order| MOHIT Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VISHAL DHAGAT| HON\'BLE SHRI JUSTICE VISHAL DHAGAT| 19-06-2020| 19-06-2020
42| CRR-198- 2020 Order| SANDEEP SINGH Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE JAGDISH PRASAD GUPTA| HON\'BLE SHRI JUSTICE JAGDISH PRASAD GUPTA| 19-06-2020| 19-06-2020
43| CRA-3049- 2020 Order| SANDEEP BOHNE Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VISHAL DHAGAT| HON\'BLE SHRI JUSTICE VISHAL DHAGAT| 19-06-2020| 19-06-2020
44| MCRC-15345- 2020 Final Order| ARVIND RUSIKAR Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VISHAL DHAGAT| HON\'BLE SHRI JUSTICE VISHAL DHAGAT| 19-06-2020| 19-06-2020
45| MCRC-16036- 2020 Order| PAVAN AMBULKAR Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE B. K. SHRIVASTAVA| HON\'BLE SHRI JUSTICE B. K. SHRIVASTAVA| 19-06-2020| 19-06-2020
46| MCRC-9264- 2020 Order| RAMBABU MALVIYA Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VISHAL DHAGAT| HON\'BLE SHRI JUSTICE VISHAL DHAGAT| 19-06-2020| 19-06-2020
47| MCRC-15533- 2020 Order| GABBAR @ GOVIND Vs STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE VIJAY KUMAR SHUKLA| HON\'BLE SHRI JUSTICE VIJAY KUMAR SHUKLA| 19-06-2020| 19-06-2020
48| MCRC-13699- 2020 Order| RAJARAM Vs THE STATE OF MADHYA PRADESH| HON'BLE SMT. JUSTICE NANDITA DUBEY| HON\'BLE SMT. JUSTICE NANDITA DUBEY| 19-06-2020| 19-06-2020
49| MCRC-17283- 2020 Order| VIRENDRA SINGH @ VEERAN Vs THE STATE OF MADHYA PRADESH| HON'BLE SHRI JUSTICE RAJEEV KUMAR SHRIVASTAVA| HON\'BLE SHRI JUSTICE RAJEEV KUMAR SHRIVASTAVA| 19-06-2020| 19-06-2020
50| MCRC-15852- 2020 Order| SUNIL KUMAR GUPTA Vs THE STATE OF MADHYA PRADESH| HON'BLE SMT. JUSTICE NANDITA DUBEY| HON\'BLE SMT. JUSTICE NANDITA DUBEY| 19-06-2020| 19-06-2020
First Previous 1 2 3 4 5 6 7 Next Last Go to Page: Page 1 of 9
来源：https://stackoverflow.com/questions/62470236/scraping-data-that-is-initially-hidden-and-appears-after-submit