how to select and extract texts between two elements?

Question

I am trying to scrape this website using scrapy. The page structure looks like this:

  <         


        

      

Answer 1

You can try to use below XPath expressions to fetch

• all text nodes for "Follows" block:

  //div[./preceding-sibling::h4[1]="Follows"]//text()
  

• all text nodes for "Followed by" block:

  //div[./preceding-sibling::h4[1]="Followed by"]//text()
  

• all text nodes for "Spin off" block:

  //div[./preceding-sibling::h4[1]="Spin-off"]//text()