how to execute scrapy shell “URL” with notebook

Question

i am trying to scrapy

and there is scrapy shell \"URL\" command, executing console ipython with a response object from the URL.

but i want to do that thing with

Answer 1

Stumbled across this unanswered question looking for the same answer. The closest thing I can come up right now is this:

import requests
from scrapy.http import TextResponse

r = requests.get('http://stackoverflow.com/')
response = TextResponse(r.url, body=r.text, encoding='utf-8')

Then you can debug response using usual methods like xpath().