Scrapy splash log-in

Question

I used to use splash form request to login in one of the site. However, developers changed it, added more javascript, and I can\'t figure out what I\'m doing wrong. I added jav

Answer 1

Borrowed idea from this topic

enter Scrapy + splash: can't select element

Instead using formdata to login, use splash and detect page's elements one by one

class MySpider(scrapy.Spider):
name = "lost"
allowed_domains = ["mydomain"]
start_urls = ['myurl']
req = 10
series = {}





def start_requests(self):                               
    script = """
    function main(splash)
        local url = splash.args.url
        assert(splash:go(url))
        assert(splash:wait(10))

        splash:set_viewport_full()

        local search_input = splash:select('input[name=mail]')   
        search_input:send_text("email")
        local search_input = splash:select('input[name=pass]')
        search_input:send_text("password")
        assert(splash:wait(5))
        local submit_button = splash:select('input[class^=primary-btn]')
        submit_button:click()

        assert(splash:wait(10))

        return {
            html = splash:html(),
            png = splash:png(),
        }
      end
    """
    yield SplashRequest(
        'myurl',
        callback = self.after_login,      ###inserting callabck
        endpoint='execute',
        args={
        'lua_source': script,
        'ua': "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36"
        }
                        )

def parse(self, response):
    script = response.body