Login with Scrapy with site using Javascript

Question

I am trying to use Scrapy to pull more information from another page. I was able to login with my last project and am modeling this one after it. The login page for this particular site is much different, however. Here is the markup for the page with the relevant information provided:

<div id="form" style="display:inline;">

    <form id="login" class="gfimaxrmform2" name="gfimaxform" action="" method="post" onsubmit="return attemptLogin(this);" >

        <label for="username">User name:</label>
        <input alt="blank" size="30" id="username" name="username" type="text" />

        <br />

        <label for="password">Password:</label>
        <input alt="blank" size="30" id="password" name="password" type="password"  />

        <br />
        <br />
        <script type="text/javascript">document.write("<input name='url' id = 'url' type='hidden' value = '"+document.URL+"'>")</script>
        <div class="formfields_belowfields">
            <button class="submit" type="submit">Login</button>
        </div>

    </form>
</div>

I am trying to use my code to login and then go to another website but I am not sure how to make it "click" submit. Here is the code I am working on:

def parse(self, response):
    return[FormRequest.from_response(response, formdata = {"username" : user, "password": password}, callback = self.after_login)]

The `after_login' will just make sure that I am logged in, however it doesn't seem to be clicking the submit button to go to the next page.

edit: The link to the page I am trying to log in to is: Here

Answer 1

I ended up using Selenium to really simplify this process. If anyone discovers a need for something in the future I would recommend using it.