1

i'm trying to scrape a website that requires login (my routers GUI), but no matter what I've done, i repeatedly get back to source of the login site, and not the page that comes after a successful login. I did some reading and came to realize that i need to return answers for the POST request. I think i found them all and returned the needed values, but still - nothing seems to do the trick. I used the help of https://curl.trillworks.com in order to be sure i have of all the relevant elements, and still didn't work. I think i don't really understood where to put my login info. I really don't know what i'm missing here, and will be more than happy to get some guidance from the masters here :)

This is the source of the login page:

<div class="formField" id="userName" help_tooltip="off">
    <label for="userName">User Name:</label>
    <input type="text" name="userName" value="" />

</div>

<div class="formField" id="origUserPwd">
    <label for="origUserPwd">Password:</label>
    <div class="passwordField">

<input type="password" autocomplete="off" name="origUserPwd" value="" />
<span class='text'><input type='checkbox' name='origUserPwdShow' tabindex='-1'/> Show password</span>
<script>
fieldPassword('origUserPwd', 'origUserPwdShow', 0, 0);
</script>

    </div>
</div>

<div class="formField" id="languageBox" help_tooltip="off">
<label for="language">Language:</label>

<select name="language" onchange='language_reload();'>
<option value='EN' selected>English</option>

</select>
<script type="text/javascript">Select_setValue(document.form.language, "EN");</script>

</div>

</fieldset>

<ul class="links">
<li><a href='cbpc/login?lang=EN' onclick='return link_onclick(this);'><img src='/images/Start16.png'> Parental Control Login</a></li>

</ul>
<div class="buttons">
    <input type="submit" name="login" onclick="return link_onclick(this);"
        value="Login"/>
</div>
</div>


</div>

<input type="hidden" name="userPwd" />
<input type="hidden" name="nonce" value="1142068318" />
<input type='hidden' name='code1' value='X.RCX45W5HZU' />
<input type='hidden' name='code2' value='C7SogPGgF2b' />
</form>

This is the responses i get from chrome after log in:

GENERAL:
Request URL: http://10.100.102.1/ui/login
Request Method: POST
Status Code: 302 Found
Remote Address: 10.100.102.1:80
Referrer Policy: no-referrer-when-downgrade

RESPONSE HEADER:
Cache-Control: no-cache
Connection: Keep-Alive
Content-Language: en
Content-Length: 0
Content-Type: text/html
Date: Mon, 08 Jun 2020 12:29:43 UTC
Expires: -1
Keep-Alive: timeout=15, max=15
Location: /ui/dboard
Pragma: no-cache
Server: ADB Broadband HTTP Server
Set-Cookie: sid=81197847; path=/ui/; HttpOnly
X-Frame-Options: DENY

REQUEST HEADER:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: en,en-US;q=0.9,he;q=0.8
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 162
Content-Type: application/x-www-form-urlencoded
Cookie: sid=81108258
Host: 10.100.102.1
Origin: http://10.100.102.1
Referer: http://10.100.102.1/ui/login
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36

FORM DATA:
userName: home
language: EN
login: Login
userPwd: 8b6c424159d2ab16de3e1c6188f0f968f0842b9bb8761fea608c9c94592eec54
nonce: 1149917371
code1: yQ4n/l
code2: RSBYEYVAX1isD

This is the code I wrote:

from bs4 import BeautifulSoup
import requests

session = requests.Session()

cookies = {
    'sid': '1251255460',
}

headers = {
    'Connection': 'keep-alive',
    'Cache-Control': 'max-age=0',
    'Upgrade-Insecure-Requests': '1',
    'Origin': 'http://10.100.102.1',
    'Content-Type': 'application/x-www-form-urlencoded',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'Referer': 'http://10.100.102.1/ui/login',
    'Accept-Language': 'en,en-US;q=0.9,he;q=0.8',
}

data = {
  'userName': 'home',
  'language': 'EN',
  'login': 'Login',
  'userPwd': '9406120598739dd0303adbe49826acccba7e6458e155d5e7baf90418c2990cfe',
  'nonce': '1129804332',
  'code1': 'A.kLTTlKZ4',
  'code2': 'seek'
}

loginurl='http://10.100.102.1/ui/login'
posturl='http://10.100.102.1/ui/dboard'

response = session.post(loginurl, headers=headers, cookies=cookies, data=data)
response = session.get(posturl)
print(response.content)

Thanks a lot for the help :)

Pelegk
  • 11
  • 2
  • I think the problem is with a cookie, and a session. You should probabaly open requests' session to get the cookie, then get sid from this cookie, then log in with this sid. – 404pio Jun 08 '20 at 13:12
  • I've opened a session request (updated the code on the main post), still doesn't work. Can you please elaborate on the sid and the coockie? – Pelegk Jun 08 '20 at 13:20
  • Ok, when I type in google "python requests login" i've got also recommendations "session cookie". So I google and I get -> https://stackoverflow.com/questions/15778466/using-python-requests-sessions-cookies-and-post . I'm not sure how is your router working, but probably you need to get a page with session, then get session, then post login data. I'm not sure. – 404pio Jun 08 '20 at 21:12

0 Answers0