Python HTML fill and submit form -
i trying write python script goes form on internal website (with name "defaultform") , fills in input name="username" field in form 'user001', input name="password" field 'pass001' , click on submit
tried doing selenium , works. want accomplish same task requests(and beautifulsoup html scraping later on)
code wrote not work!
url = 'http://server:port/dashboard/portal' payload = {'username':'user001','password':'pass001'} r = requests.get(url, params=payload) print(r.text) i check content in r.text before , after requests.get(..) , both same.
can me on how ?
edit/update: tried submit form using lxml there seems error can't seem head around
page = parse(url).getroot() page.forms[0].fields['username'] = 'user001' page.forms[0].fields['password'] = 'pass001' result = parse(submit_form(page.forms[0]).encode('utf-8')).getroot() print(result.text) this console result get
runfile('c:/users/mgreza/downloads/winpython-64bit-3.5.1.3/notebooks/temp.py', wdir='c:/users/mgreza/downloads/winpython-64bit-3.5.1.3/notebooks') traceback (most recent call last):
file "", line 1, in runfile('c:/users/mgreza/downloads/winpython-64bit-3.5.1.3/notebooks/temp.py', wdir='c:/users/mgreza/downloads/winpython-64bit-3.5.1.3/notebooks')
file "c:\users\mgreza\downloads\winpython-64bit-3.5.1.3\python-3.5.1.amd64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 841, in runfile execfile(filename, namespace)
file "c:\users\mgreza\downloads\winpython-64bit-3.5.1.3\python-3.5.1.amd64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 103, in execfile exec(compile(f.read(), filename, 'exec'), namespace)
file "c:/users/mgreza/downloads/winpython-64bit-3.5.1.3/notebooks/temp.py", line 13, in result = parse(submit_form(page.forms[0]).encode('utf-8')).getroot()
file "c:\users\mgreza\downloads\winpython-64bit-3.5.1.3\python-3.5.1.amd64\lib\site-packages\lxml\html__init__.py", line 1110, in submit_form return open_http(form.method, url, values)
file "c:\users\mgreza\downloads\winpython-64bit-3.5.1.3\python-3.5.1.amd64\lib\site-packages\lxml\html__init__.py", line 1131, in open_http_urllib return urlopen(url, data)
file "c:\users\mgreza\downloads\winpython-64bit-3.5.1.3\python-3.5.1.amd64\lib\urllib\request.py", line 162, in urlopen return opener.open(url, data, timeout)
file "c:\users\mgreza\downloads\winpython-64bit-3.5.1.3\python-3.5.1.amd64\lib\urllib\request.py", line 463, in open req = meth(req)
file "c:\users\mgreza\downloads\winpython-64bit-3.5.1.3\python-3.5.1.amd64\lib\urllib\request.py", line 1170, in do_request_ raise typeerror(msg)
typeerror: post data should bytes or iterable of bytes. cannot of type str.
please help!!!!!!!
this page trying submit
<html> <head> <base href="http://server/directory"> <link href="css/default.css" type="text/css" rel="stylesheet"> <title>xyz</title> </head> <body bgcolor="#ffffff"> <p> </p> <table width="100%" cellpadding="0" cellspacing="0"> <tbody> <tr> <td> <form method="post" name="defaultform" action="http://server/directory" onsubmit="return (isready(this));" autocomplete="off" _lpchecked="1"> <input name="action" type="hidden" value="jloginuser"> <input name="servertimestamp" type="hidden" value="1467104268529"> <input name="clienttimestamp" type="hidden" value="1467104268904"> <input name="clientip" type="hidden" value="10.221.12.67"> <table height="400" cellspacing="0" cellpadding="0" width="540" align="center" background="images/bkground.gif" border="0"> <tbody> <tr> <td> <table heigh="395" cellspacing="0" cellpadding="0" width="100%" background="images/transparent.gif" border="0"> <tbody> <tr> <td> <img height="19" src="images/transparent.gif"> </td> <td> <img src="images/logo.gif" align="top" border="0"> </td> <td> <img height="19" src="images/transparent.gif" width="8"> </td> </tr> <tr> <td width="10"> <img height="1" src="images\transparent.gif" width="10"> </td> <td valign="top" width="497"> <table cellspacing="0" cellpadding="0" width="100%" background="images/transparent.gif" border="0"> <tbody> <tr> <td valign="top" align="left" width="90"><span> </span> <table cellspacing="0" cellpadding="0" width="100%" background="images/transparent.gif" border="0"> <tbody> <tr> <td colspan="3"> </td> </tr> <tr> <td width="4"> </td> <td> <img height="1" src="images/transparent.gif" width="5"> </td> <td><span> </span> </td> </tr> <tr> <td colspan="3"> </td> </tr> </tbody> </table> </td> <td width="18"> <img height="1" src="images/transparent.gif" width="18"> </td> <td valign="top"> <table cellspacing="0" cellpadding="0" width="100%" background="images/transparent.gif" border="0"> <tbody> <tr> <td colspan="2"> </td> </tr> <tr> <td valign="top" colspan="2" height="15"> <p> <img height="15" src="images/transparent.gif" width="1"> </p> </td> </tr> <tr> <td valign="top" align="center" colspan="2"> <img src="images/integrator_login.gif"> </td> </tr> <tr> <td colspan="2"> <table cellspacing="0" cellpadding="0" width="100%" background="images/transparent.gif" border="0"> <tbody> <tr> <td valign="top" align="center"> <div id="xboxlogin"> <b class="xtop"><b class="xb1"></b><b class="xb2"></b><b class="xb3"></b><b class="xb4"></b><b class="xb5"></b></b> <div class="xboxlogincontent"> <table cellspacing="0" cellpadding="0" width="200" background="images/transparent.gif" border="0"> <tbody> <tr> <td valign="bottom" align="left" width="11" height="11"></td> <td height="11"> <img height="11" src="images/transparent.gif" width="1"> </td> <td height="11"> <img height="11" src="images/transparent.gif" width="1"> </td> <td valign="bottom" align="right" width="11" height="11"></td> </tr> <tr> <td width="11"> </td> <td class="loginform" colspan="2">please sign in</td> <td width="11"> </td> </tr> <tr> <td width="11"> </td> <td class="loginform"> </td> <td height="11"> <img height="11" src="images/transparent.gif" width="1"> </td> <td width="11"> </td> </tr> <tr> <td width="11"> </td> <td class="loginform">user id</td> <td height="11"> <img height="11" src="images/transparent.gif" width="1"> </td> <td width="11"> </td> </tr> <tr> <td width="11"> </td> <td class="loginform" colspan="2"> <input class="inputstyle" type="input" name="username" size="20" style="cursor: pointer; background-image: url("data:image/png;base64,ivborw0kggoaaaansuheugaaabaaaaascayaaabso15qaaaaaxnsr0iars4c6qaaaubjrefuobgvve2oruaqlvis4gwzeyshkhgnkmiec4zejxcmnwtwtmdh3ugcyoafhzufczfvnu4wiaie+vvq6+6qtgthgh6o4/ja7x1oicaipwj7colgsxdxsjevzat9k01cbkdwfsff/2wnuewc2yqigkzpk9glalvvwttnbqjzlnlckitaznf/mepb2birdhwhdf2hjemgarrwpa+qqoi4jle5/8xkxzrcfohg+/5icdpm13utho7q9/0wnsfwil/ouhwhrjgqr8wewvg+oxpmpadakdzvd7asc8qyhcikjjirnckwbrsmw9hcq5zv9masbeu6hjrnyrggfukacnwjkjzposik1d1gddecqobobwswzabd/d3np7ahoirvnphmpi+kc2rzbm3bcp8wuwsiot7qq0pznor6wysk0xb/agvlcwwc7ng3aaaaaelftksuqmcc"); background-attachment: scroll; background-size: contain; background-position: 98% 50%; background-repeat: no-repeat;" autocomplete="off"> </td> <td width="11"> </td> </tr> <tr> <td width="11" height="5"> </td> <td height="5"> </td> <td height="11"> <img height="11" src="images/transparent.gif" width="1"> </td> <td width="11" height="5"> </td> </tr> <tr> <td width="11"> </td> <td class="loginform">password</td> <td height="11"> <img height="11" src="images/transparent.gif" width="1"> </td> <td width="11"> </td> </tr> <tr> <td width="11"> </td> <td colspan="2"> <input class="inputstyle" type="password" maxlength="28" name="password" size="20" style="cursor: auto; background-image: url("data:image/png;base64,ivborw0kggoaaaansuheugaaabaaaaascayaaabso15qaaaaaxnsr0iars4c6qaaaubjrefuobgvve2oruaqlvis4gwzeyshkhgnkmiec4zejxcmnwtwtmdh3ugcyoafhzufczfvnu4wiaie+vvq6+6qtgthgh6o4/ja7x1oicaipwj7colgsxdxsjevzat9k01cbkdwfsff/2wnuewc2yqigkzpk9glalvvwttnbqjzlnlckitaznf/mepb2birdhwhdf2hjemgarrwpa+qqoi4jle5/8xkxzrcfohg+/5icdpm13utho7q9/0wnsfwil/ouhwhrjgqr8wewvg+oxpmpadakdzvd7asc8qyhcikjjirnckwbrsmw9hcq5zv9masbeu6hjrnyrggfukacnwjkjzposik1d1gddecqobobwswzabd/d3np7ahoirvnphmpi+kc2rzbm3bcp8wuwsiot7qq0pznor6wysk0xb/agvlcwwc7ng3aaaaaelftksuqmcc"); background-attachment: scroll; background-size: contain; background-position: 98% 50%; background-repeat: no-repeat;" autocomplete="off"> </td> <td width="11"> </td> </tr> <tr> <td width="11" height="5"> </td> <td height="5"> </td> <td height="11"> <img height="11" src="images/transparent.gif" width="1"> </td> <td width="11" height="5"> </td> </tr> <tr> <td width="6"> </td> <td class="loginform"> <input class="submit" name="submit" type="submit" value="sign in" style="font-size:10"> </td> <td width="50%" align="center"> </td> <td width="11"> </td> </tr> <tr> <td valign="top" align="left" width="11" height="11"></td> <td height="11"> <img height="11" src="images/transparent.gif" width="1"> </td> <td height="11"> <img height="11" src="images/transparent.gif" width="1"> </td> <td valign="top" align="right" width="11" height="11"></td> </tr> </tbody> </table> </div> <b class="xbottom"><b class="xb5"></b><b class="xb4"></b><b class="xb3"></b><b class="xb2"></b><b class="xb1"></b></b> </div> </td> <td width="43"> <img height="8" src="images/transparent.gif" width="43"> </td> </tr> </tbody> </table> </td> </tr> <tr> <td valign="top" align="left"> </td> <td valign="top" align="right" colspan="3"> </td> </tr> </tbody> </table> </td> </tr> </tbody> </table> </td> </tr> </tbody> </table> </td> </tr> </tbody> </table> </form> </td> </tr> </tbody> </table> </body> </html>edit/update: logs when post-ing form
request header:- accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 accept-encoding:gzip, deflate accept-language:en-us,en;q=0.8,en-gb;q=0.6 cache-control:max-age=0 connection:keep-alive content-length:140 content-type:application/x-www-form-urlencoded cookie:jsessionid=1ax740u3chasqa4rmen8ifq5b; sci_dlsso=u2vzc2lvbkleagl1ywfsegjqegx1mtn1ogftaxpva3yybw== host:10.1.28.189:5010 origin:http://server:port referer:http://server:port/dashboard/ upgrade-insecure-requests:1 user-agent:mozilla/5.0 (windows nt 6.1; wow64) applewebkit/537.36 (khtml, gecko) chrome/51.0.2704.103 safari/537.36 response header:- cache-control:no-cache content-type:text/html; charset=utf-8 expires:thu, 01 jan 1970 00:00:00 gmt last-modified:thu, 30 jun 2016 01:47:48 gmt pragma:no-cache set-cookie:jsessionid=17zql1r4hdylrfg54lardx14p;path=/dashboard/;httponly set-cookie:sci_dlsso=u2vzc2lvbklemtd6cwwxcjrozhlscmznntrsyxjkede0ca==;path=/;httponly transfer-encoding:chunked x-frame-options:sameorigin form data:- action=jloginuser&servertimestamp=1467251264480&clienttimestamp=2146&clientip=10.220.12.101&username=user001&password=pass001&submit=sign+in
this should loggeg in:
import requests lxml import html requests.session() s: xml = .get("http://server:port/dashboard/portal").getroot() form = xml.xpath("//form[@name='defaultform']")[0] print(form.xpath("./input[@name]")) data = {i.xpath("@name")[0] : i.xpath("@value")[0] in form.xpath("./input[@name]")} post_url = form.xpath("@action")[0] data["username"] = "username" data["password"] = "password" data["submit"] = "sign in" r = s.post("http://server:port/dashboard/portal"", data=data)
Comments
Post a Comment