Web 抓取請求 python

Question

Good morning everyone, I'm trying to access the html code of a web page, I've got the login working through python but only that once I print the page the html code is not as complete as if I opened it in a browser ...

loginurl = ('https://sgv.ivu-cloud.com/mbweb/j_security_check')
secure_url = ('https://sgv.ivu-cloud.com/mbweb/main/matter/desktop/main-menu')

session = HTMLSession()

payload = {
    'j_username' : creds.username,
    'j_password': creds.password
    }

s = requests.Session()
s.post(loginurl, data=payload)
r = s.get((secure_url), cookies={'from-my': 'browser'})
soup = BeautifulSoup(r.content, 'html.parser')
c = session.get('https://sgv.ivu-cloud.com/mbweb/main/matter/desktop/main-menu#duty-details?beginDate=2021-12-14&allocatedEmployeeId=22480')
d = s.get('https://sgv.ivu-cloud.com/mbweb/main/matter/desktop/main-menu#duty-details?beginDate=2021-12-14&allocatedEmployeeId=22480')



print(d)

這是答案：

    









IVU.plan Portal





        





































var mbwebBaseUrl = '/mbweb';
    <script type="text/javascript" src="/mbweb/struts/js/base/jquery-2.2.4.min.js"></script>

    <script type="text/javascript" src="/mbweb/struts/js/base/jquery-ui.min.js?s2j=4.0.3"></script>

        <script type="text/javascript"

                src="/mbweb/struts/i18n/datepicker-de.min.js?s2j=4.0.3"></script>






    $(function () {

        jQuery.struts2_jquery.version = "4.0.3";

        jQuery.struts2_jquery.loadAtOnce = true;

        jQuery.scriptPath = "/mbweb/struts/";

        jQuery.struts2_jquery.local = "de";

        jQuery.struts2_jquery.gridLocal = "de";

        jQuery.struts2_jquery.timeLocal = "de";

        jQuery.struts2_jquery.datatablesLocal = "de";

        jQuery.ajaxSettings.traditional = true;



        jQuery.ajaxSetup({

            cache: false

        });



        jQuery.struts2_jquery.require("js/struts2/jquery.ui.struts2.min.js?s2j=4.0.3");



    });


    <link id="jquery_theme_link" rel="stylesheet"

          href="/mbweb/struts/themes/smoothness/jquery-ui.css?s2j=4.0.3"

          type="text/css"/>

600

Answer 1

您可以使用您的代碼登錄並獲取 cookies。 然后您可以將 cookies 傳遞給 selenium 並繼續

from selenium import webdriver
driver = webdriver.Firefox(executable_path="driver/geckodriver.exe")
for c in s.cookies:
    driver.add_cookie({'name': c.name, 'value': c.value})

driver.get('https://sgv.ivu-cloud.com/mbweb/main/matter/desktop/main-menu#duty-details?beginDate=2021-12-14&allocatedEmployeeId=22480')

這僅在您首先使用您的代碼登錄時才有效，因此最后需要將其添加到您的代碼中