python Selenium - Unable to fetch Facebook table content

Question

. 。 Answers to this question are eligible for a +50 reputation bounty. user1584253 is looking for an : 的：

It seems like Facebook Insights page block the selenium scrapper.

I am trying to access Facebook Insights Page , here I have to fetch table under 'Pages to Watch section. I am getting timeout exception.

Below is my code:

chrome_options = webdriver.ChromeOptions()
prefs = {"profile.default_content_setting_values.notifications" : 2}
chrome_options.add_experimental_option("prefs",prefs)
driver = webdriver.Chrome(chrome_options=chrome_options)



driver.get(url)
driver.maximize_window()

time.sleep(3)

elem = driver.find_element_by_id("email") #m_login_email
elem.send_keys(usr)

elem = driver.find_element_by_id("pass") #m_login_password
elem.send_keys(pwd)

elem.send_keys(Keys.RETURN)

time.sleep(10)

users_list = [] 

delay = 60 # seconds
try:
    myElem = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, 'profile_page_insights_hubble')))
    print("Page is ready!")
except TimeoutException:
    print("Loading took too much time!")

I dumped the driver variable content in the html file and I got the blank content from where I want to scrape the data.

Where am I going wrong?, Kindly assist

EDIT: Below is the div tag from the Facebook insights page:

 <div id="profile_page_insights_hubble" data-referrer="profile_page_insights_hubble" data-veid="vc_1"><div class="_5dom _553h" id="u_0_3_9e"><div><div class="_9dq6"><div class="_fqv" style=""><div><div><div class="_9dq8 _4-u2 _4-u8"><div class="_5fez"><div><div class="_4mxp"><span class="_4mxq"><div class="_5ejd">Pages to Watch</div></span><button type="button" aria-disabled="false" class="_271k _271l _271m _1qjd _7tvm _7tv2 _7tv4" style="letter-spacing: normal; color: rgb(255, 255, 255); font-size: 12px; font-weight: bold; font-family: Arial, sans-serif; line-height: 26px; text-align: center; background-color: rgb(24, 119, 242); border-color: rgb(24, 119, 242); height: 28px; padding-left: 11px; padding-right: 11px; border-radius: 2px;"><div class="_43rl"><i aria-hidden="true" class="_271o img sp_anz1G3VvCap_1_5x sx_835655" alt="" data-visualcompletion="css-img" style="margin-right: 7px;"></i><div data-hover="tooltip" data-tooltip-display="overflow" class="_43rm">Create Post</div></div></button></div><div class="_5eje _ehw">Compare the performance of your Page and posts with similar Pages on Facebook.</div></div></div><div class="_532o"><div class="_5y1o"><div class="clearfix"><div class="_ohe lfloat"><div class="_5y1c"><button class="_4jy0 _4jy3 _4jy1 _51sy selected _42ft" type="submit" value="1">Add Pages</button></div></div><div class="_ohf rfloat"><div class="_5y1b"><div class="_5a32 _5k3z" legenddata="[object Object]"><div class="_1ovd _1ove"><div class="_kz-"><div class="_1l46 _1l47" style="background: rgb(208, 148, 171);"></div><span class="_kdk"><span class="_5j1f" data-hover="tooltip">Reactions, comments &amp; shares<i class="_5j1g img sp_svLrETnYuap_1_5x sx_784b9b" alt="" data-visualcompletion="css-img"></i><div class="_5j1h"><div class="_5j1i">The number of times that people have engaged with your posts through likes, comments, shares and more.</div></div></span></span></div></div></div></div></div></div><table class="_5k45"><thead><tr class="_5k47"><th class="b5b6ndi6 ij02y1sh tejsww75 cidj4p45 r7bn319e scpwgmsl ahb38r9s r4jidfu8 jvf90sed br743ykl majrg6er oywo8hpq"><span class="om3e55n1 b0r6k8t3 onux6t7x c0adsbae d1v569po hf30pyar lq84ybu9 b6ax4al1"><span class="qm54mken k5025e2h">Page</span></span></th><th class="b5b6ndi6 ij02y1sh tejsww75 cidj4p45 r7bn319e scpwgmsl ahb38r9s r4jidfu8 jvf90sed br743ykl majrg6er oywo8hpq"><span class="om3e55n1 b0r6k8t3 onux6t7x c0adsbae d1v569po hf30pyar lq84ybu9 b6ax4al1"><span class="qm54mken k5025e2h"><div class="_2ewi">Total Page Likes</div></span></span></th><th class="b5b6ndi6 ij02y1sh tejsww75 cidj4p45 r7bn319e scpwgmsl ahb38r9s r4jidfu8 jvf90sed br743ykl majrg6er oywo8hpq"><span class="om3e55n1 b0r6k8t3 onux6t7x c0adsbae d1v569po hf30pyar lq84ybu9 b6ax4al1"><span class="qm54mken k5025e2h"><div class="_2ewj">From last week</div></span></span></th><th class="b5b6ndi6 ij02y1sh tejsww75 cidj4p45 r7bn319e scpwgmsl ahb38r9s r4jidfu8 jvf90sed br743ykl majrg6er oywo8hpq"><span class="om3e55n1 b0r6k8t3 onux6t7x c0adsbae d1v569po hf30pyar lq84ybu9 b6ax4al1"><span class="qm54mken k5025e2h"><div class="_2ewk">Posts This Week</div></span></span></th><th class="b5b6ndi6 ij02y1sh tejsww75 cidj4p45 r7bn319e scpwgmsl ahb38r9s r4jidfu8 jvf90sed br743ykl majrg6er oywo8hpq"><span class="om3e55n1 b0r6k8t3 onux6t7x c0adsbae d1v569po hf30pyar lq84ybu9 b6ax4al1"><span class="qm54mken k5025e2h"><div class="_2ewl">Engagement This Week</div></span></span></th></tr></thead><tbody><tr class="_5k4c _5k4a"><td class="_5k49 _9b5k" colspan="1"><div class="_5y1j"><div class="_5y1d"><div direction="left" class="clearfix"><div class="_ohe lfloat"><div class="_5y1f" tabindex="0" role="button"><div class="_5y1g">1</div></div></div><div class=""><div class="_4tg1 _5xdw clearfix" direction="left"><div class="_ohe lfloat"><img class="_5y1h img _8o _8t img" src="https://scontent.fkhi10-1.fna.fbcdn.net/v/t1.6435-1/cp0/p80x80/29257663_2048742455142642_7457461319315226624_n.png?_nc_cat=1&amp;ccb=1-5&amp;_nc_sid=dbb9e7&amp;_nc_ohc=roZN3qv48ekAX_l7SV3&amp;_nc_oc=AQkWD11AXCtikk7SHXe_HvonB_dDepA3S1MHy9B6OcwhFLFUZUUnxExnzD5bxDKcTok&amp;_nc_ht=scontent.fkhi10-1.fna&amp;oh=e1ccd62a44c5bfa6aa047b23d8453fc5&amp;oe=618D67C7" alt=""></div><div class=""><div class="_42ef"><div class="_5y1i ellipsis">Meezan Bank Limited</div></div></div></div></div></div></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1k _2ewi"><div class="_5kmz _5km_"><div class="_5kn3 ellipsis">2.2m</div><div class="_5kn4 _11uq"><span data-hover="tooltip"><div class="_5abm" style="width: 100%;"><div class="_5abn" style="background-color: rgb(197, 206, 224); width: 100%;"></div><div class="_352g"></div></div></span></div></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1l _2ewj"><div class="_5cul"><div class="_5cuo"></div><span class="_5cup">100%</span></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1m _2ewk">24</div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1n _2ewl"><div class="_5kmz _5km_"><div class="_5kn3 ellipsis">127.3K</div><div class="_5kn4 _11uq"><span data-hover="tooltip"><div class="_5abm" style="width: 17%;"><div class="_5abn" style="background-color: rgb(208, 148, 171); width: 100%;"></div><div class="_352g"></div></div></span></div></div></div></td></tr><tr class="_5k4c"><td class="_5k49 _9b5k" colspan="5"><div class="_5ytl"><div class="_5ytj _5ytk"><table class="_5k45"><tbody><tr class="_5k4c _5k4a"><td class="_5k49 _9b5k" colspan="1"><div class="_5y1j"><div class=""><div direction="left" class="clearfix"><div class="_ohe lfloat"><div class="_5y1f" tabindex="0" role="button"><div class="_5y1e">YOU</div><div class="_5y1g">2</div></div></div><div class=""><div class="_4tg1 _5xdw clearfix" direction="left"><div class="_ohe lfloat"><img class="_5y1h img _8o _8t img" src="https://scontent.fkhi10-1.fna.fbcdn.net/v/t1.6435-1/cp0/p80x80/32893894_1734930716587287_2773075695138504704_n.png?_nc_cat=1&amp;ccb=1-5&amp;_nc_sid=dbb9e7&amp;_nc_ohc=44iCT34KSCoAX_sXl4e&amp;_nc_ht=scontent.fkhi10-1.fna&amp;oh=d4bf9416b13528ebee33914595b0a742&amp;oe=618C0CEF" alt=""></div><div class=""><div class="_42ef"><div class="_5y1i ellipsis">UBL - United Bank Ltd</div></div></div></div></div></div></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1k _2ewi"><div class="_5kmz _5km_"><div class="_5kn3 ellipsis">1.8m</div><div class="_5kn4 _11uq"><span data-hover="tooltip"><div class="_5abm" style="width: 81%;"><div class="_5abn" style="background-color: rgb(197, 206, 224); width: 100%;"></div><div class="_352g"></div></div></span></div></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1l _2ewj"><div class="_5cul"><div class="_5cuo"></div><span class="_5cup">100%</span></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1m _2ewk">20</div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1n _2ewl"><div class="_5kmz _5km_"><div class="_5kn3 ellipsis">474.6K</div><div class="_5kn4 _11uq"><span data-hover="tooltip"><div class="_5abm" style="width: 64%;"><div class="_5abn" style="background-color: rgb(208, 148, 171); width: 100%;"></div><div class="_352g"></div></div></span></div></div></div></td></tr></tbody></table></div><div class="_5yti"><div class="_5ytg">Keep up with the Pages you watch.</div><div class="_5yth"><span class=""><button data-tooltip-content="Promote your Page to get more Likes" data-hover="tooltip" type="button" aria-disabled="false" class="_271k _271m _1qjd _7tvm _7tv2 _7tv4" style="width: auto; max-width: 500px; letter-spacing: normal; color: rgb(255, 255, 255); font-size: 12px; font-weight: bold; font-family: Arial, sans-serif; line-height: 26px; text-align: center; background-color: rgb(24, 119, 242); border-color: rgb(24, 119, 242); height: 28px; padding-left: 11px; padding-right: 11px; border-radius: 2px;" id="js_m0"><div class="_43rl"><div class="_43rm">Get More Likes</div></div></button></span></div></div></div></td></tr><tr class="_5k4c _5k4a"><td class="_5k49 _9b5k" colspan="1"><div class="_5y1j"><div class="_5y1d"><div direction="left" class="clearfix"><div class="_ohe lfloat"><div class="_5y1f" tabindex="0" role="button"><div class="_5y1g">3</div></div></div><div class=""><div class="_4tg1 _5xdw clearfix" direction="left"><div class="_ohe lfloat"><img class="_5y1h img _8o _8t img" src="https://scontent.fkhi10-1.fna.fbcdn.net/v/t1.6435-1/cp0/p80x80/51616521_10156714708548673_669379573728673792_n.png?_nc_cat=1&amp;ccb=1-5&amp;_nc_sid=dbb9e7&amp;_nc_ohc=Vi4WocLbJ5cAX8bgbL0&amp;_nc_ht=scontent.fkhi10-1.fna&amp;oh=812ca2d12c81bb4e7b7d9b57f4ac1fcb&amp;oe=618BB88A" alt=""></div><div class=""><div class="_42ef"><div class="_5y1i ellipsis">HBL</div></div></div></div></div></div></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1k _2ewi"><div class="_5kmz _5km_"><div class="_5kn3 ellipsis">1.6m</div><div class="_5kn4 _11uq"><span data-hover="tooltip" id="js_i5"><div class="_5abm" style="width: 72%;"><div class="_5abn" style="background-color: rgb(197, 206, 224); width: 100%;"></div><div class="_352g"></div></div></span></div></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1l _2ewj"><div class="_5cul"><div class="_5cuo"></div><span class="_5cup">100%</span></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1m _2ewk">18</div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1n _2ewl"><div class="_5kmz _5km_"><div class="_5kn3 ellipsis">740.4K</div><div class="_5kn4 _11uq"><span data-hover="tooltip"><div class="_5abm" style="width: 100%;"><div class="_5abn" style="background-color: rgb(208, 148, 171); width: 100%;"></div><div class="_352g"></div></div></span></div></div></div></td></tr><tr class="_5k4c"><td class="_5k49 _9b5k" colspan="1"><div class="_5y1j"><div class="_5y1d"><div direction="left" class="clearfix"><div class="_ohe lfloat"><div class="_5y1f" tabindex="0" role="button"><div class="_5y1g">4</div></div></div><div class=""><div class="_4tg1 _5xdw clearfix" direction="left"><div class="_ohe lfloat"><img class="_5y1h img _8o _8t img" src="https://scontent.fkhi10-1.fna.fbcdn.net/v/t1.6435-1/cp0/p80x80/118588222_3322284974544530_2712089803505735278_n.png?_nc_cat=105&amp;ccb=1-5&amp;_nc_sid=dbb9e7&amp;_nc_ohc=oYdYnYx2SNgAX9nRj4T&amp;_nc_ht=scontent.fkhi10-1.fna&amp;oh=791541d2d6ae268bdc5f44d3046d63ce&amp;oe=618D7FFA" alt=""></div><div class=""><div class="_42ef"><div class="_5y1i ellipsis">Allied Bank Limited</div></div></div></div></div></div></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1k _2ewi"><div class="_5kmz _5km_"><div class="_5kn3 ellipsis">1.1m</div><div class="_5kn4 _11uq"><span data-hover="tooltip"><div class="_5abm" style="width: 49%;"><div class="_5abn" style="background-color: rgb(197, 206, 224); width: 100%;"></div><div class="_352g"></div></div></span></div></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1l _2ewj"><div class="_5cul"><div class="_5cuo"></div><span class="_5cup">100%</span></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1m _2ewk">26</div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1n _2ewl"><div class="_5kmz _5km_"><div class="_5kn3 ellipsis">18.4K</div><div class="_5kn4 _11uq"><span data-hover="tooltip"><div class="_5abm" style="width: 2%;"><div class="_5abn" style="background-color: rgb(208, 148, 171); width: 100%;"></div><div class="_352g"></div></div></span></div></div></div></td></tr><tr class="_5k4c _5k4a"><td class="_5k49 _9b5k" colspan="1"><div class="_5y1j"><div class="_5y1d"><div direction="left" class="clearfix"><div class="_ohe lfloat"><div class="_5y1f" tabindex="0" role="button"><div class="_5y1g">5</div></div></div><div class=""><div class="_4tg1 _5xdw clearfix" direction="left"><div class="_ohe lfloat"><img class="_5y1h img _8o _8t img" src="https://scontent.fkhi10-1.fna.fbcdn.net/v/t1.18169-1/cp0/p80x80/10432153_799215570136348_7145320981066523381_n.png?_nc_cat=100&amp;ccb=1-5&amp;_nc_sid=dbb9e7&amp;_nc_ohc=ga2J16rxNtYAX86jiS2&amp;_nc_ht=scontent.fkhi10-1.fna&amp;oh=f112a38e2e9c7ad5af43cea1787b6fba&amp;oe=618CB8D2" alt=""></div><div class=""><div class="_42ef"><div class="_5y1i ellipsis">Summit Bank</div></div></div></div></div></div></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1k _2ewi"><div class="_5kmz _5km_"><div class="_5kn3 ellipsis">1m</div><div class="_5kn4 _11uq"><span data-hover="tooltip"><div class="_5abm" style="width: 48%;"><div class="_5abn" style="background-color: rgb(197, 206, 224); width: 100%;"></div><div class="_352g"></div></div></span></div></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1l _2ewj"><div class="_5cul"><div class="_5cuo"></div><span class="_5cup">100%</span></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1m _2ewk">13</div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1n _2ewl"><div class="_5kmz _5km_"><div class="_5kn3 ellipsis">865</div><div class="_5kn4 _11uq"><span data-hover="tooltip"><div class="_5abm" style="width: 0%;"><div class="_5abn" style="background-color: rgb(208, 148, 171); width: 0px;"></div><div class="_352g"></div></div></span></div></div></div></td></tr><tr class="_5k4c"><td class="_5k49 _9b5k" colspan="1"><div class="_5y1j"><div class="_5y1d"><div direction="left" class="clearfix"><div class="_ohe lfloat"><div class="_5y1f" tabindex="0" role="button"><div class="_5y1g">6</div></div></div><div class=""><div class="_4tg1 _5xdw clearfix" direction="left"><div class="_ohe lfloat"><img class="_5y1h img _8o _8t img" src="https://scontent.fkhi10-1.fna.fbcdn.net/v/t1.6435-1/cp0/p80x80/120621905_2481144195513601_3367139649441934967_n.jpg?_nc_cat=1&amp;ccb=1-5&amp;_nc_sid=dbb9e7&amp;_nc_ohc=qqIxIutlZf8AX8sD7e5&amp;_nc_ht=scontent.fkhi10-1.fna&amp;oh=cb5c44d40ca39464c807b2ae47c7d1fa&amp;oe=618A11F0" alt=""></div><div class=""><div class="_42ef"><div class="_5y1i ellipsis">BankIslami Pakistan Limited</div></div></div></div></div></div></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1k _2ewi"><div class="_5kmz _5km_"><div class="_5kn3 ellipsis">938.5K</div><div class="_5kn4 _11uq"><span data-hover="tooltip"><div class="_5abm" style="width: 43%;"><div class="_5abn" style="background-color: rgb(197, 206, 224); width: 100%;"></div><div class="_352g"></div></div></span></div></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1l _2ewj"><div class="_5cul"><div class="_5cuo"></div><span class="_5cup">100%</span></div></div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1m _2ewk">18</div></td><td class="_5k49 _9b5k" colspan="1"><div class="_5y1n _2ewl"><div class="_5kmz _5km_"><div class="_5kn3 ellipsis">22.7K</div><div class="_5kn4 _11uq"><span data-hover="tooltip"><div class="_5abm" style="width: 3%;"><div class="_5abn" style="background-color: rgb(208, 148, 171); width: 100%;"></div><div class="_352g"></div></div></span></div></div></div></td></tr><tr class="_5k4c _5k4a"><td class="_5k49 _9b5k" colspan="5"><div class="_58oy" tabindex="0" role="button">See more Pages</div></td></tr></tbody></table></div></div></div></div><div></div></div></div></div></div></div></div>

Answer 1

The Element you are trying to find profile_page_insights_hubble is in an Iframe .

Need to switch to iframe to access the Element.

Try like below and confirm:

try:
    # Switch Iframe
    wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//div[@aria-label='Page Insights']//iframe")))
    
    # Look for the Element
    wait.until(EC.presence_of_element_located((By.ID,"profile_page_insights_hubble")))
    print("FOUND")
except:
    print("ERROR")

FOUND