I am trying to make a bot to scrape this page http://www.skysports.com/transfer-centre , when I open it in Chrome I can see the element I want in developer tool
but when I use the code below, it return None:
import requests
from bs4 import BeautifulSoup
page = requests.get('http://www.skysports.com/transfer-centre').text
soup = BeautifulSoup(page, 'lxml')
print(soup.find('time'))
Am I missing something?
for extracting time from the url provided, https://data.livefyre.com/bs3/v3.1/bskyb.fyre.co/363166/MTAwMDk1MTI=/init Url I got from http://www.skysports.com/transfer-centre
import requests
import time
import json
page = requests.get('https://data.livefyre.com/bs3/v3.1/bskyb.fyre.co/363166/MTAwMDk1MTI=/init').json()
contents = page['headDocument']['content']
for content in contents:
c = content['content']
t = c['updatedAt']
tt = time.strftime('%H:%M', time.localtime(t))
authorId = c['authorId']
bodyHtml = c['bodyHtml']
print('time : {} , auther: {}'.format(tt, authorId))
print('------------')
print('bodyHtml: {}' , json.dumps(bodyHtml, None, 4))
print('-----------\n\n')
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.