无法从网站上抓取一些“ div”标签

Question

I am trying to scrape job posts from this page: https://www.fl.ru .

Probably quite a newbie problem, but it turns out I can get certain tags, while others seem to be unreachable, eg:

from urllib.request import urlopen
from bs4 import BeautifulSoup

html = urlopen("https://www.fl.ru/projects/")
bsObj = BeautifulSoup(html, "lxml")

textTags = bsObj.findAll("div", class_="b-post__txt ")
print(str(textTags))

Thanks

Answer 1

If you download the page html using some downloader ( wget or curl ) you will see that the elements are not in the page at all. The elements are being generated by javascript.

For example (snippet from the source of the page):

<script type="text/javascript">document.write('<div class="b-post__body b-post__body_padtop_15 b-post__body_overflow_hidden b-layuot_width_full"> <div class="b-post__txt "> У нас есть для вас вакансия Full-stack PHP-разработчика на удаленную работу (полный рабочий день) или в офис (г. Москва).&nbsp; Работать нужно будет над нашими проектами, в том... </div> <div id="project-reason-3728923" style="display: none"> </div> </div>');</script>

You have two options: Execute the javascript (with a browser and something like selenium to drive it) or parse it manually, by using beautiful soup to get the <script> tag contents, then extracting the text inside document.write() and reparsing it with beautiful soup.

Answer 2

Many modern webpages build the DOM in the browser dynamically using Javascript, and the parts you're looking for do not exist until the browser has finished building the page.

If you're not using a browser or library that has Javascript functionality, the page elements you're looking for will simply not exist.