I want to get the content from link in python using beautifulsoup

Question

i am trying to scrap the the data from https://www.indeed.ae/jobs?q=&l=dubai

I want to grab the content(which is available on clicking each job title available on above link)

i am using python, requests, and bs4

its content from a link with in link, i want to grab all content of each job

import requests
from bs4 import BeautifulSoup
import lxml
import urllib.request

url = 'https://www.indeed.ae/jobs?q=&l=dubai'
response = requests.get(url)


soup = BeautifulSoup(response.text, 'lxml')


links = soup.find_all('a', {'class': 'jobtitle turnstileLink'})

for correct_url in (soup.find_all('a', {'class': 'jobtitle turnstileLink'})):
    correct_url = ("https://indeed.ae" + (links.get('href')))
    print(correct_url)

Answer 1

This script will print all descriptions of jobs (the page loads this information via JavaScipt, but you can use requests module to load this information):

import requests
from bs4 import BeautifulSoup


url = 'https://www.indeed.ae/jobs?q=&l=dubai'
jobdesc_url = 'https://www.indeed.ae/rpc/jobdescs'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')
jks = ','.join(jk['data-jk'] for jk in soup.select('[data-jk]'))

descriptions = requests.get(jobdesc_url, params={'jks': jks}).json()

for jk in soup.select('[data-jk]'):
    print(jk.h2.get_text(strip=True))
    print()
    print(jk.find_next('span', class_='company').get_text(strip=True))
    print('---')
    print(BeautifulSoup(descriptions[jk['data-jk']], 'html.parser').get_text())
    print('-' * 80)

Prints:

RETAIL SALES ASSOCIATE

Al Ghazi
---
Hiring a Retail Sales Associate for Dubai.

Key Responsibilities:
Attend to walk in customers, phone calls and provide customer service.
Maintain a conducive environment in the outlet through house keeping.
Promote products and ensure high levels of customer satisfaction through customer journey mapping.
Recommend and display items that match customers’ needs and preferences.

Key Requirements:
High school Diploma.
One to two years experience as a Retail sales Associate.
Presentable and well-groomed
Observe strict punctuality.
Able to work well in a team.
Able to multi-task.
--------------------------------------------------------------------------------
Human Resource Officer

Raban Al-Safina
---
The Role
We are looking for a HR Officer with 3 to 4 years UAE work experience.Candidate must be available currently in UAE and ready to join immediately. Candidates on Husband / Father Sponsor visas will be given more preference. Job Responsibilities: • Working closely with various departments, increasingly in a consultancy role, assisting line managers to understand and implement policies and procedures • Administering payroll and maintaining employee records • Interpreting and advising on employment law • Advising on pay and other remuneration issues, including promotion and benefits • Developing and implementing policies on issues like working conditions, performance management, equal opportunities, • Disciplinary procedures and absence management • Recruiting staff, which involves developing job descriptions and person specifications, preparing job adverts, checking application forms, shortlisting, interviewing and selecting candidates

Requirements

As a Human Resources (HR) officer you'll develop, advise on and implement policies relating to the effective use of staff in an organisation. • In the role your aim is to ensure that the organisation you work for employs the right balance of staff in terms of skill and experience, and that training and development opportunities are available to colleagues to enhance their performance and achieve the company's business aims. • HR officers are involved in a range of activities whatever the size or type of business. These cover areas such as: • Conditions of employment • Equality and diversity • Negotiation with external work-related agencies • Pay roll management • Recruitment • Working practices.

About the company
Raban Al-Safina Group of Companies stands as one of the most prominent industrial organizations in Iraq work in different fields in power industry, oil & gas, systems technology, and many others.
--------------------------------------------------------------------------------

...and so on.