retrieve comments using python requests module

Question

I am trying to find all the comments in a web page.

import requests

with requests.session() as r:
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0'}
    r = requests.get('https://www.example.com', verify=False, headers=headers)
    print(r)

This script returns all the source code of the page. However, I am only interested in finding the commented lines. Can anyone help with me with a regular expression to find the commented lines. Or is there a better method to finding this?

Answer 1

You might try BeautifulSoup4 , which has built-in function for identifying comments.

Here's one StackOverflow that demonstrates this: How to find all comments with Beautiful Soup