How to get title, thumbnail and description of websites (Facebook/Twitter like preview) that have a cookie consent page?

Question

What I want to do: Get the title, thumbnail picture and description of this article (German news website) for example.

When I post the link on Twitter/Facebook, they get all the right data for their preview.

When I use PHP curl or file_get_contents all I get is the data from the cookie consent page that appears when you first visit the website and did not yet accept their terms.

I haven't figured out how Twitter and Facebook are doing it. Any idea?

Do they maybe use special user agents or something like that? So that the cookie consent page isn't displayed in the first place?

Answer 1

1. Manually look for the cookie value in the inspector after accepting the cookie.

2. add a header to your scraper with the cookie value attached to it.