HTTP Design. Why servers do not send all files immediately after getting GET request?

Question

As I understand, typical navigation works like this:

1. The browser sends a request to the server
2. The server sends back an HTML file
3. The browser parses the HTML file and figures out which files it needs
4. The browser sends a separate request for each JS/CSS file
5. The server sends JS/CSS files back to the browser
6. Finally, the browser has everything to display the site

Are steps 3 and 4 really necessary? Why don't we have a server-side list of all files required for the site? This way the server can send all the files without waiting for the browser.

Here are my best attempts to explain such a design:

Explanation 1: HTML file is more important in the early stages because the browser builds a DOM tree first.

Okey. But we can have a prioritized server-side list of all files required for the site. This way the browser can build the DOM tree and download CSS concurrently.

Explanation 2: search engines don't want to download anything besides HTML.

Yes. But we can add Get-All-Files-Immediately HTTP header for that.

Explanation 3: this is a legacy. We don't want to break old sites.

Yes, we don't. But Get-All-Files-Immediately header will solve that too.

Explanation 4: this overhead is negligible.

Is it so? Let's look at Dev Tools->Network for facebook.com: :

Steps 3 and 4 happen between points A and B. This seems like a good fraction of TTFP. Am I wrong?

I am very confused as I cannot find one single good reason for such a design. What am I missing?

Answer 1

There is a list of files that the browser needs, it's in the HTML.

It's possible to preemptively send things from server to browser if the server knows the browser will request it in the future via HTTP/2 Push.

This can indeed reduce the total latency. But it also comes with challenges. How does the server know for example that the client doesn't already have the file? Clients will often cache assets like CSS and images, so if a client hits the server again, push those assets again can be wasteful.

The reality is that for most people the first roundtrip to get the HTML is not enough of a problem for it to be worth fixing.