如何欺骗爬虫/扫描仪检测网络技术？

Question

I have a PHP website and I have all the URL rewriting done. Such that the PHP extensions are hidden. But somehow the webcrawlers and security scanners / vulnerability scanners are able to find my site was developed in PHP.

How do I avoid that or How do I trick them that this website was not developed through PHP ?

Answer 1

As mentioned in a comment, I develop on a security scanner which is probably similar to the one you're trying to hide certain information from.

One of the reasons this is difficult to achieve is because security scanners don't just look at one thing, usually. The one I work on uses a very large database of fingerprints to determine if specific files or behaviors represent a certain plugin, framework, site builder, or even if the website is generated in a structurally similar way to another that uses a specific tool to generate the HTML.

Once we discover one technology, we can relate it based on those fingerprints to another website that doesn't expose all of the same information or perhaps even intentionally changes it to something misleading.

A great example of this is when people change their X-Powered-By header to something that is not representative of what they use.

Say if you ran a PHP driven website but your X-Powered-By header was "Microsoft ASP.NET" or anything else. We could assume the information is false or otherwise questionable if all of your extensions end in .php or are hidden. There are also certain behavioral nuances that exist for other technologies, such as ASP.NET, which are the existence of structural fingerprints like the _VIEWSTATE strings.

Additionally, you need to keep in mind things like URL formats, POST/PUT behavior, and even what other software you run. If you run WordPress for example, it's very probable that you're using PHP.

This is only a small example. There are thousands of rules per technology which generate more and more confidence that we're right about our guess. We have a database of products that each have unique or crossed fingerprints and it has around 10,000 identified products in it.

All of this information is collected and analyzed. If we determine a website is not representing itself correctly, it flags the website and a list of pages in question for human review, at which point an analyst will manually plug away at the website and determine its technologies by hand and figure out new fingerprints for it.

Answer 2

One-of-the legitimate ways of doing it.

---

Well most web-vulnerability scanners or crawlers make use of your website headers to find out this. Say if you do this..

<?php
var_dump(headers_list());

You will get..

array(1) {
  [0]=>
  string(23) "X-Powered-By: PHP/5.4.3"
}

So with that information a crawler can easily make up that your site was developed with PHP.

How to avoid this ?

You could make use of header_remove() in PHP for that.

As you can see from the code..

<?php
echo "<pre>";
var_dump(headers_list());
header_remove();
var_dump(headers_list());

OUTPUT :

array(1) {
  [0]=>
  string(23) "X-Powered-By: PHP/5.4.3"
}

array(0) {
}

The headers are now empty.