如何检测正在使用的电子商务软件

Question

I am making an webcrawler in C# which needs to find webshops. The problem i'm having is that I need to detect if the webpage is a webshop. If it is I need to find out what type of e-commerse software it is using. But the problem is that I don't know how you can detect it in the source code.

I also have just a Chrome plugin called builtwith which can detect all kinds of software. But I have yet to find out how they are doing that.

It would be nice if someone could help me with this problem

Answer 1

Before giving you an actual answer, it's worth noting that what you're proposing could be in violation of the terms of use for many websites out there. You should take the time to investigate what legal liability you might be exposing yourself and your organization to.

This is going to be a lot of time consuming work, but it's not difficult. Your crawler is just going to need to simply work using a rules-based approach to detect signatures in the payload of the page.

1. Find the specific software that you're intending to detect.
2. Find 2-3 sites that are definitely using the software.
3. Review the HTML payload to see what scripts, CSS, and HTML patterns they have that are common across the sites.
4. Build a code-based rule that can detect those patterns consistently. For example: if (html.Contains("widgetName")) isPlatformName = true;
5. Test that patterns across more sites that you know for certain are using that software.
6. Repeat for each software vendor.

The more complicated thing will be when the targets have multiple versions and you need to adapt your rules to know and be aware of the various versions, or when platforms are very similar.

I think the most complicated part of this is having a well-thought-out bot issue detection, reporting, and throttling architecture in place. You should probably spend the bulk of your time planning that.

That's it.

Answer 2

There are a couple different ways to determine the technologies a site is using. Firstly, if you are technically savvy, you can right click on an eCommerce page (either catalog, checkout page, etc) and look at the source code. Many platforms will have hints in the source code that will give you an idea what the site is running.

You can also look at the DNS/hosting information, which would help you determine if the eCommerce solution is hosted or SaaS (like Shopify, for example).

You can also try using InterNIC and enter the domain name. The results will return the nameservers which could point you in the right direction.

Finally, if all that sleuthing seems too difficult, there's an easier way! Try BuiltWith. It's generally pretty reliable, as long as the system you're looking up isn't custom/proprietary. Enter a domain into BuiltWith and it will show you the platform, widgets used, analytics and tracking codes, CDNs, CMS, payment processors, and more.