根据位置过滤Shodan查询结果

Question

I am attempting to parse Shodan query results and write a new JSON file with only the results that match the criteria I have set.

Example JSON entries:

{
  "matches": [
    {
      "product": "Microsoft IIS httpd",
      "hostnames": [],
      "hash": -1722221328,
      "ip": 1261462342,
      "isp": "AT&T Internet Services",
      "transport": "tcp",
      "cpe": [
        "cpe:/a:microsoft:iis:7.5",
        "cpe:/o:microsoft:windows"
      ],
      "data": "",
      "asn": "AS7018",
      "port": 631,
      "version": "7.5",
      "link": "Ethernet or modem",
      "location": {
        "city": null,
        "region_code": null,
        "area_code": null,
        "longitude": -97.822,
        "country_code3": "USA",
        "latitude": 37.751000000000005,
        "postal_code": null,
        "dma_code": null,
        "country_code": "US",
        "country_name": "United States"
      },
      "timestamp": "2017-02-28T23:55:24.306344",
      "domains": [],
      "org": "AT&T Internet Services",
      "os": null,
      "_shodan": {
        "crawler": "122dd688b363c3b45b0e7582622da1e725444808",
        "id": null,
        "module": "http-simple-new",
        "options": {}
      },
      "ip_str": "75.48.99.70"
    },
    {
      "hash": 605323305,
      "ip": 1757819678,
      "isp": "Google Cloud",
      "transport": "tcp",
      "data": "",
      "asn": "AS15169",
      "port": 9000,
      "hostnames": [
        "30.51.198.104.bc.googleusercontent.com"
      ],
      "location": {
        "city": "Mountain View",
        "region_code": "CA",
        "area_code": 650,
        "longitude": -122.0574,
        "country_code3": "USA",
        "latitude": 37.41919999999999,
        "postal_code": "94043",
        "dma_code": 807,
        "country_code": "US",
        "country_name": "United States"
      },
      "timestamp": "2017-02-28T23:51:35.997036",
      "domains": [
        "googleusercontent.com"
      ],
      "org": "Google Cloud",
      "os": null,
      "_shodan": {
        "crawler": "545144fc95e7a7ef13ece5dbceb98ee386b37950",
        "id": null,
        "module": "https-simple-new",
        "options": {}
      },
      "ip_str": "104.198.51.30"
    }
  ],
  "total": 2
}

My hope is to load the JSON file and iterate over the set of elements, removing an element if it does not match the criteria of having a location country_code of "US".

The code I have (courtesy of https://gist.github.com/madonnelly and Iterate over JsonObject properties ) is as follows:

import java.io.FileNotFoundException;
import java.io.FileReader;
import java.util.Map;
import java.util.Set;

import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;

public class ParseJSON {

    public static void main(String[] args) {
        JsonObject shodanJSON = convertFileToJSON("<Path to JSON file>");

        Set<Map.Entry<String,JsonElement>> queryResults = shodanJSON.entrySet();

        for (Map.Entry<String, JsonElement> queryResult : queryResults) {
            JsonArray locArray = queryResult.getValue().getAsJsonObject().getAsJsonArray("location");
            for (JsonElement locData : locArray) {
                if (locData.getAsJsonObject().getAsJsonPrimitive("country_code").equals("US")) {
                    System.out.println(locData.getAsString());
                }
            }
        }
    }

    public static JsonObject convertFileToJSON(String fileName) {

        // Read from File to String
        JsonObject jsonObject = new JsonObject();

        try {
            JsonParser parser = new JsonParser();
            JsonElement jsonElement = parser.parse(new FileReader(fileName));
            jsonObject = jsonElement.getAsJsonObject();
        } catch (FileNotFoundException e) {

        }
        return jsonObject;
    }
}

When I run my code I am receiving the error

Exception in thread "main" java.lang.IllegalStateException: Not a JSON Object: [{"product":"Microsoft IIS httpd","hostnames":[],"hash":-1722221328,"ip":1261462342,"isp":"AT&T Internet Services","transport":...}] at com.google.gson.JsonElement.getAsJsonObject(JsonElement.java:90) at com.cti.shodan.ParseJSON.main(ParseJSON.java:22)

I am sure I am making a ton of mistakes and am hoping someone can point out the error I am making. Thanks in advance!

Answer 1

You have some assumptions about the concrete JSON document parsing that do not meet its real structure. I'm assuming you're going to display the matched results ( $.matches ) filtering out on their subproperty values ( $.matches.*.location.country_code ).

    for ( final Entry<String, JsonElement> queryResult : shodanJsonObject.entrySet() ) {
        final JsonElement value = queryResult.getValue();
        // This is necessary to skip <"total": 2>
        if ( value.isJsonArray() ) {
            // Here comes an array, and should be iterated, rather than taken as an object
            for ( final JsonElement match : value.getAsJsonArray() ) {
                // This was the root cause, not an array
                final JsonObject location = match.getAsJsonObject().getAsJsonObject("location");
                // Previously jsonPrimitive.equals("US") -- convert the JSON primitive to a string first
                if ( location.getAsJsonPrimitive("country_code").getAsString().equals("US") ) {
                    // Previously getAsString() -- it requires a JSON string literal, just remove it
                    System.out.println(match);
                }
            }
        }
    }

With Java 8 it may be somewhat more simple:

shodanJsonObject.entrySet()
        .stream()
        .map(Entry::getValue)
        .filter(JsonElement::isJsonArray)
        .map(JsonElement::getAsJsonArray)
        .flatMap(jsonElements -> StreamSupport.stream(jsonElements.spliterator(), false))
        .peek(System.out::println)
        .map(JsonElement::getAsJsonObject)
        .map(jsonObject -> jsonObject.getAsJsonObject("location"))
        .filter(location -> location.getAsJsonPrimitive("country_code").getAsString().equals("US"))
        .forEach(jsonObject -> {
        }); // forEach is a terminal operation and it "pushes" the entire chain above

And probably the most expressive way if it's possible to use querying libraries like JsonPath:

final JsonPath jsonPath = JsonPath.compile("$.matches.*[?(@.location.country_code=='US')]");
for ( final Object match : jsonPath.<JSONArray>read(JSON) ) {
    System.out.println(match);
}