根据位置过滤Shodan查询结果

Question

我正在尝试解析Shodan查询结果，并仅使用与我设置的条件匹配的结果编写一个新的JSON文件。

JSON条目示例：

{
  "matches": [
    {
      "product": "Microsoft IIS httpd",
      "hostnames": [],
      "hash": -1722221328,
      "ip": 1261462342,
      "isp": "AT&T Internet Services",
      "transport": "tcp",
      "cpe": [
        "cpe:/a:microsoft:iis:7.5",
        "cpe:/o:microsoft:windows"
      ],
      "data": "",
      "asn": "AS7018",
      "port": 631,
      "version": "7.5",
      "link": "Ethernet or modem",
      "location": {
        "city": null,
        "region_code": null,
        "area_code": null,
        "longitude": -97.822,
        "country_code3": "USA",
        "latitude": 37.751000000000005,
        "postal_code": null,
        "dma_code": null,
        "country_code": "US",
        "country_name": "United States"
      },
      "timestamp": "2017-02-28T23:55:24.306344",
      "domains": [],
      "org": "AT&T Internet Services",
      "os": null,
      "_shodan": {
        "crawler": "122dd688b363c3b45b0e7582622da1e725444808",
        "id": null,
        "module": "http-simple-new",
        "options": {}
      },
      "ip_str": "75.48.99.70"
    },
    {
      "hash": 605323305,
      "ip": 1757819678,
      "isp": "Google Cloud",
      "transport": "tcp",
      "data": "",
      "asn": "AS15169",
      "port": 9000,
      "hostnames": [
        "30.51.198.104.bc.googleusercontent.com"
      ],
      "location": {
        "city": "Mountain View",
        "region_code": "CA",
        "area_code": 650,
        "longitude": -122.0574,
        "country_code3": "USA",
        "latitude": 37.41919999999999,
        "postal_code": "94043",
        "dma_code": 807,
        "country_code": "US",
        "country_name": "United States"
      },
      "timestamp": "2017-02-28T23:51:35.997036",
      "domains": [
        "googleusercontent.com"
      ],
      "org": "Google Cloud",
      "os": null,
      "_shodan": {
        "crawler": "545144fc95e7a7ef13ece5dbceb98ee386b37950",
        "id": null,
        "module": "https-simple-new",
        "options": {}
      },
      "ip_str": "104.198.51.30"
    }
  ],
  "total": 2
}

我的希望是加载JSON文件并遍历元素集，如果不符合将位置country_code设置为“ US”的条件，则删除该元素。

我拥有的代码（由https://gist.github.com/madonnelly提供，并通过JsonObject属性进行迭代 ）如下：

import java.io.FileNotFoundException;
import java.io.FileReader;
import java.util.Map;
import java.util.Set;

import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;

public class ParseJSON {

    public static void main(String[] args) {
        JsonObject shodanJSON = convertFileToJSON("<Path to JSON file>");

        Set<Map.Entry<String,JsonElement>> queryResults = shodanJSON.entrySet();

        for (Map.Entry<String, JsonElement> queryResult : queryResults) {
            JsonArray locArray = queryResult.getValue().getAsJsonObject().getAsJsonArray("location");
            for (JsonElement locData : locArray) {
                if (locData.getAsJsonObject().getAsJsonPrimitive("country_code").equals("US")) {
                    System.out.println(locData.getAsString());
                }
            }
        }
    }

    public static JsonObject convertFileToJSON(String fileName) {

        // Read from File to String
        JsonObject jsonObject = new JsonObject();

        try {
            JsonParser parser = new JsonParser();
            JsonElement jsonElement = parser.parse(new FileReader(fileName));
            jsonObject = jsonElement.getAsJsonObject();
        } catch (FileNotFoundException e) {

        }
        return jsonObject;
    }
}

当我运行代码时，我收到错误

线程“ main”中的异常java.lang.IllegalStateException：不是JSON对象：[{“ product”：“ Microsoft IIS httpd”，“ hostnames”：[]，“ hash”：-1722221328，“ ip”：1261462342，“ isp“：” AT＆T Internet Services“，” transport“：...}]位于com.google.gson.JsonElement.getAsJsonObject（JsonElement.java:90），位于com.cti.shodan.ParseJSON.main（ParseJSON.java： 22）

我确定我犯了很多错误，希望有人指出我犯的错误。 提前致谢！

Answer 1

您有一些关于具体JSON文档解析的假设，这些假设不符合其真实结构。 我假设您要显示匹配的结果（ $.matches ），以过滤其子属性值（ $.matches.*.location.country_code ）。

    for ( final Entry<String, JsonElement> queryResult : shodanJsonObject.entrySet() ) {
        final JsonElement value = queryResult.getValue();
        // This is necessary to skip <"total": 2>
        if ( value.isJsonArray() ) {
            // Here comes an array, and should be iterated, rather than taken as an object
            for ( final JsonElement match : value.getAsJsonArray() ) {
                // This was the root cause, not an array
                final JsonObject location = match.getAsJsonObject().getAsJsonObject("location");
                // Previously jsonPrimitive.equals("US") -- convert the JSON primitive to a string first
                if ( location.getAsJsonPrimitive("country_code").getAsString().equals("US") ) {
                    // Previously getAsString() -- it requires a JSON string literal, just remove it
                    System.out.println(match);
                }
            }
        }
    }

使用Java 8可能会更简单：

shodanJsonObject.entrySet()
        .stream()
        .map(Entry::getValue)
        .filter(JsonElement::isJsonArray)
        .map(JsonElement::getAsJsonArray)
        .flatMap(jsonElements -> StreamSupport.stream(jsonElements.spliterator(), false))
        .peek(System.out::println)
        .map(JsonElement::getAsJsonObject)
        .map(jsonObject -> jsonObject.getAsJsonObject("location"))
        .filter(location -> location.getAsJsonPrimitive("country_code").getAsString().equals("US"))
        .forEach(jsonObject -> {
        }); // forEach is a terminal operation and it "pushes" the entire chain above

如果可以使用诸如JsonPath之类的查询库，这可能是最有表现力的方式：

final JsonPath jsonPath = JsonPath.compile("$.matches.*[?(@.location.country_code=='US')]");
for ( final Object match : jsonPath.<JSONArray>read(JSON) ) {
    System.out.println(match);
}