根據位置過濾Shodan查詢結果

Question

我正在嘗試解析Shodan查詢結果，並僅使用與我設置的條件匹配的結果編寫一個新的JSON文件。

JSON條目示例：

{
  "matches": [
    {
      "product": "Microsoft IIS httpd",
      "hostnames": [],
      "hash": -1722221328,
      "ip": 1261462342,
      "isp": "AT&T Internet Services",
      "transport": "tcp",
      "cpe": [
        "cpe:/a:microsoft:iis:7.5",
        "cpe:/o:microsoft:windows"
      ],
      "data": "",
      "asn": "AS7018",
      "port": 631,
      "version": "7.5",
      "link": "Ethernet or modem",
      "location": {
        "city": null,
        "region_code": null,
        "area_code": null,
        "longitude": -97.822,
        "country_code3": "USA",
        "latitude": 37.751000000000005,
        "postal_code": null,
        "dma_code": null,
        "country_code": "US",
        "country_name": "United States"
      },
      "timestamp": "2017-02-28T23:55:24.306344",
      "domains": [],
      "org": "AT&T Internet Services",
      "os": null,
      "_shodan": {
        "crawler": "122dd688b363c3b45b0e7582622da1e725444808",
        "id": null,
        "module": "http-simple-new",
        "options": {}
      },
      "ip_str": "75.48.99.70"
    },
    {
      "hash": 605323305,
      "ip": 1757819678,
      "isp": "Google Cloud",
      "transport": "tcp",
      "data": "",
      "asn": "AS15169",
      "port": 9000,
      "hostnames": [
        "30.51.198.104.bc.googleusercontent.com"
      ],
      "location": {
        "city": "Mountain View",
        "region_code": "CA",
        "area_code": 650,
        "longitude": -122.0574,
        "country_code3": "USA",
        "latitude": 37.41919999999999,
        "postal_code": "94043",
        "dma_code": 807,
        "country_code": "US",
        "country_name": "United States"
      },
      "timestamp": "2017-02-28T23:51:35.997036",
      "domains": [
        "googleusercontent.com"
      ],
      "org": "Google Cloud",
      "os": null,
      "_shodan": {
        "crawler": "545144fc95e7a7ef13ece5dbceb98ee386b37950",
        "id": null,
        "module": "https-simple-new",
        "options": {}
      },
      "ip_str": "104.198.51.30"
    }
  ],
  "total": 2
}

我的希望是加載JSON文件並遍歷元素集，如果不符合將位置country_code設置為“ US”的條件，則刪除該元素。

我擁有的代碼（由https://gist.github.com/madonnelly提供，並通過JsonObject屬性進行迭代 ）如下：

import java.io.FileNotFoundException;
import java.io.FileReader;
import java.util.Map;
import java.util.Set;

import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;

public class ParseJSON {

    public static void main(String[] args) {
        JsonObject shodanJSON = convertFileToJSON("<Path to JSON file>");

        Set<Map.Entry<String,JsonElement>> queryResults = shodanJSON.entrySet();

        for (Map.Entry<String, JsonElement> queryResult : queryResults) {
            JsonArray locArray = queryResult.getValue().getAsJsonObject().getAsJsonArray("location");
            for (JsonElement locData : locArray) {
                if (locData.getAsJsonObject().getAsJsonPrimitive("country_code").equals("US")) {
                    System.out.println(locData.getAsString());
                }
            }
        }
    }

    public static JsonObject convertFileToJSON(String fileName) {

        // Read from File to String
        JsonObject jsonObject = new JsonObject();

        try {
            JsonParser parser = new JsonParser();
            JsonElement jsonElement = parser.parse(new FileReader(fileName));
            jsonObject = jsonElement.getAsJsonObject();
        } catch (FileNotFoundException e) {

        }
        return jsonObject;
    }
}

當我運行代碼時，我收到錯誤

線程“ main”中的異常java.lang.IllegalStateException：不是JSON對象：[{“ product”：“ Microsoft IIS httpd”，“ hostnames”：[]，“ hash”：-1722221328，“ ip”：1261462342，“ isp“：” AT＆T Internet Services“，” transport“：...}]位於com.google.gson.JsonElement.getAsJsonObject（JsonElement.java:90），位於com.cti.shodan.ParseJSON.main（ParseJSON.java： 22）

我確定我犯了很多錯誤，希望有人指出我犯的錯誤。 提前致謝！

Answer 1

您有一些關於具體JSON文檔解析的假設，這些假設不符合其真實結構。 我假設您要顯示匹配的結果（ $.matches ），以過濾其子屬性值（ $.matches.*.location.country_code ）。

    for ( final Entry<String, JsonElement> queryResult : shodanJsonObject.entrySet() ) {
        final JsonElement value = queryResult.getValue();
        // This is necessary to skip <"total": 2>
        if ( value.isJsonArray() ) {
            // Here comes an array, and should be iterated, rather than taken as an object
            for ( final JsonElement match : value.getAsJsonArray() ) {
                // This was the root cause, not an array
                final JsonObject location = match.getAsJsonObject().getAsJsonObject("location");
                // Previously jsonPrimitive.equals("US") -- convert the JSON primitive to a string first
                if ( location.getAsJsonPrimitive("country_code").getAsString().equals("US") ) {
                    // Previously getAsString() -- it requires a JSON string literal, just remove it
                    System.out.println(match);
                }
            }
        }
    }

使用Java 8可能會更簡單：

shodanJsonObject.entrySet()
        .stream()
        .map(Entry::getValue)
        .filter(JsonElement::isJsonArray)
        .map(JsonElement::getAsJsonArray)
        .flatMap(jsonElements -> StreamSupport.stream(jsonElements.spliterator(), false))
        .peek(System.out::println)
        .map(JsonElement::getAsJsonObject)
        .map(jsonObject -> jsonObject.getAsJsonObject("location"))
        .filter(location -> location.getAsJsonPrimitive("country_code").getAsString().equals("US"))
        .forEach(jsonObject -> {
        }); // forEach is a terminal operation and it "pushes" the entire chain above

如果可以使用諸如JsonPath之類的查詢庫，這可能是最有表現力的方式：

final JsonPath jsonPath = JsonPath.compile("$.matches.*[?(@.location.country_code=='US')]");
for ( final Object match : jsonPath.<JSONArray>read(JSON) ) {
    System.out.println(match);
}