Grouping and merging by value using clojure?

Question

i have a set of data like this

{
  "data": [
    {
      "target_group_id": "1234",
      "target_group_name": "abc",
      "targets": [
        {
          "target_id": "456",
          "target_name": "john"
        }
      ]
    },
    {
      "target_group_id": "56789",
      "target_group_name": "cdes",
      "targets": [
        {
          "target_id": "0987",
          "target_name": "john"
        }
      ]
    },
    {
      "target_group_id": "1234",
      "target_group_name": "abc",
      "targets": [
        {
          "target_id": "789",
          "target_name": "doe"
        }
      ]
    }
  ]
}

and want to transform by grouping and merging data by target group id so the target within the same target_group_id will be added to the existing target group and changing the key root of data from "data" into "target_groups"

{
  "target_groups": [
    {
      "target_group_id": "1234",
      "target_group_name": "abc",
      "targets": [
        {
          "target_id": "456",
          "target_name": "john"
        },
        {
          "target_id": "789",
          "target_name": "doe"
        }
      ]
    },
    {
      "target_group_id": "56789",
      "target_group_name": "cdes",
      "targets": [
        {
          "target_id": "0987",
          "target_name": "john"
        }
      ]
    }
  ]
}

is there any effective way to do it with clojure since my original code using php and take a lot of "if-clause" and "foreach"? thanks...

Answer 1

Here is how I would approach it:

(ns tst.demo.core
  (:use demo.core tupelo.core tupelo.test)
  (:require
    [clojure.string :as str]
    [tupelo.string :as ts]
    [tupelo.core :as t]))

(def data-json
"{ 'data': [
    { 'target_group_id': '1234',
      'target_group_name': 'abc',
      'targets': [
        { 'target_id': '456',
          'target_name': 'john' }
      ]
    },
    { 'target_group_id': '56789',
      'target_group_name': 'cdes',
      'targets': [
        { 'target_id': '0987',
          'target_name': 'john'  }
      ]
    },
    {
      'target_group_id': '1234',
      'target_group_name': 'abc',
      'targets': [
        { 'target_id': '789',
          'target_name': 'doe'  }
      ]
    }
  ]
} " )

with transformation:

(dotest
  (let [data-edn (t/json->edn
                   (ts/quotes->double data-json))
        d2       (t/it-> data-edn
                   (:data it) ; unnest from :data key
                   (group-by :target_group_id it ) )
        d3       (t/forv [[tgt-id entries] d2]
                   {:tgt-group-id   tgt-id
                    :tgt-group-name (:target_group_name (first entries))
                    :targets-all    (mapv :targets entries)}) ]

and results/tests:

    (is= data-edn
      {:data
       [{:target_group_id   "1234",
         :target_group_name "abc",
         :targets           [{:target_id "456", :target_name "john"}]}
        {:target_group_id   "56789",
         :target_group_name "cdes",
         :targets           [{:target_id "0987", :target_name "john"}]}
        {:target_group_id   "1234",
         :target_group_name "abc",
         :targets           [{:target_id "789", :target_name "doe"}]}]})

    (is= d2
      {"1234"
       [{:target_group_id   "1234",
         :target_group_name "abc",
         :targets           [{:target_id "456", :target_name "john"}]}
        {:target_group_id   "1234",
         :target_group_name "abc",
         :targets           [{:target_id "789", :target_name "doe"}]}],
       "56789"
       [{:target_group_id   "56789",
         :target_group_name "cdes",
         :targets           [{:target_id "0987", :target_name "john"}]}]})

    (is= d3
      [{:tgt-group-id   "1234",
        :tgt-group-name "abc",
        :targets-all    [[{:target_id "456", :target_name "john"}]
                         [{:target_id "789", :target_name "doe"}]]}
       {:tgt-group-id   "56789",
        :tgt-group-name "cdes",
        :targets-all    [[{:target_id "0987", :target_name "john"}]]}]) ))

Answer 2

Using just core clojure (with the data.json library).

First, acquire and unwrap our data:

(def data (-> "grouping-and-merging.json"
              slurp
              clojure.data.json/read-str 
              (get "data")))

When we address groups of targets, we are going to need to concatenate them. I was doing this inline, but it looks messy in the reduce, so here's a helper function:

(defn concat-targets [acc item]
  (update acc "targets" concat (item "targets")))

Then let's do the work!

(def output (->> data
                 (group-by #(get % "target_group_id"))
                 vals
                 (map #(reduce concat-targets %))
                 (assoc {} "target_groups")
                 clojure.data.json/write-str))

I'm feeling lucky that I got away with the threading macros working so well, although you'll note I had to switch from pre-threading to post-threading between the two phases. Normally I find myself wanting something like the Tupelo it-> used in Alan's answer.

I also feel like the reduce is cheating slightly - I am assuming that there won't be any subtleties and that just taking any extra keys from the first item will be sufficient.

Answer 3

Another way to do the transformation:

{"target_groups" (map merge-vector (-> "data.json"
                                       slurp
                                       json/read-str
                                       (get "data")
                                       (set/index ["target_group_id" "target_group_name"])
                                       vals))}

;; =>
{"target_groups"
 ({"target_group_id" "1234",
   "target_group_name" "abc",
   "targets"
   ({"target_id" "789", "target_name" "doe"}
    {"target_id" "456", "target_name" "john"})}
  {"target_group_id" "56789",
   "target_group_name" "cdes",
   "targets" [{"target_id" "0987", "target_name" "john"}]})}

The intermediary data structure is a sequence of set indexed by group id and group name (like using group-by ). Ie

(-> "data.json"
    slurp
    json/read-str
    (get "data")
    (set/index ["target_group_id" "target_group_name"])
    vals)

;; =>
(#{{"target_group_id" "1234",
    "target_group_name" "abc",
    "targets" [{"target_id" "789", "target_name" "doe"}]}
   {"target_group_id" "1234",
    "target_group_name" "abc",
    "targets" [{"target_id" "456", "target_name" "john"}]}}
 #{{"target_group_id" "56789",
    "target_group_name" "cdes",
    "targets" [{"target_id" "0987", "target_name" "john"}]}})

The targets (which is a vector ) are then concat together with merge-vector :

(def merge-vector
  (partial apply
           merge-with
           (fn [& xs] (if (every? vector? xs) (apply concat xs) (last xs)))))