How do I walk a complex map and remove keys based on a schema in Clojure?

Question

I'm fairly new to Clojure so apologies if this is a stupid question, but I have a map which is fairly complex (it contains nested maps, vectors etc.). Before I put it into my database, I essentially want to dissoc a bunch of things I don't need to save write speed (in this case it's a significant performance improvement). I'd like to filter out all the bits I don't need by making a schema of some sort which could be used to remove anything unnecessary. Here's an example of the sort of thing I'm trying to achieve (weird example, apologies:):

(def my-book
  {:intro {:one-liners [{:line "wowza" :rating 7} 
                        {:line "cool!" :rating 4}]}
           :how-many-lines 10
           :rubbish-one-liners [{:line "bongo" :rating 1}
                                {:line "foo"   :rating 2}]
            :other-info {:author {:name "me" :age 24}}})

(def my-schema
  [{:intro [{:one-liners {:values [:line :rating]}}
            {:values [:how-many-lines]}
            {:rubbish-one-liners {:values [:line]}}
            {:other-info {:author {:values [:name]}}}]}])

;;expected output when filtering via the schema:
(def my-output
  {:intro {:one-liners [{:line "wowza" :rating 7} 
                        {:line "cool!" :rating 4}]}
           :how-many-lines 10
           :rubbish-one-liners [{:line "bongo"}
                                {:line "foo"}]
            :other-info {:author {:name "me"}}})

I'm not really sure if that schema is at all the best way to go about it/structure, and once I do have a schema how to actually go about it given there seems to be a lot of different data structures involved - so I guess my question is, how would you advise I go about walking through whatever structure I have and removing keys based on a schema? :) thanks!

Answer 1

Clojure has select-keys , you just need to use recursion for nested structures. Try something like this:

(defn key-filter [obj schema]
  (cond 
    (vector? obj) (map #(select-keys % schema) obj)
    (map? obj) 
    (let [nw (select-keys obj (keys schema))
          res (map key-filter (vals nw) (vals schema))]
  (zipmap (keys nw) res))
    :else obj))

(def my-book {:intro {:one-liners [{:line "wowza", :rating 7} {:line "cool!", :rating 4}],
         :how-many-lines 10,
         :rubbish-one-liners [{:line "bongo", :rating 1} {:line "foo", :rating 2}],
         :other-info {:author {:name "me", :age 24}}}})

(def my-schema {:intro {:one-liners [:line], 
                        :how-many-lines [],
                        :rubbish-one-liners [:line], 
                        :other-info {:author {:name []}}}})

(key-filter my-book my-schema)

Answer 2

There is no comprehensive library that will accept a schema and coerce a data structure to match the schema by dropping extra keys, etc. Also, you have some inconsistencies between your schema & your book structure, such as which fields are vectors or not.

To start, I would use the Specter library and just hand-code up the desired transformations. An example:

(ns tst.demo.core
  (:use tupelo.core tupelo.test)
  (:require
    [com.rpl.specter :as sp]))

(dotest
  (let [book     {:intro              {:one-liners [{:line "wowza" :rating 7}
                                                    {:line "cool!" :rating 4}]}
                  :how-many-lines     10
                  :rubbish-one-liners [{:line "bongo" :rating 1}
                                       {:line "foo" :rating 2}]
                  :other-info         {:author {:name "me" :age 24}}}
        expected {:intro              {:one-liners [{:line "wowza" :rating 7}
                                                    {:line "cool!" :rating 4}]}
                  :how-many-lines     10
                  :rubbish-one-liners [{:line "bongo"}
                                       {:line "foo"}]
                  :other-info         {:author {:name "me"}}}]
    (is= expected
      (it-> book
        (sp/setval [:rubbish-one-liners sp/ALL :rating] sp/NONE it)
        (sp/setval [:other-info :author :age] sp/NONE it)))))

A live example of this code can be seen here .

Note that Specter defines a DSL that is, essentially, a separate programming language. Specter is very powerful, but it takes a little practice and trial & error to learn it properly (similar to learning Regular Expression syntax).

You could also usethe Tupelo Forest library for this task.