如何运行在同一个集合上运行的多个函数，但只遍历集合一次？ （clojure，包括示例）

Question

Bit of a weird one, but I essentially need to run two independent functions on a vector. They both map over the vector, and return a result. If I was to run them one after the other, it would mean going over the collection twice - how would I make it so that I only have to map over the collection once, and can perform both functions? The functions themselves can't be changed as they're used elsewhere independently. I might not be making much sense so an example might be better:

(defn add-one [xs] (map #(+ % 1) xs))
(defn minus-one [xs] (map #(- % 1) xs))
(def my-xs [1 2 3])
(def result {:added (add-one my-xs) :minused (minus-one my-xs)})

So, I'd essentially like to be able to calculate "result" but only have to go over "xs" once. I'm not sure it's at all possible to do this given the functions expect a collection but I thought I'd check in case there was some clojure magic I'm missing:D

EDIT - I could just use inc/dec in this example, but the point is that I need to leverage the functions which operate on a collection, as the actual functions are a lot more complex:)

Answer 1

There is no general solution. If you have two arbitrary functions which consume a sequence and operate on it in some unknown way to produce a result, you cannot avoid traversing the sequence twice.

For various kinds of constraints on the functions, combinations are possible. You've already seen in the comments how [(map f xs) (map g xs)] can be replaced with (apply map list (map (juxt fg) xs)) ; similar kinds of things can be done for consumers with a monoidal structure, like combining min and max, or if they are both just (fn [xs] (reduce fa xs)) .

Answer 2

The idea is to take the collection functions add-one and minus-one and make them applicable to one single element:

(defn singlify [fun] (fn [x] (first (fun [x]))))

;; now you can do:
((singlify #'add-one) 3)
;; => 4
;; so #'add-one became a function applicable for one element only
;; instead to an entire sequence/collection
;; - well actually we just wrap around the argument x a vector and apply
;; the sequence-function on this `[x]` and then take the result out of the
;; result list/vec/seq by a `first` - so this is not the most performant solution.
;; however, we can now use the functions and get the single-element versions
;; of them without having to modify the original functions.
;; And this solution is generalize-able to all collection functions.

Now using the comments' helpful hints, which tell that juxt makes it possible to apply two different functions on a sequence while traversing just once through it, we get

(map (juxt (singlify #'add-one) (singlify #'minus-one)) my-xs)
;; => ([2 0] [3 1] [4 2])

Using zipmap and the helpful lispy idiom (apply map #'<collector-function> <input-collection>) to transpose the result list, we can split them into a dict/map with the corresponding keywords upfront:

(zipmap [:added :minused] 
        (apply map vector 
               (map (juxt (singlify #'add-one) 
                          (singlify #'minus-one)) 
                    my-xs)))
;; => {:added [2 3 4], :minused [0 1 2]}

generalize as a function

We can generalize this as a function which takes a seq of keys, a seq of to-be-applied collection-functions and the to-be-once-only-traversed input seq/collection:

;; this helper function applies `juxt` 
;; on the `singlify`-ed versions of the collection-functions:
(defn juxtify [funcs] (apply #'juxt (map #(singlify %) funcs)))

;; so the generalized function is:
(defn traverse-once [keys seq-funcs sq]
  (zipmap keys (apply map vector (map (juxtify seq-funcs) sq))))

Using this function, the example case looks like this:

(traverse-once [:added :minused] [#'add-one #'minus-one] my-xs)
;; => {:added [2 3 4], :minused [0 1 2]}

We can now extend is as we want:

(traverse-once [:squared 
                :minused 
                :added] 
               [(fn [sq] (map #(* % %) sq)) 
                #'minus-one 
                #'add-one] 
               my-xs)
;; => {:squared [1 4 9], :minused [0 1 2], :added [2 3 4]}

Voila!

Answer 3

The functions themselves can't be changed as they're used elsewhere independently.

Refactor so that the operations collecting the values are independent of the sequence consumption, then reimplement the sequence consumption functions atop these refactorings (so the existing API is honoured, as that's a hard constraint), and then compose the independent operations as needed to avoid repeated sequence iteration.

In the case that you are consuming an upstream package from outside your organisation, open a dialog about them changing the API to allow for efficient operation in this way.

Any approaches along the lines of digging around the functions such as singlify are likely to break in subtle ways or confuse future maintainers, even assuming that the cost of boxing sequence items in a vector for reconsumption isn't a performance issue.

Answer 4

• Traverse the sequence once, using juxt to calculate all the results you want for each element - call it the base sequence.
• Return a map of sequences, each of which selects its own element from the elements of the base sequence. This is transposing on demand.

Thus:

(defn one-pass-maps [fn-map]
  (let [fn-vector (apply juxt (vals fn-map))]
    (fn [coll]
      (let [base (map fn-vector coll)]
        (zipmap (keys fn-map) (map (fn [n] (map #(% n) base)) (range)))))))

For example,

((one-pass-maps {:inc inc, :dec dec}) (range 10))
=> {:inc (1 2 3 4 5 6 7 8 9 10), :dec (-1 0 1 2 3 4 5 6 7 8)}

All the traversals are lazy. The base sequence is only realized as far as any transposed sequence travels. But if one sequence is realized up to - say - the fifth element, they all are.

Answer 5

A helpful general strategy is to express a function in an algebraic style and use various algebraic laws to optimize it. In this answer, I will focus on the case where the sequence processing can be expressed through reduce or transduce , something that maybe captures the notion of eagerly traversing a sequence, and use some "tupling laws". For brevity and clarity I will omit the handling of early termination (via reduced ), a functionality that isn't too hard to add.

First of all, I will use the modified versions of reduce and transduce shown below, which have some more desirable properties for the present case:

(defn reduce
  ([f coll] (reduce f (f) coll))
  ([f init coll] (clojure.core/reduce f init coll)))

(defn transduce [xf f & args]
  (let [f (xf f)]
    (f (apply reduce f args))))

I will also introduce polymorphic versions of juxt , map and map-indexed , which operate on vectors and maps:

(defprotocol JuxtMap
  (juxt* [fs])
  (map* [coll f])
  (map-indexed* [coll f]))

(extend-protocol JuxtMap

  clojure.lang.IPersistentVector
  (juxt* [fs]
    (fn [& xs]
      (into [] (map #(apply % xs)) fs)))
  (map* [coll f]
    (into [] (map f) coll))
  (map-indexed* [coll f]
    (into [] (map-indexed f) coll))

  clojure.lang.IPersistentMap
  (juxt* [fs]
    (fn [& xs]
      (into {} (map (juxt key (comp #(apply % xs) val))) fs)))
  (map* [coll f]
    (into {} (map (juxt key (comp f val))) coll))
  (map-indexed* [coll f]
    (into {} (map (juxt key (partial apply f))) coll)))

(letfn [(flip [f] #(f %2 %1))]
  (def map* (flip map*))
  (def map-indexed* (flip map-indexed*)))

Now we can create two functions, juxt*-rf and juxt*-xf , which satisfy the following "tupling laws", denoting composition by o and assuming that reduce , transduce and map* are curried in their first parameter and the first two don't receive an init :

• reduce o juxt*-rf = juxt* o map*(reduce)
• transduce o juxt*-xf = juxt* o map*(transduce)

Here they are:

(def juxt*-rf
  (comp
    juxt*
    (partial map-indexed*
      (fn [i f]
        (fn
          ([] (f))
          ([acc] (f (acc i)))
          ([acc x] (f (acc i) x)))))))

(def juxt*-xf
  (comp
    (partial comp juxt*-rf)
    juxt*))

Finally, let's see juxt*-xf in action:

(defn average [coll]
  (->> coll
       (transduce
         (juxt*-xf [identity (map (constantly 1))])
         +)
       (apply /)))

(average [1 2 3])
;result: 2

(defn map-many [fs coll]
  (transduce
    (juxt*-xf (map* map fs))
    conj
    coll))

(map-many {:inc inc, :dec dec} [1 2 3])
;result: {:inc [2 3 4], :dec [0 1 2]}

(transduce
  (juxt*-xf {:transp (juxt*-xf (map* map [first second]))
             :concat cat})
  conj
  [[1 11] [2 12] [3 13]])
;result: {:transp [[1 2 3] [11 12 13]], :concat [1 11 2 12 3 13]}