with Clojure threading long running processes and comparing their returns

Question

I have two different function on two very large sets of data that need to be processed, in the end coming down to two Boolean values. those values need to then be anded toge

Answer 1

Your approach is fairly normal Clojure code. One other choice is to use promises or if you need more complex processing you could consider using something like lamina or if you feel like living on the bleeding edge you could try core.async:

(ns async-example.core
  (:require [clojure.core.async :refer :all])

(defn example []
  (let [a (chan)  ; a channel for a to report it's answer
        b (chan)  ; a channel for b to report it's answer
        output (chan)] ; a channel for the reporter to report back to the repl
    (go (<! (timeout (rand-int 1000))) ; process a
        (>! a (rand-nth [true false])))
    (go (<! (timeout (rand-int 1000))) ; process b
        (>! b (rand-nth [true false])))
    (go (>! output (and (<! a) (<! b)))) ; the reporter process
    output)) ;return the channe that the result will be sent to

async-example.core> (<!! (go (<! (example))))
false
async-example.core> (<!! (go (<! (example))))
false
async-example.core> (<!! (go (<! (example))))
true

Of course this is overkill for your situation, though it's tremendously fun anyway ;-)

Answer 2

(Promise-based approach at the top, core.async-based approach lower down. Both short-circuit on first falsey value.)

---

Here's a version making use of the fact that a single promise can be delivered multiple times (although only the first delivery will succeed in setting its value; subsequent deliveries simply return nil with no side effects).

(defn thread-and
  "Computes logical conjunction of return values of fs, each of which
  is called in a future. Short-circuits (cancelling the remaining
  futures) on first falsey value."
  [& fs]
  (let [done (promise)
        ret  (atom true)
        fps  (promise)]
    (deliver fps (doall (for [f fs]
                          (let [p (promise)]
                            [(future
                               (if-not (swap! ret #(and %1 %2) (f))
                                 (deliver done true))
                               (locking fps
                                 (deliver p true)
                                 (when (every? realized? (map peek @fps))
                                   (deliver done true))))
                             p]))))
    @done
    (doseq [[fut] @fps]
      (future-cancel fut))
    @ret))

Some tests:

(thread-and (constantly true) (constantly true))
;;= true

(thread-and (constantly true) (constantly false))
;;= false

(every? false?
        (repeatedly 100000
                    #(thread-and (constantly true) (constantly false))))
;;= true

;; prints :foo, but not :bar
(thread-and #(do (Thread/sleep 1000) (println :foo))
            #(do (Thread/sleep 3000) (println :bar)))

---

Putting Arthur's and A. Webb's ideas together, you could use core.async to and the results together while short-circuiting on the first falsey value returned:

(defn thread-and
  "Call each of the fs on a separate thread. Return logical
  conjunction of the results. Short-circuit (and cancel the calls
  to remaining fs) on first falsey value returned."
  [& fs]
  (let [futs-and-cs
        (doall (for [f fs]
                 (let [c (chan)]
                   [(future (>!! c (f))) c])))]
    (loop [futs-and-cs futs-and-cs]
      (if (seq futs-and-cs)
        (let [[result c] (alts!! (map peek futs-and-cs))]
          (if result
            (recur (remove #(identical? (peek %) c)
                           futs-and-cs))
            (do (doseq [fut (map first futs-and-cs)]
                  (future-cancel fut))
                false)))
        true))))

Test with (constantly false) and (constantly true):

(thread-and (constantly true) (constantly true))
;= true
(thread-and (constantly true) (constantly false))
;= false

;;; etc.

Also note that short-circuiting does indeed work:

;;; prints :foo before returning false
(thread-and #(do (Thread/sleep 3000) false)
            #(do (Thread/sleep 1000) (println :foo)))

;;; does not print :foo
(thread-and #(do (Thread/sleep 3000) false)
            #(do (Thread/sleep 7000) (println :foo)))