Advent of Code 2024 Day 6: Janet Streams Using Fibers/Coroutines

Remember when it was all the rage to write articles about functional programming (FP) in Javascript by explaining how you can chain .map, .filter and .reduce on lists? I hated that. I think that it conditions people into thinking that FP is all about list comprehensions. And when lists aren’t convenient anymore (more on this later), they then resort to imperative programming, thinking that FP isn’t suited to that kind of task.

In Advent of Code (AoC) there are plenty of examples where eager list comprehensions won’t get you very far. An innocent looking map().filter().map() can consume all your memory and make the garbage collector go crazy if you are creating and re-creating huge lists. A straight forward solution, as I mentioned earlier, is to fall back to more imperative patterns, such as the loop macro in Janet.

I was hoping that I could use Janet’s fibers/coroutines in combination with its stdlib map and filter function to work on streams rather than eagerly evaluated lists. But that was not the case:

(->> (gen-stuff input)
     (map foo)
     (filter bar))

In my testing on AoC day 6 (the first one where you walk around in a grid), the imperative version using loop used about 80MB, the version that kicks things off with a generator and then runs this through various list comprehenions immediately took up around 10GB of memory. Here’s a concrete example:

(defn gen-nums [] (coro (for i 0 10000000 (yield i)))

(comment
  # low mem usage
  (each v (gen-nums) (print v))

  # high mem usage
  (each v (map |(string/repeat (string $) 100) (gen-nums)) (print v)))

But it turns out that it is surprisingly straight forward to have your cake and eat it too. You can use easily define streaming versions of map and filter using the coro function (or macro?):

(defn map* [f ds] (coro (each v ds (yield (f v)))))

(defn filter* [f ds] (coro (each v ds (when (f v) (yield v)))))

I’ve used these for an upated implementation of day 6, which you can find here