How to make a list in Racket that you can iterate in the same order you add elements?

Question

It's a pretty common programming use case to build up a list of things and then later to need to iterate over the list in the same order that you added to the list. A simple example may be recording compiler errors and then printing them for the user. You want the errors that were earlier in the source code, that you parsed first, to be the ones that are printed first to the screen.

But in Lisp/Scheme/Racket lists only have a head pointer, without a tail pointer. This means you can only cheaply add elements to the beginning, and you can only cheaply iterate the elements in reverse of the order that you added them. In learning Racket I've been seeing an awful lot of code that builds a list and then later iterates over (reverse the-list) . In practice for plenty of applications this should be fine, but it seems a little silly to have to add an extra N operations to your algorithm every time this comes up.

Is there a standard-idiom/most-common-solution for solving this problem? I can always roll my own list type with a tail pointer, or reimplement C++ std::vector on top of Racket's mutable vectors, but this seems common enough that there should be an already established best practice for what to do.

Answer 1

I cannot think of any way of doing it with plain Racket's lists.

There is an efficient alternative, take a look at Racket's Queues: https://docs.racket-lang.org/functional-data-structures/Queues.html

The Banker's Queue provides amortized O(1) time for enqueue, head and tail, which according to your use case, are the functions you need.

Update: There are several queues that work for your scenario, one another that is mentioned by @ben-rudgers in the comments is the Imperative Queue: https://docs.racket-lang.org/data/Imperative_Queues.html

That one also provides constant time for enqueue! and dequeue! .

Answer 2

I'm not entirely familiar with the std::vector in C++, but I believe Racket's growable vectors are fairly similar, and could be used for this. They can be imported from the data-lib package by adding (require data/gvector) : https://docs.racket-lang.org/data/gvector.html

Answer 3

You know what they say

The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.

By 'they' I mean Donald Knuth and Tony Hoare.

Comparing Queue Implementations

The code below runs #lang racket 's Bankers' Queue, Imperative Queue, and a plain old list that gets reversed. That's the sequence in which they are run. Each is run inside a timing function. A 'major garbage collection is done outside the timing function before each is run.

#lang racket
(require data/queue) ;; the imperative queue
(require (rename-in pfds/queue/bankers
                    (queue->list bq2list)))
(define (run)
  (writeln "bankers' queue for 10,000")
  (writeln "imperative queue for 1,000,000")
  (writeln "reversed list for 1,000,00")

  (collect-garbage 'major)

  ;; bankers' queue
  (time 
   (define q (queue))
   (for ((i (range 10000)))
     (enqueue i q))
   (bq2list q)
   'done)

  (collect-garbage 'major)

  ;; imperative queue
  (time 
   (define q (make-queue))
   (for ((i (range 1000000)))
     (enqueue! q i))
   (queue->list q))

  (collect-garbage 'major)

  ;; reversed list
  (time
   (define q
     (for/list ((i (range 1000000)))
       i))
   (reverse q))
  'done)

(run)

Typical Output

Welcome to DrRacket, version 6.6 [3m].
Language: racket, with debugging; memory limit: 1024 MB.
"bankers' queue for 10,000"
"imperative queue for 1,000,000"
"reversed list for 1,000,00"
cpu time: 1748 real time: 1752 gc time: 1000
cpu time: 664 real time: 664 gc time: 272
cpu time: 436 real time: 436 gc time: 180
'done
> (run)
"bankers' queue for 10,000"
"imperative queue for 1,000,000"
"reversed list for 1,000,00"
cpu time: 752 real time: 754 gc time: 8
cpu time: 660 real time: 661 gc time: 248
cpu time: 456 real time: 460 gc time: 192
'done
> (run)
"bankers' queue for 10,000"
"imperative queue for 1,000,000"
"reversed list for 1,000,00"
cpu time: 776 real time: 779 gc time: 40
cpu time: 692 real time: 693 gc time: 256
cpu time: 456 real time: 458 gc time: 184
'done
> 

Interpeting the Results

1. An idiomatic plain old reversed list tends to be fastest [at least in this naive implementation]. Not particularly surprising due to the reduced book-keeping and doing the expected (ie idiomatic) thing.

2. The imperative queue is about the same speed as a naive list. Racket's implementation uses struct 's. Since these are the basic building blocks for higher level types in the Racket ecosystem, it's not surprising for a data structure built upon them to be performant. If queue semantics are important, the queue abstraction is probably worth running a bit slower.

3. Sometimes I get impatient when programming, I've heard it's a virtue, if reducing the number of iterations of Bankers' Queue from a million to ten thousand due to my impatience constitutes a virtuous act, then maybe there's anecdotal evidence that it is. Anyway, Bankers' Queue ran about two orders of magnitude more slowly than either of the others. Of course, speed is just one measure of performance. Thread safety is another and the speed tradeoff may be worth it.