What are limitations of reader macros in Common Lisp

Question

I have my own Lisp interpreter in JavaScript that I work for some time now, and now I want to implement reader macros like in Common Lisp.

I've created Streams (almost working except for special symbols like ,@ , ` ' ) but it freezes the browser for a few seconds when it's loading the page with included scripts (lisp files that have 400 lines of code). This is because my Streams are based on substring function. If I first split tokens and then use TokenStream that iterate over tokens, it works fine.

So my question is this, is string streams really something that is in Common Lisp? Can you add reader macros that create whole new syntax like Python inside CL, this simplify to question can I implement """ macro (not sure if you can have 3 characters as reader macro) or other character that will implement template literal inside lisp for instance:

(let ((foo 10) (bar 20))
  {lorem ipsum ${baz} and ${foo}})

or

(let ((foo 10) (bar 20))
  ""lorem ipsum ${baz} and ${foo}"")

or

(let ((foo 10) (bar 20))
  :"lorem ipsum ${baz} and ${foo}")

would yield string

"lorem ipsum 10 and 20"

is something like this possible in Common Lisp and how hard would be to implement #\\{ or #\\: as reader macro?

The only way I can think of to have template literals in Lisp is something like this:

  (let ((foo 10) (bar 20))
    (tag "lorem ipsum ${baz} and ${foo}")))

where tag is macro that return strings with ${} as free variable. Can reader macro also return lisp code that is evaluated?

And another question can you implement reader macros like this:

(list :foo:bar)


(list foo:bar)

where : is reader macro and if it's before symbols it convert symbol to

foo.bar

and if it's inside it throw error. I'm asking this because with token based macros :foo:bar and foo:bar will be symbols and will not be processed by my reader macros.

and one more question can reader macro be put in one line and second line use it? This will definitely be only possible with string streams and from what I've tested not possible with interpreter written in JavaScript.

Answer 1

There are some limitations in the sense that it is pretty hard to, for instance, intervene in the interpretation of tokens in any way short of 'implement your own token interpreter from scratch'. But, well, you could if you wanted to do just that: the problem is that your code would need to deal with numbers & things as well as the existing code does and things like floating-point parsing are pretty fiddly to get right.

But the macro functions associated with macro characters get the stream that is being read, and they are free to read as much or as little of the stream as they like, and return any kind of object (or no object, which is how comments are implemented).

I would strongly recommend reading chapters 2 & 23 of the hyperspec, and then playing with an implementation. When you play with the implementation be aware that it is just astonishingly easy to completely wedge things by mucking around with the reader. At the minimum I would suggest code like this:

(defparameter *my-readtable* (copy-readtable nil))

;;; Now muck around with *my-readtable*, *not* the default readtable
;;;

(defun experimentally-read ((&key (stream *standard-input*)
                                  (readtable *my-raedtable*)))
  (let ((*readtable* readtable))
    (read stream)))

This gives you at least some chance to recover from catastrophe: if you can once abort experimentally-read then you are back in a position where *readtable* is something sensible.

Here is a fairly useless example which shows how much you can subvert the syntax with macro characters: a macro character definition which will cause ( ...) to be read as a string. This may not be fully debugged, and as I say I can see no use for it.

(defun mindless-parenthesized-string-reader (stream open-paren)
  ;; Cause parenthesized groups to be read as strings:
  ;; - (a b) -> "a b"
  ;; - (a (b c) d) -> "a (b c) d"
  ;; - (a \) b) -> "a ) b"
  ;; This serves no useful purpose that I can see.  Escapes (with #\))
  ;; and nested parens are dealt with.
  ;;
  ;; Real Programmers would write this with LOOP, but that was too
  ;; hard for me.  This may well not be completely right.
  (declare (ignore open-paren))
  (labels ((collect-it (escaping depth accum)
             (let ((char (read-char stream t nil t)))
               (if escaping
                   (collect-it nil depth (cons char accum))
                 (case char
                   ((#\\)
                    (collect-it t depth accum))
                   ((#\()
                    (collect-it nil (1+ depth) (cons char accum)))
                   ((#\))
                    (if (zerop depth)
                        (coerce (nreverse accum) 'string)
                      (collect-it nil (1- depth) (cons char accum))))
                   (otherwise
                      (collect-it nil depth (cons char accum))))))))
    (collect-it nil 0 '())))

(defvar *my-readtable* (copy-readtable nil))

(set-macro-character #\( #'mindless-parenthesized-string-reader
                     nil *my-readtable*)

(defun test-my-rt (&optional (stream *standard-input*))
  (let ((*readtable* *my-readtable*))
    (read stream)))

And now

> (test-my-rt)
12
12

> (test-my-rt)
x
x

> (test-my-rt)
(a string (with some parens) and \) and the end)
"a string (with some parens) and ) and the end"