Array "view" on an array of different type possible?

Question

Suppose, you want to create 16 bit, 32 bit, 64 bit "views" to a byte array, which behave just as if they were respectively typed arrays themselves.

The first solution coming to mind is not working (as pointed out by specification):

(defvar *r* (make-array 64 :element-type '(unsigned-byte 8)
                           :initial-contents
                            (loop for i below 64 collecting i)))

(defvar *r16* (make-array 32 :element-type '(unsigned-byte 16)
                             :displaced-to *r*))

; Evaluation aborted on #<SIMPLE-ERROR "Array element type of
; :DISPLACED-TO array does not match specified element type
; {10053520F3}>.

But it would be nice if it worked, for it would allow for very efficient lookups both in terms of bytes or words.

So, this question is asking about equally performing but working alternative ideas.

I am open to accept ffi related "hacks", if they yield what I need. CLOS solutions would probably not be "fast" enough for me.

Answer 1

You can implement your own little data structure eg using integers and ldb , dpb etc. under the hood: I'm thinking of new accessors like byte-ref and word-ref , and maybe looping constructs like do-bytes and do-words . If you need bounds and checks thereof, you could add bounds in a wrapper structure. Of course, this will not have the benefit of controlled pre-allocation (the size of the stored integer depends on the first set bit).

Answer 2

Here are two approaches to this (the second is better).

The first approach stores arrays as bytes and recreates larger quantities. This is clearly not an ideal approach (see below for why). This example

• knows that you want to talk 64-bit words in terms of 8-bit bytes;
• would make byte access fast and word access a little slower;
• has no optimization at all.

In real life if you want this to be fast you probably need to wire in word and byte sizes so that declarations can work. Obviously you can and should write macros which will generate the appropriate code for you so if you change your mind you don't need to hunt down every assumption.

What this does is to store an array of 64-bit words as a 2-dimensional array of 8-bit bytes. This makes the index arithmetic slightly easier and probably does not make access any slower (or would not if there were declarations). But you might want to experiment just a 1-dimensional array and doing the index arithmetic yourself. There are then accessors for 8-bit bytes and 64-bit words: it would be easy to write the intermediate ones.

What this would need to be fast is at least declarations and unrolling the two loops would help I expect (again, if this code is written by macros that's easy):

(defun 64-bit-word-array-length (a)
  (array-dimension a 0))

(defun byte-ref (a n b)
  (aref a n b))

(defun (setf byte-ref) (v a n b)
  (setf (aref a n b) v))

(defun 64-bit-word-ref (a n)
  ;; Unroll this loop in real life, add types
  (let ((v 0))
    (dotimes (c 8 v)
      (setf (ldb (byte 8 (* c 8)) v)
            (aref a n c)))))

(defun (setf 64-bit-word-ref) (v a n)
  (dotimes (c 8 v)
    (setf (aref a n c)
          (ldb (byte 8 (* c 8)) v))))

(defun make-64-bit-word-array (n &key (initial-element 0 initial-element-p))
  (let ((a (make-array (list n bytes/word)
                       :element-type '(unsigned-byte 8))))
    (when initial-element-p
      (dotimes (i n)
        (setf (64-bit-word-ref a i) initial-element)))
    a))

Now:

> (make-64-bit-word-array 2 :initial-element 999)
#2A((231 3 0 0 0 0 0 0) (231 3 0 0 0 0 0 0))

> (64-bit-word-ref (make-64-bit-word-array 2 :initial-element 999) 1)
999

The first approach suffers from needing to make multiple (up to 4) array references per read/store, which is a bad idea even though those array references are close to each other in memory.

So the second approach stores things as 64-bit words and then masks and shifts things out of the words. This is a better approach because it does no more than one array reference per function call, and array references are the thing that will cost time. This example:

• again knows that you are talking about 64-bit quantities and so on;
• has some rudimentary type declarations (but I have not checked performance at all)
• has example functions for 8, 16 and 32-bit access.

(deftype array-index () 
  `(integer 0 (,array-dimension-limit)))

(deftype 64-bit-word-array ()
  '(array (unsigned-byte 64) (*)))

(defun make-64-bit-word-array (n &key (initial-element 0 initial-element-p))
  (declare (type array-index n)
           (type (unsigned-byte 64) initial-element))
  (if initial-element-p
      (make-array n :element-type '(unsigned-byte 64)
                  :initial-element initial-element)
    (make-array n :element-type '(unsigned-byte 64))))

(defun 64-bit-word-array-byte-length (a)
  (declare (type 64-bit-word-array a))
  (* (array-dimension a 0) 8))

(defun 64-bit-word-array-byte-ref (a n)
  (declare (type 64-bit-word-array a)
           (type array-index n))
  (multiple-value-bind (b o) (floor n 8)
    (ldb (byte 8 (* o 8)) (aref a b))))

(defun (setf 64-bit-word-array-byte-ref) (v a n)
  (declare (type (unsigned-byte 8) v)
           (type array-index n)
           (type 64-bit-word-array a))
  (multiple-value-bind (b o) (floor n 8)
    (setf (ldb (byte 8 (* o 8)) (aref a b)) v)))

(defun 64-bit-word-array-16-bit-word-ref (a n)
  (declare (type 64-bit-word-array a)
           (type array-index n))
  (multiple-value-bind (b o) (floor n 4)
    (ldb (byte 16 (* o 16)) (aref a b))))

(defun (setf 64-bit-word-array-16-bit-word-ref) (v a n)
  (declare (type (unsigned-byte 16) v)
           (type array-index n)
           (type 64-bit-word-array a))
  (multiple-value-bind (b o) (floor n 4)
    (setf (ldb (byte 16 (* o 16)) (aref a b)) v)))


(defun 64-bit-word-array-32-bit-word-ref (a n)
  (declare (type 64-bit-word-array a)
           (type array-index n))
  (multiple-value-bind (b o) (floor n 2)
    (ldb (byte 32 (* o 32)) (aref a b))))

(defun (setf 64-bit-word-array-32-bit-word-ref) (v a n)
  (declare (type (unsigned-byte 32) v)
           (type array-index n)
           (type 64-bit-word-array a))
  (multiple-value-bind (b o) (floor n 2)
    (setf (ldb (byte 32 (* o 32)) (aref a b)) v)))

I think something derived from the second version should be within epsilon as fast as anything: the array accesses are what is expensive, while masking & shifting operations on immediate integers should be extremely cheap.