Why does array have max size 2 ^ 32 -1 in JavaScript?

Question

Max native integer value in JavaScript is 2 ^ 31 - 1 = 2147483647

You could check it using the following code:

let x = 2147483647;
console.log(x << 1) // you will get -2 instead of 2,147,483,647 * 2 = 4,294,967,296 

It means that variable bigger than 2 ^ 31 - 1 = 2147483647 , always has floating point type.

But according to the ECMAScript max array size is 2 ^ 32 - 1 = 4294967295 .

So if I write something like

let size = 3000000000;       // float type
let array = new Array(size); // passing float variable to constructor
                             // that accepts integer only
 

It means that firstly size will have float type, and then we pass this float size as array size argument. But array size obviously must be integer.

And it does not have any sense at all.

So the question is: Why does ECMAScript says that max array size is 2 ^ 32 - 1 and not 2 ^ 31 - 1 ?

Answer 1

Tl;dr:

It casts from double to uint32, that's why 2^32-1 works and is usable from within js without any issues.

---

It is in fact possible to create a holey array with exactly up to 2^32 -1 numerical slots. anything beyond will be converted to a string hash index and will not iterate the array length property beyond 2^32 -1.

one way would be this:

# node
Welcome to Node.js v16.15.0.
Type ".help" for more information.
> const a = new Array(2**31-1)
undefined
> a
[ <2147483647 empty items> ]
> a.push("foo", "bar", "asdf")
2147483650
> a
[ <2147483647 empty items>, 'foo', 'bar', 'asdf' ]
> a.length
2147483650
> a.length = 2**32 -2
4294967294
> a.push("magic", "beer", "such overflow")
Uncaught RangeError: Invalid array length
    at Array.push (<anonymous>)
> a[2**32]
'such overflow'
> a[2**32 -20] = "indexable beyond 2^31-1"
'indexable beyond 2^31-1'
> a
[
  <2147483647 empty items>,
  'foo',
  'bar',
  'asdf',
  <2147483626 empty items>,
  'indexable beyond 2^31-1',
  <17 empty items>,
  'magic',
  '4294967295': 'beer',
  '4294967296': 'such overflow'
]
> a.length
4294967295

another example without initial length:

# node
Welcome to Node.js v16.15.0.
Type ".help" for more information.
> const a = []
undefined
> a[2**32 -10] = "magic"
'magic'
> a[2**33] = "overflow"
'overflow'
> a
[ <4294967286 empty items>, 'magic', '8589934592': 'overflow' ]
>
> Object.keys(a)
[ '4294967286', '8589934592' ]
> a[5] = "foo"
'foo'
> Object.keys(a)
[ '5', '4294967286', '8589934592' ]
> a
[
  <5 empty items>,
  'foo',
  <4294967280 empty items>,
  'magic',
  '8589934592': 'overflow'
]
> a.length
4294967287

but if you fill it, you will run out of heap memory:

# node
Welcome to Node.js v16.15.0.
Type ".help" for more information.
> const a = [];
undefined
> a.length = 2**32 -1;
4294967295
> a.fill(0)

<--- Last few GCs --->
n [1537688:0x5818540]    33577 ms: Mark-sweep 238.5 (256.1) -> 238.3 (272.1) MB, 202.6 / 0.0 ms  (+ 0.4 ms in 42 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 833 ms) (average mu = 0.971, current mu = 0.868) allocation f[1537688:0x5818540]    52935 ms: Mark-sweep 1796.1 (1833.3) -> 1796.1 (1833.3) MB, 1996.9 / 0.0 ms  (+ 3.3 ms in 235 steps since start of marking, biggest step 0.1 ms, walltime since start of marking 9982 ms) (average mu = 0.911, current mu = 0.897) alloc

<--- JS stacktrace --->

FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory
 1: 0xb09c10 node::Abort() [node]
 2: 0xa1c193 node::FatalError(char const*, char const*) [node]
 3: 0xcf8dbe v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xcf9137 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xeb09d5  [node]
 6: 0x10dcbdd  [node]
 7: 0x10dcdb3 v8::internal::Handle<v8::internal::NumberDictionary> v8::internal::HashTable<v8::internal::NumberDictionary, v8::internal::NumberDictionaryShape>::EnsureCapacity<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NumberDictionary>, int, v8::internal::AllocationType) [node]
 8: 0x10dd3f4 v8::internal::Handle<v8::internal::NumberDictionary> v8::internal::Dictionary<v8::internal::NumberDictionary, v8::internal::NumberDictionaryShape>::Add<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NumberDictionary>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [node]
 9: 0x1005348  [node]
10: 0x108b615 v8::internal::JSObject::AddDataElement(v8::internal::Handle<v8::internal::JSObject>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes) [node]
11: 0x10cf36e v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::StoreOrigin) [node]
12: 0x10d3903 v8::internal::Object::SetProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::StoreOrigin, v8::Maybe<v8::internal::ShouldThrow>) [node]
13: 0xd5972d v8::internal::Builtin_ArrayPrototypeFill(int, unsigned long*, v8::internal::Isolate*) [node]
14: 0x15f2179  [node]
[1]    1537688 abort (core dumped)  node
node  26,77s user 21,04s system 39% cpu 2:02,24 total

You can also initialize an Array with up to 2^32 -1 items:

nodejs:

> new Array(2**32-1)
[ <4294967295 empty items> ]
> Array(2**32-1)
[ <4294967295 empty items> ]
> 3000000000
3000000000
> Array(3000000000)
[ <3000000000 empty items> ]

firefox:

Array(2**32-1)
(4294967295) [empty × 4294967295]
new Array(2**32-1)
(4294967295) [empty × 4294967295]

Your question was quite a bit misleading since you made it appear, as if anything beyond 2^31 was simply impossible to access while the spec explicitely states to convert the argument (IF it is of Number type) to uint32, thus allowing you to use a float to reach beyond 2^31 for indexing.

fact: https://262.ecma-international.org/12.0/#sec-array

So in essence, the engine converts your Number to uint32. since IEE 754 (JavaScripts floating point type) can carry numbers beyond 2^32-1, it will simply convert it.

It does a static cast to uint32 as seen in v8: https://github.com/v8/v8/blob/b5283a2e5bc31b254a73f2a0e59841a8654de092/src/builtins/builtins-array.cc#L176-L178

// Set "length" property, has "fast-path" for JSArrays.
// Returns Nothing if something went wrong.
V8_WARN_UNUSED_RESULT MaybeHandle<Object> SetLengthProperty(
    Isolate* isolate, Handle<JSReceiver> receiver, double length) {
  if (receiver->IsJSArray()) {
    Handle<JSArray> array = Handle<JSArray>::cast(receiver);
    if (!JSArray::HasReadOnlyLength(array)) {
      DCHECK_LE(length, kMaxUInt32);
      MAYBE_RETURN_NULL(
          JSArray::SetLength(array, static_cast<uint32_t>(length)));
      return receiver;
    }
  }

  return Object::SetProperty(
      isolate, receiver, isolate->factory()->length_string(),
      isolate->factory()->NewNumber(length), StoreOrigin::kMaybeKeyed,
      Just(ShouldThrow::kThrowOnError));
}

Same with Spidermonkey:
https://searchfox.org/mozilla-central/rev/32ca4fc265150e7d3d7aa6c6abea088768cf024b/js/src/builtin/Array.cpp#672

// Step 3.
if (!ToUint32(cx, desc.value(), &newLen)) {
  return false;
}

So the answer to your question is:

The Array uses uint32 internally in Spec (even in 5.0) and both SpiderMonkey and V8.

It casts from double to uint32, that's why 2^32-1 works and is usable from within js without any issues.

Answer 2

Why does array have max size 2^32 - 1 in JavaScript?

Quite simply: because it's specified that way. The specification authors could have chosen any maximum value.

2^32 - 1 is the biggest value that can be represented in an unsigned 32-bit integer. This is a reasonable choice because it limits required memory for storing an array length to 32 bits while maximizing the lengths that can be stored in these 32 bits.

A JavaScript engine could therefore theoretically use an unsigned 32-bit integer to store any array length (and, by extension, any valid array index). I don't know whether any JavaScript engines actually do it that way. (V8 doesn't; it uses either a signed 31-bit(!) integer or a IEE 754 float64, because... reasons too long to explain here!)

Max native integer value in JavaScript is 2 ^ 31 - 1 = 2147483647 . Variable bigger than 2 ^ 31 - 1 = 2147483647 , always has floating point type.

No, all Numbers in JavaScript are specified to be IEE 754 64-bit floating point values, there is no such thing as a "max native integer value" in JS. Some operations (namely, the bitwise operations like x << 1 ) convert their inputs to signed 32-bit integers. FWIW, unsigned right-shift converts its input to unsigned 32-bit integers and interprets its output as that too, hence eg (-1 >>> 0) === 4294967295 . This observable behavior doesn't guarantee anything about how the engine chooses to represent these values on the machine level. There is no way to tell whether 1 or 2147483647 is being stored as an integer or a float inside the engine. The JavaScript spec only guarantees that it'll behave like a float.
And, to tie this back to the question, what bitwise operations do is totally unrelated to the maximum array length.