Why is this Javascript array-access so slow?

Question

I have a 5000-element array (frames_to_boxes). Each element itself is an array of Objects of class Box:

class Box {
  constructor(x, y, width, h