Encode bounding boxes for training

Question

I am trying to train a basic convolutional network to locate objects in an image. I turn the image into a numpy array which has a shape of 480, 640, 3 (heigh, width, rgb). The target is a 2D numpy array with points of boxes. It's shape is 18, 2 (max_boxes, (x, y)).

I feed this into a fairly simple convolutional model (if it is helpful I can post the source) which was good at classifying images.

When I train the model, it quickly gets to about 0.77 accuracy with a super high loss of ~35. After this, it stops learning and when I try testing an image it is super off.

Why is this? Am I encoding the target data incorrectly or is there a problem training the network? What is the "standard" way of doing this? How should I fix it?

Example Output

Raw output:
[[[0.02862035 0.02726501]
  [0.02820634 0.02730229]
  [0.02834299 0.02714084]
  [0.02819964 0.02701551]
  [0.02831725 0.02733742]
  [0.02820959 0.02748821]
  [0.02811676 0.02731069]
  [0.0284538  0.0269504 ]
  [0.02842114 0.02728475]
  [0.02858478 0.02691348]
  [0.02842172 0.0273287 ]
  [0.02865605 0.02686743]
  [0.02795038 0.02733723]
  [0.0287733  0.02693835]
  [0.02838135 0.02726109]
  [0.02785174 0.02761061]
  [0.03038537 0.02532353]
  [0.02849848 0.0269334 ]]]
Formatted:
[[[18.317022   17.44961   ]
  [13.539045   13.105101  ]
  [ 0.02834299  0.02714084]
  [ 0.02819964  0.02701551]
  [ 0.02831725  0.02733742]
  [ 0.02820959  0.02748821]
  [ 0.02811676  0.02731069]
  [ 0.0284538   0.0269504 ]
  [ 0.02842114  0.02728475]
  [ 0.02858478  0.02691348]
  [ 0.02842172  0.0273287 ]
  [ 0.02865605  0.02686743]
  [ 0.02795038  0.02733723]
  [ 0.0287733   0.02693835]
  [ 0.02838135  0.02726109]
  [ 0.02785174  0.02761061]
  [ 0.03038537  0.02532353]
  [ 0.02849848  0.0269334 ]]]
Correct:
[[44.0, 50.0], [52.0, 51.5], [39.5, 44.5], [42.5, 46.0], [46.5, 48.0], [46.5, 48.0], [53.5, 55.5], [33.5, 53.0], [51.0, 57.0], [45.5, 54.0]]

Formatting I multiply each element by hight and width to un-normalize it:

for e in output:
    e[0] *= 640
    e[1] *= 480

Encoding Image

def format_data(X):
    X = array(X)
    n, h, w, c = X.shape
    X = X.reshape(n, h, w, c).astype("float32")
    X = X / 255 # normalize between 0 and 1
    return X

Encoding Target Data

def format_output(y, max_box_count=0):
    for e in y:
        if len(e) > max_box_count:
            max_box_count = len(e)

    y_array = []
    for e in y:
        e_zeros = zeros((max_box_count, 2))
        e_zeros[:len(e)] = e
        y_array.append(e_zeros)

    for e in y_array:
        e[0] /= 640
        e[1] /= 480

    return array(y_array).astype('float32'), max_box_count