Docker `npm install` 与 TypeScript 中的 GitHub 私有 repo 依赖项

Question

We have a Node TypeScript project that we are trying to Dockerize. The project has a dependency on another GitHub private repo that is referenced via "git@github.com:{private-repo-name}" syntax in package.json. The dependency project is also a TS project. The main project installs and builds fine when running npm install (or npm ci etc) in the clone location and shell of any local dev PC OS (eg macOS, Ubuntu LTS, etc). However when trying to Dockerize the main project we are seeing npm build script errors that apparently make no sense. The dependency project has a "prepare" script, which gets run after the npm install that is called for the dependency project, after its repo has been checked-out. The "prepare" script is npm run build and the "build" script is tsc -p . && npm run validate tsc -p . && npm run validate .

So things look like this:

Main project's package.json:

{
    "name": "main-project",
    "private": true,
    "scripts": {
        ...
    },
    "dependencies": {
        ...
        "typescript": "^4.3.4",
        "private-repo": "git@github.com:my-private-repo.git#a-branch",
    },
    "devDependencies": {
        "@types/example": "^1.0.0",
        ...
    }
}

Dependency project package.json:

{
    "name": "dependency-project",
    "main": "dist/index.js",
    "types": "dist/index.d.ts",
    "scripts": {
        "build": "tsc -p . && npm run validate",
        "prepare": "npm run build",
        "validate": "node dist/validate.js"
    },
    "private": true,
    "dependencies": {
        ...
    },
    "devDependencies": {
        "@types/example": "1.0.0",
        ...
    }
}

The overall goal is the build the Docker image in layers, but we're stumbling at the first step of just getting the first layer ( npm install of the main project) to complete without errors.

The main project's Dockerfile looks like this:

FROM node:16-alpine
ARG SSH_KEY
RUN apk add git openssh-client
COPY package.json package-lock.json ./
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
RUN ssh-agent sh -c 'echo $SSH_KEY | base64 -d | ssh-add - ; npm ci'

This methodology of handing the private key into the layer build works fine (although it is the only one of various methods, including Docker Buildkit, that we were able to get working). The repo gets checked-out and the install apparently succeeds, then the "prepare" script (and hence npm build and tsc -p ) runs.

When we run docker build --build-arg SSH_KEY=$key . everything runs fine until the following errors:

#9 27.31 npm ERR! > my-private-repo@0.0.3 prepare
#9 27.31 npm ERR! > npm run build
#9 27.31 npm ERR!
#9 27.31 npm ERR!
#9 27.31 npm ERR! > my-private-repo@0.0.3 build
#9 27.31 npm ERR! > tsc -p . && npm run validate
#9 27.31 npm ERR!
#9 27.31 npm ERR! error TS2688: Cannot find type definition file for 'cacheable-request'.
#9 27.31 npm ERR!   The file is in the program because:
#9 27.31 npm ERR!     Entry point for implicit type library 'cacheable-request'
#9 27.31 npm ERR! error TS2688: Cannot find type definition file for 'chai'.
#9 27.31 npm ERR!   The file is in the program because:
#9 27.31 npm ERR!     Entry point for implicit type library 'chai'
#9 27.31 npm ERR! error TS2688: Cannot find type definition file for 'cors'.
#9 27.31 npm ERR!   The file is in the program because:
#9 27.31 npm ERR!     Entry point for implicit type library 'cors'
#9 27.31 npm ERR! error TS2688: Cannot find type definition file for 'faker'.
#9 27.31 npm ERR!   The file is in the program because:
#9 27.31 npm ERR!     Entry point for implicit type library 'faker'
#9 27.31 npm ERR! error TS2688: Cannot find type definition file for 'lodash'.
#9 27.31 npm ERR!   The file is in the program because:
#9 27.31 npm ERR!     Entry point for implicit type library 'lodash'
#9 27.31 npm ERR! error TS2688: Cannot find type definition file for 'mocha'.
#9 27.31 npm ERR!   The file is in the program because:
#9 27.31 npm ERR!     Entry point for implicit type library 'mocha'
#9 27.31 npm ERR! error TS2688: Cannot find type definition file for 'responselike'.
#9 27.31 npm ERR!   The file is in the program because:
#9 27.31 npm ERR!     Entry point for implicit type library 'responselike'

The confusing thing is that none of packages that those "error TS2688" messages refer to are dependencies of the dependency (private repo) project (they are in the package-lock.json of the main project. We have no idea how to interpret that.

The main troubleshooting steps we have tried include:

• Using identical Node and npm versions on local dev PC OS, and running npm install of the main project [works fine].
• Following this post, trying to look for the index.d.ts file in each of the packages it is complaining about (but those packages aren't even in the dependency project): https://github.com/microsoft/TypeScript/issues/27956
• Trying to understand and check this post, to no avail: Fixing TS2688: Cannot find type definition file in node_modules

It seems like there must be something in the user context of the shell invoked in the relevant Docker layer that is causing TS to use the wrong package.json (ie the wrong dependencies), because what we are doing in the Dockerfile is very simple, and it works everywhere except in the Docker layer.

Answer 1

[Update/answer to my own question] I wasn't able to fully explain this behaviour, since it made no sense. Instead I tried using an ubuntu:20.04 instead of the node:16-alpine image. I had to add node and a bunch of dependencies, but having done that I was able to npm ci just fine, without that typescript complaint, or any other complaint.

Here's the Dockerfile in case that helps anyone (noting this is just the base/build layer):

FROM ubuntu:20.04

RUN apt-get update \
  # had to install tzdata this first to get the noninteractive to work
  && DEBIAN_FRONTEND="noninteractive" apt-get install -y --no-install-recommends tzdata \
  && apt-get install -y curl gnupg build-essential libcurl4-openssl-dev openssh-client git\
  && curl -fsSL https://deb.nodesource.com/setup_16.x | bash - \
  #&& apt-get remove -y --purge cmdtest \
  && apt-get update \
  && apt-get install -y nodejs \
  && rm -rf /var/lib/apt/lists/* \
  && rm -rf /var/lib/apt/lists.d/* \
  && apt-get autoremove \
  && apt-get clean \
  && apt-get autoclean

RUN adduser --disabled-password --gecos "" --uid 1000 node

RUN chown -R node:node /home/node

USER node

# had to set mode to 0700 otherwise couldn't open .ssh director to write known_hosts file
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
RUN mkdir -p ~/app/node_modules && chown -R node:node /home/node/app

WORKDIR ~/app

COPY package.json package-lock.json ./

# had to allow uid=1000 access for this to work
RUN --mount=type=ssh,uid=1000 npm ci

COPY --chown=node:node . .

RUN npm run build

Hope that helps someone else, since it took my about a week of "spare" time to get to this point!