User not added as collaborator is able to push/pull to private GitHub repository

Question

I have two GitHub accounts (say personnel and work ). I configured my ~/.ssh/config file to work with multiple accounts . I created ssh key for work account and added the public key to my work GitHub account.

I created an organisation account and created a private repo. I can clone/push/pull to this private repo from my work account. Every time I push/pull I need to enter my password so I added ssh key running command ssh-add ~/.ssh/id_rsa_work and I can push/pull without entering password again and again. But the problem is after running ssh-add ~/.ssh/id_rsa_work somehow I can clone/push/pull this private repository from my personnel account which I think shouldn't be happening.

I am not sure what went wrong but it would be great if someone could guide me to right direction and tell me what I am doing wrong. Feel free to ask the relevant information if needed (Not a git/GitHub expert)

Config file details:

#Default GitHub (personnel account)
Host github.com
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa

#Work Github
Host github_work
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa_work

Command to add remote using pesonnel account:

git remote add origin git@github.com:Organisation/hello.git  

for work account:

git remote add origin git@github_work:Organisation/hello.git 

Answer 1

First, make sure both remote repositories are referenced with SSH URLs

cd /paath/to/repo
git remote -v

If one of them is using HTTPS, then it would use the credentials (username/password) stored/cached in a credential helper, while the other would use the right passphrase-protected private key.

Second, you can try and test the same approach with private keys not passphrase-protected, to avoid having to deal with ssh-agent, just for testing.