git workflow: public and private repositories where folder contents diverge but won't be merged

Question

I'm looking for a practical way to work on a private and public repository where the private repository's contents in a specific folder may diverge from that in the public branch. This folder must exist in both branches, and its content tracked in both repositories but must never be merged.

This is the simplified setup of the repository:

/folderA  <-- public content
/folderB  <-- mixed public & private content
/folderB/private.files <-- this file is different in public & private repos
/folderB/newPrivateFolder <-- private repo may add more private-only folders here

These are my seemingly simply requirements, yet I haven't found a good working solution:

• private.files must exist in the public repo with default content for the project to work
• changes to private.files in public repo should not be merged with private repo
• private.files changes must be tracked in private repo as it's needed by team members
• changes to private.files in the private repo must never be merged with public repo
• additional files/folders added to folderB in private repo must never be added to public repo
• private repo commits should be isolated, the private repo's history should not be merged with the public repo

The private repo is a duplicate of the public repo.

What I have tried:

• include public repo as submodule or subtree
  • can't "override" private folder's contents because changes go directly to public repo when including it as submodule or subtree
  • plus: a bit pointless to include the entire project as subtree since I want two diverging versions of the same repository
• sparse-checkout
  • merging still merges all files/folders, even those not checked out in the local branch
• attributes merge filter
  • only applies when merging content, but will still allow adding/removing files

What I haven't tried yet:

• two completely disconnected repositories (not a duplicate), and somehow merging them while ensuring folderB contents are "clean"
• local merge repo where all private repo branch changes in a specific folder are undone (how?) before merging with the public repo branch (well, actually I have tried this and it seemed to work with sparse-checkout - but once pushed all private changes went straight to the public repo)

Anything else I could try?

Perhaps there's already a solution to this issue - but I've been looking at dozens of SO and dozens more articles on the web, yet there's seemingly no solution to this diverging folder contents issue.

Answer 1

in similar situations (deployment to differing installations with small local but permanent modification) i have used a master branch that includes nothing or dummy files in the folders in question. then for every local configuration (that would be private and public in your case) i created a branch off master and did the relevant changes there.

development takes places on master and occasionally i switch to the other branches (private / public) and merge from master to these branches, never the other way round.

the different configurations (private / public) only share code from master, not the local modifications, i can keep them for an indefinite time, and i still have one and only one central place for changes/development (the master branch).

Answer 2

It seems so simple now that I figured it out. In essence you just delete folderB on merge and replace it with its public contents, then commit.

Basic setup requires:

• public repo filled with initial "default" content in folderB
• duplicate public repo into private repo using this guide (apply only the first code box)
• both public and private repo connected in a local repo as branches with their respective origins - this local repo is where merges happen
• do your work in the private branch

Now when the private repo has made modifications & additions to both folderA and folderB and you want to merge all changes except those in folderB back to the public repo, you merge as follows (I suggest an additional merge branch to safely experiment with it):

• merge private branch into public (or merge) branch - do not commit yet
• run the following commands in the repo (this assumes a merge-branch):
  • git checkout merge-branch
  • git merge -s recursive -Xtheirs --no-edit --no-commit private-branch
  • rm -rf folderB
  • git reset <tree-ish> -- folderB/
  • git checkout --theirs <tree-ish> -- folderB/
• now commit as usual

What this does:

• checkout changes to the merge branch (recommended to use one, and it should be up to date with origin)
• merge accepts all changes from the private branch (merge "theirs"), no edit prevents merge from wanting input from us (commit message) and no commit prevents the commit because we want to clean the working dir before committing
• rm removes the entire folderB and its contents (including subfolders) from the working directory. This prevents any added files/folders from being added to the branch's index on commit because, well, you just removed them.
• reset uses a "tree-ish" (ie branch or in my case I use a commit SHA) to reset the index and working dir of the folderB
• checkout then pulls the current state of folderB to restore it to its HEAD state as it is in the public repo. This means the entire contents of folderB remain unchanged from the public repo's perspective.

I'm not sure if both reset AND checkout are necessary but I'm doing it anyway. Feel free to experiment.

As a consequence, the commit history of any changes to folderB in the private branch will not be added to the public repo if you clean out the working directory after the merge and before committing.

I tried experimenting with a post-merge hook to do the job, but it doesn't seem to run if you don't commit. Besides it is easier to just put all commands in one script.

Merging public repo changes back to the private repo works normally, no additional steps are required.

Answer 3

As .git does NOT support .ignore (via remote or branch differences); here is my simple work-around in NPM:

"scripts": {
    "deploy:public": "git add . && git commit -am 'master src committed'",
    "deploy:private": "git add . && git rm -r --cached src/* && git commit -am 'master src omitted.'" }