Skip to content

Commit 63ca6c1

Browse files
author
David Ferlay
authored
Merge pull request #239 from davidferlay/mirroringjob
Adding mirroring script
2 parents 553a200 + 65ac0c4 commit 63ca6c1

File tree

3 files changed

+195
-0
lines changed

3 files changed

+195
-0
lines changed
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Example of .gitlab-ci.yml jobs
2+
# Use with scripts/mirroring/mirror_current_branch.sh
3+
4+
stages:
5+
- mirror
6+
7+
.mirroring_template: &mirroring_template
8+
stage: mirror
9+
dependencies:
10+
- prepare:front # Where front dependencies are installed and assets are builded
11+
- prepare:back # Where back dependencies are installed
12+
allow_failure: true
13+
retry:
14+
max: 2
15+
16+
Mirror to repo XXX:
17+
<<: *mirroring_template
18+
script:
19+
- echo " - Start of CI script"
20+
- date; pwd; ls -lah;
21+
- mkdir -p ~/.ssh
22+
- ls -lah ~
23+
- echo "$REMOTE_REPO_PRIVATE_KEY" > ~/.ssh/id_rsa
24+
- chmod 0600 ~/.ssh/id_rsa
25+
- ls -lah ~/.ssh
26+
- apk add --no-cache openssh-client git rsync
27+
- ssh-keyscan -H "$REMOTE_REPO_IP" >> ~/.ssh/known_hosts
28+
- git version
29+
- rsync --version
30+
- export TARGET_GIT_REPO="${REMOTE_REPO_URL_1}" # Gitlab custom variable to update for each repo to mirror
31+
- echo -e "TARGET_GIT_REPO = $TARGET_GIT_REPO"
32+
- export REMOTE_REPO_TYPE="${REMOTE_REPO_TYPE}"
33+
- echo -e "REMOTE_REPO_TYPE = $REMOTE_REPO_TYPE"
34+
- chmod +x scripts/mirroring/mirror_current_branch.sh
35+
- ./scripts/mirroring/mirror_current_branch.sh
36+
- echo " - End of CI script"
37+
38+
Mirror to repo YYY: # Job can then be duplicated to clone to multiples repos (just use a different REMOTE_REPO_URL_N variable for each)
39+
...
40+

scripts/mirroring/README.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Mirroring script
2+
3+
## What
4+
5+
- This script mirrors the current branch from a repo to another one using a Gitlab CI job
6+
- You can also mirror to multiple repositories at the same time using multiple jobs in the same pipeline
7+
- Additionally, branches non-existing in local repo are deleted from remote repo(s)
8+
9+
## Why
10+
11+
Why not use the built-in mirroring functionality Gitlab-ci and other repository services usualy offer ?
12+
13+
- Because it doesn't work when your git repo is protected behind a basic authentication
14+
- Because mirroring to multiple repos at once is sometime a premium (paid) feature
15+
16+
## Setup
17+
18+
2 files are required :
19+
- `.gitlab-ci.yml`
20+
- `mirror_current_branch.sh`
21+
22+
1. Define a mirroring CI job like "Mirror to repo XXX" in .gitlab-ci.yml, as shown in .gitlab-ci.mirroring_example.yml
23+
- To include artefact dependencies in mirroring, this CI job should be positioned after all dependencies have been built and installed and use the [dependencies](https://docs.gitlab.com/ee/ci/yaml/#dependencies) key word
24+
1. In Gitlab UI, add the following custom CI/CD variables :
25+
- REMOTE_REPO_IP : IP or domain name of target git repos
26+
- REMOTE_REPO_PRIVATE_KEY : SSH private key matching public key added to git user
27+
- REMOTE_REPO_TYPE : Possible values : "PLATFORM.SH" only for now, or leave empty if appropriate
28+
- REMOTE_REPO_URL_1 : Git repo to which mirror current branch (you can have multiple ones)
29+
- GIT_USER_EMAIL : Email to be used by git user (used to commit)
30+
- GIT_USER_NAME : Name to be used by git user (used to commit)
31+
Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
#!/usr/bin/env sh
2+
# Use with Gitlab CI jobs from scripts/mirroring/.gitlab-ci.mirroring_example.yml
3+
4+
echo -e "\n- Start of mirroring script"
5+
6+
# Defining functions # For local use only, NOT FOR USE IN CI
7+
8+
CURRENT_BRANCH_FUNC()
9+
{
10+
git rev-parse --abbrev-ref HEAD
11+
}
12+
13+
14+
CURRENT_COMMIT_FUNC()
15+
{
16+
git log -1 --pretty=%B
17+
}
18+
19+
# Defining variables
20+
echo -e "- Defining variables...\n"
21+
22+
MIRRORING_DIR=$(pwd)/files_to_mirror
23+
echo -e "MIRRORING_DIR = $MIRRORING_DIR"
24+
25+
# TARGET_GIT_REPO_1=XXX@XXX.git # For local use only, NOT FOR USE IN CI
26+
# For CI use, var is moved to CI job itself, so that same script can be used to clone on multiple repos
27+
echo -e "TARGET_GIT_REPO = $TARGET_GIT_REPO"
28+
29+
# CURRENT_BRANCH=$(CURRENT_BRANCH_FUNC) # For local use only, NOT FOR USE IN CI
30+
CURRENT_BRANCH="$CI_COMMIT_REF_NAME" # For CI use only, using Gitlab predefined variable
31+
echo -e "CURRENT_BRANCH = $CURRENT_BRANCH"
32+
33+
# CURRENT_COMMIT=$(CURRENT_COMMIT_FUNC) # For local use only, NOT FOR USE IN CI
34+
CURRENT_COMMIT="$CI_COMMIT_MESSAGE" # For CI use only, using Gitlab predefined variable
35+
echo -e "CURRENT_COMMIT = $CURRENT_COMMIT"
36+
37+
# GIT_USER_EMAIL="XXX@XXX.com" # For local use only, NOT FOR USE IN CI
38+
echo -e "GIT_USER_EMAIL = $GIT_USER_EMAIL" # For CI use only, using Gitlab custom variable
39+
40+
# GIT_USER_NAME="XXX CI/CD" # For local use only, NOT FOR USE IN CI
41+
echo -e "GIT_USER_NAME = $GIT_USER_NAME" # For CI use only, using Gitlab custom variable
42+
43+
# Saving list of branches in GitLab before switching directory
44+
git fetch
45+
LOCAL_BRANCHES=$(git branch -r)
46+
47+
# Preparing mirrorring dir
48+
echo -e "- Preparing mirrorring dir...\n"
49+
mkdir "$MIRRORING_DIR"
50+
cd "$MIRRORING_DIR"
51+
52+
# Initialising external git repo
53+
echo -e "- Initialising external git repo...\n"
54+
git init && git config --local core.excludesfile false && git config --local core.fileMode true
55+
git remote add origin $TARGET_GIT_REPO
56+
git fetch origin
57+
git checkout -b $CURRENT_BRANCH
58+
git config --local user.email "$GIT_USER_EMAIL"
59+
git config --local user.name "$GIT_USER_NAME"
60+
61+
62+
# Cleaninng orphan branches in remote repo, compared to local repo
63+
echo -e "- Syncing git branches..."
64+
REMOTE_BRANCHES=$(git branch -r)
65+
REMOVED_BRANCHES=$(echo "$REMOTE_BRANCHES" | grep -v "$(echo "$LOCAL_BRANCHES" | sed ':a;N;$!ba;s/\n/\\|/g')" | sed 's/origin\///g;s/\n/ /g')
66+
67+
if [ ! -z "$REMOVED_BRANCHES" ]
68+
then
69+
echo -e "- Removing branches from remote git repo :"
70+
echo -e "$REMOVED_BRANCHES"
71+
if [ "$REMOTE_REPO_TYPE" -eq "PLATFORM.SH" ]; then
72+
echo -e "Use platform.sh CLI to remove environments along with git branches"
73+
curl -sS https://platform.sh/cli/installer | php
74+
"$HOME/"'.platformsh/bin/platform' environment:delete --delete-branch -y $REMOVED_BRANCHES
75+
else
76+
echo -e "Use git to remove branches"
77+
git branch -D $REMOVED_BRANCHES
78+
git push origin --delete $REMOVED_BRANCHES
79+
fi
80+
fi
81+
82+
83+
# Copying files to mirrorring dir
84+
echo -e "- Copying files to mirrorring dir...\n"
85+
rsync -av --progress ../. . --exclude .git/
86+
87+
88+
# Making sure everything needed will be included in commit
89+
echo -e "- Making sure everything needed will be included in commit...\n"
90+
mv .gitignore .gitig
91+
echo -e "- Deleting all .gitignore files except root one...\n"
92+
find . -name '.gitignore' -type f | wc -l
93+
find . -name '.gitignore' -type f -exec rm {} +
94+
mv .gitig .gitignore
95+
96+
echo -e "- Deleting all .git directories except root one...\n"
97+
mv .git .got
98+
find . -name '.git' -type d | wc -l
99+
find . -name '.git' -type d -exec rm -rf {} +
100+
mv .got .git
101+
102+
103+
# Preventing platform.sh error "Application name 'app' is not unique"
104+
if [ "$REMOTE_REPO_TYPE" -eq "PLATFORM.SH" ]; then
105+
echo -e "- Preventing platform.sh error "Application name 'app' is not unique"...\n"
106+
sed -i "s|name: 'app'|name: 'XXX'|g" ../.platform.app.yaml
107+
fi
108+
109+
110+
# Commiting to external repo
111+
echo -e "- Commiting to external repo...\n"
112+
git status
113+
git add -A
114+
git status
115+
git commit --quiet -m "$CURRENT_COMMIT"
116+
git push origin $CURRENT_BRANCH --quiet -f
117+
118+
119+
# Cleaning mirrorring dir
120+
echo -e "- Cleaning mirrorring dir...\n"
121+
cd ..
122+
rm -rf "$MIRRORING_DIR"
123+
124+
echo -e "- End of mirroring script"

0 commit comments

Comments
 (0)