How to synchronize Git services

Keywords: Linux GitLab git ssh Python

I haven't written a blog for a long time. Although I haven't written a few articles, I'm still active in the Internet industry... Needless to say, it's useless. Let's share how to synchronize Git services.
Git service we usually use gitlab rake tool to do regular backup. When there is a problem, we can use backup recovery. Is there a need for another environment to deploy a set of GIT service, and need to synchronize with the previous set of GIT service? Even if we do cold backup, we can save recovery time and achieve the purpose of regular recovery exercise.
At present, we meet such a demand in another environment. Of course, we can study and learn the Git API to realize it. However, because the current demand does not require high synchronization and real-time performance, I plan to use a direct, simple and crude way to realize it. In fact, I will take Git backup to another environment for automatic recovery, and we will give the expect tool to solve the interaction problem.
Here are Python and Shell scripts to share as reference learning. Synchronization time can adjust the timing backup time and recovery time. The content is similar and the ideas are basically the same:

#!/usr/bin/python
# -*- coding: utf-8 -*-

import os
import sys
import pexpect
import paramiko

# IP address and user name of backup server
git_src_host = "xxx.xxx.xxx.xxx"
git_src_user = "xxx"
git_src_port = "22"
# Address changes according to the environment
private_key  = paramiko.RSAKey.from_private_key_file('/xxx/xxx/xxx/.ssh/id_rsa')

git_bak_dir  = "/var/opt/gitlab/backups"

# Copy backup to xxx.xxx.xxx.xxx

try:
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect(
        hostname=git_src_host,
        username=git_src_user,
        port=git_src_port,
        password=private_key)

    cmd=("find " + git_bak_dir + " -type f -ctime -1")
    stdin, stdout, stderr = ssh.exec_command(cmd)

    for item in stdout.readlines():
        chown_cmd="chown git:git " + git_bak_dir + "/*;chmod 700 " + git_bak_dir
        rsync_cmd=('rsync -avzP '
                    + git_src_user + '@'
                    + git_src_host + ':'
                    + item.strip('\n') + ' '
                    + git_bak_dir)

        #print(rsync_cmd)
        os.system(rsync_cmd)
        os.system(chown_cmd)

    ssh.close()
except Exception, e:
    print e

# Restore backup

ls_cmd="ls -t " + git_bak_dir + " | \
        head -n 1 | \
        sed -r 's/^([0-9]+_[0-9]+_[0-9]+_[0-9]+_.*)_gitlab_backup.tar/\\1/g'"

rs=os.popen(ls_cmd)
restore_file=rs.read()
# print(restore_file)

os.system("gitlab-ctl stop unicorn;gitlab-ctl stop sidekiq")

expectations=["Do you want to continue (yes/no)?","Do you want to continue (yes/no)?",]
child = pexpect.spawn('gitlab-rake gitlab:backup:restore BACKUP=' + restore_file,timeout=1800)
child.logfile = sys.stdout
while True:
    try:
        i = child.expect(expectations)
        if i == 0:
            child.sendline('yes')
        elif i == 1:
            child.sendline('yes')
    except pexpect.EOF:
        print('Exiting fexpect for EOF.')
        break

os.system("gitlab-ctl restart")
#!/bin/bash
#

git_src_host="xxx.xxx.xxx.xxx"
git_src_user="xxx"

git_bak_dir="/var/opt/gitlab/backups"
git_bak_file=`ssh ${git_src_user}@${git_src_host} "find ${git_bak_dir} -type f -ctime -1"`

# Copy backup to xxx.xxx.xxx.xxx

for bakfiles in $git_bak_file ; do

    echo "------------------------------------"
    rsync -avzP ${git_src_user}@${git_src_host}:${bakfiles} ${git_bak_dir}
    echo $bakfiles
    echo "------------------------------------"

done

chown git:root $git_bak_dir
chmod 700 $git_bak_dir

# Restore backup

restore_file=`ls -t $git_bak_dir | head -n 1 | sed -r 's/^([0-9]+_[0-9]+_[0-9]+_[0-9]+_.*)_gitlab_backup.tar/\1/g'`

gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq

/bin/expect <<EOF
set time 30
spawn gitlab-rake gitlab:backup:restore BACKUP=$restore_file
expect "Do you want to continue (yes/no)?" { 
    send "yes\r";
}
expect "Do you want to continue (yes/no)?" { 
    send "yes\r";
}
expect "Do you want to continue (yes/no)?" { 
    send "yes\r";
}
expect "Do you want to continue (yes/no)?" { 
    send "yes\r";
}
expect "Do you want to continue (yes/no)?" { 
    send "yes\r";
}
expect eof
EOF

#gitlab-ctl reconfigure
gitlab-ctl restart

Posted by oliverw92 on Mon, 13 Apr 2020 07:56:26 -0700