Migrating from Gitlab to self hosted Gitblit

Published: Nov 30, 2020

Approx. time: 3 min read

Tags: Tools Linux Script Git Python

With GitLab being a powerful project and source code management tool it can be difficult to manage on your own, when self hosting, or has many features which are not needed by every user.

At the moment, I tend to host many services on my own and having hosted GitLab was fairly easy to do, but the system resources required by GitLab grew with every release and 90% of the features provided were not utilized.

After looking into some other solutions (e.g: Gitolite, webgit, gitosis, gitea, …), I found Gitblit which all features I need, with out beeing as resource hungry as GitLab.

For my Gitblit GO instance, I set up a small Alpine Linux VM, adapted the Running Gitblit behind Apache to work with Nginx and started on migrating all repositories to my Gitblit instance.

Gitblit provides multiple ways to administer repositories:

Web frontend
JSON RPC
- used by Gitblit Manager (GUI)
SSH
- provided by powertools plugin

The easiest way to automate repository creation turned out to be with the Gitblit Powertools plugin.

The script is intended to be executed within a Linux environment, it might be possible to adapt it to support Windows and to have a SSH identity configured for Gitblit SSH.

Use the requirements.txt to install modules needed by Python (pip install [--user] -r requirements.txt).

File content: requirements.txt

paramiko==2.7.2
python-gitlab==2.5.0
rich==9.2.0

Run the code python gl2gb.py after adjusting the settings to your needs.

File content: gl2gb.py

#!/usr/bin/env python

import time
import os
from subprocess import run, DEVNULL
import tarfile
import gitlab
import paramiko
from rich.progress import Progress

progress = Progress(transient=True)
progress.start()

ssh = paramiko.SSHClient()
ssh.load_system_host_keys()

# TODO: Change the next five lines to match your needs.
GITLAB_HOST = 'https://gitlab.com'
GITLAB_TOKEN = 'PersonalAccessToken_api'
GITBLIT_HOST = 'gitblit'  # Change: Gitblit host address
GITBLIT_PORT = 29419  # Change: Gitblit ssh port
GITBLIT_USER = 'user'

GITBLIT_DIR = '/'  # or 'project_name/' or f'~{GITBLIT_USER}' for private
GITBLIT_URI = f'ssh://{GITBLIT_USER}@{GITBLIT_HOST}:{GITBLIT_PORT}'

gl = gitlab.Gitlab(GITLAB_HOST, private_token=GITLAB_TOKEN)
ssh.connect(GITBLIT_HOST, port=GITBLIT_PORT)

progress.log('Fetching project list...')
for project in progress.track(gl.projects.list(all=True)):
    if os.path.exists(f'repos/{project.path}.done'):
        progress.log(f'{project.name}: Already exported and cloned')
        continue

    progress.log(f'{project.name}: Creating export...')
    export = project.exports.create()
    export.refresh()
    while export.export_status != 'finished':
        time.sleep(0.25)
        export.refresh()

    progress.log(f'{project.name}: Downloading export...')
    with open(f'exports/{project.path}.tgz', 'wb') as f:
        export.download(streamed=True, action=f.write)

    # progress.log(f'Preparing {project.name} for re-import')
    with tarfile.open(f'exports/{project.path}.tgz', 'r:gz') as t:
        progress.log(f'{project.name}: Extracting ./project.bundle...')
        try:
            t.extract('./project.bundle')
        except KeyError:
            continue

        progress.log(f'{project.name}: Cloning into bare repository...')
        ret = run(['/usr/bin/git', 'clone', '--mirror', './project.bundle',
                   f'repos/{project.path}'],
                  stderr=DEVNULL, stdout=DEVNULL)

        progress.log(f'{project.name}: Creating new repository...')
        ssh.exec_command(f'gb repos new {GITBLIT_DIR}/{project.path}')

        progress.log(f'{project.name}: Setting remote...')
        ret = run(['/usr/bin/git', '--git-dir', f'repos/{project.path}',
                   'remote', 'set-url', 'origin',
                   f'{GITBLIT_URI}/{GITBLIT_DIR}/{project.path}'],
                  stderr=DEVNULL, stdout=DEVNULL)

        progress.log(f'{project.name}: Pushing local repository to remote...')
        ret = run(['/usr/bin/git', '--git-dir', f'repos/{project.path}',
                   'push'],
                  stderr=DEVNULL, stdout=DEVNULL)

        os.remove('./project.bundle')
        os.rename(f'repos/{project.path}', f'repos/{project.path}.done')

progress.stop()

Depending on the size and amount of your repositories, the migration can take a few moments. The repositories are downloaded and stored in the subfolder ./exports/, the processed repository is stored in ./repos/. After importing a repository, the directory within ./repos/ is renamed by appending the suffix .done. Should the program run into an exception during processing, it will skip all repositories already completed and pick up at the last unsuccessful one. If you encounter a problem, try to fix it, make the code more robust or configure it correctly.

Update #1: A more recent version of the program can be downloaded/cloned here.