GitLab Runner Deployment Setup

This guide shows how to deploy OpportunityDAO using a GitLab Runner installed directly on your EC2 instance. This is the most secure method as it eliminates the need to store SSH keys in GitLab.

Why GitLab Runner on EC2?

Security Benefits:

✅ No SSH keys stored in GitLab variables
✅ Runner executes as local user with proper permissions
✅ No network transfer of credentials
✅ Direct access to Docker and system resources
✅ Faster deployments (no file transfer over SSH)

vs SSH Key Approach:

❌ SSH keys in GitLab variables (even if protected)
❌ Network latency for file transfers
❌ rsync complexity
❌ Key rotation overhead

Installation on EC2

Step 1: Install GitLab Runner

# On your EC2 instance
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

# Install
sudo apt-get install gitlab-runner

# Verify installation
gitlab-runner --version

Step 2: Get Registration Token

On your self-hosted GitLab:

Go to your project: OpportunityDAO/reprise
Navigate to: Settings → CI/CD → Runners
Expand "Specific runners" section
Copy the registration token

Or for group/instance runners (if you're admin):

Admin Area → CI/CD → Runners → Register an instance runner

Step 3: Register Runner

# Interactive registration
sudo gitlab-runner register

# You'll be prompted for:
# GitLab instance URL: https://your-gitlab-instance.com
# Registration token: (paste token from step 2)
# Description: ec2-opportunitydao-production
# Tags: ec2-production,docker
# Executor: shell

Non-interactive registration:

sudo gitlab-runner register \
  --non-interactive \
  --url "https://your-gitlab-instance.com" \
  --registration-token "YOUR_REGISTRATION_TOKEN" \
  --executor "shell" \
  --description "ec2-opportunitydao-production" \
  --tag-list "ec2-production,docker" \
  --run-untagged="false" \
  --locked="false"

Step 4: Configure Runner User

The runner needs access to Docker and deployment directory:

# Add gitlab-runner user to docker group
sudo usermod -aG docker gitlab-runner

# Give ownership of deployment directory
sudo chown -R gitlab-runner:gitlab-runner /var/www/opportunitydao

# Verify docker access
sudo -u gitlab-runner docker ps

# Test write access
sudo -u gitlab-runner touch /var/www/opportunitydao/test.txt

Step 5: Configure Runner Settings

Edit runner config:

sudo nano /etc/gitlab-runner/config.toml

Update the runner configuration:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "ec2-opportunitydao-production"
  url = "https://your-gitlab-instance.com"
  token = "GENERATED_TOKEN"
  executor = "shell"

  # Optional: Restrict to specific projects
  # allowed_images = []

  [runners.custom_build_dir]

  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

  # Environment variables available to all jobs
  environment = ["GIT_STRATEGY=fetch"]

Restart runner:

sudo gitlab-runner restart
sudo gitlab-runner verify

Updated GitLab CI/CD Pipeline

Replace .gitlab-ci.yml with this runner-based version:

stages:
  - build
  - deploy

variables:
  APP_PATH: /var/www/opportunitydao

# Build stage runs on EC2 runner
build:
  stage: build
  tags:
    - ec2-production
  script:
    - cd $APP_PATH
    - git fetch origin
    - git checkout main
    - git pull origin main
    - npm ci
    - npx prisma generate
    - npm run build
  artifacts:
    paths:
      - .next/
    expire_in: 1 hour
  only:
    - main

# Deploy stage runs on EC2 runner
deploy:
  stage: deploy
  tags:
    - ec2-production
  script:
    - cd $APP_PATH

    # Pull latest changes (already done in build, but just in case)
    - git pull origin main

    # Run database migrations
    - docker-compose exec -T app npx prisma migrate deploy || echo "Migrations will run after container start"

    # Rebuild and restart containers
    - docker-compose down
    - docker-compose up -d --build

    # Wait for containers to be healthy
    - sleep 10

    # Run migrations inside container
    - docker-compose exec -T app npx prisma migrate deploy

    # Verify deployment
    - docker-compose ps

    # Clean up old Docker images
    - docker system prune -f
  only:
    - main
  environment:
    name: production
    url: https://opportunitydao.app
  when: manual  # Change to 'on_success' for automatic deployment

# Rollback option
rollback:
  stage: deploy
  tags:
    - ec2-production
  script:
    - cd $APP_PATH
    - git checkout HEAD~1
    - docker-compose down
    - docker-compose up -d --build
    - docker-compose exec -T app npx prisma migrate deploy
  only:
    - main
  when: manual
  environment:
    name: production
    action: rollback

# Health check job
health-check:
  stage: deploy
  tags:
    - ec2-production
  script:
    - curl -f http://localhost:3000/ || exit 1
    - docker-compose ps | grep "Up"
  only:
    - main
  needs:
    - deploy

Simplified Alternative (No Docker Build Stage)

If you want even simpler deployment:

stages:
  - deploy

variables:
  APP_PATH: /var/www/opportunitydao

deploy:
  stage: deploy
  tags:
    - ec2-production
  script:
    - cd $APP_PATH
    - git pull origin main
    - docker-compose up -d --build
    - docker-compose exec -T app npx prisma migrate deploy
    - docker system prune -f
  only:
    - main
  when: manual

Environment Variables on EC2

Since runner executes locally, it reads from .env file:

# Ensure .env exists
cat /var/www/opportunitydao/.env

# Permissions should be restrictive
chmod 600 /var/www/opportunitydao/.env
chown gitlab-runner:gitlab-runner /var/www/opportunitydao/.env

No need to add to GitLab CI/CD Variables!

Testing the Setup

Test Runner Connection

# On EC2
sudo gitlab-runner verify

# Expected output:
# Verifying runner... is alive

Test Docker Access

sudo -u gitlab-runner docker ps

Test Deployment Manually

# Switch to runner user
sudo su - gitlab-runner

# Test deployment commands
cd /var/www/opportunitydao
git pull origin main
docker-compose ps

Trigger Pipeline from GitLab

Make a small change and push to main
Go to CI/CD → Pipelines
You should see pipeline running with tag ec2-production
Click "Deploy" button when ready

Monitoring

View Runner Logs

# Follow runner logs
sudo gitlab-runner --debug run

# Or view service logs
sudo journalctl -u gitlab-runner -f

View Job Output in GitLab

All job output appears in GitLab's job console, including:

Git operations
Docker build output
Migration results
Error messages

Troubleshooting

Runner Not Picking Up Jobs

Check runner is active:

sudo gitlab-runner verify
sudo gitlab-runner list

Check tags match:

Pipeline uses tags: [ec2-production]
Runner is registered with tag ec2-production

Check runner is not paused:

In GitLab: Settings → CI/CD → Runners
Ensure runner has green indicator

Permission Errors

# Docker permission denied
sudo usermod -aG docker gitlab-runner
sudo gitlab-runner restart

# File permission errors
sudo chown -R gitlab-runner:gitlab-runner /var/www/opportunitydao

Docker Compose Not Found

# Ensure docker-compose is in PATH for gitlab-runner user
sudo -u gitlab-runner which docker-compose

# If not found, install
sudo apt install docker-compose

# Or install via pip
sudo apt install python3-pip
sudo pip3 install docker-compose

Git Authentication Issues

If repository is private and runner can't pull:

# Option 1: Use deploy token
# In GitLab: Settings → Repository → Deploy Tokens
# Create token with 'read_repository' scope

# Add to git config on EC2
git config --global credential.helper store
git config --global user.name "GitLab Runner"
git config --global user.email "runner@example.com"

# First pull will ask for credentials (use deploy token)
cd /var/www/opportunitydao
git pull

# Or Option 2: Use SSH deploy key
# Generate key as gitlab-runner user
sudo -u gitlab-runner ssh-keygen -t ed25519 -C "gitlab-runner"
# Add public key to GitLab: Settings → Repository → Deploy Keys

Security Considerations

Runner Isolation

Since runner runs as gitlab-runner user:

✅ Cannot access other users' files
✅ Limited sudo access (only for docker if configured)
✅ No shell access for other users
❌ Can access deployment directory (by design)

Audit Logging

# Monitor what runner is doing
sudo journalctl -u gitlab-runner -f

# Check git operations
cd /var/www/opportunitydao
git log --all --oneline -20

# Check Docker operations
docker ps -a
docker logs opportunitydao-app

Lock Runner to Specific Project

In GitLab: Settings → CI/CD → Runners

Edit runner → Check "Lock to current projects"

This prevents other projects from using this runner.

Advantages Over SSH Key Method

Feature	SSH Key Method	GitLab Runner
SSH key storage	In GitLab variables	Not needed
File transfer	rsync over network	Git pull locally
Speed	Slower (network)	Faster (local)
Setup complexity	Medium	Medium
Security	Good (if done right)	Better (no key exposure)
Debugging	Harder (remote)	Easier (local logs)
Deployment access	Via SSH	Direct local

Migration from SSH Key Method

If you already have SSH key setup:

Install GitLab Runner (above steps)
Update .gitlab-ci.yml (use runner version)
Test deployment with manual trigger
Once working, remove SSH key from GitLab variables
Revoke SSH key from EC2 authorized_keys

Backup and Disaster Recovery

Backup Runner Configuration

sudo cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.backup

Re-register After Instance Recovery

If EC2 is rebuilt:

# Install runner
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt-get install gitlab-runner

# Re-register (see Step 3)
sudo gitlab-runner register

# Restore configuration if you have backup
sudo cp /path/to/config.toml.backup /etc/gitlab-runner/config.toml
sudo gitlab-runner restart

Performance Optimization

Parallel Jobs

If you want faster builds, enable concurrent jobs:

sudo nano /etc/gitlab-runner/config.toml

Change:

concurrent = 4  # Allow 4 jobs simultaneously

Build Caching

Cache node_modules between builds:

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - node_modules/
    - .next/cache/

Summary

GitLab Runner on EC2 is recommended because:

✅ No SSH keys in GitLab - Most secure approach
✅ Faster deployments - No network file transfer
✅ Simpler CI/CD - Direct execution, no SSH complexity
✅ Better logging - Full job output in GitLab UI
✅ Easier debugging - Local access to runner logs

Next steps:

Install GitLab Runner on EC2 (10 minutes)
Register runner with your GitLab instance
Update .gitlab-ci.yml to use runner
Test deployment
Remove SSH key from GitLab (if previously configured)

Why GitLab Runner on EC2?​

Installation on EC2​

Step 1: Install GitLab Runner​

Step 2: Get Registration Token​

Step 3: Register Runner​

Step 4: Configure Runner User​

Step 5: Configure Runner Settings​

Updated GitLab CI/CD Pipeline​

Simplified Alternative (No Docker Build Stage)​

Environment Variables on EC2​

Testing the Setup​

Test Runner Connection​

Test Docker Access​

Test Deployment Manually​

Trigger Pipeline from GitLab​

Monitoring​

View Runner Logs​

View Job Output in GitLab​

Troubleshooting​

Runner Not Picking Up Jobs​

Permission Errors​

Docker Compose Not Found​

Git Authentication Issues​

Security Considerations​

Runner Isolation​

Audit Logging​

Lock Runner to Specific Project​

Advantages Over SSH Key Method​

Migration from SSH Key Method​

Backup and Disaster Recovery​

Backup Runner Configuration​

Re-register After Instance Recovery​

Performance Optimization​

Parallel Jobs​

Build Caching​

Summary​