semaphore/README.md

# Semaphore UI Runner Setup Guide

This guide explains how to set up and manage Semaphore UI runners for parallel task execution.

## Overview

Semaphore UI supports remote runners to execute tasks in parallel across multiple machines. This setup allows you to:
- Run multiple tasks simultaneously
- Distribute workload across different servers
- Scale your automation infrastructure

## Current Configuration

- **Semaphore UI URL**: https://ansible.phx-erp.de
- **Max Parallel Tasks**: 3
- **Registration Token**: `Z*!!MPPi68C3Vganq2BbV*PBNTxxtaLCfk9M.XG@z!h`
- **Database**: PostgreSQL (ansible-postgres container)

## Prerequisites

- Docker and Docker Compose installed
- Access to the Semaphore UI web interface
- Network connectivity between runner and Semaphore server

## Setting Up a New Runner

### Method 1: Using the Existing Docker Container (Recommended)

If you want to add another runner using the same container setup:

#### Step 1: Access the Semaphore Container
```bash
# Navigate to your project directory
cd /opt/phx

# Access the Semaphore container
docker exec -it ansible-semaphore /bin/bash
```

#### Step 2: Register the Runner
```bash
# Inside the container, register a new runner
echo "Z*!!MPPi68C3Vganq2BbV*PBNTxxtaLCfk9M.XG@z!h" | semaphore runner register --config /etc/semaphore/config.json --stdin-registration-token
```

#### Step 3: Start the Runner
```bash
# Start the runner in the background
semaphore runner start --config /etc/semaphore/config.json &
```

#### Step 4: Verify Runner Status
```bash
# Check if the runner is running
ps aux | grep "semaphore runner"
```

### Method 2: Setting Up a Remote Runner on Another Machine

#### Step 1: Install Semaphore Runner
On the remote machine where you want to run tasks:

```bash
# Download the latest Semaphore binary
wget https://github.com/semaphoreui/semaphore/releases/latest/download/semaphore_linux_amd64.tar.gz

# Extract the binary
tar -xzf semaphore_linux_amd64.tar.gz

# Make it executable
chmod +x semaphore

# Move to a system path
sudo mv semaphore /usr/local/bin/
```

#### Step 2: Create Runner Configuration
Create a configuration file for the remote runner:

```bash
# Create runner config directory
sudo mkdir -p /etc/semaphore

# Create runner configuration file
sudo tee /etc/semaphore/config.runner.json > /dev/null <<EOF
{
  "runner": {
    "registration_token": "Z*!!MPPi68C3Vganq2BbV*PBNTxxtaLCfk9M.XG@z!h",
    "api_url": "https://ansible.phx-erp.de"
  }
}
EOF
```

#### Step 3: Register the Remote Runner
```bash
# Register the runner with the Semaphore server
echo "Z*!!MPPi68C3Vganq2BbV*PBNTxxtaLCfk9M.XG@z!h" | semaphore runner register --config /etc/semaphore/config.runner.json --stdin-registration-token
```

#### Step 4: Start the Remote Runner
```bash
# Start the runner as a service
semaphore runner start --config /etc/semaphore/config.runner.json
```

### Method 3: Using Docker on Remote Machine

#### Step 1: Create Docker Compose for Runner
On the remote machine, create a `docker-compose.runner.yml`:

```yaml
version: '3.8'
services:
  semaphore-runner:
    image: semaphoreui/semaphore:latest
    container_name: semaphore-runner
    restart: unless-stopped
    environment:
      - SEMAPHORE_RUNNER_REGISTRATION_TOKEN=Z*!!MPPi68C3Vganq2BbV*PBNTxxtaLCfk9M.XG@z!h
      - SEMAPHORE_RUNNER_API_URL=https://ansible.phx-erp.de
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock  # If you need Docker-in-Docker
      - ./runner-config:/etc/semaphore
    command: >
      sh -c "
        echo 'Z*!!MPPi68C3Vganq2BbV*PBNTxxtaLCfk9M.XG@z!h' | 
        semaphore runner register --stdin-registration-token &&
        semaphore runner start
      "
```

#### Step 2: Start the Runner Container
```bash
# Start the runner container
docker-compose -f docker-compose.runner.yml up -d
```

## Managing Runners

### Viewing Registered Runners
```bash
# Access the Semaphore container
docker exec -it ansible-semaphore /bin/bash

# List registered runners (if supported by your version)
semaphore runner list --config /etc/semaphore/config.json
```

### Stopping a Runner
```bash
# Stop runner process
docker exec ansible-semaphore pkill -f "semaphore runner start"

# Or if running on remote machine
pkill -f "semaphore runner start"
```

### Unregistering a Runner
```bash
# Unregister a runner
semaphore runner unregister --config /etc/semaphore/config.json
```

## Troubleshooting

### Common Issues

#### 1. Runner Registration Fails
**Error**: `registration token cannot be empty`

**Solution**: Ensure you're using the correct token and method:
```bash
echo "Z*!!MPPi68C3Vganq2BbV*PBNTxxtaLCfk9M.XG@z!h" | semaphore runner register --config /etc/semaphore/config.json --stdin-registration-token
```

#### 2. HTTP 400 Errors When Starting Runner
**Error**: `error status code 400` when checking for new jobs

**Possible Causes**:
- Network connectivity issues
- Incorrect API URL configuration
- Server not ready to accept runners

**Solutions**:
1. Verify network connectivity to `https://ansible.phx-erp.de`
2. Check if Semaphore UI is accessible
3. Ensure the server is properly configured for remote runners

#### 3. Runner Configuration Issues
**Error**: Configuration validation fails

**Solution**: Verify your configuration file structure:
```json
{
  "runner": {
    "registration_token": "Z*!!MPPi68C3Vganq2BbV*PBNTxxtaLCfk9M.XG@z!h",
    "api_url": "https://ansible.phx-erp.de"
  }
}
```

### Debugging Runner Issues

#### View Runner Logs
```bash
# View recent logs
docker logs ansible-semaphore --tail 50

# Follow logs in real-time
docker logs ansible-semaphore --follow

# View logs with timestamps
docker logs ansible-semaphore --timestamps

# View logs from specific time
docker logs ansible-semaphore --since 10m

# Filter for runner-specific errors
docker logs ansible-semaphore | grep -E "(runner|error|400|invalid)"

# Filter for specific error types
docker logs ansible-semaphore | grep "invalid public key"
docker logs ansible-semaphore | grep "error status code"
```

#### Enable Debug Logging
```bash
# Start runner with debug logging
semaphore runner start --config /etc/semaphore/config.json --log-level DEBUG
```

#### Check Runner Status Inside Container
```bash
# Access container and check processes
docker exec -it ansible-semaphore /bin/bash
ps aux | grep runner

# Check system logs inside container
journalctl -f  # if systemd is available
```

## Parallel Task Configuration

### Current Settings
Your Semaphore configuration supports:
- **Max Parallel Tasks**: 3 (configurable in `config.json`)
- **Max Tasks per Template**: 3
- **Remote Runner Support**: Enabled

### Increasing Parallel Capacity
To handle more parallel tasks:

1. **Add More Runners**: Set up additional runner machines
2. **Increase Limits**: Update `max_parallel_tasks` in `config.json`
3. **Optimize Resources**: Ensure runners have adequate CPU/memory

### Example: Scaling to 5 Parallel Tasks
```bash
# Update configuration
sed -i 's/"max_parallel_tasks": 3/"max_parallel_tasks": 5/' /opt/phx/semaphore-config/config/config.json

# Restart Semaphore
docker-compose restart semaphore
```

## Security Considerations

1. **Token Security**: Keep your registration token secure
2. **Network Security**: Use VPN or secure networks for remote runners
3. **Access Control**: Limit runner access to necessary resources only
4. **Regular Updates**: Keep Semaphore and runners updated

## Monitoring and Maintenance

### Health Checks
```bash
# Check if runners are responding
curl -k https://ansible.phx-erp.de/api/health

# Monitor runner processes
docker exec ansible-semaphore ps aux | grep runner
```

### Log Rotation
Configure log rotation to prevent disk space issues:
```bash
# Add to crontab for log cleanup
0 2 * * * docker exec ansible-semaphore find /tmp -name "*.log" -mtime +7 -delete
```

## Support and Resources

- **Semaphore Documentation**: https://docs.semaphoreui.com/
- **GitHub Repository**: https://github.com/semaphoreui/semaphore
- **Configuration Reference**: https://docs.semaphoreui.com/administration-guide/configuration/
- **Runner Documentation**: https://docs.semaphoreui.com/administration-guide/runners/

## Quick Commands Reference

```bash
# Register runner
echo "TOKEN" | semaphore runner register --config /etc/semaphore/config.json --stdin-registration-token

# Start runner
semaphore runner start --config /etc/semaphore/config.json

# Stop runner
pkill -f "semaphore runner start"

# Unregister runner
semaphore runner unregister --config /etc/semaphore/config.json

# Check runner status
ps aux | grep "semaphore runner"

# View logs
docker logs ansible-semaphore
```

---

**Note**: Replace `TOKEN` with your actual registration token: `Z*!!MPPi68C3Vganq2BbV*PBNTxxtaLCfk9M.XG@z!h`