Troubleshooting Guide¶

This guide covers common issues and their solutions when running MarchProxy.

Quick Diagnostics¶

Health Check Commands¶

# Check all service health
curl http://localhost:8000/healthz    # Manager
curl http://localhost:8080/healthz    # Proxy
curl http://localhost:9090/targets    # Prometheus
curl http://localhost:3000/api/health # Grafana

# Get detailed status
curl http://localhost:8000/healthz/detailed
curl http://localhost:8080/admin/status

Service Status¶

# Docker Compose
docker-compose ps
docker-compose logs --tail=50 manager
docker-compose logs --tail=50 proxy

# Kubernetes
kubectl get pods -n marchproxy
kubectl logs -n marchproxy deployment/marchproxy-manager --tail=50
kubectl logs -n marchproxy deployment/marchproxy-proxy --tail=50

Common Issues¶

Installation Issues¶

Docker Permission Denied¶

Symptoms: Permission denied errors when running Docker commands

Solution:

# Add user to docker group
sudo usermod -aG docker $USER

# Restart session or run
newgrp docker

# Or run with sudo temporarily
sudo docker-compose up -d

Port Already in Use¶

Symptoms: "Port already in use" or "bind: address already in use"

Solution:

# Find what's using the port
sudo lsof -i :8000
sudo netstat -tulpn | grep :8000

# Kill the process or change ports in docker-compose.yml
# Edit docker-compose.yml and change ports:
# - "8001:8000"  # Use port 8001 instead of 8000

eBPF Compilation Errors¶

Symptoms: "Failed to load eBPF program" or compilation errors

Solution:

# Install required kernel headers
sudo apt install linux-headers-$(uname -r)  # Ubuntu/Debian
sudo dnf install kernel-devel                # RHEL/CentOS

# Check kernel version (4.18+ required)
uname -r

# Install additional eBPF tools
sudo apt install libbpf-dev clang llvm      # Ubuntu/Debian
sudo dnf install libbpf-devel clang llvm    # RHEL/CentOS

# Restart proxy after installing headers
docker-compose restart proxy

Database Issues¶

Database Connection Failed¶

Symptoms: "Connection refused" or "could not connect to database"

Solution:

# Check database status
docker-compose logs postgres

# Verify database is running
docker-compose ps postgres

# Check connection string
docker-compose exec manager env | grep DATABASE_URL

# Test database connection manually
docker-compose exec postgres psql -U marchproxy -d marchproxy -c "\l"

# Reset database if corrupted
docker-compose down
docker volume rm marchproxy_postgres_data  # WARNING: Destroys data
docker-compose up -d

Database Migration Errors¶

Symptoms: Migration errors or schema mismatch

Solution:

# Check migration status
docker-compose exec manager python3 -c "
from applications.marchproxy.models import db
print('Tables:', db._adapter.tables())
"

# Run migrations manually
docker-compose exec manager python3 -c "
from applications.marchproxy.models import db
db.commit()
"

# Reset database schema (WARNING: Destroys data)
docker-compose exec postgres psql -U marchproxy -d marchproxy -c "
DROP SCHEMA public CASCADE;
CREATE SCHEMA public;
"
docker-compose restart manager

Proxy Issues¶

Proxy Registration Failed¶

Symptoms: Proxy not showing up in manager interface

Solution:

# Check proxy logs
docker-compose logs proxy

# Verify manager connectivity
docker-compose exec proxy curl http://manager:8000/healthz

# Check cluster API key
docker-compose exec proxy env | grep CLUSTER_API_KEY

# Verify network connectivity
docker-compose exec proxy ping manager

# Check proxy configuration
curl http://localhost:8080/admin/config

High Memory Usage¶

Symptoms: Proxy consuming excessive memory

Solution:

# Check memory usage
docker stats marchproxy_proxy_1

# Tune Go garbage collector
docker-compose up -d --scale proxy=1 -e GOGC=50

# Reduce connection limits
# Edit proxy configuration:
MAX_CONNECTIONS=1000
WORKER_THREADS=2

# Monitor memory with pprof
curl http://localhost:8080/debug/pprof/heap > heap.prof
go tool pprof heap.prof

Performance Issues¶

Symptoms: Low throughput or high latency

Solution:

# Check acceleration status
curl http://localhost:8080/admin/acceleration

# Enable XDP if supported
docker-compose up -d -e ENABLE_XDP=true

# Optimize network settings
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.wmem_max=134217728
sudo sysctl -w net.netfilter.nf_conntrack_max=1048576

# Check for resource constraints
docker-compose exec proxy top
docker-compose exec proxy iostat -x 1

License Issues¶

License Validation Failed¶

Symptoms: "Invalid license" or "License expired" messages

Solution:

# Check license status
curl http://localhost:8000/api/v1/license/status

# Verify license key format (PENG-XXXX-XXXX-XXXX-XXXX-ABCD)
docker-compose exec manager env | grep LICENSE_KEY

# Check license server connectivity
docker-compose exec manager curl https://license.penguintech.io/health

# Update license key
# Edit .env file with new license key
# Restart services
docker-compose restart manager proxy

Feature Not Available¶

Symptoms: Enterprise features not working on Community edition

Solution:

# Check license type
curl http://localhost:8000/api/v1/license/status

# Verify proxy count (max 3 for Community)
curl http://localhost:8000/api/v1/proxies/count

# Upgrade to Enterprise license
# Contact [email protected] for Enterprise license

Network Issues¶

Services Cannot Reach Internet¶

Symptoms: Backend services cannot connect to external resources

Solution:

# Check proxy routing
curl http://localhost:8080/admin/routes

# Test service connectivity
docker-compose exec proxy curl -v http://httpbin.org/ip

# Check iptables rules
sudo iptables -L -n
sudo iptables -t nat -L -n

# Verify DNS resolution
docker-compose exec proxy nslookup google.com

# Check network interfaces
docker-compose exec proxy ip addr show

TLS Certificate Issues¶

Symptoms: SSL/TLS errors or certificate validation failures

Solution:

# Check certificate validity
openssl x509 -in /path/to/cert.crt -text -noout

# Verify certificate chain
openssl verify -CAfile ca.crt server.crt

# Test TLS connection
openssl s_client -connect proxy.company.com:443 -servername test.company.com

# Regenerate certificates
# Use manager interface to generate new certificates
# Or use OpenSSL:
openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 365 -nodes

Authentication Issues¶

Symptoms: Login failures or session issues

Solution:

# Reset admin password
docker-compose exec manager python3 -c "
from applications.marchproxy.models import db, auth
# Reset admin user password
admin_user = db.auth_user[1]  # Usually admin is ID 1
admin_user.update_record(password=auth.get_or_create_db().password_validator('newpassword123'))
db.commit()
print('Admin password reset to: newpassword123')
"

# Check session configuration
docker-compose exec manager env | grep SECRET_KEY

# Clear browser cookies and cache
# Or try incognito/private browsing mode

# Check authentication logs
docker-compose logs manager | grep -i auth

SAML/OAuth2 Issues (Enterprise)¶

Symptoms: SSO authentication failures

Solution:

# Check SAML configuration
curl http://localhost:8000/api/v1/auth/saml/metadata

# Verify IdP connectivity
curl -v https://your-idp.example.com/.well-known/saml-configuration

# Check OAuth2 settings
curl http://localhost:8000/api/v1/auth/oauth2/config

# Review authentication logs
docker-compose logs manager | grep -i saml
docker-compose logs manager | grep -i oauth2

Performance Troubleshooting¶

Monitoring Performance¶

Built-in Metrics¶

# Proxy metrics
curl http://localhost:8080/metrics

# Manager metrics
curl http://localhost:8000/metrics

# System metrics
curl http://localhost:9100/metrics  # Node exporter if installed

Performance Testing¶

# Basic throughput test
iperf3 -c your-proxy-ip -p 80 -t 30

# HTTP load testing
ab -n 10000 -c 100 http://localhost:8080/

# Connection testing
curl -w "@curl-format.txt" -o /dev/null -s http://localhost:8080/

# Create curl-format.txt:
cat > curl-format.txt << EOF
     time_namelookup:  %{time_namelookup}\n
        time_connect:  %{time_connect}\n
     time_appconnect:  %{time_appconnect}\n
    time_pretransfer:  %{time_pretransfer}\n
       time_redirect:  %{time_redirect}\n
  time_starttransfer:  %{time_starttransfer}\n
                     ----------\n
          time_total:  %{time_total}\n
EOF

Resource Optimization¶

Memory Optimization¶

# Reduce Docker memory limits
# Edit docker-compose.yml:
services:
  manager:
    mem_limit: 512m
  proxy:
    mem_limit: 1g

# Optimize Go garbage collection
environment:
  GOGC: 50
  GOMEMLIMIT: 1GiB

CPU Optimization¶

# Limit CPU usage
# Edit docker-compose.yml:
services:
  proxy:
    cpus: '2.0'
    cpu_percent: 75

# Enable CPU affinity for better performance
# Check CPU cores
nproc

# Pin proxy to specific cores
docker-compose exec proxy taskset -cp 0,1 1

Log Analysis¶

Collecting Logs¶

Centralized Logging¶

# Export all logs
docker-compose logs --no-color > marchproxy-logs.txt

# Get logs from specific timeframe
docker-compose logs --since="2024-01-01T00:00:00" --until="2024-01-02T00:00:00"

# Filter logs by service
docker-compose logs manager | grep ERROR
docker-compose logs proxy | grep -i "connection"

Log Levels¶

# Increase log verbosity
# Edit .env file:
LOG_LEVEL=debug

# Restart services
docker-compose restart

# Reduce log noise
LOG_LEVEL=warn

Common Log Patterns¶

Error Patterns to Look For¶

# Connection errors
grep -i "connection.*failed\|connection.*refused\|timeout" logs.txt

# Authentication errors
grep -i "authentication.*failed\|invalid.*credentials\|unauthorized" logs.txt

# Database errors
grep -i "database.*error\|sql.*error\|migration.*failed" logs.txt

# License errors
grep -i "license.*invalid\|license.*expired\|license.*error" logs.txt

# Performance warnings
grep -i "high.*latency\|memory.*usage\|cpu.*usage" logs.txt

Getting Help¶

Information to Provide¶

When requesting help, include:

Environment Information:

# System info
uname -a
docker version
docker-compose version

# MarchProxy version
curl http://localhost:8080/version
curl http://localhost:8000/version

Configuration:

# Sanitized environment variables (remove secrets)
docker-compose config | grep -v -E '(PASSWORD|SECRET|KEY)'

Logs:

# Recent logs with timestamps
docker-compose logs --timestamps --tail=100

Error Details:
Exact error messages
Steps to reproduce
When the issue started
Any recent changes

Support Channels¶

Community: GitHub Issues
Documentation: docs.marchproxy.com
Enterprise Support: Contact [email protected]

Emergency Procedures¶

Complete System Reset¶

# WARNING: This destroys all data
docker-compose down
docker volume prune -f
docker network prune -f
docker-compose up -d

Backup and Restore¶

# Backup database
docker-compose exec postgres pg_dump -U marchproxy marchproxy > backup.sql

# Restore database
docker-compose exec -T postgres psql -U marchproxy marchproxy < backup.sql

# Backup configuration
cp .env .env.backup
docker-compose config > docker-compose.backup.yml

For issues not covered in this guide, please check our GitHub Issues or contact support.

Troubleshooting Guide¶

Quick Diagnostics¶

Health Check Commands¶

Service Status¶

Common Issues¶

Installation Issues¶

Docker Permission Denied¶

Port Already in Use¶

eBPF Compilation Errors¶

Database Issues¶

Database Connection Failed¶

Database Migration Errors¶

Proxy Issues¶

Proxy Registration Failed¶

High Memory Usage¶

Performance Issues¶

License Issues¶

License Validation Failed¶

Feature Not Available¶

Network Issues¶

Services Cannot Reach Internet¶

TLS Certificate Issues¶

Authentication Issues¶

Cannot Login to Manager¶

SAML/OAuth2 Issues (Enterprise)¶

Performance Troubleshooting¶

Monitoring Performance¶

Built-in Metrics¶

Performance Testing¶

Resource Optimization¶

Memory Optimization¶

CPU Optimization¶

Log Analysis¶

Collecting Logs¶

Centralized Logging¶

Log Levels¶

Common Log Patterns¶

Error Patterns to Look For¶

Getting Help¶

Information to Provide¶

Support Channels¶

Emergency Procedures¶

Complete System Reset¶

Backup and Restore¶