# 🚨 Booking System Operations Runbook

**Print Date:** _____________  **Version:** 1.0  **Owner:** DevOps Team

---

## 🔥 EMERGENCY PROCEDURES

### Kill Switch - Stop All Bookings
```bash
# Immediate shutdown
export BOOKING_ENABLED=false
php artisan config:clear

# Or via .env file
echo "BOOKING_ENABLED=false" >> .env
php artisan config:clear
```

**User Message:** "Booking is temporarily disabled for maintenance. Please try again later."  
**Expected Response:** HTTP 503 with booking disabled message

### Rollback Deployment
```bash
# Quick rollback (if using symlinks)
ln -sfn /releases/previous /current
systemctl reload php-fpm nginx

# Or git rollback
git checkout HEAD~1
composer install --no-dev --optimize-autoloader
php artisan migrate:rollback
```

**⚠️ Data Safety:** Always backup database before rollbacks!

### Emergency Hold Expiry
```bash
# Expire ALL holds for an event (nuclear option)
mysql -e "UPDATE seat_reservations SET expires_at = NOW() WHERE event_id = {EVENT_ID} AND status = 'held'"

# Or via API
curl -X POST /api/__test/database/expire-holds -d '{"event_id": 123}'
```

---

## 📊 MONITORING DASHBOARDS

### Key URLs
- **Health Check:** `/api/health` (JSON)
- **Metrics:** `/api/metrics` (JSON) or `/metrics` (Prometheus)
- **Current Status:** `/api/__test/state` (Testing only)

### Critical Metrics Thresholds
| Metric | Warning | Critical | Action |
|--------|---------|----------|---------|
| `stale_holds_over_ttl` | > 0 | > 10 | Run cleanup script |
| `hold_conflicts` rate | > 10% | > 25% | Check capacity |
| `availability_p95_ms` | > 300ms | > 1000ms | Scale/optimize |
| `hold_p95_ms` | > 600ms | > 2000ms | Database tuning |

### Grafana Queries (if using)
```promql
# Active holds
booking_active_holds

# Conflict rate over time
rate(booking_hold_conflicts_total[5m]) / rate(booking_holds_created_total[5m])

# Performance trends
histogram_quantile(0.95, booking_availability_response_time)
```

---

## 🚦 ALERT RESPONSES

### Alert: `stale_holds_over_ttl > 0`
**Severity:** Warning  
**Action:**
```bash
# Check stale holds
curl -s /api/health | jq '.data.stale_holds'

# Clean up manually
mysql -e "DELETE FROM seat_reservations WHERE status='held' AND expires_at < NOW()"
```

### Alert: `hold_conflicts spiking`
**Severity:** Warning  
**Possible Causes:**
- High demand event (normal)
- Bot activity
- Performance degradation

**Investigation:**
```bash
# Check recent conflict patterns
curl -s /api/metrics | jq '.data.metrics'

# Look at logs
tail -f storage/logs/metrics.log | grep CONFLICT
```

### Alert: `confirm_idempotent_hits > 30%`
**Severity:** Warning  
**Indicates:** Potential client retries or abuse

**Action:**
```bash
# Check for retry patterns
grep "idempotent" storage/logs/metrics.log | tail -20

# Consider rate limiting adjustment
```

---

## 🔧 ON-CALL DIAGNOSTIC QUERIES

### Database Health
```sql
-- Active holds by event
SELECT event_id, COUNT(*) as active_holds 
FROM seat_reservations 
WHERE status = 'held' AND expires_at > NOW() 
GROUP BY event_id;

-- Stale holds (should be 0)
SELECT COUNT(*) as stale_holds 
FROM seat_reservations 
WHERE status = 'held' AND expires_at < NOW();

-- Recent booking activity (last hour)
SELECT 
    DATE_FORMAT(created_at, '%Y-%m-%d %H:%i') as time_bucket,
    status,
    COUNT(*) as count
FROM seat_reservations 
WHERE created_at > DATE_SUB(NOW(), INTERVAL 1 HOUR)
GROUP BY time_bucket, status 
ORDER BY time_bucket DESC;
```

### API Health Check
```bash
# Quick health verification
curl -s /api/health | jq '.'

# Performance check
time curl -s /api/seats/availability/1 > /dev/null

# Rate limit status
curl -I /api/seats/hold | grep -i rate
```

### Application Logs
```bash
# Error patterns
grep -i "error\|exception\|failed" storage/logs/laravel.log | tail -10

# Booking-specific issues
grep -i "booking\|seat\|hold" storage/logs/laravel.log | tail -10

# Metrics tracking
tail -f storage/logs/metrics.log
```

---

## 🛠️ COMMON FIXES

### Performance Degradation
```bash
# Clear all caches
php artisan config:clear
php artisan route:clear
php artisan view:clear

# Restart services
systemctl restart php-fpm
systemctl restart nginx

# Check database connections
mysql -e "SHOW PROCESSLIST;"
```

### Rate Limiting Issues
```bash
# Clear rate limit cache
php artisan cache:forget booking_rate_*

# Check current limits
grep RATE_LIMIT .env
```

### Database Connectivity
```bash
# Test connection
php artisan tinker
>>> DB::connection()->getPdo();

# Check MySQL status
systemctl status mysql
mysql -e "SELECT 1;"
```

---

## 📞 ESCALATION CONTACTS

| Issue Type | Primary | Secondary | SLA |
|------------|---------|-----------|-----|
| P1: Booking Down | On-Call Engineer | Tech Lead | 15 min |
| P2: Performance | DevOps Team | Backend Team | 1 hour |
| P3: Monitoring | Platform Team | DevOps Team | 4 hours |

---

## 🔄 ROUTINE MAINTENANCE

### Daily Checks
- [ ] Review `/api/health` status
- [ ] Check for stale holds
- [ ] Verify backup completion
- [ ] Monitor error rates

### Weekly Tasks
- [ ] Performance review
- [ ] Log rotation
- [ ] Database optimization
- [ ] Capacity planning

### Monthly Tasks
- [ ] Security patches
- [ ] Performance baseline updates
- [ ] Disaster recovery test
- [ ] Documentation updates

---

**📋 Checklist for New On-Call Engineer:**
- [ ] Access to monitoring dashboards
- [ ] Database read/write access
- [ ] Application deployment keys
- [ ] Escalation contact list
- [ ] Test emergency procedures

**🚨 Remember:** When in doubt, use the kill switch first, investigate second!