🧪 Labs • Commands • SOPs • Fix notes

Labs

Hands-on tasks and repeatable command snippets. When something breaks, the fix goes here as a mini case-study.

Copy/paste ready Small wins Verify after changes Document the fix

Core lab routines

Daily “green-check”

Fast validation when you first log in.

sudo nginx -t && sudo systemctl reload nginx
sudo systemctl status nginx --no-pager
sudo systemctl status docker --no-pager
sudo systemctl status tailscaled --no-pager
sudo systemctl status cloudflared --no-pager
df -h
free -h
uptime
docker ps

Daily Ops Status

Logs first

When something looks off, start here before changing anything.

# nginx
sudo tail -n 120 /var/log/nginx/error.log
sudo tail -n 120 /var/log/nginx/access.log

# cloudflared (systemd)
sudo journalctl -u cloudflared -n 120 --no-pager

# tailscale (systemd)
sudo journalctl -u tailscaled -n 120 --no-pager

# docker overview
docker ps
docker logs --tail=120 <container_name>

Logs Triage Troubleshooting

Monitoring lab

Monitoring Stack Write-up

Prometheus + Grafana + exporters + alerting — health checks and troubleshooting.

Prometheus Grafana Observability

Monitoring quick checks

Run on the monitoring host (from /opt/monitoring).

cd /opt/monitoring
docker compose ps

# Node Exporter metrics (should return lots of metrics)
curl -s http://localhost:9100/metrics | head

# Prometheus readiness (internal only; do not publish)
curl -s http://localhost:9095/-/ready && echo

# Confirm targets up (via API)
curl -s "http://localhost:9095/api/v1/query?query=up" | head -c 400 ; echo

Targets Readiness API

SOP snippets

Nginx: validate + reload

sudo nginx -t && sudo systemctl reload nginx
sudo systemctl status nginx --no-pager

Nginx Safe reload

Docker: compose ops

cd /opt/monitoring
docker compose ps
docker compose logs --tail=120 prometheus
docker compose restart prometheus

Docker Compose

Ports: what’s listening

sudo ss -tulpn | head -n 30
sudo lsof -i -P -n | head -n 30

Networking Ports

Disk space: quick read

df -h
sudo du -h /var/www/mywebsite/html --max-depth=2 | sort -h | tail -n 20

Storage Capacity

Case study format

Template: “Fixed it” note

## Problem
- Symptoms:
- Error message:

## Cause
- (what it ended up being)

## Fix (steps)
1)
2)
3)

## Verify
- Commands run:
- Expected output:

Habit: paste the last ~50 lines of the relevant log, then write the verified fix steps.