Maintenance¶

Regular maintenance tasks to keep your Omakase homelab running smoothly.

Daily Tasks¶

Monitor Services¶

Check service health:

docker compose ps

View recent logs:

docker compose logs --tail=100

Check Backup Status¶

Verify daily backups completed:

docker compose logs backup | grep -i "backup completed"

Weekly Tasks¶

Update Docker Images¶

Pull latest images:

make pull

Restart services with new images:

make restart

Review Security Alerts¶

Check CrowdSec decisions:

docker exec crowdsec cscli decisions list

Review blocked IPs:

docker exec crowdsec cscli alerts list

Check Disk Usage¶

Monitor storage:

df -h ${DATA_DIR}

Check Docker disk usage:

docker system df

Monthly Tasks¶

Clean Unused Resources¶

Remove unused containers, images, volumes:

make clean

Or manually:

docker system prune -a --volumes

Review Logs¶

Archive old logs if needed:

find ${DATA_DIR}/*/logs -name "*.log" -mtime +30 -exec gzip {} \;

Test Backup Restore¶

Perform test restore to verify backups:

docker exec restic-backup restic restore latest --target /tmp/restore-test

Update Documentation¶

Review and update service documentation as configurations change.

Quarterly Tasks¶

Security Audit¶

Review user access in Authelia
Rotate sensitive credentials
Review CrowdSec security collections
Update security policies

Performance Review¶

Check resource usage:
```
docker stats
```
Identify resource-heavy services
Optimize configurations if needed

Dependency Updates¶

Review and test major version updates: 1. Check Renovate PRs 2. Read changelogs 3. Test in development environment 4. Deploy to production

As-Needed Tasks¶

Add New Service¶

Follow the Adding Services guide.

Rotate Secrets¶

Generate new secret:
```
make pwgen
```
Update in Infisical
Restart affected services:
```
docker compose restart <service>
```

Scale Resources¶

Adjust CPU/memory limits in compose files:

deploy:
  resources:
    limits:
      cpus: '2.0'
      memory: 2G

Monitoring Checklist¶

Daily: - [ ] Check service status - [ ] Verify backup completion - [ ] Review error logs

Weekly: - [ ] Update images - [ ] Review security alerts - [ ] Check disk usage

Monthly: - [ ] Clean unused resources - [ ] Test backup restore - [ ] Review logs

Quarterly: - [ ] Security audit - [ ] Performance review - [ ] Dependency updates

Maintenance Windows¶

For major updates that require downtime:

Notify users (if shared homelab)

Backup current state:

docker exec restic-backup restic backup /data

Stop services:
```
make down
```
Perform maintenance
Start services:
```
make up
```
Verify functionality:
```
docker compose ps
docker compose logs
```

Automation¶

Consider automating routine tasks:

Watchtower for Auto-Updates¶

Warning: Not recommended for production. Use Renovate instead for controlled updates.

Monitoring Alerts¶

Set up alerts for: - Service failures - Disk space warnings - Backup failures (via Telegram) - Security incidents (CrowdSec)

Emergency Procedures¶

Service Down¶

Check logs:
```
docker compose logs <service>
```
Restart service:
```
docker compose restart <service>
```
If persistent, restore from backup

System Resources Exhausted¶

Identify resource hog:
```
docker stats
```
Stop non-critical services:
```
docker compose stop <service>
```
Clean up:
```
make clean
```

Security Incident¶

Check CrowdSec alerts
Review service logs

Block malicious IPs:

docker exec crowdsec cscli decisions add --ip <ip> --duration 24h

Rotate compromised credentials