Ops runbook

Bring-up, recovery, and common operator tasks. All commands assume cwd /home/kushal/apps/sanchalan_setu/attendance_dev unless noted.

1. First-time bring-up on a fresh host

1. Ensure SDS shared infra is running: docker compose -f /opt/sds-dms/docker-compose.yml ps (postgres-1, redis-1).
2. Run scaffold bootstrap: ./scripts/bootstrap.sh — generates .env, creates attendance_dev Postgres role + DB, runs migrations.
3. Wire central creds: sudo /usr/local/sbin/sds-ops/wire-attendance-audit.sh — pulls DB_AUDIT_PASSWORD + REDIS_PASSWORD from the system-of-record (sanchalan_prod .env) and writes them in.
4. Wire host nginx: sudo /usr/local/sbin/sds-ops/install-attendance-nginx.sh.
5. Bring containers up: docker compose up -d.
6. Verify: curl https://attendance.rajyasabha.digital/api/healthz → {"ok":true}.
7. Optional seed: sudo /usr/local/sbin/sds-ops/seed-dev-data.sh — 2,300 synthetic employees + 13 holidays + 7 fences + leave policies.

2. Daily checks

3. Common tasks

Add a new admin user (dev)

docker compose exec app php artisan tinker --execute='
  $u = \App\Models\User::firstOrCreate(
    ["email"=>"alice@rajyasabha.digital"],
    ["name"=>"Alice","password"=>bcrypt("dev"),"role"=>"admin","active"=>true]
  );
  $emp = \App\Models\AttendanceEmployeeRegister::where("email","alice@rajyasabha.digital")->first();
  if ($emp) $u->update(["employee_id"=>$emp->id]);
  echo "user_id={$u->id} employee_id={$u->employee_id}\n";
'

Mint an API token (dev)

docker compose exec app php artisan tinker --execute='
  $u = \App\Models\User::where("email","you@rajyasabha.digital")->firstOrFail();
  echo $u->createToken("local",["attendance.punch","attendance.leave"])->plainTextToken,"\n";
'

Revoke all tokens for a user

docker compose exec app php artisan tinker --execute='
  \App\Models\User::where("email","compromised@rajyasabha.digital")->first()?->tokens()->delete();
'

Deactivate a device

// From admin web
//   /admin/devices → click the red ban icon next to the row
// Or via API
curl -X POST -H "Authorization: Bearer $TOK" \
  https://attendance.rajyasabha.digital/api/v1/devices/{id}/deactivate

Reset / rotate IMEI salt

Rotating the salt invalidates all device device_imei_hash values. Plan accordingly:

1. Generate new salt: openssl rand -hex 16.
2. Update .env: IMEI_SALT=<new>.
3. Truncate attendance_devices.device_imei_hash via tinker (set NULL).
4. Rebuild + redeploy mobile APK with the matching IMEI_SALT_FRONTEND.
5. Force a re-register: app's first /v1/punch failure → user prompted to re-bind the device.

4. Recovery

Audit container in restart loop

Symptom: docker compose ps shows Restarting. Logs show Predis\Client returned:

// AuditConsume::redis(): Return value must be of type Redis, Predis\Client returned

Fix: ensure REDIS_CLIENT=phpredis in .env (not predis). The image has the phpredis ext compiled. Recreate: docker compose up -d --force-recreate audit.

Worker / audit unhealthy

Symptom: (unhealthy) on workers. Cause: image-level HEALTHCHECK pings apache, which workers don't run.

Fix: docker-compose.yml overrides the healthcheck with pgrep -f <cmd> for worker + audit. Already in place.

Punch always returns rejected_geofence

Most likely the SSID in attendance_geo_fences.allowed_ssids doesn't match the device's actual SSID. Check:

SELECT id, name, allowed_ssids FROM attendance_geo_fences WHERE active;

Either update the fence with the actual SSID, or unset allowed_ssids to NULL (then any SSID inside the polygon is accepted).

Replay-guard false positive

Symptom: legitimate punches return verdict=duplicate.

Cause: client is reusing the nonce. Fix in client (regenerate per punch). To clear historical: the partial unique index only constrains verdict='accepted' — replays of rejected nonces still pass.

Sanctum token issued but 401s on call

Most likely the user has no employee_id. Most controller actions abort 401 if $req->user()->employee_id is null. Fix: set employee_id on the User row (auto-set by SSO if email matches an active employee_register row).

5. Maintenance

Run migrations

docker compose exec app php artisan migrate

Clear caches

docker compose exec app php artisan config:clear
docker compose exec app php artisan route:clear
docker compose exec app php artisan view:clear

Regenerate OpenAPI spec

docker compose exec app php artisan l5-swagger:generate
# UI: /api/documentation

Run tests

# All
docker compose exec app vendor/bin/phpunit
# Just unit
docker compose exec app vendor/bin/phpunit --testsuite=Unit
# Just feature
docker compose exec app vendor/bin/phpunit --testsuite=Feature

Smoke test (full e2e against live)

/tmp/punch-smoke.sh "$TOKEN"     # 9 steps: register, punch, replay, verdict matrix
/tmp/admin-smoke.sh              # admin login + 8 pages + CSV + leave approve + logout

Cut face-template DEKs over to Vault transit

Default is KEK_BACKEND=laravel_crypt (DEKs wrapped with APP_KEY). To migrate face-template DEKs to Vault transit envelope encryption — server-side rotation, per-row blast radius, no plaintext leaves Vault:

1. (One-time per host, root) sudo /usr/local/sbin/sds-ops/wire-attendance-vault-transit.sh — creates transit/keys/attendance, extends sanchalan-rw policy with encrypt/decrypt/rewrap caps, mints a fresh AppRole secret-id into /home/kushal/sds/setus/sanchalan-setu/.vault-approle. Idempotent; re-runs rotate the secret-id.
2. Copy the printed VAULT_ROLE_ID + VAULT_SECRET_ID into attendance_dev/.env; set KEK_BACKEND=vault_transit.
3. Recreate the PHP containers (compose restart doesn't re-read env_file): docker compose up -d.
4. Sanity check: docker compose exec app php artisan attendance:rewrap-face-deks --dry-run reports the rows that would migrate.
5. Backfill existing rows: docker compose exec app php artisan attendance:rewrap-face-deks (add --include-retired for completeness, --limit=N to stagger). The underlying GCM blob is untouched; only the wrapped DEK changes, so blob_sha256 stays valid and in-flight matches are unaffected.
6. Vault key rotation later: vault write -f transit/keys/attendance/rotate; old DEKs keep decrypting (version is in the ciphertext prefix). Use attendance:rewrap-face-deks again to re-envelope under the latest key version.
7. If Vault is sealed (post-restart), face enrolment + match will 5xx until an operator unseals. Existing punches without face requirement are unaffected.

6. Production cutover checklist

1. Set APP_ENV=production, APP_DEBUG=false.
2. Set AUTH_MODE=sso; populate SBIAM_CLIENT_ID + SBIAM_BASE_URL + SBIAM_REDIRECT_URI + SBIAM_MOBILE_CLIENT_ID.
3. Operator registers two clients in sb-iam: web (with browser redirect_uri) + mobile (public PKCE, with sanchalan-attendance://auth/callback).
4. Replace Tailwind CDN with built CSS (npm run build).
5. Bake vendor/ + assets into the image — drop the host bind mount of ./backend.
6. Switch storage bind to a uid-aligned entrypoint chown (replaces chmod 777).
7. Provision IMEI_SALT from Vault at boot, not via .env.
8. Register attendance in /infra/ §24 as status=live: infra-doc-add-deployment.sh --service=attendance --env=prod --status=live.
9. Wire daily audit-chain anchor cron: sudo /usr/local/sbin/sds-ops/audit-chain-anchor-sign.sh in crontab.
10. Cap punch signature field length (currently nullable text — add a max).