# Operations

Source: https://docs.externa.qiick.io/docs/operations

Externa depends on a **long-running queue worker** and Laravel’s **scheduler** for file zips/uploads, AI attachments, activity-log retention, and AI sync sources. Without them, async features stall and temporary storage grows. For product automation (listeners / jobs / webhooks vs Directus Flows), see [Automation without Flows](/docs/automation-without-flows).

## Queue worker (required)

Default in `.env.example` for the **full** stack: `QUEUE_CONNECTION=redis` with [Horizon](/docs/horizon). Cloners without Redis should use `QUEUE_CONNECTION=database` (see [Minimal vs full stack](/docs/minimal-vs-full-stack), [Production without Redis](/docs/production-without-redis), and [Configuration](/docs/configuration)). Upgrades: [Upgrade](/docs/upgrade).

Jobs that expect a worker include:

- Multi-file / folder **zip** preparation
- Large / bulk **file duplicates**
- **Collection import** jobs started from AI tools
- **Outbound webhook** delivery (`DeliverOutboundWebhookJob`) — see [Outbound webhooks](/docs/webhooks)

With Redis + Horizon, those jobs are consumed by `php artisan horizon` instead of `queue:listen`. Queue health lives on the dashboard **Health** tab (native cards — no primary “Open Horizon” CTA); failed/retry deep-dive: `/horizon` (super-admin).

### Local

Minimal (`composer run dev`) starts:

```bash
php artisan queue:listen --tries=1 --timeout=0
```

Full stack with **Herd Redis + Herd Reverb** (recommended):

```bash
php84 artisan horizon
php84 artisan pulse:work
```

Herd Reverb is already on `:8080` — match `REVERB_*` / `VITE_REVERB_*` and skip `reverb:start`. Standalone all-in-one: `composer run dev:full` (includes `reverb:start`; conflicts if Herd already binds `:8080`). See [Redis prerequisite](/docs/redis-prerequisite) and [Minimal vs full stack](/docs/minimal-vs-full-stack).

**Docker Compose:** Horizon / Reverb / scheduler / Pulse are separate services — `docker compose up --build` ([Installation — Docker](/docs/installation#docker-compose-local-full-stack)).

### Production

Prefer Horizon when using Redis:

```bash
php artisan horizon
```

Also run a Reverb process (or managed WebSocket) and `pulse:work` when Pulse redis ingest is enabled. Or a plain worker:

```bash
php artisan queue:work --tries=3 --timeout=300
```

Use your process manager (Supervisor, systemd, Forge, Cloud Run sidecars, etc.). Restart workers after each deploy so they pick up new code.

> `QUEUE_CONNECTION=sync` runs jobs inline and will block HTTP requests on large zips/imports. Keep a real queue driver and a worker.

## Scheduler

Defined in `routes/console.php`:

| Command                         | Frequency   | Purpose                                               |
| ------------------------------- | ----------- | ----------------------------------------------------- |
| `activitylog:clean`             | daily       | Purge old Spatie activity log rows per package config |
| `files:cleanup-uploads`         | hourly      | Remove expired partial/chunked uploads                |
| `files:cleanup-zips`            | hourly      | Remove prepared zips past `FILES_ZIP_TTL_MINUTES`     |
| `ai:cleanup-attachments`        | daily       | Delete expired AI chat attachments + files            |
| `ai:run-sync-sources`           | everyMinute | Run due AI collection sync sources                    |
| `collections:process-schedules` | everyMinute | Promote due drafts / soft-delete due unpublish_at     |

The scheduler only fires when something invokes Laravel’s schedule runner every minute:

```bash
* * * * * cd /path/to/externa-core && php artisan schedule:run >> /dev/null 2>&1
```

On platforms with a native scheduler (Laravel Cloud, Forge, etc.), enable the equivalent cron / schedule feature.

### Manual runs

```bash
php artisan activitylog:clean
php artisan files:cleanup-uploads
php artisan files:cleanup-zips
php artisan ai:cleanup-attachments
php artisan ai:run-sync-sources
php artisan schedule:run   # runs due events once
```

## Permissions sync

When `PermissionEnum` changes (or after deploy of permission-related code):

```bash
php artisan permissions:sync
# remove DB permissions no longer in the enum:
php artisan permissions:sync --prune
```

Also available from the admin UI via `POST /settings/permissions/sync` (`can-edit-permissions`). Sync upserts Spatie permission rows and regenerates `resources/js/enums/permission-enum.ts`.

After first install / empty DB, prefer `php artisan db:seed` (permissions → roles → super admin). See [Configuration](/docs/configuration).

## Health check

Auth-free probes (not the admin Pulse/Health UI):

| Endpoint            | Meaning                                                                            |
| ------------------- | ---------------------------------------------------------------------------------- |
| `GET /up`           | Laravel framework liveness                                                         |
| `GET /health/live`  | Process up — JSON `{ "status": "ok" }`                                             |
| `GET /health/ready` | DB (+ Redis when required) — JSON `{ "status", "checks" }`; **503** when not ready |

Use `/health/ready` for Compose/Kubernetes readiness and load balancers. Details and Compose `healthcheck` example: [Deployment — Health probes](/docs/deployment#health-probes).

## Jobs monitor (admin UI)

Settings → **Jobs** (`/settings/jobs`):

| Permission        | Capability                             |
| ----------------- | -------------------------------------- |
| `can-show-jobs`   | View pending sample + failed jobs list |
| `can-manage-jobs` | Retry / delete failed jobs             |

Reads Laravel `jobs` and `failed_jobs` (database queue). Domain status UIs (AI import job UUID, zip preparation) stay in their existing screens — this page is queue ops only, not a workflow builder.

Routes: `jobs.index`, `jobs.retry`, `jobs.destroy` in `routes/admin.php`. Controller: `JobMonitorController`.

## Performance settings

Settings → **Performance** (`/settings/performance`):

| Permission                    | Capability                         |
| ----------------------------- | ---------------------------------- |
| `can-manage-project-settings` | View status, flush targeted caches |

Operators see nested ops status (cache store, Redis reachable, queue connection, app env/debug), bootstrap cache read-only (config/routes/events/packages cached yes/no), public API cache parameters (TTL 120s safety net, epoch counter), and observability quick links when permitted (Jobs, Pulse, Horizon). Four targeted flush actions let you invalidate specific subsystems without a full `artisan optimize:clear`:

1. **Public API cache** — bumps the global epoch counter so all collection responses miss immediately (next hit rebuilds with fresh permission matrix).
2. **Permission matrices** — forgets cached collection/file permission matrices for every role (Forces lazy rebuild on next role check).
3. **Spatie permissions** — flushes Spatie's permission registrar cache (when `PermissionEnum` changes or roles/permissions are mutated).
4. **Dashboard metrics** — forgets the 13 dashboard metric keys (fresh metrics on next dashboard load).

Bootstrap cache status is **read-only** — no artisan optimize buttons from the UI; prefer `php artisan optimize` / `php artisan optimize:clear` in your deploy script or CI (or do not cache at all in local).

Observability links appear when the user has the relevant permission:

- **Jobs** — `can-show-jobs` → `/settings/jobs`
- **Pulse** — `viewPulse` gate (super-admin) → `/pulse`
- **Horizon** — `viewHorizon` gate (super-admin) → `/horizon`

Routes: `performance.edit`, `performance.flushPublicApi`, `performance.flushPermissionMatrices`, `performance.flushSpatiePermissions`, `performance.flushDashboardMetrics` in `routes/admin.php`. Controller: `PerformanceSettingsController`.

## Observability

### Laravel Pail

Dev script includes:

```bash
php artisan pail --timeout=0
```

Tail application logs in real time during local development. In production, rely on your log stack (`LOG_CHANNEL`, centralized logging).

### Activity log

Spatie Laravel Activitylog records auth and domain mutations (configurable via `config/activitylog.php`, env `ACTIVITYLOG_ENABLED`).

- UI: `/activity-logs` (`can-show-activity-logs`)
- Cleanup: `activitylog:clean` (daily schedule)
- AI: `QueryActivityLogs` tool for users with show permission

### Notifications

Unread counts are shared on every Inertia page (`notifications.unread_count`). With Reverb + Echo, the sidebar bell listens on the private user channel; the **60s unread poll runs only when** `BROADCAST_CONNECTION` is `log`/`null`. JSON helpers:

- `GET /notifications`
- `GET /notifications/unread-count`
- `POST /notifications/read`

(all `auth` + `verified`). Details: [Reverb & Echo](/docs/reverb-and-echo) · [Activity & notifications](/docs/activity-and-notifications).

## Dependency vulnerability CI

Maintainers: CI runs Composer + npm production audits on every PR (see [Contributing — Dependency audits](/docs/contributing#dependency-audits-ci)). Treat red audit jobs like red tests — upgrade or document an accepted risk before merge. Host/app reporting: [`SECURITY.md`](https://github.com/qiick-io/externa-core/blob/develop/SECURITY.md).

## Code scanning & secret scanning

For maintainers of the public GitHub repo:

| Control                                   | Where                                                                  |
| ----------------------------------------- | ---------------------------------------------------------------------- |
| CodeQL (JS/TS; PHP when pack supports it) | Actions workflow `codeql`; alerts under **Security → Code scanning**   |
| Secret scanning + push protection         | Repo **Settings → Code security** (enabled on `qiick-io/externa-core`) |

Triage CodeQL on first land and after noisy upgrades. Push protection blocks commits that contain known secret patterns — rotate if anything leaked. Details: [Security checklist](/docs/security-checklist#code-scanning-and-secrets) · [`SECURITY.md`](https://github.com/qiick-io/externa-core/blob/develop/SECURITY.md).

## Operational checklist

1. Worker process up and processing the `default` queue.
2. Cron / platform scheduler running `schedule:run` every minute.
3. `storage:link` present if serving public/assets URLs.
4. `permissions:sync` after permission enum deploys.
5. `/up` monitored.
6. Disk space watched under `storage/app` (uploads, zips, AI attachments, logs).
7. Run pending migrations before relying on revisions / M2M junction shape (`collection_item_revisions`, junction object migration).
8. Prefer `LIGHTHOUSE_QUERY_CACHE_MODE=opcache` (default) so GraphQL parse cache does not store AST objects in the database cache driver.
9. Walk the [Security checklist](/docs/security-checklist) before production.
10. After deploy, smoke the Public API / GraphQL with [externa-bruno](https://github.com/qiick-io/externa-bruno) (duplicate **Local**, set staging `base_url` + key) — see [Public CMS API](/docs/public-cms-api#bruno-collection-externa-bruno).