fj-stoker

fj-stoker is an autoscaling daemon for disposable Forgejo Actions runners on Google Compute Engine. VMs are created on demand and destroyed when the work is done.

Configuration

Copy fj-stoker.example.toml and pass its path with --config. The default path is /etc/fj-stoker/config.toml.

deployment_id identifies the state database and all managed Forgejo and GCE resources. Use a lowercase slug and keep it unchanged while managed resources exist.

Each [worker_profiles.<selector>] table maps one Forgejo runs-on selector to an immutable GCE instance-template name and its capacity, boot, idle, drain, and maximum-lifetime policy. Drain timeout defaults to 300 seconds and maximum lifetime to four hours when omitted. The selector is the durable profile ID and a GCE ownership label, using the same lowercase marker syntax as deployment_id. Jobs with no configured selector are ignored; a job matching more than one profile is rejected.

Supply the Forgejo API token and GCP service-account JSON through credential files. Under systemd, use credentials named forgejo-token and gcp-service-account:

[Service]
DynamicUser=yes
StateDirectory=fj-stoker
LoadCredential=forgejo-token:/etc/fj-stoker/forgejo-token
LoadCredential=gcp-service-account:/etc/fj-stoker/gcp-service-account
ExecStart=/usr/bin/fj-stoker --run --config /etc/fj-stoker/config.toml

For local development, pass credential files explicitly or place both files in $CREDENTIALS_DIRECTORY under those names:

cargo run -- --config fj-stoker.example.toml \
    --forgejo-token-file /path/to/forgejo-token \
    --gcp-credentials-file /path/to/gcp-service-account \
    --state-file .state/state.sqlite3

The SQLite database defaults to state.sqlite3 under $STATE_DIRECTORY when systemd provides it, otherwise /var/lib/fj-stoker/state.sqlite3. --state-file overrides both. New database files use mode 0600; SQLite uses WAL mode, full synchronous durability, an application identifier, and a schema version. A database created for one deployment ID is rejected when opened under another. Lease intent records the worker-profile selector and deterministic instance name before external resources are created.

With no mutation flag, the daemon performs authenticated read-only probes of the repository Actions job queue, directly owned repository runners, and deployment-owned GCE instances; validates ownership and duplicate claims; computes and logs a read-only lifecycle plan; then exits. --reconcile-once applies one plan, including orphan adoption and runner-first cleanup. --probe-runner-lifecycle creates, verifies, and deletes a temporary offline repository runner to test runner-administration permissions.

--dispatch-once selects at most one waiting job, persists lease intent, creates a marked Forgejo runner, persists that identity, merges the immutable instance template's complete metadata and labels with per-lease bootstrap values, creates the worker, waits for the GCE operation, and exits. Failures leave durable identity for reconciliation.

--run starts the continuous autoscaling loop in the foreground. Each cycle inventories and validates the three ledgers, applies reconciliation, then considers at most one dispatch against freshly loaded durable capacity. Cycles begin immediately and ordinarily wait poll_interval_seconds, which defaults to 15 and must be between 1 and 3600. Consecutive failed cycles use exponential backoff from that interval, capped at the greater of five minutes or the configured interval, and a successful cycle resets the delay. Retries occur at the complete-cycle boundary; mutating HTTP requests are not repeated after ambiguous failures. SIGINT and SIGTERM stop the loop during backoff or an in-flight bounded API operation.

Logging

Under systemd, fj-stoker sends structured events directly to journald, preserving event levels and fields. Outside systemd it writes compact, uncolored logs to the terminal. The default maximum level is info; use --log-level debug for additional diagnostics.

Read the service log with:

journalctl -u fj-stoker.service --since today -o short-iso

Journald storage may be volatile depending on the host configuration. To retain logs across reboots, configure persistence and an explicit storage budget, for example in /etc/systemd/journald.conf.d/fj-stoker-retention.conf:

[Journal]
Storage=persistent
SystemMaxUse=1G
MaxRetentionSec=90day

These limits apply to the host journal as a whole. Forgejo stores workflow output and artifacts; the daemon log records scaling decisions, worker state transitions, cloud operations, and failures. Credentials and runner tokens must never be logged.

Development

The minimum supported Rust version is 1.85.

cargo build
cargo test
cargo clippy --all-targets --all-features -- -D warnings
cargo fmt --all -- --check