Autoscaler

The Crow CI Autoscaler dynamically provisions cloud servers to execute pipelines, then terminates them when idle.

sequenceDiagram
    participant Queue as Build Queue
    participant AS as Autoscaler
    participant Cloud as Cloud Provider
    participant Agent as Agent (VM)
    participant Server as Crow Server

    Queue->>AS: Pending build
    AS->>Cloud: Provision VM
    Cloud->>Agent: VM ready
    Agent->>Server: Register & connect
    Agent->>Agent: Execute pipeline
    Note over AS,Agent: Idle timeout
    AS->>Cloud: Terminate VM

Supported Providers

Provider	Configuration Reference
AWS	flags.go
Azure	flags.go
Hetzner Cloud	flags.go
Linode	flags.go
Scaleway	flags.go
Vultr	flags.go

Additional providers with a Go SDK can be added — contributions welcome!

Setup

1. Deploy alongside the server — the autoscaler listens for build triggers

2. Configure server connection — provide server address and authentication tokens

3. Configure scaling limits — set min/max agents and workflows per agent

4. Configure gRPC — set the autoscaler’s gRPC endpoint, and forward it to spawned agents via CROW_AGENT_ENV

5. Configure cloud provider — set provider credentials and instance settings

Tip

Provisioning Windows VMs through the autoscaler? Windows hosts are supported on AWS and Azure. Set CROW_OS_TYPE=windows, and note that CROW_AGENT_IMAGE is then treated as a URL to a crow-agent_windows_amd64.zip artifact, not a container image. See Windows Agents for the full picture, including what Windows hosts cost and recommended container images for your steps.

A pool intended to run the Hyper-V backend additionally needs an instance type offering nested virtualization, which rules out most of them.

Complete Example

# docker-compose.yaml
services:
  crow-autoscaler:
    image: codefloe.com/crowci/crow-autoscaler:<version>
    restart: always
    depends_on:
      - crow-server
    environment:
      # Server connection
      - CROW_SERVER=crow-server:9000
      - CROW_TOKEN=${CROW_TOKEN} # Admin API token
      - CROW_AUTOSCALER_TOKEN=${CROW_AUTOSCALER_TOKEN}

      # Scaling limits
      - CROW_MIN_AGENTS=0
      - CROW_MAX_AGENTS=2
      - CROW_WORKFLOWS_PER_AGENT=5

      # Autoscaler's own gRPC connection to the server
      - CROW_GRPC_ADDR=grpc.crow.example.com
      - CROW_GRPC_SECURE=true

      # Timeouts
      - CROW_AGENT_IDLE_TIMEOUT=10m
      - CROW_AGENT_SERVER_CONNECTION_TIMEOUT=10m

      # Cloud provider (Hetzner example)
      - CROW_PROVIDER=hetznercloud
      - CROW_HETZNERCLOUD_API_TOKEN=${HETZNER_TOKEN}
      - CROW_HETZNERCLOUD_LOCATION=fsn1
      - CROW_HETZNERCLOUD_SERVER_TYPE=cax41
      - CROW_HETZNERCLOUD_IMAGE=ubuntu-24.04
      - CROW_HETZNERCLOUD_NETWORKS=my-network
      - CROW_HETZNERCLOUD_SSH_KEYS=my-key
      - CROW_HETZNERCLOUD_FIREWALLS=my-firewall

      # Agent image (optional — auto-detected from server version if omitted)
      # - CROW_AGENT_IMAGE=codefloe.com/crowci/crow-agent:v5.3.2

      # Optional: agent environment
      - CROW_AGENT_ENV=CROW_LOG_LEVEL=debug,CROW_HEALTHCHECK=false

Configuration Reference

Server Connection

Variable	Description
CROW_SERVER	Server address (internal or public URL)
CROW_TOKEN	Admin API token for agent management
CROW_AUTOSCALER_TOKEN	Registration token for autoscaler

Scaling

Variable	Default	Description
CROW_MIN_AGENTS	0	Minimum agents always running
CROW_MAX_AGENTS	1	Maximum concurrent agents
CROW_WORKFLOWS_PER_AGENT	1	Parallel workflows per agent

Tip

Monitor resource usage during initial builds to tune CROW_WORKFLOWS_PER_AGENT. The optimal value depends on your instance type and workflow requirements.

Timeouts

Variable	Default	Description
CROW_AGENT_IDLE_TIMEOUT	10m	Time before idle agent is terminated
CROW_AGENT_SERVER_CONNECTION_TIMEOUT	10m	Max time without server connection
CROW_AGENT_PROVISION_TIMEOUT	5m	How long a new agent usually takes to connect; see Provisioning Progress

Tip

Match idle timeout to your provider’s billing cycle. If billed hourly, consider keeping agents alive longer to handle subsequent builds.

gRPC Connection

The autoscaler needs its own gRPC connection to the Crow server for queue and agent management. These variables configure the autoscaler’s connection, not the agents it spawns. For configuring spawned agents, see Spawned Agent Environment below.

Variable	Description
CROW_GRPC_ADDR	gRPC address the autoscaler dials (no protocol prefix)
CROW_GRPC_SECURE	Set true when the autoscaler’s gRPC target uses TLS

Agent Image

Variable	Default	Description
CROW_AGENT_IMAGE	auto	Container image for spawned agents

When CROW_AGENT_IMAGE is not set, the autoscaler queries the Crow server’s /version endpoint and uses the matching agent image automatically — for example, if the server reports version v5.3.2, the autoscaler uses codefloe.com/crowci/crow-agent:v5.3.2.

Set this variable explicitly only if you need to pin a specific agent version or use a custom image.

Agent Configuration

Variable	Default	Description
CROW_AGENT_ENV	none	Environment variables passed to spawned agents (comma-separated KEY=value pairs)
CROW_FILTER_LABELS	none	Only count queued tasks matching these labels (comma-separated key=value, a task must match all) toward scaling decisions. Required for multiple autoscalers.

Example agent environment:

CROW_AGENT_ENV=CROW_AGENT_LABELS=tier=heavy,CROW_LOG_LEVEL=debug,CROW_HEALTHCHECK=false

Note

CROW_AGENT_LABELS accepts multiple labels as a comma-separated list, e.g. CROW_AGENT_LABELS=tier=heavy,gpu=true. Even though it lives inside the comma-separated CROW_AGENT_ENV, the autoscaler reassembles the full label list, so no labels are lost. Label keys must be lower-case so they can be distinguished from the upper-case environment variable names that share CROW_AGENT_ENV.

Spawned Agent Environment

The autoscaler does not automatically forward its own connection settings to the agents it spawns. Every variable a spawned agent needs — including how to reach the Crow server — must be set via CROW_AGENT_ENV.

A typical remote setup forwards the server endpoint and TLS flag:

- CROW_AGENT_ENV=CROW_SERVER=grpc.crow.example.com:443,CROW_GRPC_SECURE=true,CROW_LOG_LEVEL=info

Note that the spawned agent reads CROW_SERVER as a gRPC address, not as the HTTP URL the autoscaler itself uses. The example above is the form the agent expects (host:port), passed through CROW_AGENT_ENV.

Provisioning Progress

While an autoscaler starts an agent for a pending workflow, the workflow shows that an agent is being provisioned, how long that has taken so far, and how long it usually takes. The usual time is the autoscaler’s CROW_AGENT_PROVISION_TIMEOUT. Set it to how long your hosts really take: a Linux VM is ready within a few minutes, while a Windows host that installs a hypervisor can take 20. Once that time has passed without the agent connecting, the workflow says provisioning appears stuck.

The host can also report what it is doing, and the workflow lists those phases with the time each began. The autoscaler passes every host the URL to report to as CROW_PROVISIONING_URL, next to the agent’s token in CROW_AGENT_SECRET. The Hyper-V bootstrap reports its phases this way, and a custom bootstrap can do the same:

# report a phase; add "failed": true and a "detail" when one fails
curl -fsS -X POST "$CROW_PROVISIONING_URL" \
  -H "Authorization: Bearer $CROW_AGENT_SECRET" \
  -H 'Content-Type: application/json' \
  -d '{"phase": "Installing Docker"}'

The request authenticates with the agent’s own token, so a host can only report for its own agent. Reporting is best effort: a bootstrap should carry on when a report fails.

Anyone who can view a pipeline sees the phase names on its waiting workflows, so keep them generic. The detail text, such as the error of a failed phase, is left off workflows and only returned with the agent by the agents API. The server keeps the last 20 phases per agent.

gRPC Proxy Setup

Remote agents need a TLS-secured gRPC endpoint. Configure your reverse proxy to forward to the server’s gRPC port (default: 9000).

Note

Use Let’s Encrypt or similar for SSL certificates.

Nginx

server {
    listen 443 ssl http2;
    server_name grpc.crow.example.com;

    ssl_certificate /etc/ssl/certs/crow.crt;
    ssl_certificate_key /etc/ssl/private/crow.key;

    location / {
        grpc_pass grpc://crow-server:9000;
    }
}

Caddy

grpc.crow.example.com {
    reverse_proxy h2c://crow-server:9000
}

Traefik

http:
  routers:
    crow-grpc:
      rule: Host(`grpc.crow.example.com`)
      service: crow-server
      tls:
        certResolver: letsencrypt
  services:
    crow-server:
      loadBalancer:
        servers:
          - url: h2c://crow-server:9000

Hybrid Setup

Combine static agents (always-on) with autoscaled agents (on-demand) for cost efficiency.

Agent Type	Use Case
Static	Fast, lightweight builds; always available
Autoscaled	Resource-intensive builds; cost-optimized

Example: Run a small static agent alongside the server for quick jobs. The autoscaler provisions powerful VMs only when the static agent is at capacity.

Use labels to route workflows:

Static agent configuration:

CROW_AGENT_LABELS=tier=standard

Workflow targeting autoscaled agents (.crow.yaml):

labels:
  tier: heavy

The autoscaler checks for available agents before provisioning. If a static agent can handle the workload, no new VM is created.

Multiple Autoscalers

A single Crow server can use multiple autoscalers simultaneously. Each autoscaler runs as an independent process with its own registration token, provider configuration, and scaling limits.

Why Use Multiple Autoscalers

Multiple autoscalers let you target different cloud providers from a single server, for example, Hetzner for Linux builds and Azure for Windows builds. You can also provision different instance sizes, using small VMs for unit tests and large VMs for integration tests. Multi-region setups are possible too, placing agents in eu-west for European teams and us-east for US teams. For cost optimization, non-urgent work can run on spot or preemptible instances while time-sensitive builds use on-demand capacity. Finally, you can serve different architectures by provisioning amd64 agents from one provider and arm64 agents from another.

Note

Currently, a single autoscaler can only provision a single instance type. This logic may change in the future.

If you need multiple instance types to choose from, you need to register multiple autoscalers.

How It Works

Each autoscaler reports its capabilities to the server via heartbeat. The server uses two mechanisms to route workflows to the right autoscaler:

1. Agent labels (CROW_AGENT_LABELS inside CROW_AGENT_ENV) — the autoscaler reports these to the server, which uses them to determine whether the autoscaler can provision agents for a given workflow. A workflow’s labels: must match an autoscaler’s reported labels for that autoscaler to handle it.

2. Filter labels (CROW_FILTER_LABELS) — the autoscaler uses these locally to decide which queued tasks count toward its scaling decisions. Without this, every autoscaler would see all pending tasks and try to scale up for work meant for a different autoscaler.

Caution

When running multiple autoscalers, always set CROW_FILTER_LABELS on each one. Without it, each autoscaler counts all pending tasks — not just the ones it can handle — and will over-provision agents.

Example: Dual-Provider Setup

Register two autoscalers on the server and note their tokens.

# docker-compose.yaml
services:
  # Small instances for standard builds
  autoscaler-standard:
    image: codefloe.com/crowci/crow-autoscaler:<version>
    restart: always
    environment:
      - CROW_SERVER=crow-server:9000
      - CROW_TOKEN=${CROW_TOKEN}
      - CROW_AUTOSCALER_TOKEN=${AUTOSCALER_TOKEN_STANDARD}
      - CROW_GRPC_ADDR=grpc.crow.example.com
      - CROW_GRPC_SECURE=true
      - CROW_MAX_AGENTS=4
      - CROW_WORKFLOWS_PER_AGENT=3
      - CROW_FILTER_LABELS=tier=standard
      - CROW_AGENT_ENV=CROW_AGENT_LABELS=tier=standard
      - CROW_PROVIDER=hetznercloud
      - CROW_HETZNERCLOUD_API_TOKEN=${HETZNER_TOKEN}
      - CROW_HETZNERCLOUD_SERVER_TYPE=cax21
      # ... other Hetzner settings

  # Large instances for heavy builds
  autoscaler-heavy:
    image: codefloe.com/crowci/crow-autoscaler:<version>
    restart: always
    environment:
      - CROW_SERVER=crow-server:9000
      - CROW_TOKEN=${CROW_TOKEN}
      - CROW_AUTOSCALER_TOKEN=${AUTOSCALER_TOKEN_HEAVY}
      - CROW_GRPC_ADDR=grpc.crow.example.com
      - CROW_GRPC_SECURE=true
      - CROW_MAX_AGENTS=2
      - CROW_WORKFLOWS_PER_AGENT=1
      - CROW_FILTER_LABELS=tier=heavy
      - CROW_AGENT_ENV=CROW_AGENT_LABELS=tier=heavy
      - CROW_PROVIDER=aws
      - CROW_AWS_INSTANCE_TYPE=c5.2xlarge
      # ... other AWS settings

Workflows select their tier with labels:

# .crow.yaml — lightweight job
labels:
  tier: standard

steps:
  - name: lint
    image: golangci/golangci-lint
    commands:
      - golangci-lint run

# .crow.yaml — resource-intensive job
labels:
  tier: heavy

steps:
  - name: integration
    image: golang
    commands:
      - go test -race -count=1 ./...