deployment.md

Deploying EKOS for a team

One server, one container, one volume. This guide takes you from nothing to a console that several people and agents use, with separate tenants, roles, and backups.

It describes what exists today. What is not built is listed at the end; read that before planning around it.

What you are deploying

Piece	What it is
The image	The ekos binary, the console API and the console UI, on port 8000
The EKOS home	One volume, /var/lib/ekos: every project's config and compiled data, the console's database and run logs. This is all the state there is
Your sources	Repositories and document folders on the host, mounted read-only at /sources. EKOS never writes to them
An identity provider	Any OIDC provider (Keycloak, Entra ID, Okta, Google). EKOS stores no passwords
A reverse proxy	For TLS. The container speaks plain HTTP on loopback

Levels of organisation: tenant → project → source → object. A tenant is a team or a customer. A project is one ledger. A source is one repository or folder of a project.

1. First start

git clone https://github.com/alexeyban/EKOS && cd EKOS
cp deploy/.env.example deploy/.env

In deploy/.env set at least:

EKOS_SOURCES=/srv/repos                 # the host directory holding your sources
EKOS_CONSOLE_SESSION_SECRET=...         # openssl rand -hex 32

EKOS_CONSOLE_OIDC_ISSUER=https://id.example.com/realms/main
EKOS_CONSOLE_OIDC_CLIENT_ID=ekos
EKOS_CONSOLE_OIDC_CLIENT_SECRET=...
EKOS_CONSOLE_OIDC_REDIRECT_URI=https://ekos.example.com/api/auth/callback

EKOS_CONSOLE_OWNERS=you@example.com     # platform owners, comma-separated
EKOS_CONSOLE_PRODUCTION=true

docker compose -f deploy/docker-compose.yml up -d --build

Tagged releases publish ghcr.io/alexeyban/ekos:<version>; until the next tag, --build builds the image from your checkout. The build compiles the binary and takes several minutes.

A server without internet access

Build the image on a machine that has access, save it, and load it on the server:

# on the connected machine, in the checkout
docker compose -f deploy/docker-compose.yml build          # tags ghcr.io/alexeyban/ekos:latest
docker save ghcr.io/alexeyban/ekos:latest | gzip > ekos-image.tar.gz
tar czf ekos-deploy.tar.gz deploy/docker-compose.yml deploy/.env.example docs/deployment.md

# on the server
docker load < ekos-image.tar.gz
tar xzf ekos-deploy.tar.gz && cp deploy/.env.example deploy/.env    # then edit it as above
docker compose -f deploy/docker-compose.yml up -d                   # no --build

Compose uses the loaded image because one with that name exists locally; it pulls only when none does. The identity provider must still be reachable from the server.

A server without Docker

The full package (RFC 0185) runs the same console from its own bundled Python. On the server, as the user that will own the home:

cat > answers.toml <<'EOF'
[console]
mode = "team"
host = "127.0.0.1"
port = 8000
oidc_issuer = "https://id.example.com/realms/main"
oidc_client_id = "ekos"
oidc_redirect_uri = "https://ekos.example.com/api/auth/callback"
owners = "you@example.com"
production = true
[start]
at_login = true      # a systemd user unit; `loginctl enable-linger <user>` keeps it running
open = false
EOF
bin/ekos setup --answers answers.toml

The OIDC client secret goes in the keychain where the server has one (oidc_client_secret in the answers), otherwise in the user unit's environment (EKOS_CONSOLE_OIDC_CLIENT_SECRET). Without a session secret the console keeps a generated one in control/session.key, readable by its owner only, so sessions survive a restart. Settings changed later on the console's Settings page apply after Restart now, which works here because ekos console run starts it again.

To remove a server installed this way, run bin/ekos uninstall on it (--purge --yes also deletes the home). The console's Settings → Uninstall EKOS page is offered in local mode only: it opens a page on the console's own machine, which a team's owner reaches from elsewhere.

People who want the ekos command on their own machine (for a project of their own, or for ekos mcp serve beside an editor) do not need the image: scripts/package-release.sh builds an offline install package for Linux x86_64 with the installer and an INSTALL.md; see the README's Installation section.

EKOS_CONSOLE_PRODUCTION=true makes the console refuse to start without OIDC, an owner and a session secret, turns the two development tokens off, and gives nobody access except through a granted role. Without it the console is a development setup: static tokens work, and every signed-in user can read the default tenant.

Put a reverse proxy with TLS in front of 127.0.0.1:8000. Do not change the port binding to 0.0.0.0.

The container must be able to reach the identity provider at the issuer URL, and that URL must be the same one browsers use. If sign-in answers "the identity provider could not be reached", check that from inside the container. With an https:// redirect URI the session cookie is marked Secure.

To try it without an identity provider, leave the OIDC and production lines out and set EKOS_CONSOLE_CONSOLE_TOKEN and EKOS_CONSOLE_CONSOLE_WRITE_TOKEN instead. The write token acts as a platform owner. Do not expose such a console.

Changing settings later

A platform owner can change the console's settings on its Settings page: sign-in, owners, MCP limits and the rest. They are saved in /var/lib/ekos/control/console.toml, in the volume. A variable set in deploy/.env wins over the page and is shown there read-only, so the deployment file stays the authority for what it sets. Settings apply when the console starts again; the page's Restart now works because the Compose restart policy starts it again.

Secrets (the OIDC client secret, the session secret) cannot be typed on that page in a container: they go to an OS keychain, and a container has none. Keep them in deploy/.env.

2. Tenants

Sign in as an owner and open Admin.

• Create a tenant for each team or customer. Its id is lowercase letters, digits, -, _.
• Set its source roots: the directories under /sources its projects may read, one per line. A tenant with none can create nothing. Only an owner sets them, so a tenant's admin cannot point a project at another team's repository.

The tenant default exists from the start. In production mode it follows the same rules as any other.

3. People and roles

Grant a role to an e-mail address in Admin → Roles. The person gets it at their first sign-in with that address.

Role	May
viewer	read everything in a project, including runs and logs
reviewer	also decide business semantics, conflicts and the source-model review, and sign it off (RFC 0213)
operator	also run the pipeline, manage schedules, edit the project config
admin	everything in the tenant: projects, members, groups, tokens, audit log

• A role is granted on the whole tenant or on one project. admin is tenant-wide.
• reviewer and operator are independent. Give both to someone who does both.
• A group holds a role for its members. Give it a claim value and everyone whose sign-in carries that value in the provider's group claim (EKOS_CONSOLE_OIDC_ROLE_CLAIM, default groups) is a member without being listed. Many providers (Dex, Keycloak, Okta) send that claim only when asked: set EKOS_CONSOLE_OIDC_SCOPES=openid email profile groups. Without it such a group stays empty and nothing says why.
• A project or tenant you have no role in looks exactly like one that does not exist.

Removing a role takes effect on that person's next request.

Sign-in by the platform: a public demo (RFC 0251)

On Azure Container Apps the platform can do the sign-in (GitHub, Google, …) instead of an OIDC issuer. Set platform_auth = "aca" (EKOS_CONSOLE_PLATFORM_AUTH) and turn on the app's authentication with allow unauthenticated requests: the console reads the person from the headers Container Apps sets (X-MS-CLIENT-PRINCIPAL-IDP / -NAME) and calls them <provider>:<name> — github:octocat, google:ann@example.com. Name owners the same way in EKOS_CONSOLE_OWNERS. The sign-in page offers platform_auth_providers (default github,google); API tokens keep working, so agents and Superset reach /mcp/… as before.

Only use it where the console is reachable only through that authentication: Container Apps strips those headers from outside requests, but another app in the same environment could send them. It cannot be combined with OIDC or local mode, and static tokens are refused.

To give everyone who signs in read access: create a group (Admin → Groups), bind viewer on the project to it, and put the group's id in signed_in_group. Every signed-in person is in that group; a token never is. Remove the binding to close the project again.

A read-only showcase (read_only = true, EKOS_CONSOLE_READ_ONLY): every principal — owners and API tokens included — may only read. Runs, schedules, review decisions, ekos.toml, the Admin and Settings pages are refused, and so are three pages that act with read rights alone: Commands (read-class CLI commands), Chat (a paid model) and ekos.toml. The UI hides them and shows "read-only". MCP reads still work, so a BI tool keeps answering; ekos_sql_generate keeps its own caps. To change the console's set-up, switch it off, make the change, and switch it on again.

4. Projects

A tenant admin adds a project in Add a workspace → New project from a source directory, with a path such as /sources/payments-api. Then an operator presses Init & compile in the project's bar (RFC 0189: build, recover, resolve, compile, commit and a coverage report as one run; identity conflicts are reported in the log and left for review), or runs single stages from the Run tab. Each stage of a chained run may take up to six hours (run_stage_timeout_seconds on the Settings page); a single command keeps a 30-minute limit.

A project of several repositories and folders is created from the command line:

docker compose -f deploy/docker-compose.yml exec ekos \
  ekos --tenant acme project add shop \
    --source api=/sources/payments-api --source web=/sources/storefront \
    --source handbook=/sources/handbook

It appears in the console after a restart. Naming the sources is what lets you limit access by source later, and a project compiled without names cannot gain them.

Sources and databases (RFC 0223)

A project's Sources tab lists its directories and databases. An operator adds a directory (it must lie under the tenant's source roots) or a database. Kind chooses Oracle, PostgreSQL, ClickHouse (RFC 0235), SQL Server or Informix (RFC 0252 §3.5), which sets the connect form: Oracle host:port/service, PostgreSQL host[:port]/database, ClickHouse http[s]://host[:port], SQL Server host[:port]/database (add ?trust-server-certificate=true only for a server whose certificate is self-signed; the connection is encrypted either way), Informix host[:port]/database@server (Client SDK) or drda://host[:port]/database (Data Server Driver). Then a read-only user, the password and the schemas to read (for ClickHouse, the databases; for Informix, the table owners) and the profile tier (p0 statistics, p1 row counts, p2 one aggregate scan per table, for every kind); for Oracle also, optionally, the iDempiere dictionary. The first directory added to a project created from one directory names that directory as a source too, so it keeps being read.

The password is kept in the server's keychain under a variable the console names (EKOS_DB_<TENANT>_<PROJECT>_<DATABASE>) and given to that tenant's processes always — unlike model keys, it does not depend on Allow metered model calls. ekos.toml holds the variable's name. Only a signed-in person can enter one; an API token cannot. A server without a keychain (a container) refuses with instructions: set the variable in deploy/.env, grant it in Admin → Tenants, and add the database with ekos database add … --password-env <VAR>. Removing a database (admin) deletes the password the console kept for it.

Test connection reads the catalog read-only (no profiling), checks that the server refuses a write, and puts the database's census next to what was compiled: tables, views, routines, triggers. PostgreSQL, ClickHouse and SQL Server need nothing installed on the server. SQL Server has no read-only session setting, so EKOS asks for read-only routing (ApplicationIntent=ReadOnly), runs every statement in a transaction it always rolls back, and reports the login's own rights (HAS_PERMS_BY_NAME) as the write check: give it a login that is only in db_datareader, plus VIEW DEFINITION for routine and view text. The Oracle driver needs Instant Client: set LD_LIBRARY_PATH (and TNS_ADMIN if you use one) in the console's environment; the console passes these to every ekos it starts.

Informix needs an ekos built with --features informix (the release packages and the image are not; it links unixODBC — apt install unixodbc, or unixodbc-dev to build) and an ODBC driver that speaks to Informix:

• the IBM Informix Client SDK (CSDK, from IBM; needs an IBM account): install it, set INFORMIXDIR in the console's environment, and EKOS loads $INFORMIXDIR/lib/cli/iclit09b.so (or name the driver with ?driver= in ekos.toml). Connect as host:9088/database@server, where server is the instance's INFORMIXSERVER name; add ?db-locale=en_US.utf8 for a UTF-8 database;
• or the IBM Data Server Driver for ODBC and CLI (a free download, no account: linuxx64_odbc_cli.tar.gz from IBM's public driver site), over DRDA: connect as drda://host:9089/database?driver=<clidriver>/lib/libdb2.so. The server must listen for DRDA (drsoctcp in sqlhosts). This driver's libdb2.so uses a 32-bit SQLLEN under unixODBC; EKOS reads it one row at a time so that it still reads correctly.

The ?driver= option names a shared library to load, so the console's form does not accept it: add such a database with ekos database add on the server. Informix signs users in through the operating system: give EKOS an OS user with CONNECT on the database and SELECT on the tables to read — and revoke the rights a database without ANSI mode grants PUBLIC on every new table, or the write check reports the user as able to write.

5. Limits inside a project

A viewer can be limited to part of one project. In Admin → Roles, grant viewer on one workspace and choose:

• only… a source, or a path pattern in it (src/**);
• all except… a source or a path pattern (secrets/**).

Know what this does before relying on it:

• A limited person gets the graph, search, object pages and impact analysis, filtered. The dashboard, run history and logs, the configuration, semantics and conflicts are closed to them, because those show more than the ledger.
• Anything the compiler could not tie to a source is hidden from them: commit authors, crates, rollups, business-semantics items, conflicts.
• Operators and admins cannot be limited. Running the pipeline shows every source.
• A rule hides objects. It does not rewrite a visible document that mentions a hidden one.

6. Agents

Create an API token in Admin → API tokens. It is shown once. A token is viewer or operator, on the tenant or one project, and can carry the same limits as a person. It can never review or administer.

An MCP client connects to the project through the console:

{"url": "https://ekos.example.com/mcp/acme/shop",
 "headers": {"Authorization": "Bearer ekos_…"}}

The agent sees the tools its role allows, limited to what its rules let it see. Every call is in the tenant's audit log: who, which tool, allowed or denied. Arguments are not recorded.

A project's MCP server starts on first use and stops after 15 idle minutes; at most 8 run at once (EKOS_CONSOLE_MCP_IDLE_SECONDS, EKOS_CONSOLE_MCP_MAX_SERVERS). Each holds its project's index in memory, so size the cap to the machine.

SQL written by EKOS for an agent (RFC 0250)

ekos_sql_generate runs EKOS's own text-to-SQL agent (the one measured on BIRD) for an MCP client — the EKOS Superset fork uses it. It calls a model, so it is off until the project says so:

[sql-generate]
mcp = true
allow-metered = true   # only for a provider other than a local Ollama model
daily-calls = 50       # the whole project, per UTC day; over it the tool answers `limit`
daily-tokens = 200000

Through the console it also needs Allow metered model calls on the tenant, and each caller has a daily allowance: mcp_metered_per_person_per_day (default 5) for a signed-in person, mcp_metered_per_token_per_day (default 50) for an API token, which stands for a service that limits its own users (Settings → Public demo). The console chat never offers this tool. With one [[databases]] entry the agent probes that database read-only; it never runs the answer.

Extension pages

A console started by an extension's binary (ekos-lakehouse console start, or EKOS_BIN pointing at it) shows that extension's pages as extra workspace tabs (RFC 0198). A page is data the binary declares: panels from its read commands, buttons that run its catalogue commands with their usual class, role and confirmation. A restricted reader does not see these tabs. A job a button starts has its tenant's environment, like every other job: a secret it needs (a source database's password) is granted to the tenant below, or kept in an owner-only file the project names — EKOS Lakehouse's installer does the latter (its docs).

7. Model credentials

Some pipeline passes call a language model. A project's ekos.toml names the environment variable holding the key; it never holds the key.

1. Put the keys in deploy/.env, one variable per tenant that pays for its own:

   ACME_ANTHROPIC_KEY=sk-ant-...
   GLOBEX_OPENAI_KEY=sk-...

2. In Admin → Tenants, under Model credentials, list the variable names for each tenant and tick Allow metered model calls.

A tenant's pipeline and MCP server receive its own named variables and no other tenant's, and none of the console's settings. With the box unticked they receive none, so a hosted provider fails for want of a key. A local model server still works: point ekos.toml at http://host.docker.internal:11434.

Calls to a hosted model are billed by that provider. A project can cap what one command may spend with [llm] budget-usd (RFC 0197: a call that would pass the budget, or whose price is unknown, is refused), and ekos llm usage reports what was spent; the console itself sets no cap.

Outside production mode the default tenant keeps the console's whole environment (minus the console's own settings and other tenants' named variables), as before this existed.

The chat model

The console's Chat tab (RFC 0187) uses one model, set by an owner in Settings → Chat: chat_provider (ollama, openai, anthropic), chat_model, chat_base_url and chat_api_key_env, the name of the variable holding the key. In a container put the key itself in deploy/.env under that name. A hosted model is paid per call: a tenant's people can chat with it only while that tenant's metered calls are on (Admin → Tenants), the same switch as for its projects' model calls. A model on the server itself (Ollama, a local OpenAI-compatible server) always may.

Public-demo safety (RFC 0249)

A console that serves strangers (the public demo) screens every question before it is answered. It is off by default: moderation_enabled = true in Settings → Public demo turns on POST /api/moderation/screen, GET /api/moderation/limits and, for the platform owner, GET /api/moderation/events, GET /api/moderation/restricted and POST /api/moderation/unban.

• Order of checks: the address rate, the person's standing (banned, blocked), the address block, the daily question limit, deterministic signals, then — only for what they did not catch — a model. A hard signal (overriding the rules, probing for a key, hidden text, a forbidden schema) bans at once and blocks the address for 24 hours; a soft one (deleting or overloading) is refused, and the third in a row bans. A request only the model judged unsafe blocks the person for 24 hours; a second one bans. A model that cannot be reached refuses without penalty and never allows.
• moderation_model names the intent-check model (empty: deterministic checks only); moderation_key_env the variable holding its key (default OPENCODE_API_KEY); moderation_trusted_hops how many proxies in front of the console are trusted for X-Forwarded-For (0: the socket peer — behind Container Apps set it to the number of proxies); moderation_limits the published numbers as JSON, e.g. {"questions_per_day": 30, "block_hours": 24} (an unknown name is an error).
• What is sent to the model: the one message, as an escaped JSON string in a delimited block. No row values, no schema, no earlier answers. That text goes to the model's provider, and for Claude models on OpenCode Zen the provider keeps requests for 30 days (OpenCode's data policy): the rules page must say so.
• The log keeps a 200-character escaped excerpt and a SHA-256 of each refused message. How long it is kept is a data-protection decision that is not made yet.
• web/api/tools/redteam_live.py measures a model on the red-team corpus (it spends tokens; it is never run by the tests).

Chat channels: Telegram, Slack, Teams (RFC 0253)

People who know what a code or a table means can answer EKOS where they already are. A bot in Telegram, Slack or Microsoft Teams answers questions with the person's own access, takes files and links into the project's inbox, records what they tell it (only when they press Save, in their own words) and delivers the open questions they were picked to answer (RFC 0254).

The service starts only when a channel is configured. Each secret is the name of a variable, set in Settings → Channels; its value goes in the environment (deploy/.env) or the server's keychain under that name (the Settings page's variables; a container has no keychain, so there they go in deploy/.env), never in console.toml:

Channel	Settings	Transport
Telegram	channels_telegram_token_env (the BotFather token)	long polling: no inbound endpoint, works in local mode
Slack	channels_slack_bot_token_env (xoxb-…), and channels_slack_app_token_env (xapp-…, Socket Mode) or channels_slack_signing_secret_env (Events API)	Socket Mode needs no inbound endpoint; the Events API posts to /api/channels/slack/events and /api/channels/slack/interactivity, each request checked against the signing secret (and refused when older than five minutes)
Microsoft Teams	channels_teams_app_id, channels_teams_app_secret_env	Bot Framework posts to /api/channels/teams/messages (an HTTPS endpoint the bot's registration names); every request's JWT is checked against the Bot Framework's published keys, issuer, audience = the app id, expiry and service URL

Slack scopes: app_mentions:read, chat:write, im:history, im:read, im:write, channels:read/groups:read (members of a channel, for the group rule below) and files:read. In local mode the console takes loopback connections only, so use Telegram or Slack Socket Mode.

• Linking. An account the console does not know gets the link instructions and nothing else. The person writes /ekos link to the bot, which answers with a one-time code (valid channels_link_code_minutes, only its hash is kept); they enter it on the console's Channels page while signed in. From then on the bot acts with that person's roles and read scope. Signed-in people only: a development token cannot be linked. Claims from the sign-in provider (OIDC groups by claim) do not travel to a channel; role bindings to the user or a named group do.
• Projects. An owner binds a Slack or Teams channel, or a Telegram group, to one project on the Channels page (POST /api/channels/bindings). A direct message uses the person's own choice, /ekos project <name>.
• Group chats. An answer that read the ledger goes only to the asker (a Telegram direct message, a Slack ephemeral message, a Teams personal chat) unless every member of the channel is a linked person who sees the whole project. Telegram cannot list a group's members, so there it is always private.
• Material. A file or a lone link goes to the project's inbox as the sender's (ekos inbox add --as); from someone without the reviewer role it waits as pending (Inbox tab). A public deployment (moderation_enabled or read_only) takes no material. Largest file: inbox_upload_max_mb (the project's [inbox] max-file-mb also applies).
• Testimony. /ekos tell <what you know> asks the chat model (read tools only) what the statement is about; it answers with a card of proposals, each quoting the person's words — a proposal without such a quote is dropped. Save records the person's own text with ekos testimony add --as <them>; a button press counts only from the account the card was sent to, and recording needs the reviewer role. /ekos edit <card> <text> corrects the words first. The chat model is the console's (chat_provider), with the same metering rule as the Chat tab.
• Questions digest. channels_digest_cron (cron, UTC) sends each person their open questions (ekos questions digest, at most [questions] per-person-per-day) to the channel they wrote from last, each with Answer and Not me. Owners can send it now from the Channels page. Nothing is sent from a public deployment.
• Erasure. Purging an inbox item deletes the original file; the compiled, redacted text stays in the ledger, which cannot forget. Only inbox originals are erasable.

8. Backup and restore

Everything is in the volume. A backup is one file.

# every tenant, and the console's database
docker compose -f deploy/docker-compose.yml exec ekos ekos home backup
# → /var/lib/ekos/backups/ekos-backup-<time>.tar.zst

# one tenant's projects only
docker compose -f deploy/docker-compose.yml exec ekos ekos --tenant acme home backup

Copy the file off the machine: it is written inside the volume it backs up.

docker compose -f deploy/docker-compose.yml cp \
  ekos:/var/lib/ekos/backups/ekos-backup-20261013T101500Z.tar.zst /srv/backups/

• A project is copied under its ledger's write lock. If a pipeline stage is writing, the backup stops with that project's name and writes nothing; run it again afterwards. Reads, including agents' queries, are not interrupted.
• The console database is copied as a consistent snapshot while the console runs.
• Not in a backup: your sources (they are not in the volume), the binary, the trash.
• The archive is not encrypted. It holds the ledgers and the console database (token hashes, not tokens). Protect it as you would the volume.
• On one server there is no schedule: run it from the host's cron. In a cloud (section 10) the console backs up to object storage on a schedule.

The image can also write a backup straight to object storage and restore from there (RFC 0232), with the machine's cloud identity; no key is ever on the command line:

ekos home backup --to s3://my-bucket/ekos/          # or gs://…, https://<account>.blob.core.windows.net/<container>/ekos/
ekos home restore --from s3://my-bucket/ekos/ --force   # the newest backup there
ekos home restore --from s3://my-bucket/ekos/ekos-backup-20261030T021700Z.tar.zst --force

To restore onto a new machine or a new volume, start the deployment first, then:

docker compose -f deploy/docker-compose.yml cp /srv/backups/ekos-backup-….tar.zst ekos:/tmp/b.tar.zst
docker compose -f deploy/docker-compose.yml exec ekos ekos home restore /tmp/b.tar.zst --force
docker compose -f deploy/docker-compose.yml restart

--force is needed even on a new volume: the console created its own empty database when it started, and a full backup holds a console database. The empty one is moved to trash/. Restart afterwards; until then the console is still using the database it started with.

• Without --force nothing that exists is replaced. If any project in the archive, or the console database, is already there, the restore stops and changes nothing.
• --force replaces: each existing project, and the console database, is first moved to trash/ in the volume. Nothing is deleted.
• --tenant acme restores that tenant's projects only. Role bindings are in the console database, which a per-tenant archive does not include.
• A project's config holds the path of its sources. Restore onto a machine where the sources are mounted at the same path, or edit the project's ekos.toml.

9. Upgrading

docker compose -f deploy/docker-compose.yml exec ekos ekos home backup
git pull                                   # or change the image tag
docker compose -f deploy/docker-compose.yml up -d --build

• The console upgrades its own database at start. Take the backup first: there is no downgrade.
• A ledger is append-only and is read by newer versions. Some upgrades change what is indexed and rebuild the search index at the first write; the release notes say when.
• From RFC 0229 (tantivy 0.27) the search index is written in a newer format. A newer EKOS reads an older index as it is; an older EKOS cannot read an index a newer one wrote, and runs without search (it says so) until the index is rebuilt. Restore the backup to go back.
• Re-running the pipeline is never needed for an upgrade. It is how new compiler features reach an existing project, and for a project with model-backed passes it costs money.

10. In a cloud: Azure, Google Cloud, AWS (RFC 0232)

deploy/terraform deploys the same image to any of the three clouds, three ways each:

compute	Azure	Google Cloud	AWS
vm	VM + managed disk, Caddy	Compute Engine + persistent disk, Caddy	EC2 + EBS, Caddy
container	Container Apps + Azure Files NFS	Cloud Run + Filestore	ECS Fargate + ALB + EFS
kubernetes	AKS + Helm chart	GKE Autopilot + Helm chart	EKS Auto Mode + Helm chart

cd deploy/terraform/examples/azure        # or gcp, aws
cp terraform.tfvars.example terraform.tfvars   # compute, OIDC, owners, repositories
export TF_VAR_secrets='{"OPENAI_API_KEY":"sk-…"}' TF_VAR_git_token=… TF_VAR_oidc_client_secret=…
terraform init && terraform apply
terraform output url

A key a person adds to Key Vault themselves (Azure, compute = "container"). TF_VAR_secrets puts a value through Terraform, so it lands in the state. To keep a key out of it, add the secret to the deployment's vault by hand and name it in external_secrets: console variable → Key Vault secret name, e.g. external_secrets = { ANTHROPIC_API_KEY = "anthropic-api-key" }. The app reads it through its managed identity (versionless: a new value is picked up by the next revision). Add the secret before the apply that names it — a revision whose reference cannot be resolved does not start:

read -rs KEY && az keyvault secret set --vault-name "$(terraform output -json key_vault | jq -r .name)" \
  --name anthropic-api-key --value "$KEY" -o none; unset KEY

The image. The modules, the examples and the chart default to the latest release, ghcr.io/alexeyban/ekos:1.1.0. That release predates this section: its image has no ekos-entrypoint and no ekos-git-sync (the git sync container does not start), no ekos home backup --to (scheduled backups fail and nothing is restored on first start), no instance lease, and no secrets read from mounted files (Kubernetes, VM). Until a release contains RFC 0232, build the image from main (docker build -t <registry>/ekos:<tag> .), push it to your registry and set image. On Azure, az acr build builds it in the registry without a local Docker, and the example's acr_id (the registry's resource id) lets the deployment pull it.

What each deployment creates and how it behaves:

• Secrets go to Key Vault / Secret Manager / Secrets Manager, never into the image, a file in the repository or the VM's disk. The platform injects them (Container Apps, Cloud Run, ECS) or mounts them as files the image's entrypoint turns into variables (Kubernetes CSI, the VM's boot script). Rotate one in the store and restart the console.

• One identity (managed identity, service account, IAM role; workload identity on Kubernetes) may read those secrets and write ekos/ in the backup bucket. Nothing else.

• Sign-in: give oidc and owners and the console starts in production mode. The redirect URI to register with the provider is <url>/api/auth/callback. Without OIDC the console uses two generated tokens: terraform output -json static_tokens.

• Sources: git_repos = ["shop=https://github.com/acme/shop.git#main"] runs the git sync beside the console; it clones into /sources/shop and keeps it at the branch's tip every five minutes. Create the project for /sources/shop in the console as usual.

• Backups run every night (backup_cron, UTC) to the deployment's bucket and expire after backup_retention_days. terraform output backup_url is what ekos home restore --from takes.

• A new deployment over an existing bucket restores itself. When the console starts over an empty home (no project, no console database) and backup_url is set, it restores the newest backup there before it opens its database, and logs the home was empty; restored …. An empty location means a first deployment, and the console starts empty. Any other failure — a wrong bucket, missing credentials — stops the start: a console that cannot read its backups must not start empty, because its first nightly backup would become the newest one. The restore has to fit the platform's start allowance (Cloud Run 240 s, Container Apps 10 min, ECS 5 min, Kubernetes startupSeconds, 30 min by default; a VM has none). For a larger home, set EKOS_CONSOLE_BACKUP_RESTORE_ON_EMPTY=false and run ekos home restore --from <backup_url> once by hand.

• One console at a time. Container platforms briefly run a new revision beside the old one; the console's instance lease (EKOS_CONSOLE_INSTANCE_LEASE) makes the new one take over and the old one stop. A platform that restarts the stopped console (Container Apps does) finds it waiting while the new one is alive, instead of taking the lease back; consoles older than 21230c3 did take it back, so the first rollout onto a newer image may need the old revision deactivated by hand. Never scale the console beyond one replica.

• Kubernetes without a domain is reached with kubectl -n ekos port-forward svc/ekos 8000:80. With a domain:

  • GKE: a reserved global address and a Google-managed certificate. Point the domain's A record at terraform output ingress_ip; the certificate is issued once it resolves (up to an hour).
  • EKS: give certificate_arn (an ACM certificate for the domain) and the module creates an IngressClass for Auto Mode's built-in ALB. Point the domain at the address kubectl -n ekos get ingress shows.
  • AKS: no preset. The application routing add-on's NGINX is supported only until November 2026 and its Gateway API successor cannot be enabled from the azurerm provider yet, so give the cluster's ingress class and TLS in helm_values (for example ingress: {className: …, tlsSecretName: …}); Terraform refuses a domain without them.

  helm_values always apply over the module's own values, so they can replace a preset. The chart (deploy/helm/ekos) can be installed on any cluster on its own.

• exposure = "private" keeps the console off the internet: Container Apps gets an internal environment and a private DNS zone for its domain in the VNet, Cloud Run takes VPC traffic only, ECS gets an internal load balancer, EKS an internal ALB, GKE the internal Application Load Balancer (with a proxy-only subnet, proxy_only_cidr; give ingress.tlsSecretName in helm_values, as Google-managed certificates are external only). On AKS the module adds no ingress, so the one in helm_values decides. Reaching it — a VPN, peering, a bastion — is yours. A VM cannot be private (Terraform refuses it), and Key Vault, Secret Manager, Secrets Manager and the backup buckets keep their public endpoints, restricted to the deployment's identity: with private endpoints, Terraform run from outside the network could no longer write the secrets.

What it costs. List prices for the module defaults in the examples' regions (Azure westeurope, Google Cloud europe-west3, AWS eu-central-1), on demand, one console all month, light traffic, 10 GB of backups; checked on 2026-10-31. Estimates, not quotes:

compute	Azure	Google Cloud	AWS
vm	~$205 (D4s_v5)	~$150 (e2-standard-4)	~$200 (m7i.xlarge)
container	$115–280 (Container Apps active vs idle time; premium NFS 228 GiB)	$330–375 (Filestore's 1 TiB minimum + an always-on 2 vCPU instance)	~$145 (Fargate 2 vCPU / 8 GiB, ALB, EFS by GB stored)
kubernetes	~$235 (one D4s_v5 node, Free tier control plane)	$145–155 (cluster fee + Autopilot Pod requests)	$180–195 (control plane fee + one Auto Mode node)

• The git sync adds about $60 a month on Cloud Run (its 1 vCPU is always allocated) and $8–28 on Container Apps; nothing on a VM, Fargate or a node with room.
• An AKS Standard tier (uptime SLA) adds $73. Every public IPv4 address on AWS costs about $4.
• Not included: internet egress, model API calls, DNS, taxes and support.
• Not confirmed from an official page: Cloud Run's Tier 2 rate and Filestore's price in Frankfurt, Autopilot's Frankfurt rates (scaled from us-central1), which instance EKS Auto Mode picks. Check the providers' calculators before quoting a figure.

What has been checked. terraform validate with the real providers; terraform test plans every module with mocked providers (which resources each compute and exposure creates, the presets, the platform limits on start probes); helm lint and helm template with each preset; the image's backup to and restore from S3-compatible storage, git sync, entrypoint and first-start restore against MinIO. Applied for real once: Azure, compute = "vm" (2026-10-31, through Azure Pipelines with a federated service connection and a private ACR image): HTTPS from Let's Encrypt, Key Vault secrets, git sync, a project compiled from the console, a backup to Blob Storage and a restore on first start all worked; then everything was destroyed. Google Cloud, AWS and Azure kubernetes have not been applied yet. Azure container (Container Apps) was applied the same day: the rollout lease, ignore-patterns for file names, a 504 for a slow knowledge server and the environment's logs_destination were fixed from what it showed (devlog_316).

• A private registry on Azure: set acr_id to the registry's resource id. The deployment's identity gets AcrPull (and AKS's kubelet); the VM signs docker in with that identity at every start, Container Apps uses its registry identity. No registry password exists.
• On a VM, the home disk's size is its speed: a 32 GiB premium disk is a P4 (120 IOPS). The first Azure run's 142k-object commit took 48 minutes on a network disk even at P20, because each synchronous write waits on the disk's latency: keep home_size_gb at 128 or more, and exclude data dumps (one INSERT per row) from a project's sources.
• On Azure Files NFS (Container Apps), a commit costs ~100 ms per object: 142k objects took 3½ hours, 3k objects 5 minutes. Name generated SQL and data dumps in [observe] ignore-patterns (exact file or directory names, not globs).

What is not built

• One console. It is a single writer over its home, on one machine, VM, container or pod (section 10). No PostgreSQL; the home is a disk or an NFS share, not object storage.
• Isolation between tenants is by directory and permission check, inside one process and one OS user. It is not a sandbox.
• No TLS in the container (the cloud deployments put it in front), no OAuth for MCP clients, no rate limits, no spending limits.
• No encrypted backups beyond the bucket's own encryption at rest, and no deletion of a tenant or a user. Scheduled backups exist only with backup_url (the cloud deployments set it).
• Limits inside a project leave a limited reader without a dashboard, semantics or conflicts, and cannot apply to operators.
• Sources are local directories. The console does not clone; mount a checkout, or run the git sync beside it (ekos-git-sync, section 10).
• Evaluation runs (ekos eval run) write beside the sources and do not work for a project kept in the home.
• The image is linux/amd64.
• Channels (RFC 0253): the adapters are tested against the platforms' documented request and response shapes, not yet against live Telegram, Slack or Teams workspaces. No voice messages; no Slack modal or Teams task module for Edit (a /ekos edit message instead); no automatic Teams linking by AAD object id; the Azure module does not create the Bot resource yet.

Where things are

The seven-step operator workflow (sources → review → requirements → extraction → bronze, silver and gold) is described in operator-workflow.md.

A project	/var/lib/ekos/data/<tenant>/<project>/ (ekos.toml, state/)
Console database, run logs	/var/lib/ekos/control/
Removed or replaced projects	/var/lib/ekos/trash/
Backups	/var/lib/ekos/backups/
Settings	deploy/.env; every one is described in deploy/.env.example
Design	RFCs 0178–0183 in ekos/docs/rfcs/
An offline install package of the CLI	scripts/package-release.sh → dist/ekos-<version>-release.tar.gz
Cloud deployments	deploy/terraform (modules and examples for Azure, Google Cloud, AWS), deploy/helm/ekos; RFC 0232
A check of a whole deployment	deploy/e2e/run.sh: the image, a test identity provider, two tenants, a group from a claim, an MCP client, the pages in a browser, backup and restore