| title | Cloud archives |
|---|---|
| nav_order | 14 |
| permalink | /cloud-archives/ |
{: .no_toc }
Use a separately deployed Worker service as an authenticated read and snapshot-publication boundary for a Gitcrawl archive. {: .fs-6 .fw-300 }
- TOC {:toc}
Gitcrawl stores the Worker endpoint and archive identifier in its config and calls the remote service. The service itself is deployed separately from Gitcrawl.
Cloud mode does not replace the local or Git-backed portable-store workflows. A cloud-mode configuration sends supported read commands, including status and direct search, to the remote archive without creating a local SQLite database.
gitcrawl init \
--remote URL \
--archive gitcrawl/openclaw__openclawBearer-authenticated remote endpoints must use HTTPS. Plain HTTP is accepted only for loopback development endpoints.
gitcrawl remote login --endpoint URL --json
gitcrawl whoami --json
gitcrawl remote archives --json
gitcrawl remote status --jsonremote login starts the service's GitHub OAuth flow, verifies organization and team membership server-side, and stores the returned signed bearer token in the operating-system keyring.
For non-browser bootstrap, name an environment variable that contains a GitHub token:
gitcrawl remote login \
--endpoint URL \
--github-token-env GITHUB_TOKEN \
--jsonThe Worker verifies the GitHub token against the same organization and team policy. Gitcrawl stores only the remote session token.
An Access-protected endpoint requires two credentials: the human Access application token for the gateway, and a reader session for the archive service. A browser's Access cookie is not automatically available to the CLI.
Use Cloudflare's supported CLI login with your existing permitted identity. In a POSIX shell:
cloudflared access login https://archive.example.invalid
export CRAWL_ACCESS_TOKEN="$(cloudflared access token --app=https://archive.example.invalid)"
gitcrawl --config reader.toml remote login \
--endpoint https://archive.example.invalid \
--access-token-env CRAWL_ACCESS_TOKEN --json
gitcrawl --config reader.toml whoami --jsonConfigure the intended archive in that reader config before login. The remote must
support POST /v1/auth/access/session. Gitcrawl verifies the exchanged identity and
its exact reader-only role before saving the Worker session to a separate keyring
account bound to the endpoint, archive, and config path. Legacy reader keyring entries remain
unchanged.
Gitcrawl saves only the Access environment variable's name and endpoint binding
under [cloudflare_access]. Keep the variable available to later reader commands
and renew it through cloudflared when its session expires. Gitcrawl sends the
Access token only within the configured API endpoint and rejects redirects,
changed endpoint/archive bindings, and service credentials. Never print tokens or
put them in command arguments or config files. This login does not grant publisher
roles or change the deployment's Access policies.
After cloud initialization and login, supported read commands use the configured archive:
gitcrawl status --json
gitcrawl search openclaw/openclaw --query "manifest cache" --json
gitcrawl threads openclaw/openclaw --numbers 42,73 --include-closed --jsonthreads requires the service's gitcrawl.threads.list query. It preserves full
native fields, --numbers, --include-closed (local closure), and --limit.
Pages and split selections share one completed publication pin. With no limit,
it returns all matching rows; backend errors fail without a local fallback.
Commands that require a writable local database fail in cloud mode instead of silently creating one.
Publication starts from a local SQLite archive:
gitcrawl cloud publish \
--remote URL \
--archive gitcrawl/openclaw__openclaw \
--jsonBefore any upload or ingest, Gitcrawl checks the configured credential through the advertised /v1/whoami route and requires both publisher and reader roles. This also applies to stage-only publication.
The publisher freezes one local SQLite image and uses its SHA-256 digest as the snapshot identity. Repositories, threads, revisions, fingerprints, summaries, durable clusters, and pull request detail/file rows are exported from that same image. Gitcrawl negotiates the remote snapshot contract before changing R2, uploads a digest-scoped gzip bundle, and sends D1 data in row- and encoded-byte-bounded batches.
Gzip is lossless transport, not portable-store compaction. The cloud SQLite image retains complete canonical issue and pull request bodies, comments, revisions, review comments, and pull request patch text.
The remote must advertise gitcrawl.snapshot.staging.v1. Pass --stage-only to upload and validate an immutable snapshot without changing the archive served to unpinned readers:
gitcrawl cloud publish \
--remote URL \
--archive gitcrawl/openclaw__openclaw \
--stage-only \
--jsonA later publish verifies the candidate through the publisher-only status projection. It skips repeated ingest only when the digest, source sync, schema, resolved publication profile, immutable warnings, generation timestamp, and coverage match.
Cutover requires reader-authenticated GET /sqlite. Gitcrawl validates the cutover acknowledgement, polls the scoped reader projection until its digest, profile, generation, and dataset coverage match, rechecks the publisher metadata, downloads the bound SQLite image, and verifies its hash before reporting success. Without --stage-only, a successful publish moves unpinned reads to the complete snapshot.
Incomplete local enrichment fails before remote mutation. --allow-incomplete is the explicit override. --observation-order publishes durable fetch ordering only after the remote operator fence is enabled.
To publish a raw archive with explicitly reported source and enrichment gaps:
gitcrawl cloud publish \
--remote URL \
--archive gitcrawl/openclaw__openclaw \
--admission-policy=archive-v1 \
--observation-order \
--stage-only \
--jsonThis opt-in policy requires the remote capabilities
gitcrawl.archive-admission.v1 and gitcrawl.observation-order.v1.
An older remote or a disabled observation fence fails preflight before upload.
Do not combine the policy with --allow-incomplete. Omitting the policy retains
the strict default.
Archive admission still requires SQLite integrity, compatible canonical tables, repositories, referential closure, full bodies, and the existing privacy scrub. Use the full runtime archive, not a lossy portable export. Native portable profiles that declare excerpts or excluded patches/history are rejected even when current bodies fit the excerpt limit. Empty patch rows in a full runtime are not proof of loss. The existing 4 GiB SQLite, 512 MiB gzip, eight-part, staging, and cutover limits still apply. Admission does not request summaries, embeddings, or other model work.
The JSON result includes typed admission evidence and stable warnings.
Repository inventory observations distinguish unsupported, unknown, missing,
partial, complete, and empty observations. Child observations also report stale
reservations relative to the archived thread. Complete observations describe
the recorded scope, not current GitHub state. Workflow freshness and current
remote freshness remain unknown; export clocks never establish either.
Source gaps, PR detail/file gaps, and all six enrichment metrics remain visible.
Incomplete current revisions retain their actual coverage counts and
complete: false; admission does not make stale enrichment current for readers.
The sanitized SQLite copy stores the same deterministic evidence in
portable_metadata.cloud_admission_v1 before hashing. Existing source markers
are not trusted, and strict exports remove them. The original database is not
changed. Canonically sorted warnings are part of immutable snapshot identity,
including stage-only replay and reader verification. Warnings alone never
enable admission.
Digest-scoped bundles can contain private issue and pull request text. Bundle metadata declares both message-body and source-code sensitivity because patch text is retained.
Publication excludes raw API payloads, blob-backed payloads, local run diagnostics, source-code indexes, and machine-local paths. These are not part of the shared archive contract.
Gitcrawl intentionally has no remote deletion command. Operators should enable publication only when the remote deployment has bounded lifecycle rules for failed, superseded, and uncut staged bundles. --stage-only does not move retention responsibility back to the client.
| Command | Purpose |
|---|---|
gitcrawl init --remote URL --archive id |
Configure cloud mode |
gitcrawl remote login |
Authenticate and store a remote session token |
gitcrawl whoami |
Print the current remote identity |
gitcrawl remote archives |
List archives visible to that identity |
gitcrawl remote status |
Inspect the configured archive |
gitcrawl cloud publish |
Publish a local snapshot to the Worker remote |