Document RPC BACKFILL flag and refresh sample config - #2789
Conversation
The sample config in configuring.mdx predates two flags. Add both, in alphabetical order, with the usage text from the RPC source. - BACKFILL = false - SERVE_LEDGERS_FROM_DATASTORE = false Add a section for BACKFILL. It gives the version, the default, the datastore prerequisite, and the retention window that sets the size. It links to Data Lake Integration for datastore setup. Partially addresses #2602
There was a problem hiding this comment.
Pull request overview
Documents RPC backfill configuration and startup behavior.
Changes:
- Adds
BACKFILLand datastore-serving flags. - Adds startup backfill prerequisites, defaults, and datastore guidance.
Recommendation: NEEDS-CHANGES — correct the incomplete generated-config claim, avoid guaranteeing exact backfill coverage, and label the code fence.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| # Populates database with `history-retention-window` ledgers synchronously on | ||
| # startup. This defaults to a week of ledgers if unspecified | ||
| BACKFILL = false |
There was a problem hiding this comment.
Correct on the facts. I checked v27.1.1 options.go and counted the TOML keys the generator emits.
The generator emits 66 keys. The block on main has 57. Nine are missing:
BACKFILLSERVE_LEDGERS_FROM_DATASTORENETWORKSTELLAR_CAPTIVE_CORE_HTTP_QUERY_PORTSTELLAR_CAPTIVE_CORE_HTTP_QUERY_THREAD_POOL_SIZESTELLAR_CAPTIVE_CORE_HTTP_QUERY_SNAPSHOT_LEDGERSbuffered_storage_backend_configdatastore_configingest_load_test_config
Two of those are not in your list: NETWORK and ingest_load_test_config.
No key in the block is absent from the generator, so nothing here is stale or invented. The block is only incomplete.
This PR adds the first two keys. Seven stay missing. Filling them by hand risks wrong defaults, so it needs a real gen-config-file run. The PR description already flags that as a separate pass.
|
|
||
| Set `BACKFILL = true` to populate the database with a trailing window of history synchronously on startup, before live ingestion begins. This option was added in RPC v25.1.0 and defaults to `false`. | ||
|
|
||
| `HISTORY_RETENTION_WINDOW` sets how many ledgers the node populates. The default is `120960` ledgers, which is about 7 days. |
There was a problem hiding this comment.
Correct. I read cmd/stellar-rpc/internal/ingest/backfill.go at v27.1.1. The window is a target, not a promise.
Evidence:
setBoundscaps the count:nBackfill = min(retentionWindow, currentTipLedger).- The start ledger is clamped to the oldest ledger in the datastore:
max(currentTipLedger-nBackfill+1, dsInfo.sequences.First). - Existing local rows shrink the work. The backward phase is skipped when the local DB tail already covers the left edge.
- The forward fill stops at a checkpoint:
bounds.frontfill.Last = PrevCheckpoint(currentTipLedger). verifyBoundsonly warns when the result is short by more thanledgerThreshold(384 ledgers). It returnsnil.
The source agrees. RunBackfill says it checks that "retention window requirements are (at least approximately) met".
I will reword the sentence to name it a target window, not an exact count.
|
Preview is available here: |
Verified the sample block against the real generator. Built the gen-config-file body from stellar-rpc v28.0.0 and diffed it. - comment out BACKFILL and SERVE_LEDGERS_FROM_DATASTORE, because the generator comments out zero-value defaults - add NETWORK - add STELLAR_CAPTIVE_CORE_HTTP_QUERY_PORT and STELLAR_CAPTIVE_CORE_HTTP_QUERY_SNAPSHOT_LEDGERS - correct MAX_GET_LEDGERS_EXECUTION_DURATION from 5s to 10s Every environment-independent key now matches the generator exactly.
|
Preview is available here: |
Copilot flagged the retention sentence as an over-promise. It is right. Verified in internal/ingest/backfill.go at v27.1.1 and v28.0.0: - nBackfill = min(retentionWindow, currentTipLedger) - start clamped by dsInfo.sequences.First - forward fill ends at PrevCheckpoint(currentTipLedger) - ledgers already in the local database are not refetched Upstream's own comment says the window is "at least approximately" met. So call the value a target and name the three limits. Also label the error code fence as text.
|
Pushed fixes for two of the three review comments. Retention wording. The over-promise flag was correct. I checked
Upstream's own comment says the window is met "at least approximately". The section now calls the value a target and names the three limits. I stopped short of documenting exact thresholds or the failure path. @ElliotFriend and @kalepail reserved coverage boundaries for RPC-team review, so that wording should come from them. Code fence. Labelled as Sample config completeness. I regenerated the block instead of reasoning about it. I built the Every environment-independent key now matches the generator exactly, in value and comment state. That pass also caught two things: my two new keys needed commenting out, and Four keys still differ, and they cannot be fixed here. Their defaults depend on the host: defaultStellarCoreBinaryPath, _ := exec.LookPath("stellar-core")
defaultUintCPU := uint(runtime.NumCPU())
defaultUint16CPU := uint16(runtime.NumCPU())The block currently holds container values, such as |
|
Preview is available here: |
Addresses the mechanical half of #2602, as scoped by @ElliotFriend and @kalepail in the issue thread.
What changed
configuring.mdxtells readers the TOML block is the output ofgen-config-file. The block had drifted. I verified it against the real generator and corrected it.Added flags:
# BACKFILL = false# SERVE_LEDGERS_FROM_DATASTORE = false# NETWORK = ""STELLAR_CAPTIVE_CORE_HTTP_QUERY_PORT = 11628STELLAR_CAPTIVE_CORE_HTTP_QUERY_SNAPSHOT_LEDGERS = 4Corrected value:
MAX_GET_LEDGERS_EXECUTION_DURATIONfrom"5s"to"10s"(source:DefaultValue: 10 * time.Second)New section Backfilling History on Startup gives the version, the default, the datastore prerequisite, and the retention window that sets the size. It links to Data Lake Integration instead of restating it, per @kalepail's note.
How I verified
I could not run
docker run stellar/stellar-rpc:latest gen-config-file. Instead I clonedstellar-rpcatv28.0.0and built thegen-config-filebody againstinternal/config, which needs no Rust preflight linkage. Then I diffed its output against the doc block key by key.Result: every environment-independent key now matches the generator exactly, in value and in comment state.
internal/config/options.goBACKFILLdefaultfalseName: "backfill",DefaultValue: falseValidate:errors whencfg.Backfill && !cfg.ServeLedgersFromDatastoreSERVE_LEDGERS_FROM_DATASTOREdefaultfalseDefaultValue: falseHISTORY_RETENTION_WINDOWdefault120960SevenDayOfLedgers = OneDayOfLedgers * 7,OneDayOfLedgers = 17280MAX_GET_LEDGERS_EXECUTION_DURATIONdefault10sDefaultValue: 10 * time.SecondBoth usage strings and the startup error message are copied verbatim from the source.
Version:
BACKFILLfirst appears inv25.1.0.v25.0.0has nobackfilloption. This matches @ElliotFriend's finding.Why the block cannot be regenerated verbatim on a laptop
Four defaults depend on the machine, not on the release:
They feed
STELLAR_CORE_BINARY_PATH,PREFLIGHT_WORKER_COUNT,PREFLIGHT_WORKER_QUEUE_SIZEandSTELLAR_CAPTIVE_CORE_HTTP_QUERY_THREAD_POOL_SIZE. The current block holds container values, such as/usr/bin/stellar-coreand8workers. My run produced""and15. Pasting my output would put my CPU count into the docs.So a faithful full regeneration has to run inside the Docker image. I left these four untouched, and left out
STELLAR_CAPTIVE_CORE_HTTP_QUERY_THREAD_POOL_SIZEfor the same reason. Please regenerate them in the image when convenient.Two more differences I found and did not change:
DB_PATH: the doc showsstellar_rpc.sqlite, the generator emitssoroban_rpc.sqlite. The source carries// TODO: deprecate and rename to stellar_rpc.sqlite, so the doc matches the intent. Your call.HISTORY_ARCHIVE_URLS: the generator comments this out, the doc leaves it live. Live is arguably better, because operators must set it.Left for RPC team review
The issue also asks for startup-behavior and coverage-boundary wording. I did not write it. Both reviewers said it needs RPC-team input rather than source reading, and
options.gocannot settle it:BACKFILLPlease keep #2602 open for that half.
Checks
prettierpasses with the repo configscripts/check-relative-links.sh --stagedreports "All links check out"