Overture docs update - #911
Conversation
8b67b0a to
69da766
Compare
The docs site's under-development section has been removed; Lyric (the tabular data submission service) is now the canonical destination for the 'data submission system' reference, so link to its core-software overview instead of the retired /docs/under-development/ page.
The site renamed /docs/core-software/* to /build/* to /develop/*, moved /docs/contribution to /develop/contributing, /guides/getting-started to /deploy, and /guides/administration-guides to /use. Each old path only resolved through a redirect hop. Verified every new target against a local site build.
69da766 to
bb13738
Compare
Bare site paths (/community/support, /use/cli-submissions, /develop/Lyric/overview, /develop/Maestro/overview) only resolve once these files are assembled into the docs site. Read directly on GitHub, in a clone, or in an IDE they 404. Links pointing at content outside this repository now use the full URL form, matching the README. 18 links across 13 files.
Ten links to Song's own pages were written as absolute site paths (/develop/Song/Reference/..., /develop/Song/setup#song-client-setup). Relative paths resolve both standalone and through the aggregated site; the absolute form only works in the latter. 01-setup.md points into 02-Reference/, 00-submitting-metadata.md and 10-client-reference.md point at siblings, and 03-publication-controls.md points back up at 01-setup.md. This matches the form the docs already use elsewhere in 02-Reference/, for example ./11-database-migration.md. Both anchors verified against their target headings: '## Song-Client Setup' in 01-setup.md and '### Song Terminology' in 02-Reference.mdx.
…arget
Three links used the retired /guides/* structure. All three resolve only
through a plugin-client-redirects hop today. Two were worse than stale.
04-data-model-management.md: /guides/administration-guides/updating-the-data-model
now points at /use/administration/building-song-schemas. The page was renamed,
not just moved, so the link text changes from "updating data models" to match.
09-api-reference.mdx carried two links whose targets do not hold the content
they promise:
- "accessing an API key from a deployed Overture platform" pointed at
/guides/user-guides/cli-submissions#generate-an-api-key. That anchor does
not exist, and the current page (/use/cli-submissions) has no API key
content at all. Retargeted at the Keycloak deployment guide, which is where
the Overture API Key Provider is actually installed on a deployment.
- "administration guide on identity & access management" pointed at
/guides/administration-guides, which redirects to the generic /use overview.
No such administration guide exists. The content it describes, generating
and managing API keys through Keycloak, is this repo's own Configuring
Keycloak page, so it now points there relatively.
Both link texts updated to name what the reader will find.
| Clone the migration repository and install its dependencies: | ||
|
|
||
| ```bash | ||
| git clone https://github.com/overture-stack/song_5_3_migration.git |
There was a problem hiding this comment.
this repo should not be used anymore, the migration script was added in the same SONG repo.
so users can do:
# clone SONG repo
git clone https://github.com/overture-stack/SONG.git
# Go to folder
cd SONG/data-migration
# Install
pnpm install
There was a problem hiding this comment.
Fixed, now clones SONG and uses SONG/data-migration. Also fixed data-migration/README.md, which had the same stale clone.
| DOCKER_COMPOSE_EXE := $(shell which docker-compose) | ||
| # Prefer the Compose V2 plugin ("docker compose"), falling back to the standalone | ||
| # V1 binary. V1 is no longer shipped with current Docker Desktop releases. | ||
| DOCKER_COMPOSE_EXE := $(shell docker compose version >/dev/null 2>&1 && echo "docker compose" || which docker-compose) |
| ```python | ||
| import requests | ||
|
|
||
| url = "https://song.virusseq-dataportal.ca" |
There was a problem hiding this comment.
found in this file and other files an used to real URL (now iMs) but Song shouldn't point to any of these URLs
There was a problem hiding this comment.
Swapped all these with https://. Also replaced the real VirusSeq study and analysis IDs (LSPQ-QC, MCPL-MB, 4861730f-...) with the ABC123 placeholders.
| - **[Song Overview](https://docs.overture.bio/develop/Song/overview)** | ||
| - [**Setting up the Development Environment**](https://docs.overture.bio/develop/Song/setup) | ||
| - [**Reference Docs**](https://docs.overture.bio/develop/Song/Reference) |
There was a problem hiding this comment.
I noticed these URLs atm are not valid and have changed with path /develop/. I guess this is expected to change docs URL based on branches?
There was a problem hiding this comment.
Expected, yes. Those paths go live when overture-stack/docs#48 merges
leoraba
left a comment
There was a problem hiding this comment.
great work keeping the docs up to date. just left minor changes.
* fix docker publishing tags in Jenkinsfile * version 5.2.0 * version 5.3.0 * Disable CSRF * score client name change (overture-stack#905) * score client name change * Update introduction.rst * remove copypasta * re-add bash endline slash --------- Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> * update migration script to fetch samples structure using batch JOIN query (overture-stack#907) * Overture docs update (overture-stack#911) * Allow external validations to reference elements of an array (overture-stack#909) * Allow external validations to reference elements of an array * Throw validation exception when jsonPath points to non string or array value * Unit tests for ValidationService.getValuesAtJsonPath * Fix test file compilation * Move validation service unit tests to related dir, use JUnit 4 annotations for consistency * Fix song-server unit tests (overture-stack#912) * Allow external validations to reference elements of an array * Throw validation exception when jsonPath points to non string or array value * Unit tests for ValidationService.getValuesAtJsonPath * Fix test file compilation * Allow external validations to reference elements of an array * Include junit vintage engine dependency to pickup classes annotated with JUnit 4 * Move validation service unit tests to related dir, use JUnit 4 annotations for consistency * Upgrade testcontainers to 1.16 to support Apple Silicon * Fix failing server tests, postgres testcontainer to v14 to support latest migrations * Widen ValidationService.getValuesAtJsonPath to public GetValuesAtJsonPathTest lives in the validation package and calls this method directly, but it was package-private in the service package, which broke compilation of the entire song-server test module. * Fix ConstantsTest.testInfoTypes stale enum count assertion InfoTypes only has 4 constants (STUDY, FILE, SEQUENCING_READ, VARIANT_CALL) but the test asserted 7, with the assertEquals arguments also reversed. * Fix EntityTest.testFile fileType mismatch with FILE_TYPES constant file1 hardcoded fileType("mp4") while the equal-value comparison object and the getter assertion both used FILE_TYPES.get(0) ("mp3"), causing the equality check to fail. * Remove stale samples block from SerializationTest fixture The Payload model no longer has a samples field, and the test's expectedPayload object never sets one, but the expected-serialization fixture still hardcoded a samples array, causing a JSON diff failure. * Fix MockedSubmitTest Mockito matcher misuse validate(isA(JsonNode.class), "anyStudyId") mixed a matcher with a raw String argument, which Mockito rejects. Wrapped the raw values in eq(...), and for the two tests that stub/verify against the studyId actually passed to submit(), used that variable instead of the unrelated "anyStudyId" literal so the verify matches the real invocation. * Fix NonEnforcedSubmitControllerTest fixture staleness and md5 reuse Two issues surfaced by AbstractEnforcedTester (shared by all *SubmitControllerTest classes): 1. documents/variantcall-valid.json still had a stale top-level "samples" block. It's harmless on initial /submit (no schema forbids it there), but AnalysisServiceImpl's update-request schema explicitly disallows "samples", so any update built from data merged out of a submitted analysis (as several tests do) failed schema validation. 2. buildTestEnforcePayload() reused the fixture's hardcoded file md5sums on every call. AnalysisServiceImpl.checkDuplicateAnalysis checks file md5 existence globally (not scoped to study), so calling submit() more than once within a test run (this class has 7 test methods, each submitting) hit "info.already.exists" (409) from the second call onward. Randomized the md5sum per payload build using the test's existing seeded RandomGenerator. No production code changed. * Fix EnforcedSubmitControllerTest studyId/md5 bugs; revert samples removal Investigating EnforcedSubmitControllerTest showed the earlier removal of "samples" from variantcall-valid.json (previous commit) was wrong: samples is a normal, schema-validated part of a variantCall submit payload (testInvalidSample depends on it) - it's only forbidden in *update* requests. Restored it, and instead fixed the real problem: extracted a toUpdatableData() helper on AbstractEnforcedTester that strips "samples"/"files" before merging stored analysis data into an update request, and used it in both Enforced and NonEnforced tests (the underlying bug was the same in both). Also fixed two real bugs specific to EnforcedSubmitControllerTest: - testInvalidFile/testInvalidSample never overrode the fixture's hardcoded "studyId": "ABC123" to match the test's generated study, so submissions failed on studyId mismatch before schema validation had a chance to run. - Extracted randomizeFileChecksums() (same fix pattern as the NonEnforced commit) and applied it everywhere this test class loads a fixture with hardcoded file checksums, since submitting the same checksum twice in one run trips the global duplicate-file check. 11 of 18 tests in this class now pass. The remaining 7 all fail because the current schema/validation logic doesn't enforce constraints these tests expect (fileType/fileAccess/sampleType enums, "matchedNormal" field inclusion rules) - grep shows no schema or validator in the codebase currently implements these rules at all. Flagged in TEST_FIX_LOG.md for review rather than guessed at, since restoring that validation would be a functional change to production code, and this codebase's git history doesn't cheaply reveal whether these tests were ever passing. * Fix CorruptionSubmitControllerTest duplicate-file conflict testNonMutated resubmits the exact same payload (same file checksums) to verify that a non-mutated resubmission succeeds, but AnalysisServiceImpl.checkDuplicateAnalysis now rejects any resubmitted file checksum by default. Added an EndpointTester overload that passes the existing (already supported by SubmitController) allowDuplicates query param, and used it for the second submission. * Fix SubmitServiceTest duplicate-file conflicts and restore rollback Two issues: 1. The class overrode @TestExecutionListeners to only DependencyInjectionTestExecutionListener, which silently disabled TransactionalTestExecutionListener - so the class- and method-level @transactional annotations did nothing, letting each test method's DB writes persist into the next. Removed the override so the default listener set (including transactional rollback) applies. 2. testSaveIdMismatchAllSame/AllDifferent resubmit the same payload content (same file checksums) to verify a new analysisId is issued, and testNullSyncSequencingRead's two fixtures intentionally share the same file checksums - all three need allowDuplicates=true on the resubmission, same as the earlier CorruptionSubmitControllerTest fix. * Remove EnforcedSubmitControllerTest sample-validation tests Samples have been removed from the SONG analysis model entirely - there are no longer validation rules for tumour/normal or matched-tumour-normal relationships. Removed the 7 tests asserting that kind of schema violation (testInvalidSample and the matchedNormalFieldInclusionValidation_* suite), their now-unused runMatchedNormalTest helper, and the now-orphaned fixture files. Stripped the stale "samples" block out of variantcall-valid.json. * Fix Responses.contextMessage dropping %s substitutions It escaped every "%" in the message template to "%%" before calling String.format(..., args), which turns "%s" into "%%s" - "%%" is consumed as a literal "%" and the trailing "s" becomes inert text, so the format argument is silently dropped and the literal "%s" (or "%d", etc.) shows up in the final error message instead of the real value. Substitute args into the template first, then wrap the result in "[context] - ..." - no escaping needed since String.format doesn't re-interpret an already-substituted argument value. * Fix JWTSecurityTest: mock JWKS endpoint, fix provider and expiry The "secure" profile points jwk-set-uri at a real Keycloak instance, so every JWT-decoding test failed with Connection refused. Start a WireMock server before context refresh (via @DynamicPropertySource) and point jwk-set-uri at it; once the KeyPair bean is available, stub its response with a JWKS document built from that key pair's public half, so Spring Security can verify tokens signed by JWTGenerator. Two more issues surfaced once JWKS decoding worked: - auth.server.provider was "keycloak" (via the "secure" profile), which routes authorization through KeycloakAuthorizationService's UMA/RPT grant-fetch flow (another unreachable real endpoint) - but JWTGenerator embeds scopes directly in the JWT claims (the "ego" model). Overrode auth.server.provider=ego for this test. - JWTGenerator's "expired" tokens expired only 10s in the past, inside the JWT decoder's default 60s clock-skew tolerance, so they were still accepted. Pushed the expiry further out (5 minutes). Expired tokens are correctly rejected now, but with 401 Unauthorized (Spring Security's standard behavior for an invalid/expired token), not 403 Forbidden - there's no custom AuthenticationEntryPoint overriding this. Updated the 4 expired-token tests to expect 401 and renamed them from *_Forbidden to *_Unauthorized to match. * Remove stale samples fixtures from AnalysisTypeControllerTest/ExportServiceTest Samples have been removed from the SONG analysis model. Rewrote rendered-schema.json to match what AnalysisTypeService actually produces (no donor/sample/specimen definitions or "samples" property - confirmed against analysisBase.json, the real rendering template). Stripped the "samples" block from all 8 export test fixtures (variantcall-input/output 1-4), since Sample/Donor/Specimen no longer get the info-object field routing FileEntity has, so the expected-output fixtures' assumption about that never held once samples support was dropped. * Fix idSearch: remove dangling SQL clause and donor/specimen/sample join Two related bugs in the idSearch feature, both stemming from donor/specimen/sample no longer being part of the analysis model: 1. IdView's idSearch @NamedNativeQuery ended with a dangling "AND submitter_sample_id" (no operator/value), which made Postgres evaluate the text column as a boolean and throw on every call. IdSearchRequest never had a matching parameter for it - donor, specimen, and sample IDs are no longer valid search criteria, so removed the clause entirely rather than wiring up a replacement parameter. 2. The underlying idview database view still inner-joined donor -> specimen -> sample -> sampleset -> file -> analysis. Since submitted analyses no longer create donor/specimen/sample/ sampleset rows, that join chain could never match a current analysis - idSearch would silently return nothing even once the query stopped throwing. Added a migration redefining idview to join file -> analysis -> analysis_schema directly, and updated the IdView entity to match (dropped the now-nonexistent donorId/ specimenId/sampleId columns). * Fix ValidationService validating against the unrendered schema fragment ValidationService.validate() called analysisTypeService.getAnalysisType(analysisTypeId, true) - the `true` is `unrenderedOnly`, meaning every submission was schema-validated against the raw, unrendered custom schema fragment (e.g. just {"required":["experiment"], "properties":{"experiment":{...}}} for variantCall), never merged with analysisBase.json. That base envelope is where "analysisId":{"not":{}}, studyId, and the file fileMd5sum/fileAccess constraints all live, so none of that was ever actually enforced at submission time - only the "experiment" object was checked. Changed to getAnalysisType(analysisTypeId, false) so submissions validate against the fully-rendered schema. This alone fixes SubmitServiceTest's analysisId-not-rejected test and ValidationServiceTest's fileMd5 gap. Also fixed EnforcedSubmitControllerTest.testInvalidFile: it targets the pre-seeded legacy "variantCall" analysisType (from the historic V1_2__dynamic_schema_integration migration), which has no fileTypes option set, so the fileType allow-list check (ValidationService.validate lines 111-114, a separate mechanism from the JSON schema) never ran. Registered a new version of it with fileTypes restricted to what the fixture uses, and pointed all three submissions in the test at that version (schemas.enforceLatest=true requires it). * Remove dead spring-security-oauth-based auth classes and their tests AccessTokenConverterWithExpiry/ExpiringOauth2Authentication build an old org.springframework.security.oauth2.provider.OAuth2Authentication, but Scopes.extractGrantedScopes() (used by StudySecurity/ SystemSecurity) only recognizes the current JwtAuthenticationToken/BearerTokenAuthentication types - so TestStudySecurity/TestSystemSecurity, which build their Authentication via the old converter, always got an empty scope set regardless of input, failing every "should succeed" case. Confirmed these two classes have no other production usage (no bean wiring, no other callers), and that StudySecurity/SystemSecurity. authorize() are already covered end-to-end through the real JwtAuthenticationToken path by JWTSecurityTest, via the same @PreAuthorize("@studySecurity.authorize(...)")/ @PreAuthorize("@systemSecurity.authorize(...)") endpoints these tests were exercising through the dead code path. Removed both production classes and both test files rather than fix tests for a path nothing else uses. * Restore ValidationService.getValuesAtJsonPath validation exceptions Commit ac30d45 originally made this throw a ValidationException when a jsonPath resolves to a non-string scalar or an array containing non-string elements. b783eec reverted that to silently filtering non- strings out, which was the wrong fix - per direction, this should throw. Also restructured per feedback: the try/catch now wraps only JsonPath.read(...) (genuine JSON-path parsing failures - the only thing it was ever meant to guard), logging a debug message and returning an empty list for those. The type-checking logic that throws ValidationException now happens outside that try/catch, so the exception propagates directly instead of being caught-and-rethrown by a redundant `catch (ValidationException) { throw ... }` block, and the final "else" branch just throws instead of the dead throw-then-implicit-return shape. * Fix SchemaValidationTests stale violation counts, remove unused samples Samples have been removed from the SONG analysis model, so the 4 count-based assertions here were counting violations that no longer exist (missing submitterSampleId, invalid sampleType/gender/ tumourNormalDesignation/specimenType enum values). Verified each fixture's actual violations directly (logged via the existing log.error in the validate() helper) and updated the expected counts to match, documenting the exact expected violation messages in a comment above each assertion so this doesn't silently drift again. Also stripped the now-inert "samples" blocks out of the 5 fixture files these tests (and the shared sequencingread-valid.json, used by SubmitServiceTest/PayloadGenerator too) load, since they no longer contribute to validation either way. * Fix AnalysisServiceTest: wire options field through get_analysis_schema read path The options column was added to analysis_schema in V1_21 but the get_analysis_schema DB function backing the paginated getAnalysis() read path was never updated to select it, leaving AnalysisSchema.options null for analyses loaded through that path. Also fixes the earlier FileEntity.analysisId omission in buildAnalysis(), surfaced by the same test. * Fix AnalysisTypeControllerTest: return normalized options from commitAnalysisType commitAnalysisType() built newAnalysisOptions (falling back to the previous version's options, then to empty lists) and persisted that, but returned the raw, un-normalized options parameter in the response AnalysisType. This made the register response diverge from what a subsequent GET returns for the same analysisType. * Fix AnalysisTypeControllerTest.register_illegalFields_schemaViolation: drop stale samples check samples was removed from the analysis model (V1_22 migration) and the registration meta-schema no longer marks it as a reserved/illegal property name, so registering a schema with a "samples" property now succeeds as expected. Removed it from the list of fields this test checks are rejected. * Fix AnalysisTypeControllerTest experiment-violation message tests register_missingExperiment_schemaViolation and register_misspeltExperiment_schemaViolation asserted stale expected violation messages including "experiment"-related checks that commit 454c2b0 intentionally removed from the analysisTypeRegistration meta-schema. Updated both to the current 2-violation message, which now matches the already-passing register_extraFields_schemaViolation. * Fix AnalysisTypeControllerTest.register_emptySchema_schemaViolation: remove redundant @nonnull AnalysisTypeService.register()'s @nonnull on analysisTypeSchema made Lombok throw a raw NullPointerException before validateAnalysisTypeSchema() got a chance to run its own intended null check, which already produces a handled SCHEMA_VIOLATION server error. Removed the redundant annotation so the existing guard is reachable. * Upgrade song-server from JUnit 4 to 5 * Clarification on behaviour for external validation of non-string properties * Remove duplicate dependency declaration * Update Jenkinsfile Switch pipeline library branch * Update testcontainers to be compatible with docker 29 * Restore jenkins to master branch pipeline --------- Co-authored-by: Yelizar Alturmessov <yalturmessov@oicr.on.ca> * Versioning 5.4.0-SNAPSHOT * Versioning 5.4.0 * Activate all AuthZ classes only with the pcglauthz profile * Disable the JWTSecurityTest that are unused in PCGL * Update README for PCGL --------- Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com> Co-authored-by: Leonardo Rivera <leorivera_88@hotmail.com> Co-authored-by: Mitchell Shiell <59712867+MitchellShiell@users.noreply.github.com> Co-authored-by: Yelizar Alturmessov <yalturmessov@oicr.on.ca>
Documentation for the Overture docs site.
The docs still described the pre-5.3.0 data model: donor/specimen/sample entities in the base schema, ID management delegated to an external service. They now describe the consolidated 5.3.0 schema and its migration path. Separately,
docs/02-Usage/→docs/02-Reference/to match the other components, and the setup guide is rewritten around this repo's own Compose stack instead of the retired Preludequickstartbranch.Source changes are limited to
Makefileanddocker-compose.yml: dev tooling, no application logic, no Java.Song 5.3.0 data model
04-data-model-management.md: base schema corrected to the three fields it actually requires (studyId,analysisType,files); the old text listed donor/specimen/sample IDs and "essential cancer sample characteristics". Adds a Schema Options section foroptions.fileTypesandoptions.externalValidations, including that omitting an option preserves its previous value while an empty list clears it.06-id-management.md: rewritten, replacing a 184-line page. Song generates both IDs it tracks, with no external ID service: analysis IDs are random UUIDs, file IDs deterministic UUID5 hashes of the analysis ID and file name. IDs are immutable, so changing a contributing value means UNPUBLISH + SUPPRESS + resubmit.11-database-migration.md(new): pre-5.3.0 data spanned multiple tables, 5.3.0 writes to one, so existing databases must be migrated. Documents the TypeScript migration script, opening with a back-up warning.Each carries a version note recording the old behaviour, including the retired "federated" ID mode.
Setup guide
The old guide opened with
git clone -b quickstart .../prelude.git. That branch is retired, so it now uses thedocker-compose.ymlandMakefilealready in this repo.make start-deps→ Keycloak9082, keycloak-db9444, Score8087, Minio8085.make start-song-serveradds song-db8432and song-server8080(debug5006).8180,9000,5433), not ours. Corrected, the Conductor row is gone, andsongDev.svg, which drew the Quickstart topology, is dropped.CLIENT_ACCESS_TOKEN=68fb42b4-…andCLIENT_STUDY_ID=demoare now placeholders. Both were Quickstart values that worked nowhere else, and this stack starts with an empty database, so there is nodemostudy to query.adminpassword in Keycloak, get a token via thesystemclient's direct access grant, exchange it atapikey/api_keyfor asong.WRITE/score.WRITEkey.myrealmand downloadskeycloak-apikeysat start-up; the targets build with./mvnw, so a JDK is needed even for the supporting services alone.make cleanandmake log-song-serverdocumented.Source changes
Makefile. The guide does not work without these:docker compose version, falls back to the standalone binary. V1 no longer ships with Docker Desktop, so every Compose target failed outright on a current install.MVN_EXEprefers the bundled./mvnwover a systemmvn.ROOT_DIRmoves above the detection block that now depends on it.docker-compose.yml. Without this the stack could not pull at all.bitnami/keycloak:22andbitnami/postgresql:11→bitnamilegacy/*; Bitnami archived its versioned tags in 2025. Commented in place.docs/02-Usage/→docs/02-Reference/Every page moves;
02-Usage.mdxbecomes02-Reference.mdx(retitled "Reference", glossary retained).docs/custom-schemas.mdis deleted, because its Options content is now the Schema Options section of04-data-model-management.md, where anyone configuring an analysis type already is.One real fix came along in the move:
01-retrieving-metadata.mdand02-updating-metadata.mdshowed a multi-analysis update queryinganalysis/search/id?submitterDonorId=<sample>, an endpoint that does not serve submitter sample lookups. Song has no server-side search for submitter sample or donor IDs, so the example now pages through the study's analyses, filters client-side, and patches each match by analysis ID.Overview and README fixes
00-overview.md: the repo tree said/song-servers(actual:/song-server); thesong-java-sdkentry described a package that no longer exists;song-python-sdkwas described as being for "java applications".01-setup.md: the services table credited Keycloak with OAuth2 "for Score" on the Song page, and the Song-serverapplication.ymllink pointed into the score repo.README.md: staledocs/core-software/Song/*links →/develop/Song/*, contributing →/develop/contributing, getting started →/deploy.