Skip to content

[survey2ddi 2/6] Port normalize_responses: the LimeSurvey response-data path #11

Description

@jstet

Step 2 of 6 for the survey2ddi retirement. Plan: HANDOVER_SURVEY2DDI.md. Independent of the xlsform2ddi --data issue; both are needed before survey2ddi's CLIs can be deprecated.

The gap

lstsvToDdiXml emits metadata. buildDataCsv (#8) emits a data CSV from submissions keyed by bare question name or group/name path. Neither shape is what a LimeSurvey response export produces, so there is no LimeSurvey full-mode path in this library.

limesurvey2ddi/transform.py in CorrelAid/survey2ddi does the missing step — normalize_responses(variables, rows):

  • Bracket subkeys. LimeSurvey exports a select_multiple as one column per subquestion, keyed with a bracket suffix. _match_choice(subkey, choices) maps that suffix back to the parsed choice code so it can line up with the <name>_<choice> DDI variable.
  • The underscore quirk. _norm(name) strips underscores and lowercases, "mirrors LimeSurvey's export behaviour" — LimeSurvey mangles field names on export, so matching must be done on the normalized form.

Both are LimeSurvey-specific and belong nowhere near the Kobo path.

What to build

src/pipelines/lstsv2ddi/data.ts, mirroring how xlsform2ddi/data.ts sits beside variables.ts:

lstsvToDataCsv(tsv: string, responses: Record<string, unknown>[], options?): string

Simplest honest shape: a key-normalizing adapter that re-keys LimeSurvey rows onto DDI variable names, then delegates to the existing buildDataCsv — the RFC 4180 writing and the column plan are already correct and must not be duplicated. Do not widen readCell in xlsform2ddi/data.ts with LimeSurvey heuristics; the two platforms' quirks stay separate.

Then add --data / --data-out to the lstsv2ddi CLI command, same flags and semantics as the xlsform2ddi issue.

Verification

  • Fixture-based unit test: a LimeSurvey response export (bracketed multi columns, underscore-mangled names) → correct 0/1 binaries and correct scalar columns
  • Header order equals <var name=""> order of lstsvToDdiXml on the same TSV
  • Byte-compare against Python normalize_responses + build_data_csv on one fixture, accounting for the known column-order divergence in the Python emitter (documented in the handover doc)
  • tests/fixtures/lstsv/ in survey2ddi has three usable structure TSVs

Acceptance criteria

  • Bracket-subkey → choice-code matching ported, with the _norm normalization
  • lstsv2ddi --data writes a CSV whose header matches the XML
  • LimeSurvey-specific keying lives in lstsv2ddi/, not in the shared emitter
  • Kobo path behaviour unchanged (existing data.test.ts still green)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions