feat(xlsform2ddi): emit the DDI response-data CSV (buildDataCsv)

[jstet]

Closes #5.

What

buildDdiXml describes a dataset it could not produce: submissions fed <caseQnty> and the <fileDscr> filename, but nothing in the library wrote the data file. Consumers kept a Python runtime around for that one call — formtransform-app boots Pyodide and the survey2ddi wheel purely for survey2ddi_core.data.build_data_csv.

New src/pipelines/xlsform2ddi/data.ts:

Export	Purpose
buildDataCsv(variables, submissions)	RFC 4180 CSV — CRLF, minimal quoting, header row + submissions in input order
getDdiColumnNames(variables)	DDI variable names in <var name=""> order
remapSubmissionsToDdi(variables, submissions)	rows re-keyed onto those column names

Also exported from the package root: extractVariables, choicesByListFromRows, normalizeChoices — building a Variable[] to hand to buildDataCsv was otherwise impossible from outside the library.

const xml = buildDdiXml(survey, choices, { settings: settings[0], submissions });
const csv = buildDataCsv(
  extractVariables(survey, choicesByListFromRows(choices)),
  submissions,
);

The one hard contract: column order equals <var name=""> order

So schema↔data alignment is a zip, not a lookup. The column plan walks the same buckets dataDscr does — grid-group members, select_multiple binaries, _other patterns, standalone vars — which is why splitDataVars and its types are now exported from ddi/codebook.ts. Consequences:

• select_multiple → one 0/1 column per choice, named <name>_<choice>
• _other multi patterns drop the other binary and keep the _other text column, matching emitOtherPatternVars
• note variables get no column
• a unit test asserts header order == <var name=""> order on a survey exercising all four buckets, so the two walks cannot drift silently

Deliberate divergence from the Python emitter

survey2ddi_core.data.get_canonical_columns documents XML order but emits input order. Its CSV header therefore does not match its own XML once a grid, select_multiple, or _other pattern reorders the buckets (ddi_xml.py buckets exactly the way codebook.ts does). Fixed here rather than replicated.

Cell values are byte-identical to Python. Verified against the real survey2ddi_core.data module on a flat survey — quoting, CRLF, None → empty cell, space-joined multi-select expansion:

q_free,age,comment
"say ""hi""",30,"a,b"
"line1
line2",,

cmp clean. On a survey with a select_multiple the two differ in column order only, per the above.

Submission keys — no Variable.data_key

Issue #5 asked whether to add the Python data_key field. Not needed: readCell accepts either the bare question name or the slash-joined group path (group/name, nested groups joined) that Kobo's CSV export uses, bare name winning when a row carries both. That covers the Kobo identity case and the LimeSurvey flattening without a field the caller has to populate correctly.

Validation surface

None, per the issue's preferred option — the remap is mechanical and a malformed row is the caller's problem. No skipValidation toggle.

Testing

• 18 new unit tests in tests/ts/unit/pipelines/xlsform2ddi/data.test.ts
• full suite: 725 passed (48 files)
• npm run validate green (lint, typecheck, format, knip, dup, arch, markdownlint)
• README + src/pipelines/README.md document the emitter and the ordering contract

The Python byte-comparison was run locally against CorrelAid/survey2ddi but not committed — the repo has no survey2ddi Python dependency and adding one to pin a package this library is meant to replace seemed like the wrong trade. Happy to add it as a tests/live/ script if you want it in CI.

Next

Unblocks formtransform-app#10: drop Pyodide, the survey2ddi wheel, pyodide.config.json, scripts/setup-pyodide.mjs, src/lib/pyodide.*, and the pyodide + xlsform2lstsv deps. The full-mode Kobo branch becomes parseXLSData + buildDdiXml + buildDataCsv.

🤖 Generated with Claude Code