feat!: implement more expressive templates via liquidjs - #428
Merged
Conversation
Templates are parsed once and rendered per recipient, so a basic
template can render at 140k renders/s whereas a richer one with
sevearl conditionals and loops renders at ~18k renders/s. 1M-contact
campaign spends under one minute of CPU time for rendering, which is
negligible against 1M SES calls.
The engine is sandboxed: {% include %}, {% render %} and {% layout %}
are rejected because they resolve partials off the file system, and
parse/render/memory are capped so a runaway loop can't stall a worker.
Rendering stays lenient, a template that fails to parse falls back to
plain {{variable}} substitution rather than failing the send. Syntax
errors are surfaced at write time instead: POST/PATCH of templates and
campaigns, and POST /v1/send with an inline subject/body, now return
400 with the line and column.
Adding multiple tests to assess the proper behaviour and to keep
some performance guarantees (for both memory & CPU).
Backwards compatible with existing template system for most scenarios.
See template-language.mdx Notes for existing templates for more info,
but basically: 0/false/NaN are now rendered while before they produced
empty string. Also {{ name ?? 'hi' }} produced 'hi' with quotes, which
seems like a bug, now emits hi with no quotes. Body copy containing
balanced {% %} markup is now executed rather than printed literally —
wrap it in {% raw %} to keep it as text.
verifyDomain and getDomainVerificationAttributes were already mocked, but deleteIdentity and disableFeedbackForwarding were not, so the suite issued live AWS SES requests on every run. DomainService catches and logs failures from both, so the calls stayed invisible in test output while still going over the network. With real SES credentials in .env they would have executed DeleteIdentity and SetIdentityFeedbackForwardingEnabled against a live AWS account. DomainService.test.ts drops from ~40s to 23s.
Contributor
|
Yes please; I asked for the same here https://discord.com/channels/990573272561242162/990573273144229900/1502479340414304346 I actually were maintaining my own liquid patch on top but didn't create a PR yet; but this one looks better anyway so has my thumbs up 👍 |
Member
|
Could be an interesting change. I notice that I do not have edit rights on the PR. Would you want me to leave feedback for you to implement? |
Contributor
Author
|
TL;DR You should have edit rights now. To me is fine if you can do the changes and/or you give me feedback. Both work for me. Long story: |
- vitest 4 deprecated forks - exposed garbage collector for better memory assertions - performance tests scaled down for CI machines - setup.ts to include default AWS SES so tests work if .env misses them
Contributor
Author
|
Fixed the performance tests and added some minor fixes/changes to the test config. I also synched the branch to the latest changes in Performance tests were failing due to CI running shared build machines, which are slower. The fixes on the test setup is just so that if you are missing some vars in the .env, the tests still run. If those env vars are there, then it uses them. Another minor change is a fix on some params that were being silently ignored by vitest 4 so I replaced with maxWorkers to keep same spirit (nonetheless, this is only for the tests) Hopefully tests should pass now. |
A template that trips the render or memory ceiling falls back to the legacy placeholder renderer, but nothing recorded that it had failed. The per-email call sites re-compile the same source for every recipient, so each contact re-entered the engine and spent the budget again before falling back — on a large campaign that is a lot of CPU burned on a send that is already degraded. Latch the failure on the cached parse result, which is shared across compileTemplate calls, so only the first recipient pays. The latch is per template and cleared with the cache. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Syntax checking inline subject/body turned a lenient renderer into a hard 400
on the transactional path. Those bodies are generated by whatever system calls
the API — Handlebars output, framework markup, a stray `{{` in a code sample —
and they sent fine before Liquid existed. Rendering already handles them the
way the old renderer did (unknown placeholders drop to empty, unparseable
markup falls back to plain substitution), so a live integration should not
start failing over markup that still renders identically.
Validation stays on the authoring surfaces, templates and campaigns, where a
human is editing and a whole audience is downstream of the mistake.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The API rejects an unparseable template on save, but a 400 after the fact is a
poor way to learn you mistyped a tag — and its "line 12, column 30" is unusable
in the visual editor, whose HTML is a single line with no gutter.
Validation is pure and already ships in @plunk/shared, so run it client-side on
a debounce and show the problem above the editor. In HTML mode the CodeMirror
gutter makes line and column meaningful, so they are shown. In visual mode they
are not, so the offending {{ }} or {% %} block is extracted from the source and
shown instead — the only locator that means anything there.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`{{ }}` already rendered as a chip, but `{% %}` rendered as prose — same font,
weight and colour as the sentence around it. The data layer was visible and the
control flow was invisible, so a template with a few conditionals read as one
undifferentiated block of text.
Decorate logic tags too, deliberately styled apart from the blue value chips
rather than in a second accent colour: outputting a value and branching are
different kinds of thing and should not look like variations of one. Matching is
on the delimiters rather than a list of tag names, so if/else/for/case/assign
and anything Liquid gains later are all covered.
Also resets lastIndex per text node. The shared /g regex kept its offset between
nodes, so a document with several text nodes silently skipped matches.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n menus
The {{ menu works because it is populated from the project's real contact
fields: nothing has to be remembered. Control flow had no equivalent, so typing
{% now offers the shapes people actually reach for, labelled in plain language
with the Liquid shown underneath. Both tags are inserted together with the caret
between them, so an unbalanced block, the most common reason a template fails to
save, is not reachable through the menu.
Building it surfaced problems in the existing {{ menu, so both now share one
list component and one row grammar: what you are choosing, what it writes, and
what we know about it. Specifically:
- Rows lead with the field name rather than raw {{syntax}}, and carry the type
and coverage the fields endpoint already returned and the UI discarded. A
{{plan}} that only 4% of contacts have renders blank for the rest, and the
only way to discover that was to send the campaign.
- Hover no longer paints a second highlight competing with the keyboard
selection. Pointer movement moves the selection, so exactly one row is ever
marked and it is always the one Enter takes.
- The selected row uses the same neutral-100 highlight as Select and
DropdownMenu instead of inventing a second vocabulary for the same gesture.
- Arrowing past the fold scrolls the selection into view. It previously moved
invisibly.
- Field names are escaped before being interpolated into the list markup. They
come from contact data keys, which are supplied through the API and CSV
import, so a crafted key could inject markup into the dashboard.
- The list is a listbox with options and aria-selected rather than anonymous
divs, and Tab accepts alongside Enter.
- Logic blocks are type-aware: no == "value" on a boolean, no numeric
comparison on a name. Results are capped per field so a query matching
several fields shows several fields rather than one field's variants.
Two supporting fixes:
- Suggestion lists rebind command on update. It closes over the match range,
which grows with every keystroke, so the one captured at onStart deleted the
trigger and left the query behind as literal text. This affected {{ too.
- The syntax strip stays quiet while a menu is open. A half-typed {%ema is not
valid Liquid, and flagging it contradicts the menu offering to finish it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Picking a block put both tags on one line with the caret between them, so writing anything multi-line meant pressing Enter from a spot that reads like the inside of a chip. The block arrived complete but with no room in it. Blocks are now described as lines, where an empty string is an editable gap, and the insertion picks its shape from context: on an otherwise empty line the block is written across paragraphs with the gap already open and the caret in it, and mid-sentence the same lines collapse to the inline pair as before. An if/else opens both of its gaps. Inserting a multi-line block mid-sentence would break the sentence, and an inline pair on an empty line is the awkward case above, so neither shape works everywhere — which is why this reads the caret's surroundings rather than picking one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Syntax validation is structurally blind to the most common template mistake.
`{% if emai %}` parses, and is then silently false for every contact forever:
the campaign sends, that block renders nothing, and nobody finds out.
The fields endpoint already reports every field a project's contacts carry,
along with what share of them carry it, so the editor can check references
against reality and say two useful things:
- emai matches no field on any contact, and email is one character away.
- trialEndsAt exists, but only 12% of contacts have it, so the other 88% see
this block as empty. That number was previously undiscoverable without
sending the campaign.
Warnings only, shown below the syntax strip and never blocking a save. A field
that legitimately has no data yet is a real scenario: you write the template
before the data starts flowing.
The risk in a lint like this is crying wolf, since a warning that fires on
correct templates teaches people to ignore the strip. So it stays quiet for
loop variables, assigned and captured names, filters and their arguments,
string literals, the legacy ?? fallback, raw blocks, and the HTML entities the
rich-text editor escapes markup into — > would otherwise report a missing
field called gt. It also says nothing at all until the field list has loaded,
when every reference would look like a mistake.
Names bound anywhere silence that name everywhere rather than tracking scope
per block. That can miss a genuine mistake, which is the better failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The menu covered if/unless/for and stopped there, so the multi-way branch people reach for most after a plain conditional — case/when — could only be written from memory, which is the thing this menu exists to avoid. Added, each still inserting complete and balanced: - case/when/else, for picking a version per value. Offered for text, number and date fields but not booleans, where a multi-way branch on two states is noise. - contains, for matching within a value. - for/else, which renders the fallback when the list is empty. - comment, for notes that never send. - raw, for showing template markup as literal text. The docs already tell authors to reach for it and the menu did not offer it. Also a below-a-value comparison for numbers and dates, which only had above. The block definitions are now checked against the same parser the API validates with, both filled and left empty. A typo in a closing tag would otherwise ship as a menu entry that produces an unsaveable template, and nothing in the type system would catch it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Templates used as workflow steps are edited on the same screen as campaign templates, and are rendered with the trigger's event alongside contact data (WorkflowExecutionService). Reading `event.plan` is correct there, but the field lint only knew about contact fields and reported it as a missing one — a warning firing on a working template, which is the failure mode this lint most needs to avoid. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
|
Thanks for merging & accepting the contribution! 🙌 🙌 🙌 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Implement liquidjs templating system as per #426
Tests and documentation have been updated. I manually ran tests locally.
Templates are parsed once and rendered per recipient, so a basic template can render at 140k renders/s whereas a much richer template with several conditionals and loops renders at ~18k renders/s (worse-case scenario).
Even on a 1M-contact campaign, it would take less than 1 minute of CPU time for rendering, which is still negligible against 1M SES calls. I mean, worse-case scenario less than a minute for 1M emails, whereas best-case scenario less than 8 seconds (note: tested on an AMD Ryzen AI 9 365).
The engine is sandboxed: {% include %}, {% render %} and {% layout %} are rejected because they resolve partials off the file system, and parse/render/memory are capped so a runaway loop can't stall a worker.
Rendering stays lenient, a template that fails to parse falls back to plain {{variable}} substitution rather than failing the send. Syntax errors are surfaced at write time instead: POST/PATCH of templates and
campaigns, and POST /v1/send with an inline subject/body, now return 400 with the line and column.
Adding multiple tests to assess the proper behaviour and to keep some performance guarantees (for both memory & CPU).
Backwards compatible with existing template system for most scenarios.
See template-language.mdx Notes for existing templates for more info (in the commited files), but basically:
I think is very unlikely these create issues with existing templates.
Type of Change
Marking as breaking change since it is not 100% backwards compatible, even though my personal guess is that it might stay compatible with 99% campaigns or even higher, but I have no data to back this up, thus, breaking change it is.
feat:New feature (MINOR version bump)fix:Bug fix (PATCH version bump)feat!:Breaking change - new feature (MAJOR version bump)fix!:Breaking change - bug fix (MAJOR version bump)docs:Documentation update (no version bump)chore:Maintenance/dependencies (no version bump)refactor:Code refactoring (no version bump)test:Adding tests (no version bump)perf:Performance improvement (PATCH version bump)PR Title Format
Testing
I created new automated tests both for template system itself and also for campaign service but also performance test to assess memory and CPU performance stays within reasonable boundaries.
The important caveat here is that I have not tested this in a live environment, given the change, it's totally worth to try it.
Most scenarios for the use of variables should be backwards compatible. See notes.
Checklist
Related Issues
Closes #426