Machine translation decodes HTML entities in source strings: <script> is stored as <script>, or deleted outright when safe-html is set #21004

Describe the issue

For the engines that mix in XMLMachineTranslationMixin (deepl, google-translate-api-v3, microsoft-translator), Weblate wraps every machinery call in an escape/unescape pair:

• cleanup_text() applies escape_text() → html.escape() (weblate/machinery/base.py:486-497, :1296)
• uncleanup_text() applies unescape_text() → html.unescape() (weblate/machinery/base.py:511, :1292)

That round-trip is only lossless if the provider returns the escaping at the level Weblate sent it. HTML-mode providers do not guarantee that, and in practice they renormalise. When they do, Weblate's unconditional unescape on the return path removes one level of escaping that was never re-added, and a source string containing a literal HTML entity comes back decoded.

Observed on a source string whose correct translation is inert text:

source   To disable it, write <script> in the template.
cleaned  To disable it, write &amp;lt;script&amp;gt; in the template.   <- Weblate escapes
reply    Um es zu deaktivieren, schreiben Sie &lt;script&gt; in die Vorlage.   <- provider returns one level decoded
stored   Um es zu deaktivieren, schreiben Sie <script> in die Vorlage.        <- Weblate unescapes again

Inert, escaped text in the source has become live markup in the target.

Whether the provider should have returned < rather than < is arguable, and this report does not rest on it. What is not arguable is where the damage happens: at the moment Weblate receives the reply the content is still an escaped entity, and therefore still inert. Weblate performs the final decode that turns it into live markup. The defect is that uncleanup_text() unescapes unconditionally, assuming the escaping level it sent survives the round trip, with no check that it did.

That the provider's reply is still escaped, inert and recoverable at the moment Weblate receives it can be shown directly, because the value is cached before the unescape is applied. (Not that it is the correct translation — the provider has already normalised one escaping level. The point is narrower: what arrives is still inert text, and what Weblate does to it next is what makes it dangerous.) Instrumenting a single google-translate-api-v3 call through the machinery layer:

cached provider result   Um es zu deaktivieren, schreiben Sie <script> in die Vorlage.
returned to the caller   Um es zu deaktivieren, schreiben Sie <script> in die Vorlage.

The cache entry and the returned value come from the same single provider call (provider_calls_first: 1). A second translate() with the network hard-blocked returns the corrupted form again from that same intact cache entry, so the corruption is applied on every read, not once at fetch time. Raw data: round2-gap-probe.json → translation_cache.

With the safe-html flag set, this turns into silent data loss. The BleachHTML autofix (weblate/trans/autofixes/html.py, present in DEFAULT_AUTOFIX_LIST) runs on save, sees the now-live <script>, and nh3.clean(..., clean_content_tags={"script","style"} - tags) removes the element and everything after it:

source   To disable it, write <script> in the template.
stored   Um es zu deaktivieren, schreiben Sie

The rest of the sentence is gone, and it is gone from the translation file on disk, not just from the UI. Forcing a repository commit and reading the working copy out of the container confirms the database and the file agree:

POST /api/components/{p}/{c}/repository/  {"operation": "commit"}   -> 200

database                                       file on disk
                                               /app/data/vcs/.../json-safe/de.json
escaped_script "Um es zu deaktivieren, ..."    "Um es zu deaktivieren, schreiben Sie "
escaped_tag    "Das -Tag kennzeichnet ..."     "Das -Tag kennzeichnet Text als fett."

<b> is removed the same way (The <b> tag makes text bold. → Das -Tag kennzeichnet Text als fett.), because extract_html_tags() is given the source, which contains no real tags — only entities — so the allowed-tag set is empty and clean_content_tags=CLEAN_CONTENT_TAGS - tags therefore covers everything:

# weblate/utils/html.py:345-360   HTMLSanitizer.clean
tags, attributes = extract_html_tags(source)
text = nh3.clean(text, link_rel=None, tags=tags, attributes=attributes,
                 clean_content_tags=CLEAN_CONTENT_TAGS - tags)

weblate.trans.autofixes.html.BleachHTML is active in a default installation — confirmed present in settings.AUTOFIX_LIST on the instance used here. Raw data: ondisk-loss-verify.json.

In every case autotranslate returns 200 {"details": "Automatic translation completed, 4 strings were updated."} and no check fires on the resulting unit. There is no signal to the user or to automation that anything was lost.

The provider is inconsistent even within a single string, so this cannot be compensated for by assuming a fixed escaping level. German, google-translate-api-v3, one request:

cleaned  Escape <, > and & before saving.
reply    Vor dem Speichern die Escape-Taste (<, > und &) drücken.
stored   Vor dem Speichern die Escape-Taste (<, &gt; und &amp;) drücken.

The first entity came back single-escaped and was corrupted; the other two came back double-escaped and survived.

Across a deliberately entity-heavy set of 7 sources × 4 locales, one attempt each, for the three engines that use the HTML/XML mixin:

Repeating every call showed the outcome is deterministic (110 of 112 repeat pairs byte-identical; the 2 exceptions were DeepL wording variation that preserved the entities in both attempts).

Please read those ratios as "this is reproducible on demand", not as a corruption rate for real projects. The corpus was built to concentrate literal entities, so the denominator is not representative of ordinary translation content. The claim is that the outcome is deterministic per string, not that a given share of any real project is affected.

(Weblate applies no escaping at all to the google-translate v2 engine, which sends format=text, so it does not traverse this code path. Entity-bearing sources are also mangled there, but by a different mechanism, and it is not evidence about this one.)

It is not a file-format problem — and Weblate already has a mechanism that prevents it.

The same four source strings, the same engine, the same locale, uploaded as four different file formats:

JSON, Android and gettext produce byte-identical target strings — three unrelated serialisations, same output — which places the defect in the machinery layer, not in any format's parser or writer.

XLIFF is the interesting one, because it shows a mechanism that prevents this already exists in the codebase. XLIFF units carry an automatic xml-text flag (RichXliffUnit.add_flags), and with that flag Weblate highlights the entity references themselves and protects them as non-translatable spans before the call:

source                To disable it, write <script> in the template.

no flags   on wire    To disable it, write <script> in the template.
                      highlights: []                                   -> corrupted

xml-text   on wire    To disable it, write <span translate="no" id="21">&amp;amp;lt;</span>script<span translate="no" id="31">&amp;amp;gt;</span> in the template.
                      highlights: [(21,25,'&lt;'), (31,35,'&gt;')]      -> survives

So the escape/unescape round trip is only lossy for entity spans that were never protected. xml-text protects them and the corruption disappears; every format without that flag is exposed. Raw data: xliff-flag-probe.json, entity-format-matrix.json.

Two caveats, so this is not overstated. Protection preserves the entity but not the surrounding whitespace — the XLIFF targets came back as < script >, with spaces the provider inserted around the protected spans. And safe-html defeats the protection completely: with that flag set, XLIFF's targets are byte-identical to JSON's, so all four formats converge on the same corrupted output.

checks was empty on every unit in all eight components, corrupted or not.

I already tried

• I've read and searched the documentation.
• I've searched for similar filed issues in this repository.

(Searched for prior art on entity handling, double escaping, escape_text/unescape_text, and safe-html removing translation content.)

Nearest existing items, none of which cover this:

• Ampersands are sent to machine translation as html escapes #12936 / fix(machinery): really fix machinery XML unescaping #12938 introduced the forced uncleanup that this report is about. It was the right fix for its case — raw ampersands coming back escaped — and its regression coverage is written around that case, not around source text that already contained a literal entity before translation. Any fix here needs to keep Ampersands are sent to machine translation as html escapes #12936 fixed.
• Automatic translation fails for a component, if a contained string contains an rst inline literal with backslash #17048 / fix(machinery): do not interpret replacement string for placeholders #17050 fixed a different defect in the same restoration function (re.sub interpreting escape sequences in the replacement string) by switching to a callable replacement. It made restoration stop corrupting the replacement; it does not touch the escaping level.
• Discussion HTML entities get converted to characters in translations #6657 and HTML-escaped strings are shown as HTML #11995 concern entity display and file-format handling, not the machinery round trip.
• Compare number of HTML/XML entities in check #6478 proposed an entity-count check. That would be a useful oracle for exactly this failure, but it was closed as stale.
• There are HTML entities instead of literal characters in Automatic suggestions for DeepL #6958 concerns the inverse DeepL behaviour (entities being introduced), not entities being decoded.

Most relevant of all, and the reason I think the safe-html half of this is not controversial:

• Entering < in a translation clears the translation field and saves an empty string #18967 — "Entering < in a translation clears the translation field and saves an empty string". Same end state as here (the safe-html autofix destroying translated content), reached by a different route: a human typing into an Android resource unit rather than machine translation writing to it. It was accepted as a defect and fixed by PR fix(formats): improved android safe-html flag handling #18997, at the file-format boundary. The machinery path is untouched by that fix, which is why this is adjacent rather than duplicate — but the principle stated in it applies directly here: "Under no circumstances should Weblate clear text the translator has entered." Machine translation output that a reviewer is about to see deserves the same guarantee.
• Erroneous "XML tags in translation do not match source" error when using entities #17093 — "Erroneous 'XML tags in translation do not match source' error when using entities", closed as not planned. Adjacent entity-fidelity concern on the human-editing path, in the opposite direction: a check firing when it should not, rather than content vanishing with no check at all.

I could not find any existing report covering the machinery escape/unescape path specifically.

Steps to reproduce the behavior

A self-contained script using only the public REST API is attached below. Manually:

1. Configure google-translate-api-v3.
2. Create a JSON component with check_flags set to safe-html.
3. Add a source string containing a literal escaped tag, e.g.
   To disable it, write <script> in the template.
4. Add a de translation and run
   POST /api/translations/{p}/{c}/de/autotranslate/ with
   {"mode": "translate", "q": "state:empty", "auto_source": "mt", "engines": ["google-translate-api-v3"], "threshold": 10}.
5. Read the unit back. The target is Um es zu deaktivieren, schreiben Sie — everything from the escaped tag onwards has been removed. checks is empty.
6. Repeat with check_flags empty. The target is now
   Um es zu deaktivieren, schreiben Sie <script> in die Vorlage. — the escaped tag has been promoted to live markup instead of deleted.

Reproduction script:

# repro-mt-entity-roundtrip.py -- public REST API only, creates and deletes a
# disposable project.
#   export WLTOKEN=<admin token>; export WEBLATE_URL=http://localhost:8080
#   python3 repro-mt-entity-roundtrip.py                 # safe-html
#   python3 repro-mt-entity-roundtrip.py --no-safe-html  # without the flag

Output on 2026.7.1, safe-html, 3 locales × 4 strings → 6/12 units corrupted:

=== de: autotranslate HTTP 200 {'details': 'Automatic translation completed, 4 strings were updated.'}
  escaped_script   CORRUPTED  checks=[]
    source: 'To disable it, write <script> in the template.'
    target: 'Um es zu deaktivieren, schreiben Sie '
  escaped_tag      CORRUPTED  checks=[]
    source: 'The <b> tag makes text bold.'
    target: 'Das -Tag kennzeichnet Text als fett.'

Without safe-html, 7/12 corrupted:

  escaped_script   CORRUPTED  checks=[]
    source: 'To disable it, write <script> in the template.'
    target: 'Um es zu deaktivieren, schreiben Sie <script> in die Vorlage.'

Expected behavior

A source string containing a literal HTML entity should round-trip through machine translation with its escaping level intact — <script> in the source should stay <script> in the target, not become <script> and not disappear.

Concretely, any of:

• Shield entity spans instead of relying on double-escaping, building on the mechanism that already exists. Under xml-text, Weblate already highlights entity references and protects them as non-translatable spans, and units with that flag do not exhibit this corruption. Extending that protection to entity spans generally, independent of the flag, would address this without inventing a new mechanism. To be clear this is a starting point rather than a finished fix: xml-text protection is not lossless (the provider inserted whitespace around the protected spans) and safe-html defeats it entirely, so it demonstrates feasibility, not a drop-in solution. This is deliberately not a proposal to drop the unconditional unescape — that unescape is what fixes Ampersands are sent to machine translation as html escapes #12936, and removing it would regress that. The narrower change is to stop pre-existing entity spans from entering the escape/unescape pair at all.
• Detect the mismatch rather than absorb it. If the provider's reply does not contain the escaping level that was sent, that is a signal the round-trip failed — surfacing it as a check or a machinery error would at least make the loss visible instead of silent. Compare number of HTML/XML entities in check #6478's entity-count idea would serve here.
• At minimum, make the safe-html autofix not delete content that machine translation itself introduced in the same operation. Removing an entire clause with no check, no warning and a success-shaped API response is the part that turns a quality problem into a data-loss problem.

Screenshots

N/A

Exception traceback

No exception is raised — that is central to the report. Every call returns 200 and reports success.

How do you run Weblate?

Docker container

Weblate version

2026.7.1 (image weblate/weblate:2026.7.1.1)

Still present on main (verified at 8a20ef28): cleanup_text, uncleanup_text, uncleanup_text_item, escape_text, unescape_text and make_re_placeholder are byte-identical to the tested tag; force_uncleanup and the uncleanup_results call are unchanged apart from being relocated into a new _apply_downloaded_translations helper, which still writes the provider result to cache before uncleanup runs. weblate/checks/placeholders.py, weblate/machinery/googlev3.py and weblate/trans/autofixes/html.py are byte-identical. That is a method-level comparison, not a claim about everything else that changed in the intervening commits.

Weblate deploy checks

N/A — reproduced on a minimal, disposable single-node instance for isolation.

Additional context

Found while building an edge-case corpus for machine translation. Note that an earlier round of this work measured provider APIs directly and drew the opposite conclusion — that Weblate fails to unescape Google v3 output. That was wrong: XMLMachineTranslationMixin.unescape_text() does unescape, and with force_uncleanup = True it runs on every result. The actual defect is the reverse — Weblate unescapes output that the provider had already decoded once.

Happy to share the full instrumented transcripts (source → cleanup_text output → provider reply → stored value, for 5 engines across 4 locales) if useful for a regression test.