Migrate from Wattsi to Bikeshed

@tabatkins

The HMTL spec is built using Wattsi, which is different in many ways to Bikeshed. The differences can be a bit of hassle for people who are used to Bikeshed, although one does get used to Wattsi's syntax after a while. Bikeshed source can be more compact autolinks and some Markdown syntax is used, but doesn't have to be. See whatwg/html#11026 for an initial discussion of this topic.

Known blockers/requirements:

Performance. @tabatkins has a plan to make Bikeshed's parser pass much faster using lxml.
A tool to faithfully convert the source to Bikeshed. In order to support converting in-flight PRs, it has to work reliably over time and not just at a single point in time. I have been experimenting in Add experimental support to build using Bikeshed #296 understand what this would entail.
Stability guarantees so that the HTML build isn't broken by Bikeshed changes. This can hopefully be addressed in Bikeshed's CI.
Multiple document support speced/bikeshed#269

Discuss!

cc @annevk @domenic @domfarolino @sideshowbarker @zcorpan

Some notes on cross-references in Wattsi and Bikeshed after experimenting in #296.

In Wattsi:

<dfn>s have a single "topic" string coming from either a data-x attribute or the text content. The topic string isn't necessarily human readable, it's often the intended ID. The data-x="" attribute can appear on a child element, and is effectively hoisted. The for attribute is translated to data-dfn-for but otherwise ignored.
Links are created by <span>, <i>, and <code> elements. There are subtle differences, but basically the "topic" is extracted and the <dfn> looked up using that.

Typical Wattsi usage:

Something defined as <dfn data-x="concept-response-type">type</dfn>.

It is linked as <span data-x="concept-response-type">type</span>.

In Bikeshed:

<dfn>s have a set of "linking texts" extracted from the lt attribute (split on "|") or the text content. The for attribute is typically used.
Links are created by <a>foo</a> or shorthands like [=foo=] for concepts and {{foo}} for code.

Typical Bikeshed usage:

Something defined as <dfn for=response>type</dfn>.

It is linked as <a for=response>type</a> or [=response/type=], or [=type=] if unambiguous.

Many cases can be translated just fine.

The biggest challenge so far is translating the pattern <dfn export data-x="concept-response-type">type</dfn>. <dfn export lt="concept-response-type">type</dfn> doesn't work because the output would be data-lt="concept-response-type", and we shouldn't change the exported linking texts. The workaround I've found is using local-lt="concept-response-type" but it's more verbose and incompatible with in compact links to the definition like [=response/type=].

I think some heuristics may be needed to add for attributes, but I don't know how reliable it can be.

We already annotate exported <dfn>s with appropriate for attributes and such today. How much work would it be to annotate the remainder of <dfn> elements in the same manner? Because then we have an answer for how each data-x is to be converted (and we need to keep that string as well, to be clear, but as ID).

Yeah, I think all those are gonna need a manual pass to turn them into for attributes, so <dfn id=concept-response-type for=response>type</dfn>. A bunch are already done that way, and it'll be a strict improvement for the ecosystem to have them namespaced better anyway.

Links are created by <span>, <i>, and <code> elements. There are subtle differences

What are the differences? Is it enough to just replace span with a, and i/code with an a child? (Or parent, whatever Wattsi currently does for those when it adds links.)

@tabatkins

Performance. @tabatkins has a plan to make Bikeshed's parser pass much faster using lxml.

To be clear here, after a bit of tweaking a few weeks ago to kill some accidental quadratics, Bikeshed is already on par with Wattsi for performance. Exact numbers depend on environment, but it seems to process in ~1min, which is about how long Wattsi takes. A decent chunk (~20%?) of that is due to the duplicate parsing pass I'm still performing, which'll go away with the conclusion of the parser-rewrite project this year. There's even more potential for improvement with a larger project I have in the wings to replace LXML as the tree model entirely, but that might not happen this year, and if it does it'll be late in the year.

In order to support converting in-flight PRs, it has to work reliably over time and not just at a single point in time. I have been experimenting in #296 understand what this would entail.

I feel like this isn't actually required, at least not to the extent that this text makes it seem. Converting in-flight PRs is more or less a one-time cost; after that, the source and all extant PRs are all in Bikeshed and we dont' have to worry about it again. Plus, most PRs are fairly small; even if we chose to commit to "Tab manually converts all extant PRs by hand", it's not much work.

The only reason it would have to work "over time" is if you're concerned about edits that are currently in the process of being written, but haven't yet been committed to the repo as a PR, at the time of the switchover. Again, this seems like a fairly small set of things, and amenable to just handling manually as needed.

[blockers]

The other big blocker is multipage output, which I'm working on right now and expect to have done by the end of the month.

Stability guarantees so that the HTML build isn't broken by Bikeshed changes. This can hopefully be addressed in Bikeshed's CI.

Yeah, I already test a whole bunch of specs as part of my CI, so I can see when something is going to break. HTML would definitely become a part of that.

We already annotate exported <dfn>s with appropriate for attributes and such today. How much work would it be to annotate the remainder of <dfn> elements in the same manner? Because then we have an answer for how each data-x is to be converted

There's a total of of 6794 <dfn> elements now, excluding <dfn data-x=""> which aren't processed. Breakdown:

4298 with neither export nor for
2069 without export but with for
290 with export but no for (is this the category that was ~done?)
137 with both export and for

So that's over 4500 <dfn>s with no for attribute.

(and we need to keep that string as well, to be clear, but as ID).

Yeah, this is what I've done. I think it'll be a lot of IDs in the source however, so I'm thinking about an HTML mode for Bikeshed to generate IDs in the style of the HTML spec, so that the generated IDs are correct more of the time.

~1min, which is about how long Wattsi takes.

This is not accurate. Wattsi takes ~4 seconds. Even the build server version (which includes upload/download times and is running on a relatively slow machine) takes 14 seconds.

Huh, I was basing those assumptions on the runtime of the "build" step in https://github.com/whatwg/html/actions/runs/13847732392/job/38749434294, but it does look like that's doing a bunch of other stuff. There's no sub-step timing information, so I can't tell what's what. Any chance of a breakdown in the relative times of those things, so I have a better benchmark?

I imagine most of that time is spent on:

Downloading Docker images and setting up Docker environments
The HTML validator
The Prince PDF generator, including setting up a server to serve the files

The CI time is not really relevant as CI builds are not what people iterate based on. I'd suggest doing local Wattsi builds and timing those.

Here's an example on one of my slower machines.

domenic@Domenic-White:/mnt/c/Users/d/OneDrive - domenic.me/Code/GitHub/whatwg/html$ time wattsi --single-page-only source deadb33f ../html-build/out default ../html-build/.cache/mdn-spec-links-html.json
Parsing MDN data...
Parsing...
Generating HTML variant...
Saving index-html

real    0m10.117s
user    0m2.269s
sys     0m0.600s

Cool, I'll check that out.

I will say, tho, that 4s is, uh, not obtainable with Python. Absolutely out of the question. Even if Bikeshed did literally nothing but run the source thru the initial html parser and then immediately quit, that's currently about 20s of time for a document the size of HTML.

My planned projects will be knocking substantial chunks off the runtime still. I think 40s is very obtainable by the end of the year, and possibly down to 30s. I've also done zero real perf hacking on the parser so far, and am very interested in what possibilities might exist there. But it would be good to know, for the sake of this issue, whether a 1min build time is an acceptable tradeoff for the benefit of moving to a more widely-used build system.

(A longer-term project is, of course, learning Rust and porting Bikeshed to it. That should give some massive benefits. If it's important I can bump the priority of trying to do that, but it'll still be a longer term project, with a 2026 completion likely.

I will say, tho, that 4s is, uh, not obtainable with Python. Absolutely out of the question. Even if Bikeshed did literally nothing but run the source thru the initial html parser and then immediately quit, that's currently about 20s of time for a document the size of HTML.

Yeah, I sympathize, but that's why I think it's always been a long shot to consider moving HTML to Bikeshed. (Instead of, e.g., improving Wattsi to accept more Bikeshed syntax.)

But it would be good to know, for the sake of this issue, whether a 1min build time is an acceptable tradeoff for the benefit of moving to a more widely-used build system.

It would not be. We do frequent rapid iteration and need our build tools to support this. I think 15 seconds on a slow machine like the one I'm testing now would be my hard limit.

I think we can instead get many of the same benefits from projects that improve the existing HTML build tooling so that people writing the HTML spec don't need to be as aware of the differences, or deal with the more burdensome things that Bikeshed-style preprocessors can do for you. E.g., #293.

Separate from performance:

An issue we discussed at WHATNOT is that just doing a minimal conversion isn't really worth it, if the result is a source file that looks basically like a Wattsi source file. We'd want to be properly using a large set of Bikeshed-isms, e.g. the definition-scoping mentioned above, but also things like [=short/link=]s, Markdown paragraphs and other syntax, etc. We tentatively identified the goal of describing our desired subset of Bikeshed features to use as an initial step, so getting that discussion started somewhere would be good.

This also seems important to establish ahead of time since I suspect such features will increase the processing time.

Stability guarantees so that the HTML build isn't broken by Bikeshed changes. This can hopefully be addressed in Bikeshed's CI.

Yeah, I already test a whole bunch of specs as part of my CI, so I can see when something is going to break. HTML would definitely become a part of that.

This is not the kind of stability Philip and I discussed. Our concerns with Bikeshed stability stem from issues like its CI being constantly red, its PRs not receiving any code review from a second person, or the biggest breaking changes in Bikeshed history (the new parser) being released with no release notes or notification, or the whole definition panels saga.

(Briefly: we tried to maintain our definition panel styling ourselves, for uniformity across the WHATWG spec ecosystem and because we believe that fewer inline styles and scripts is better. Bikeshed started changing the markup structure with no warning, breaking definition panels for all WHATWG specs. We tried to keep up for a few weeks, but eventually had to give up and let Bikeshed put a bunch of inline styles and scripts into the documents because the markup structure kept changing too rapidly.

As a consequence of this, one day there was a huge regression in definition panels usability. Over the next few weeks, some of the regression was fixed, but the unwanted animations stayed around, and our requests to have them reverted were rebuffed.)

@foolip

It would not be. We do frequent rapid iteration and need our build tools to support this. I think 15 seconds on a slow machine like the one I'm testing now would be my hard limit.

Okay, that's good to know. @foolip, that changes the calculus for you significantly, then.

We'd want to be properly using a large set of Bikeshed-isms, e.g. the definition-scoping mentioned above, but also things like [=short/link=]s, Markdown paragraphs and other syntax, etc. [...] This also seems important to establish ahead of time since I suspect such features will increase the processing time.

I agree that being able to properly use Bikeshed's features would be useful. Based on my own explorations in this area, your fear about cost should be unfounded. I tested with "15 copies of DOM stapled together", which uses all the Bikeshed features and is roughly the same source size as HTML, and it's comparable in runtime to the roughly-bikeshedded HTML. Basically those features are either cheap enough to not make a difference, or you're already paying for them regardless of whether you use them.

some tangential details

The vast majority (about 75%) of processing time in Bikeshed is in the initial parsing: out of a ~60s runtime for the bikeshedded html spec on my laptop, it's spending 27s in *my* html parser, 16s in LXML's html parser, and 4s in my markdown parser. All the rest of Bikeshed's processing is 13s in total. I can certainly squeeze a little more blood out of that last "everything else" bit (I just knocked a second off with some trivial changes I spotted from the profile I generated to get these numbers...), but most of the perf win will be in improving parsing time, most importantly killing the extra LXML parse. That said, the LXML parse is still 16s, and that's with a mature project largely written in C, so I suspect that's about the lower bound I can hit; Python just costs a lot. We'll see, tho; I do have some big ideas here that might just be impossible for LXML to pursue, given their desired API shapes. In any case, moving to Rust will likely be the only "real" solution for knocking this out, since it's just a lot of string munging and struct creation.

its CI being constantly red

You've mentioned this in the past, and I've explained why this is wrong in both its details and its overall gist.

The CI is always green when I cut releases, which is what's important for users. Virtually all usage of Bikeshed is via the pypi releases.
The vast majority of real-world time is spent with green CI anyway
The vast majority of CI redness is just lint issues, which I almost always immediately clear anyway. Most of the rest is code on branches, which mixes with the rest of the things in the Actions tab, and is totally okay to be red during development.
"being red" is what CI is for, to a first approximation. As long as the audience being exposed to the redness is minimal and prepared for it, this is 100% okay.

I would appreciate it if you would stop bringing this up, as it's simply inaccurate.

its PRs not receiving any code review from a second person

An unfortunate consequence of Bikeshed being a 1-person dev team in practice, yeah. Would be nice to have more engineers on it, but that hasn't been something worth pursuing headcount for.

the biggest breaking changes in Bikeshed history (the new parser) being released with no release notes or notification,

This was postmortem'd, and changes were made to my process to ensure it didn't happen again. (The issue was a mistake in how I was testing, so I missed some very significant changes that should have prevented the release). I'll note that the subsequent re-release of the new parser was indeed smooth, and another similarly-large change recently (the upgrade to move all special inline syntaxes into the parser) went off extremely smoothly, with virtually all "issues" downstream users encountered being existing errors getting caught that were previously being silently ignored, sometimes resulting in actual errors making it into specs.

the whole definition panels saga

Yup, I wasn't particularly happy with that either; some of that usability regression should have been caught by manual testing. I was trusting a new dev on the project to do things well; growing pains.

That said, I don't see any "rebuffing" in that issue. I disagreed and thought (still think!) some aspects of the new behavior were reasonable, and there were no further responses on the thread so I assumed it was acceptable. If this is incorrect and y'all are still annoyed with any of this, please leave a comment to that effect. As you well know from spec editting, silence is usually taken as acceptance.

I'm assuming that we'll drop the "HTML in Bikeshed" project for now until/unless Bikeshed is able to deliver more comparable perf numbers. But when that is the case, I'm happy to discuss what more can be done for stability.

I would appreciate it if you would stop bringing this up, as it's simply inaccurate.

I don't plan to stop, sorry. It's not inaccurate, as one can see from a glance at https://github.com/speced/bikeshed/commits/main/. And CI being red is not what CI is for; it's a sign of a project with insufficient review structure and engineering rigor. It's certainly not acceptable in the majority of open source projects, as you can confirm by checking around the ecosystem.

@domenic

@domenic @tabatkins I expect there will be some additional work needed on performance. I think the most useful comparison is how long it takes to run bikeshed vs wattsi from within build.sh, and I think that will dominate the time taken for ./build.sh --fast. On my machine it's currently 32 vs. 3 seconds.

However, the Bikeshed source produced by #296 is still quite broken and produces 45k lines of errors. Just printing the errors takes some time, and I'd like to look at the performance with a more realistic input after some more work.

If it should turn out that parsing is a big chunk of the time, then the LXML approach might help a lot. In a quick test, parsing source-whatwg-complete.bs with lxml.html.parse and walking the tree with LXML takes 0.27 seconds. So ultimately it should come down to how fast Bikeshed is with everything that comes after HTML parsing.

We can also look into a --fast mode for Bikeshed that just skips some work that isn't as important for local iteration.

@tabatkins

The other big blocker is multipage output, which I'm working on right now and expect to have done by the end of the month.

@tabatkins I'll add it to the list up top. Do you have a tracking issue for this?

I'm not going to argue with you further about this, as you're not meaningfully engaging with what I'm saying. It's not currently relevant, and I can deal with it when and if it becomes so.

Do you have a tracking issue for this?

Yup, speced/bikeshed#269

and produces 45k lines of errors.

Hmm, I should do a little testing to see how much that really is costing. I purposely don't pay attention to the cost of printing errors, because they should be rare and it's much better for them to be useful even if it's expensive to construct the message. I can turn off a handful of the common ones and see what that's costing.

If it should turn out that parsing is a big chunk of the time, then the LXML approach might help a lot.

So this turned out to be really important. I was despairing about ever getting a really significant win, enough to bring us back into the realms requested here, since the lxml parser was only about 40% faster than mine even tho it was written in C. But it turns out I was never using the lxml parser, I was using the html5lib parser, which is pure Python and not very optimized (it just uses the lxml tree structure). Swapping out for the actual lxml html parser made that ~16s of parsing time drop down to about 0.25s. It's nowhere near spec-compliant, but now that my parser checks for a lot of document structure issues in the first place, that's not nearly as concerning. It's producing some slightly different document structure so I've got some bugs to iron out, but I'm already quite happy with this - it drops total time down to about 40s on my laptop now, from 60s. (I've made a few more tweaks that account for the additional handful of lost seconds.)

That, then, suggests that there's incredible amounts of win available if I shift my parser into C, or ideally Rust. I know Rust-in-Python has gotten a lot more love in the last few years and should be quite usable these days. So I'll make that a Q2 project. If it goes well, it means I could virtually eliminate all my parsing costs, dropping the entire parse step to 1s or less on the html spec. I'd be left with just the 14s or so of "everything else", and I know there's still juice to be wrung out there. (I might even be able to transfer some of it over to Rust as well!) There's also definitely a few spots that could be turned off for a "fast path" by default when running locally (but turned back on in CI), like verifying that arg names aren't misspelled (0.8s right now) or that 2119 keywords aren't misused (1.2s right now).

So that shifts my entire outlook from "ugh, this is probably impossible" to "this is extremely possible and very likely even in the short term". I'll go ahead and keep my plans for now, and just shift my next quarter's project a bit to accommodate.

@tabatkins

I have tried out @tabatkins's in-progress https://github.com/speced/bikeshed/tree/real-lxml and profiled with and without it to see what it will mean from HTML. (This is using #296 to make the input file.)

Here's the original profile, with html5parser taking ~19% of the time:

On the real-lxml branch that is gone,

Without the profiler the total time goes from ~33 to ~29 seconds on my machine, so it's solid improvement.

Most of the remaining time is spent in:

~30% in earlyParse() where a the source parsed, rewritten and reserialized before being sent to the "real" HTML parser. In particular the <{element}> wouldn't survive the parser, so some of this is strictly necessary, but maybe could be optimized.
~12% in processAutolinks() which includes looking up references
~7% in syntax highlighing, which could be made optional with a "fast" mode

I've also looked into splitting the source into many files, since building a small file is already fast. Splitting using the split-filename attributes there would be 60 files.

The biggest of them is parsing.bs and builds in about 2 seconds. But building all 60 files in parallel (using GNU parallel) still takes 14 seconds, so overall it's only ~2x as fast, less than I was hoping to see.

However, with a build system around this, incremental builds could be very fast, as most of the time only one file is edited. In the most straightforward implementation one would have to have both multipage input and output, but with enough acrobatics perhaps the source could be split and the singlepage output assembled transparently without a real parser. I feel this direction is somewhat promising, but am interested to hear from other editors about preferred source organization and workflows.

Alright, I have the build time on my machine down to 9.5 seconds for the full spec. What I did in foolip/bikeshed@6abba08 just skipping stuff, not optimizing anything:

Skip earlyParse() which converts some Bikeshed-isms into HTML and adds bs-* attributes for later error messages
Skip Markdown parsing in markdown.parse(...)
Skip a bunch of time-consuming things like adding self-links and syntax highlighting that aren't critical when editing the spec

Going down this route would mean defining a subset of Bikeshed that can be used. It's possible some features could be added back and made fast, depending on what features would be the most useful in HTML. Certainly the {{foo}} syntax would be useful.

Would you mind posting a current html.bs file for me to play around with, so I have a more realistic example than "15 copies of DOM stapled together"?

Summarizing some bits of the off-issue convo I've been having with foolip:

skip earlyParse() - can't be done, too much necessary work is done there to make things work correctly later. This is especially true in the real-lxml branch, because the native lxml parser isn't remotely spec-compliant like the html5lib one was, and it breaks a lot of specs in fun ways as a result (for example, omitting end tags usually works badly; earlyParse() is now maintaining a view of the open elements and auto-inserting closing tags as necessary)

However, I have done literally zero perf work on earlyParse() so far. I have no idea if there is low-hanging fruit I could fix for some quick perf wins. I've been focusing solely on features and correctness so far, so we should wait and see what I can actually wring out of it before doing anything drastic in this space.
skip markdown - this is possible, since HTML currently doesn't use any Markdown. It would also let us skip the serialization step from earlyParse() nodes to strings (currently necessary because the Markdown parser works on strings), as we could instead go straight to building the lxml tree via SAX events, which are a trivial mapping from the earlyParse nodes.
skip self-links, syntax-highlighting, etc - this is all very possible; there's a lot of features you don't necessarily need in a local build that we could turn off (and build times on CI are irrelevant as long as they're not ridiculous; the current Bikeshed performance is more than sufficient for that)
processAutolinks() currently doesn't cache its own results in any way, so multiple identical autolinks each duplicate all the search/filtering work the first one performs. I'd bet we can get some nice wins out of just doing some naive caching here.

Everything but the first bullet point is very easy/fast to prototype and test.

I've sent speced/bikeshed#3186 to skip the Markdown parsing.

Would you mind posting a current html.bs file for me to play around with, so I have a more realistic example than "15 copies of DOM stapled together"?

Oops, I never did this. Here it is: source-whatwg-complete.bs.gz

Experimental support is now landed in #296 and https://github.com/whatwg/html/tree/bikeshed-experiment has the manual fixes to HTML that are still needed. I hope to present at TPAC what the current status is and the roadmap ahead.

Navigation Menu

Repository navigation

Migrate from Wattsi to Bikeshed #297

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Migrate from Wattsi to Bikeshed #297

Description

Activity

foolip commented on Mar 14, 2025

annevk commented on Mar 14, 2025

tabatkins commented on Mar 14, 2025

tabatkins commented on Mar 14, 2025

foolip commented on Mar 14, 2025

domenic commented on Mar 14, 2025

tabatkins commented on Mar 14, 2025

domenic commented on Mar 14, 2025

tabatkins commented on Mar 14, 2025

domenic commented on Mar 14, 2025

tabatkins commented on Mar 15, 2025

domenic commented on Mar 15, 2025

foolip commented on Mar 17, 2025

foolip commented on Mar 17, 2025

tabatkins commented on Mar 17, 2025

tabatkins commented on Mar 18, 2025

foolip commented on Sep 10, 2025

foolip commented on Sep 10, 2025

foolip commented on Sep 11, 2025

tabatkins commented on Sep 11, 2025

tabatkins commented on Sep 11, 2025

foolip commented on Oct 8, 2025

foolip commented on Oct 8, 2025

foolip commented on Nov 6, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions