[teofr]

Single line comments (regular and natspec) are lexed separately, essentially rolling back #2226, the natspec module now is in charge of filtering out trivia from the lexer stream, while collecting relevant NatSpec comment (that may span several lexemes in the case of single line) on a NatSpecStack.

This PR degrades the parser between 1% and 3% in Ir, #2254 improves on that and brings down regressions to around 1.5%-2%. #2269 further improves on it with only a handful of benchmarks regressing around 1%, however I'm not sure placing such a strong restriction on a public function is a good idea, and it may be targetting the benchmarks too much.

@OmarTawfik what do you think of the tradeoff? I think some of the regression is real work, a lot of it is hindering the optimiser by making the parser/lexer loop more complex, as shown by the two other PRs

[changeset-bot]

⚠️ No Changeset found

Latest commit: c8e0e5b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[Copilot]

🟡 Changes recommended

A local doc comment triggers a warning that becomes a compilation failure under CI’s -Dwarnings configuration.

1 open finding

• Use a regular comment to avoid unused_doc_comments build failure · New

What changed in this PR

Restores per-line lexing for single-line comments while merging consecutive NatSpec lines during parser token filtering.

Changes:

• Simplifies comment lexer definitions across Solidity, pragma, and Yul contexts.
• Introduces parser-side NatSpec collection and merging.
• Updates lexer and parser coverage for comments and line endings.

File	Description
parser/​tests.rs	Tests merged NatSpec lines and separators.
parser/​natspec.rs	Implements NatSpec collection and token filtering.
parser/​natspec_stack.rs	Removes the superseded stack module.
parser/​mod.rs	Integrates the new NatSpec token stream.
lexer/​tests.rs	Updates expectations for per-line comments.
lexer/​mod.rs	Exposes lexemes internally.
lexer/​contexts.generated.rs	Regenerates simplified lexer patterns.
definition.rs	Defines per-line comment scanners.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Bencher Report

Branch	teofr/split-single-line-comments
Testbed	ci

⚠️ WARNING: Truncated view!

The full continuous benchmarking report exceeds the maximum length allowed on this platform.

🚨 2 Alerts

🐰 View full continuous benchmarking report in Bencher

[teofr]

Rebased and spent some more time looking into this.

---

3. Keeping the /// comment actual range, without the line break. Nothing needs it, and memory_safety.rs trims it twice today already.

is implemented, it's the right call.

---

After #2280 (comment), next_lexeme is already not inlined, so this PR is now "free" (or rather, #2280 paid the price).

Given that, I left this PR to be a functional change, without performance work. I think we may be spending too much time into how rustc and LLVM are arranging our code in library benchmarks, that is also not a faithful representation of how a final application binary will behave.

cc @OmarTawfik what do you think?

---

As a note, and a learning from this, the skip feature in logos is not really well behaved, since it creates a chance for stack overflow on recursive regex matching functions. This can be fixed with the state_machine_codegen, but the resulting lexer is not as performant.