(towards #3083) Extend treesitter frontend by sergisiso · Pull Request #3558 · stfc/PSyclone

sergisiso · 2026-08-13T10:14:58Z

This PR focuses on extending the number of treesitter nodes that are recognised and handled, and adds many small Fortran snippets in the tests that are already parsable, but it is still:

ignoring the Reader arguments
not isolating fparser (so its performance penalty is still paid)
not tested with real world applications

…lting nodes

…sed PSyir

LonelyCat124 · 2026-08-13T10:21:28Z

@sergisiso At some point (after this PR is merged) can I have a look at trying to get comments into this as well? Partly to try to undesrtand how this works and partly to try to keep our current level of comment support.

codecov · 2026-08-13T11:00:24Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (bb366f9) to head (5a717cb).
⚠️ Report is 24 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##            master     #3558    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files          397       397            
  Lines        55758     56545   +787     
==========================================
+ Hits         55758     56545   +787

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

… to states that the pinned version does not produce

…ntain strings

sergisiso · 2026-08-17T12:00:11Z

@LonelyCat124 This is ready for review, I keep finding things that could be better but the PR is already huge and at some point I need to cut the PR. It supports a lot more Fortran now but it is not mature enough to replace workflows yet (real_application->transformation_script->backend is still fragile).

Note that I used AI assistance, I spent quite a bit of my effort getting the high level fronted design as I wanted (this is the _process_node, handler selection, using generators, utility methods, scoping with contextmanager, meaning of errors, ... I tried to document all these) but code inside the handlers and test is largely generated (just asking to follow certain patterns, e.g. 'test should start with real Fortran snippet and assert against the produced PSyIR'). Feel free to say if any large sections need to be changed.

LonelyCat124

I added the few comments I managed today, I'm going to be off the rest of the day again but I'll try to finish the review off tomorrow.

LonelyCat124 · 2026-08-18T10:33:06Z

-''' PSyIR TreeSitter Fortran reader '''
+'''
+
+PSyIR fronted to ingest Fortran using the TreeSitter parse generator.


s/fronted/frontend

Also not sure "ingest" is a good word? But will leave that decision to yhou.

Updated this line.

LonelyCat124 · 2026-08-18T10:33:49Z

+https://github.com/stadelmanma/tree-sitter-fortran/blob/master/grammar.js
+
+Note that psyclone is pinned to a particular version of treesitter (in
+pyproject.toml), use that branch instead of master to follow the grammar.


Do we have a plan on how we keep up-to-date with treesitter? How fast moving is the grammar?

Their grammar seems relatively stable nowadays, I would just update when we need a new feature.

I decided that pinning to a particular version (similar to what we do with fparser), to have a fixed grammar, and avoid lots of defensive checks that we would need to do if we cannot guarantee the exact children of each tsnode.

LonelyCat124 · 2026-08-18T12:12:30Z

+    tsnode: Optional['TSNode'], types: Union[str, Container[str]]
+) -> Generator['TSNode']:
+    ''' Provides a generator to iterate over the provided tsnode
+    chidlren of the given type(s).


LonelyCat124 · 2026-08-18T12:55:17Z

                )
-                children.append(code_block)
+            return None
+        if expect is not _NodeExpectation.LIST:


Can we do this check at the start of the routine instead since it doesn't rely on the result at all.

LonelyCat124

I put a couple more comments - I'll have to review the rest later this week or next week when back to 100% to make sure I'm not missing anything.

LonelyCat124 · 2026-08-19T10:21:40Z

+    '''
+    check_types = (types,) if isinstance(types, str) else types
+    if tsnode:
+        for child in tsnode.children:


I'm not sure what the best way to do yield functions is - should the bulk of the cost be on the first call or is it expected for each call to be similar? Since you could do

for child in [child for child in tsnode.children if child.type in check_types]: yield child

And have all the computation's cost be at the start? Not sure which is better.

Actually I don't even know if list comprehensions are done lazily with that or not...

I googled a bit, so the list comprehension is not done lazily. If we did a generator statement (with ( instead of [) then it would be done lazily (and thus be similar to your current implementation) so that suggests to me that we should make the generator be lazy and not precompute it, so happy to leave this as is if you are.

Yes, the idea of the generator was precisely to not precompute things so when we search a tree we stop at the instance found instead of iterating the whole tree.

LonelyCat124 · 2026-08-19T12:16:04Z

@sergisiso One comment I do have is that it might be worth splitting the tests up into more files (by input/output nodes) - I don't like having one large "generic test" file for treesitter like we have for fparser, I think some more modular would make it much easier.

LonelyCat124

@sergisiso I didn't go through the tests yet, but I did finish fortran_tree_reader.py now. Most of the comments are either questioning the input source to reach certain failure states, or poor documentation or typehinting.

LonelyCat124 · 2026-08-24T08:28:57Z

+        )
+        return code_block
+
+    def _get_handler(self, tsnode: 'TSNode') -> Callable:


Return type question here - We reach the end of this function and not return anything. First should we explicitly return None (so we always have coverage)? Return type should be Callable | None or Optional as you prefer.

LonelyCat124 · 2026-08-24T08:30:14Z

-    def _translation_unit(self, tsnode: 'TSNode') -> nodes.Node:
+        # If at this point we still don't have a handler, it is unsupported
+        raise NotImplementedError(
+            f"Unsupported '{tsnode.type}' tree-sitter node.") from None


Why does this raise from None?

LonelyCat124 · 2026-08-24T08:35:57Z

+
+    def _translation_unit_handler(
+        self, tsnode: 'TSNode'
+    ) -> nodes.Node:


returns a FileContainer not just nodes.Node - probably worth being specific wherever we can.

LonelyCat124 · 2026-08-24T08:37:19Z

-    def _module_handler(self, tsnode: 'TSNode') -> nodes.Node:
+    def _module_handler(
+        self, tsnode: 'TSNode'
+    ) -> nodes.Node:


nodes.Container

LonelyCat124 · 2026-08-24T08:38:13Z

+        with self._using_scope(container.symbol_table):
+            visibility_map = self._process_access_statements(tsnode.children)
+
+            # This nodes are already processed


Can you add what processes these to the comment?

The comment isn't quite accurate either as internal_procedures are handled after, not already processed.

LonelyCat124 · 2026-08-24T13:18:02Z

+            return nodes.IntrinsicCall.create(intrinsic, args)
+        except (TypeError, ValueError):
+            raise NotImplementedError(
+                f"Unsupported operands for {intrinsic.name}") from None


Test from this requires monkeypatching, is this actually reachable code we want to create a Codeblock for?

LonelyCat124 · 2026-08-24T13:20:52Z

+        '''
+        ident = next(children_of_type(tsnode, "identifier"), None)
+        if ident is None:
+            raise NotImplementedError(


What happens if we just allow this?

LonelyCat124 · 2026-08-24T13:22:14Z

+        if before:
+            lower = self._process_nodes(
+                before[0], _NodeExpectation.EXPRESSION)
+        if not after:


Is this valid input source?

LonelyCat124 · 2026-08-24T13:22:32Z

Split the tests please, I've not otherwise reviewed them yet.

LonelyCat124 · 2026-08-24T13:24:09Z

Generally I'd like more comments/docstring explanations, A lot of the code is fairly understandable, but there is little discussion of the treesitter structure that it relies on, which is useful both for developers to understand from this file, but also makes it easier to diagnose issues if treesitter structures were to change in the future.

LonelyCat124

@sergisiso I didn't go through the tests yet, but I did finish fortran_tree_reader.py now. Most of the comments are either questioning the input source to reach certain failure states, or poor documentation or typehinting.

sergisiso added 18 commits June 25, 2026 09:58

Start implementing treesitter declarations parsing

81a9dac

Merge remote-tracking branch 'origin/master' into continue_ts

d2167a0

#3083 Add more treesitter handlers

5290f8c

#3083 Improve treesitter frontend implementation

cc94d6c

#3083 Improve treesitter frontend implementation

14b3080

Merge remote-tracking branch 'origin/master' into continue_ts

efabd3b

Rename treesitter frontend support class name

a959e80

Merge remote-tracking branch 'origin/master' into continue_ts

240b333

Merge remote-tracking branch 'origin/master' into continue_ts

9e47059

Introduce treesitter redirection handlers to simplify code

0ad138c

Add treesitter support for creating CodeBlock.Structure.STATEMENTs

a2eb6f8

In treesitter add a process_node expect argument to validate the resu…

1e12e35

…lting nodes

Remove treesitter unneeded class lists of node types

7544ca2

Improve treesitter comments and store CommonDeclarationAttr as proces…

32cd259

…sed PSyir

Clean up tree sitter implementation

149a0b0

Improve treesitter testing

ebfe782

Merge remote-tracking branch 'origin/master' into continue_ts

571f176

Improve treesitter frontend utilities names and docstrings

5087395

sergisiso self-assigned this Aug 13, 2026

sergisiso added the in progress label Aug 13, 2026

Fix flake8

15919c5

sergisiso added 7 commits August 13, 2026 14:39

Fix some issues in the Treesitter frontend and improve its test coverage

7081ef2

Improve some aspects of the treesitter frontend generated PSyIR

a6eeecc

Fix python 3.9 issues

20c80df

Make sure treesitter codeblocks result in valid Fortran code

4337f61

#3083 Bring to master

8dc9d06

Improve treesitter capturing of UnsupportedFortranType symbols

677e735

#3083 Pin treesitter to a single version and remove defensive testing…

51def63

… to states that the pinned version does not produce

sergisiso added 2 commits August 17, 2026 12:05

#3083 Simplify treesitter generator utility

705dcd4

Fix get_fortran_lines for specific fparser codeblocks that already co…

5e68154

…ntain strings

sergisiso requested a review from LonelyCat124 August 17, 2026 11:57

sergisiso added ready for review and removed in progress labels Aug 17, 2026

LonelyCat124 reviewed Aug 18, 2026

View reviewed changes

#3083 Clean up treesitter frontend

5a717cb

LonelyCat124 reviewed Aug 19, 2026

View reviewed changes

LonelyCat124 requested changes Aug 24, 2026

View reviewed changes

LonelyCat124 added reviewed with actions and removed ready for review labels Aug 24, 2026

Uh oh!

Conversation

sergisiso commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LonelyCat124 commented Aug 13, 2026

Uh oh!

codecov Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sergisiso commented Aug 17, 2026

Uh oh!

LonelyCat124 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LonelyCat124 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LonelyCat124 commented Aug 19, 2026

Uh oh!

LonelyCat124 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LonelyCat124 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

sergisiso commented Aug 13, 2026 •

edited

Loading

codecov Bot commented Aug 13, 2026 •

edited

Loading