Infrastructure Enhancement

[6cdh]

Many features still have significant room for improvement. However, these enhancements often require additional infrastructure that we currently lack, such as:

• A dedicated lexer
• AST traversal for pre-expanded code (result of read-syntax)
• AST traversal for fully expanded code (result of expand)
• Cross-module analysis (dependency graph)
• Language-specific analysis (e.g., improved support for Typed Racket, Rhombus, the little typer's language, the little prover's language)

Adding these new infrastructure could be complex and challenging.

To move forward effectively, I propose starting by creating a enhancement list for each feature.

For example, the enhanced feature list can be:

• code completion when expansion never succeed (requires lexer + AST traversal for pre-expanded code), this should also helps for cases like issue textDocument/completion behaving strangely #163.
• cross module variable renaming (requires cross-module analysis and dependency graph, etc).
• show (function/struct/class/parameter/...) definition for hover request for local project (requires AST traversal for pre-expanded code, and maybe comment/scribble inside those definitions).
• mitigated unused variables diagnostic when upstream can't fix fast, like issue false positive "unused variable" for sig-elems in define-signature #166 (requires AST traversal for pre-expanded code).
• better formatting (requires lexer).

We can take these features one by one, gradually adding and evolving the infrastructure while delivering concrete feature improvements.

This approach will be easier after completing #168 (Extract logic into a library refactor).

[dannypsnl]

About dependencies graph, I found if collect requires information only then it can saves 99% times for each check-syntax (I modify the DrRacket tool lib to test this), so it’s actually able to check every files in a project for deps graph.

[dannypsnl]

Not sure if https://docs.racket-lang.org/syntax/module-helpers.html#%28mod-path._syntax%2Fmoddep%29 is enough to get dependencies graph, but it's worth to give it a try.

[dannypsnl]

Get dependencies graph

drracket/private/syncheck/traversals.rkt's annotate-basic recursively walks the entire fully-expanded syntax tree of a module (every lambda/let body) to produce Check Syntax's arrows/hover data. When we only need dependencies graph, that's mostly wasted work. Here I benchmark four ways to get dependency info (on the same target file) for further development.

A. Full traversal (build-trace%)

What the langserver currently based on.

B. Requires-only walk

Expand once, then scan only the top-level #%plain-module-begin forms for
#%require. Modified from drracket/private/syncheck/traversals.rkt

(define (top-level-forms mod-stx)
  (syntax-case mod-stx (module #%plain-module-begin)
    [(module name lang (#%plain-module-begin form ...))
     (syntax->list #'(form ...))]))

(for/sum ([f (in-list (top-level-forms expanded))])
  (syntax-case f (#%require)
    [(#%require spec ...)
     (for ([s (in-list (syntax->list #'(spec ...)))])
       (module-path-index-resolve (module-path-index-join (syntax->datum s) #f)))
     (length (syntax->list #'(spec ...)))]
    [_ 0]))

C. module-compiled-imports

Get (or compile) the compiled code and ask it what it imports. Measured cold (no cache, forces a full compile) and warm (cache already built).

(require syntax/modcode syntax/moddep)

(define code (get-module-code target-file))
(module-compiled-imports code)

Results (on target: racket-langserver/main.rkt, median of 10 runs)

method	total from scratch (ms)	marginal if expand already done (ms)	specs	memory (MB)
A. full traversal	914.110	9.620	828 entries	0.204
B. requires-only walk	904.545	0.187	13	−0.338
C-cold. module-compiled-imports	104.863	n/a	14	0.319
C-warm. module-compiled-imports	0.582	n/a	14	−0.319