Move main LSP loop off Tokio to an owned thread

[lionel-]

Branched from #1274
Progress towards #1212

Fixes a source of deadlocks when the main loop blocks on a Salsa write until it gains exclusive access, but background tasks hold a DB handle and are scheduled by Tokio to run on the same worker thread as the main loop. We could wrap writes in block_in_place() instead, which instructs Tokio to move all pending tasks to a different worker, but moving the main loop off tokio altogether is really the safer move.

From the documenting comment:

        // Since the main loop owns the Salsa DB and writes to it, we run on its
        // own thread instead of a Tokio worker. Salsa writes are potentially
        // blocking until the writer gains exclusive access. If background tasks
        // holding clones of the DB are stuck on the same thread as the main
        // loop, the LSP deadlocks. This can be avoided by wrapping writes in
        // `block_in_place()` but the safer structure is to have it run on an OS
        // thread that we're in control of.

Fixes a bunch of issues observed in frontend tests: https://github.com/posit-dev/positron/actions/runs/27755066106?pr=14254

We still use tokio for the auxiliary loop and the blocking worker pool, for the time being.

[DavisVaughan]

Do you really need a tokio::task::JoinSet anymore for 1 task?

[lionel-]

That's a nice way of cancelling on abort. Dropping a task handle doesn't cancel it, so we'd need a custom Drop method to call the handle's abort() method.

[DavisVaughan]

Dropping a task handle doesn't cancel it

oh i was not aware of that

[DavisVaughan]

i would just remove this

[lionel-]

Let's keep it as a token of the nicer loop structure

[DavisVaughan]

This one doesn't feel that useful? We log at the beginning of every event type

[DavisVaughan]

Why do we want to know this on every iteration? Why do we want to know this at all? It feels like itll be expected for the db to have outstanding holds in background threads (i.e. its working on diagnostics when we get a did-change update)

[lionel-]

This is not every iteration, this is only when we're about to write to the DB.

This is meant to help debugging hangs from user logs, we can remove this later once proven stable.

[DavisVaughan]

I feel like the previous while look was pretty nice?

You'd just put the lsp::log_warn!("process_diagnostics_queue: channel closed, task exiting"); part after the while loop exits, i think?

[lionel-]

Good point

[DavisVaughan]

Since we don't say Generating diagnostics or Finished diagnostics via tracing::trace! (we only do that via lsp::log_info!), i feel like this isn't actually that useful on the kernel side (i know its not new from you, but maybe we can just remove it)

How do we decide where to log?

[lionel-]

I decided to keep it there because diagnostics code logs at kernel level, so we get a section marker.

In general, the current split is that eventing and dispatch belong to the LSP, handling belongs to the kernel. This is a remnant from r_task()-heavy handling and we have nicer logging facilities on the kernel side (context, profiling).
But: when the LSP becomes standalone, everything moves to the LSP.