Reduce ProgramCache lock contention and wake only waiters for the same type

[RonYoung666]

Summary

internal/caching/pcache.go currently serializes all cache-miss compilation through a single ProgramCache.m mutex. Because Compute() holds this mutex while executing the potentially expensive compute() function, only one goroutine can compile a program at a time, even when goroutines are compiling different Go types.

This creates unnecessary startup latency and can also cause goroutines waiting for an already-compiled type to remain blocked behind unrelated type compilations.

Current Behavior

ProgramCache.Compute() currently does the following:

self.m.Lock()
defer self.m.Unlock()

if val = self.Get(vt); val != nil {
    return val, nil
}

if val, err = compute(vt, ex...); err != nil {
    return nil, err
}

atomic.StorePointer(&self.p, unsafe.Pointer((*_ProgramMap)(atomic.LoadPointer(&self.p)).add(vt, val)))
return val, nil

This means the global cache mutex protects both:

1. RCU map writes
2. The entire compilation/computation process

As a result, all cache misses are globally serialized.

Problems

1. Different types cannot compile concurrently

During cold start, many goroutines may marshal/unmarshal different struct types for the first time. Even if these types are unrelated, only one goroutine can execute compilation at a time because compute() runs while holding ProgramCache.m.

This can increase startup latency and cause unnecessary blocking under high concurrency.

2. Waiters for an already-compiled type can be delayed by unrelated compilation

Example sequence:

1. Goroutine 1 marshals struct type A, enters Compute(A), acquires the lock, and starts compiling A.
2. Goroutine 2 marshals struct type A, enters Compute(A), and waits for the lock.
3. Goroutine 3 marshals struct type B, enters Compute(B), and waits for the lock.
4. Goroutine 1 finishes compiling A, updates the cache, and releases the lock.
5. Goroutine 3 acquires the lock first and starts compiling B.
6. Goroutine 2 continues waiting, even though A has already been compiled.

The double-check inside Compute() prevents duplicate compilation of A, so this is not a correctness issue. However, it is a latency issue: goroutine 2 may wait for an unrelated compilation of B before it can observe that A is already available.

Expected Behavior

• Cache hits should remain lock-free.
• For the same *rt.GoType, only one goroutine should execute compute().
• Goroutines waiting for the same *rt.GoType should all be woken when that type finishes compiling.
• Different *rt.GoType values should be allowed to compile concurrently.
• Failed compilation should not be cached, preserving current behavior.

Proposed Solution

Use a per-type in-flight call table, similar to a local singleflight implementation.

Suggested internal structure:

type programCall struct {
    done chan struct{}
    val  interface{}
    err  error
}

type ProgramCache struct {
    m       sync.Mutex
    p       unsafe.Pointer
    pending map[*rt.GoType]*programCall
}

Suggested Compute() behavior:

1. First check the RCU cache via Get(vt) without locking.
2. Acquire ProgramCache.m and double-check the cache.
3. If another goroutine is already compiling the same vt:
   • get the existing programCall
   • release the mutex
   • wait on <-call.done
   • return call.val, call.err
4. If no compilation is pending for vt:
   • create a programCall with done := make(chan struct{})
   • store it in pending[vt]
   • release the mutex
   • execute compute(vt, ex...) outside the mutex
5. After compute finishes:
   • reacquire the mutex
   • if successful, update the RCU map
   • store val/err on the call
   • delete pending[vt]
   • close call.done to wake all waiters for this type

The important part is that close(call.done) acts as a broadcast to all goroutines waiting for the same type. This avoids the missed-wakeup issue and does not require waiters to hold the global mutex while waiting.