fixed performance bottlenecks and concurrency issues across core packages

[valentin-kaiser]

Reduce repeated work, allocations and lock contention across core packages, and fix correctness issues found through profiling and integration testing.

• Optimize logging, error traces, configuration reads, version parsing, language matching and AES cipher reuse.
• Add opt-in memory cache sharding and native serialization; fix expiration handling and concurrent access.
• Fix router and JSON-RPC races, reduce request overhead and improve response buffering.
• Correct RabbitMQ scheduling, dequeue timeouts and reconnection; add consumer prefetch and bounded finished-job retention.
• Stream database backups and restores, improve MySQL/PostgreSQL dump encoding and batching, redact backup passwords and close replaced connection pools.
• Enforce SMTP message size limits and reduce message buffering allocations.
• Add benchmarks, integration and race regression tests, containerized test services and a manual benchmark workflow with allocation regression checks.

Behavior changes: queues retain the newest 10,000 finished jobs by default, with configurable unlimited retention. Native cache serialization shares nested values and requires callers to treat them as read-only. PostgreSQL restores preserve existing rows with ON CONFLICT DO NOTHING.

Validation coverage includes concurrent access, database backup/restore round trips and SMTP delivery, authentication and size limits. CI expands race detection and reads the Go version from go.mod.

[Copilot]

🟡 Changes recommended

Concurrent cache/router state and restore/retention behavior contain correctness issues that should be resolved before approval.

11 open findings

• Restore callback hides SQL errors and allows partial commits · New
• Limiter snapshot race can cause nil dereference · New
• Sharded cache can exceed configured MaxSize · New
• Stale cache miss can overwrite concurrent environment registration · New
• Redirected regular files are not recognized as interactive · New
• Ring eviction can delete newer job incarnation · New
• Benchmark keys collide across packages · New
• Benchmark writer headers accumulate across requests · New
• Documentation references incorrect compose file · New
• Compose command uses nonexistent file path · New
• Documented compose filename does not exist · New

What changed in this PR

This PR optimizes hot paths across core packages while adding concurrency safeguards, bounded retention, streaming backups, and performance regression tooling.

Changes:

• Reduces allocations and lock contention in logging, routing, caching, configuration, RPC, security, and version handling.
• Adds queue retention, RabbitMQ consumer improvements, database streaming, SMTP limits, and race regression coverage.
• Introduces broad benchmarks, service containers, and allocation checks.

File	Description
web/​xrpc/​codec.go	Optimizes simple JSON strings
web/​xrpc/​codec_test.go	Generalizes descriptor test helper
web/​xrpc/​codec_bench_test.go	Adds codec benchmarks
web/​stack_bench_test.go	Benchmarks complete web stack
web/​router.go	Adds lock-free routing snapshots
web/​router_race_test.go	Tests concurrent registration
web/​router_ip_test.go	Tests IP filtering concurrency
web/​router_bench_test.go	Adds router benchmarks
web/​response.go	Lazily allocates response state
web/​middlewares.go	Avoids disabled log construction
web/​middleware_bench_test.go	Benchmarks middleware overhead
web/​jrpc/​server_bench_test.go	Benchmarks unary JSON-RPC
web/​jrpc/​jrpc.go	Fixes validation race and allocations
web/​jrpc/​concurrent_test.go	Tests concurrent RPC calls
version/​version.go	Caches and optimizes version parsing
version/​cache_test.go	Tests cached version behavior
version/​bench_test.go	Adds version benchmarks
test/​compose.yaml	Defines integration-test services
security/​bench_test.go	Adds security benchmarks
security/​aes.go	Reuses expanded AES-GCM ciphers
security/​aes_cache_test.go	Tests cipher reuse and concurrency
scripts/​bench.sh	Adds benchmark runner
scripts/​allocs-check.sh	Adds allocation regression checks
queue/​task.go	Reduces task-copy allocations
queue/​task_bench_internal_test.go	Benchmarks task scheduling
queue/​retention.go	Adds finished-job retention
queue/​retention_test.go	Tests retention behavior
queue/​rabbitmq_test.go	Expands RabbitMQ integration coverage
queue/​queue.go	Uses timers for retries
queue/​parallel_bench_test.go	Benchmarks queue contention
queue/​memory.go	Applies memory-queue retention
ptr/​bench_test.go	Benchmarks pointer helpers
mail/​smtp_integration_test.go	Adds SMTP integration coverage
mail/​server.go	Enforces message-size limits
machine/​bench_test.go	Benchmarks machine identification
logging/​zerolog.go	Adds atomic logger state
logging/​stream.go	Uses a ring log buffer
logging/​standard.go	Optimizes standard logging
logging/​registry.go	Makes registry state atomic
logging/​race_test.go	Adds logging race tests
logging/​log/​log.go	Reuses no-op adapter
logging/​log/​bench_test.go	Benchmarks logging facade
logging/​interactive.go	Expands output-mode detection
logging/​bench_test.go	Adds adapter benchmarks
logging/​adapter.go	Adds level-enabled helper
i18n/​middleware.go	Caches language matchers
i18n/​middleware_cache_test.go	Tests matcher invalidation
i18n/​i18n.go	Invalidates matcher on registration
i18n/​bench_test.go	Adds localization benchmarks
flag/​flag.go	Caches environment variable names
flag/​bench_test.go	Tests and benchmarks name caching
etcd/​lock.go	Avoids unnecessary lock sessions
etcd/​etcd_test.go	Generalizes etcd test helpers
etcd/​bench_test.go	Adds etcd benchmarks
database/​server_test.go	Adds database round-trip tests
database/​reconnect_test.go	Tests replaced-pool closure
database/​queries.go	Optimizes query construction
database/​queries_test.go	Tests typed query constructors
database/​overhead_bench_test.go	Benchmarks wrapper overhead
database/​middleware.go	Avoids disabled logging work
database/​dump.go	Streams SQL statement parsing
database/​dump_test.go	Tests dump parsing and encoding
database/​backup_server_bench_test.go	Benchmarks server backups
database/​backup_bench_test.go	Benchmarks SQLite backups
config/​large_bench_test.go	Benchmarks large configurations
config/​get_bench_test.go	Benchmarks configuration reads
config/​core.go	Publishes configuration atomically
cache/​tiered.go	Uses atomic hit counters
cache/​tiered_bench_test.go	Benchmarks tiered caching
cache/​shard_test.go	Tests sharding and native values
cache/​redis.go	Reduces Redis stats contention
cache/​redis_bench_test.go	Adds Redis benchmarks
cache/​memory_race_test.go	Adds cache race regressions
cache/​memory_bench_test.go	Benchmarks cache configurations
cache/​cache.go	Adds sharding and native serialization
apperror/​trace_cache_test.go	Tests trace caching
apperror/​extra_bench_test.go	Adds error-path benchmarks
apperror/​chain_bench_test.go	Benchmarks wrapped error chains
apperror/​apperror.go	Caches caller trace resolution
.github/​workflows/​test.yaml	Expands race-enabled CI
.github/​workflows/​bench.yaml	Adds manual benchmark workflow

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.