Context & Architectural Distinction
In userver, two distinct mechanisms protect exception handling performance under heavy load:
- The Phdr Cache /
_dl_find_object: Solves CPU lock contention and slow iteration overhead during stack unwinding. When system _dl_find_object is available (glibc 2.35+ with libgcc/libunwind support), userver's internal dl_iterate_phdr cache becomes redundant.
MLockDebugInfo (mlock): Solves Major Page Faults (Disk I/O pauses). It pins the main executable's code (PT_LOAD) and exception frame (PT_GNU_EH_FRAME) sections into physical RAM, ensuring that an unexpected spike in exceptions never triggers a blocking disk read.
The Problem with the Current Implementation
Currently, in userver's source code, both optimizations are tightly coupled behind a single preprocessor macro (USE_PHDR_CACHE):
#if !defined(USERVER_DISABLE_PHDR_CACHE) && defined(__linux__) && ...
#define USE_PHDR_CACHE 1
#endif
Because of this, if a user sets USERVER_DISABLE_PHDR_CACHE=ON to leverage the system's fast _dl_find_object, MLockDebugInfo is completely disabled as well (falling back to empty no-op stubs).
This is problematic because even with a fast native _dl_find_object, the main binary's pages can still be swapped out to disk under high memory pressure, leading to severe latency spikes (p99/p99.9 degradation) due to page faults. We want to disable the phdr cache (avoiding its memory overhead and dlopen restrictions) while keeping mlock active for the main binary.
Furthermore, relying on hardcoded compiler version checks (__GNUC__ >= 12) to toggle these features is fragile due to downstream backports (e.g., Ubuntu 22.04 ships with glibc 2.35 and a backported libgcc_s.so supporting _dl_find_object, despite defaulting to GCC 11).
Proposed Solution
- Decouple
MLockDebugInfo from USE_PHDR_CACHE:
Refactor engine/impl/exception_hacks.cpp so that MLockDebugInfo can run independently of whether the phdr cache or dlopen hooks are active. mlock only needs a single initial pass via dl_iterate_phdr (or system traversal) during startup to pin the main executable.
- Smart CMake Auto-Detection via
readelf:
Replace static compiler version checks with a dynamic runtime symbol check of libgcc_s.so.1 or libunwind.so inside CMake to set the default of USERVER_DISABLE_PHDR_CACHE:
execute_process(
COMMAND ${CMAKE_CXX_COMPILER} -print-file-name=libgcc_s.so.1
OUTPUT_VARIABLE LIBGCC_PATH
OUTPUT_STRIP_TRAILING_WHITESPACE
)
set(UNWINDER_HAS_DL_FIND_OBJECT OFF)
if(EXISTS "${LIBGCC_PATH}" AND NOT IS_DIRECTORY "${LIBGCC_PATH}")
execute_process(
COMMAND readelf -sW "${LIBGCC_PATH}"
COMMAND grep -q "_dl_find_object"
RESULT_VARIABLE READELF_RESULT
)
if(READELF_RESULT EQUAL 0)
set(UNWINDER_HAS_DL_FIND_OBJECT ON)
endif()
endif()
If _dl_find_object is detected in the runtime library, default USERVER_DISABLE_PHDR_CACHE to ON (disabling the redundant cache and lifting dlopen restrictions), while retaining mlock for the main binary.
Benefits
- Zero Latency Spikes: Combines the best of both worlds — fast lock-free lookups via system
_dl_find_object and guaranteed residency in RAM for the main binary via mlock.
- Accuracy & Backport Compatibility: Correctly identifies runtime capabilities on LTS distributions (like Ubuntu 22.04) via binary symbol inspection instead of abstract version numbers.
- Flexibility: Lifts unnecessary restrictions on
dlopen on modern systems without sacrificing memory-pinning protections.
Context & Architectural Distinction
In
userver, two distinct mechanisms protect exception handling performance under heavy load:_dl_find_object: Solves CPU lock contention and slow iteration overhead during stack unwinding. When system_dl_find_objectis available (glibc 2.35+ withlibgcc/libunwindsupport),userver's internaldl_iterate_phdrcache becomes redundant.MLockDebugInfo(mlock): Solves Major Page Faults (Disk I/O pauses). It pins the main executable's code (PT_LOAD) and exception frame (PT_GNU_EH_FRAME) sections into physical RAM, ensuring that an unexpected spike in exceptions never triggers a blocking disk read.The Problem with the Current Implementation
Currently, in
userver's source code, both optimizations are tightly coupled behind a single preprocessor macro (USE_PHDR_CACHE):Because of this, if a user sets
USERVER_DISABLE_PHDR_CACHE=ONto leverage the system's fast_dl_find_object,MLockDebugInfois completely disabled as well (falling back to empty no-op stubs).This is problematic because even with a fast native
_dl_find_object, the main binary's pages can still be swapped out to disk under high memory pressure, leading to severe latency spikes (p99/p99.9degradation) due to page faults. We want to disable the phdr cache (avoiding its memory overhead anddlopenrestrictions) while keepingmlockactive for the main binary.Furthermore, relying on hardcoded compiler version checks (
__GNUC__ >= 12) to toggle these features is fragile due to downstream backports (e.g., Ubuntu 22.04 ships with glibc 2.35 and a backportedlibgcc_s.sosupporting_dl_find_object, despite defaulting to GCC 11).Proposed Solution
MLockDebugInfofromUSE_PHDR_CACHE:Refactor
engine/impl/exception_hacks.cppso thatMLockDebugInfocan run independently of whether thephdrcache ordlopenhooks are active.mlockonly needs a single initial pass viadl_iterate_phdr(or system traversal) during startup to pin the main executable.readelf:Replace static compiler version checks with a dynamic runtime symbol check of
libgcc_s.so.1orlibunwind.soinside CMake to set the default ofUSERVER_DISABLE_PHDR_CACHE:If
_dl_find_objectis detected in the runtime library, defaultUSERVER_DISABLE_PHDR_CACHEtoON(disabling the redundant cache and liftingdlopenrestrictions), while retainingmlockfor the main binary.Benefits
_dl_find_objectand guaranteed residency in RAM for the main binary viamlock.dlopenon modern systems without sacrificing memory-pinning protections.