Skip to content

CMake: Auto-detect _dl_find_object support in libgcc/libunwind for USERVER_DISABLE_PHDR_CACHE default #1314

Description

@root-kidik

Context & Architectural Distinction

In userver, two distinct mechanisms protect exception handling performance under heavy load:

  1. The Phdr Cache / _dl_find_object: Solves CPU lock contention and slow iteration overhead during stack unwinding. When system _dl_find_object is available (glibc 2.35+ with libgcc/libunwind support), userver's internal dl_iterate_phdr cache becomes redundant.
  2. MLockDebugInfo (mlock): Solves Major Page Faults (Disk I/O pauses). It pins the main executable's code (PT_LOAD) and exception frame (PT_GNU_EH_FRAME) sections into physical RAM, ensuring that an unexpected spike in exceptions never triggers a blocking disk read.

The Problem with the Current Implementation

Currently, in userver's source code, both optimizations are tightly coupled behind a single preprocessor macro (USE_PHDR_CACHE):

#if !defined(USERVER_DISABLE_PHDR_CACHE) && defined(__linux__) && ...
#define USE_PHDR_CACHE 1
#endif

Because of this, if a user sets USERVER_DISABLE_PHDR_CACHE=ON to leverage the system's fast _dl_find_object, MLockDebugInfo is completely disabled as well (falling back to empty no-op stubs).

This is problematic because even with a fast native _dl_find_object, the main binary's pages can still be swapped out to disk under high memory pressure, leading to severe latency spikes (p99/p99.9 degradation) due to page faults. We want to disable the phdr cache (avoiding its memory overhead and dlopen restrictions) while keeping mlock active for the main binary.

Furthermore, relying on hardcoded compiler version checks (__GNUC__ >= 12) to toggle these features is fragile due to downstream backports (e.g., Ubuntu 22.04 ships with glibc 2.35 and a backported libgcc_s.so supporting _dl_find_object, despite defaulting to GCC 11).


Proposed Solution

  1. Decouple MLockDebugInfo from USE_PHDR_CACHE:
    Refactor engine/impl/exception_hacks.cpp so that MLockDebugInfo can run independently of whether the phdr cache or dlopen hooks are active. mlock only needs a single initial pass via dl_iterate_phdr (or system traversal) during startup to pin the main executable.
  2. Smart CMake Auto-Detection via readelf:
    Replace static compiler version checks with a dynamic runtime symbol check of libgcc_s.so.1 or libunwind.so inside CMake to set the default of USERVER_DISABLE_PHDR_CACHE:
execute_process(
    COMMAND ${CMAKE_CXX_COMPILER} -print-file-name=libgcc_s.so.1
    OUTPUT_VARIABLE LIBGCC_PATH
    OUTPUT_STRIP_TRAILING_WHITESPACE
)

set(UNWINDER_HAS_DL_FIND_OBJECT OFF)
if(EXISTS "${LIBGCC_PATH}" AND NOT IS_DIRECTORY "${LIBGCC_PATH}")
    execute_process(
        COMMAND readelf -sW "${LIBGCC_PATH}"
        COMMAND grep -q "_dl_find_object"
        RESULT_VARIABLE READELF_RESULT
    )
    if(READELF_RESULT EQUAL 0)
        set(UNWINDER_HAS_DL_FIND_OBJECT ON)
    endif()
endif()

If _dl_find_object is detected in the runtime library, default USERVER_DISABLE_PHDR_CACHE to ON (disabling the redundant cache and lifting dlopen restrictions), while retaining mlock for the main binary.


Benefits

  • Zero Latency Spikes: Combines the best of both worlds — fast lock-free lookups via system _dl_find_object and guaranteed residency in RAM for the main binary via mlock.
  • Accuracy & Backport Compatibility: Correctly identifies runtime capabilities on LTS distributions (like Ubuntu 22.04) via binary symbol inspection instead of abstract version numbers.
  • Flexibility: Lifts unnecessary restrictions on dlopen on modern systems without sacrificing memory-pinning protections.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions