- DotNet Memory Leak App
- 1. Repository Overview
- 2. Key Features
- 3. Motivation
- 4. Project Structure
- 5. Deployment Guide (OpenShift / MicroShift)
- 6. Usage & Diagnostic Workflow
- 6.1. Triggering a Memory Leak
- 6.2. Monitoring Application Logs
- 6.3. Collecting Crash Dumps
- 6.3.1. Option A: Automatic Crash Dumps
- 6.3.2. Option B: On-Demand Sidecar via Deployment Patching
- 6.3.3. Option C: On-Demand Dumps via Ephemeral Debug Container (kubectl debug)
- 6.3.4. Option D: Secure On-Demand Dumps via Shell-less Ephemeral Container
- 6.3.5. Option E: Deploying with a Hardened Security Context
- 6.3.6. kubectl debug Important Limitation: Pod Instability After Repeated Debugging
- 6.3.7. baseOS access through nsenter
- 6.4. Limits & LimitaRanges
- 6.5. Testing and Validation
- 7. Security & Troubleshooting Considerations
- 8. External References & Further Reading
- 9. Contributing
- 10. License
This repository does contains a .NET web service specifically engineered to simulate, monitor, and analyze memory leaks within a containerized environment. Designed for deployment on platforms like OpenShift and MicroShift, it provides a robust framework for understanding .NET memory management in cloud-native settings. The application is configured to automatically generate crash dumps upon encountering Out-Of-Memory (OOM) conditions, facilitating in-depth post-mortem analysis.
- Controlled Memory Leak Simulation: Provides an endpoint to trigger a continuous memory allocation pattern.
- Automated Crash Dump Generation: Configured to automatically create
.NETcrash dumps (ELFformat) when the application experiences an OOM or other critical failures. - On-Demand Diagnostic Tooling: Integrates
.NETCLI diagnostic tools (dotnet-dump,dotnet-trace,dotnet-counters,dotnet-gcdump) via dedicated debug containers for live analysis. - Security-Hardened Deployment: Demonstrates production-ready security practices including non-root containers, read-only filesystems, network policies, and minimal RBAC.
- Red Hat UBI RHEL9 Compliant: Uses exclusively Red Hat Universal Base Images (UBI) RHEL9 for .NET 8.0 applications, ensuring enterprise support and security compliance.
- MicroShift Optimized: Specifically designed for single-node MicroShift deployments with appropriate resource limits and storage configurations.
- Air-Gapped Deployment Support: Includes instructions and examples for deploying in environments without direct internet access to container registries.
Effective memory management is paramount for application stability and performance. This repository aims to:
- Provide a controlled, reproducible environment to observe and understand
.NETmemory leak behavior in containers. - Enable practical crash dump analysis for debugging
.NETapplications in cloud-native environments, aligning with Microsoft's diagnostic recommendations for.NETapps on Containers. - Showcase robust deployment strategies for
.NETapplications on OpenShift/MicroShift, emphasizing security and operational best practices. - Facilitate debugging in restricted environments, including air-gapped scenarios, by demonstrating image loading and offline tooling.
This repository contains a comprehensive .NET memory leak simulation and diagnostic toolkit with the following key components:
Program.cs: Main application entry point with ASP.NET Core web API endpoints/triggerMemoryLeak: Endpoint to initiate controlled memory allocation/readme: Endpoint to render this README as a web page/crash: Endpoint to trigger a segmentation fault for coredump testing/healthz: Liveness health check endpoint (returns HTTP 200 OK when healthy)/readyz: Readiness health check endpoint (returns HTTP 200 OK when ready)/swagger: Swagger UI documentation (available in development mode)
MemoryLeakManager.cs: Static class managing memory allocation and trackingDotNetMemoryLeakApp.csproj: .NET 8.0 project configuration with diagnostic tooling dependencies
Containerfile: Multi-stage container build using Red Hat UBI RHEL8 .NET 8.0 images (production runtime-only, no debug tools)- Use case: Standard production deployments with automatic OOM dump generation
- Image tag:
v1-base
Containerfile-debug: Specialized debug container using Red Hat UBI RHEL8 .NET 8.0 SDK for secure dump collection- Use case: On-demand debugging with .NET diagnostic tools (dotnet-dump, dotnet-trace, etc.)
- Image tag:
v1-debug
Containerfile-debug-shellless: Shell-less debug container with automated dump collection using Go utility- Use case: Secure debugging in environments that forbid interactive shells
- Image tag:
v1-debug-shellless
Containerfile.hardened: Hardened production container with enhanced security configurations- Use case: Deployments requiring maximum security posture
- Image tag:
v1-hardened
configure_core_dump.sh: Shell script for core dump configuration (deprecated for container use)
deployment.yaml: Security-hardened production deployment configurationdeployment-secure.yaml: Additional hardened security deployment variantkustomization.yaml: Kustomize configuration for resource managementns.yaml: Namespace definition (dotnet-memory-leak-app)serviceaccount.yaml: Service account configurationrbac.yaml: Minimal namespace-scoped RBAC definitionsnetworkpolicy.yaml: Network isolation policy for ingress/egress controlscc-and-rbac-secure.yaml: Security context constraints for hardened deploymentsvc.yaml: Service definitionroute.yaml: OpenShift route configurationpvc.yaml: Persistent volume claim for dump storage (PVC name:coredump-pvc)limitrange.yaml: Resource limits and constraintspatch/: JSON patch files for dynamic sidecar injectionargocd-application.yaml: ArgoCD Application manifest for GitOps deploymentargocd-application-acm.yaml: ArgoCD Application manifest with Red Hat ACM integrationhost-coredump/: Host-level coredump collection manifests (PV, PVC, CronJob)
tools/pid-finder/main.go: Go utility for automated process discovery and dump collectionDotNetMemoryLeakApp.http: HTTP request examples for testing endpoints
appsettings.json/appsettings.Development.json: Application configurationProperties/launchSettings.json: Development launch settings
This guide walks you through deploying the DotNet Memory Leak App to an OpenShift or MicroShift cluster.
Before proceeding, ensure you have:
- MicroShift installed and running, or access to an OpenShift cluster.
kubectlorocCLI configured and connected to your cluster.- A container registry (e.g., Quay.io, or local Podman storage) available for pushing images.
- Storage: Ensure your cluster has a default StorageClass (e.g., topolvm for MicroShift) for PVC creation.
- Network Policies: If your cluster enforces NetworkPolicies, ensure the provided policy is compatible with your CNI.
MicroShift Specific Notes:
- MicroShift uses topolvm as the default storage provisioner
- Routes are available by default; NodePort is an alternative if Routes are disabled
- Ephemeral containers require Kubernetes 1.25+ (check with
kubectl version)
In an internet-connected environment, build and push the container image to your chosen registry:
# Navigate to the project root directory
cd DotNetBuggyApp-main
# Build the image using the optimized Containerfile.hardened
# (Ensure your Containerfile.hardened includes the necessary tool installations and TMPDIR setup as discussed)
podman build -t quay.io/your-namespace/dotnet-memory-leak-app:v1 -f Containerfile.hardened .
# Push the image to your container registry
podman push quay.io/your-namespace/dotnet-memory-leak-app:v1
For air-gapped deployments, save the image as a tarball:
podman save -o dotnet-memory-leak-app.tar quay.io/your-namespace/dotnet-memory-leak-app:v1
Transfer dotnet-memory-leak-app.tar to your air-gapped environment's worker nodes.
Navigate to the kubernetes directory within your project:
cd DotNetBuggyApp-main/kubernetes
Load the saved image (Air-Gapped Only): In an air-gapped system, load the image into the local Podman storage on your worker nodes:
sudo podman load -i dotnet-memory-leak-app.tar
sudo podman images # Confirm the image is available
(If not air-gapped, skip this step. Kubernetes will pull the image from the registry.)
Apply the Kubernetes manifests using kustomize. This will create the namespace, service account, roles, PVC, deployment, service, and route.
IMPORTANT NOTE: The kustomization.yaml is configured to deploy multiple deployment variants for comparison and testing purposes:
deployment.yaml: Standard deployment with basic securitydeployment-secure.yaml: Hardened deployment with enhanced security contextdeployment-host-coredump.yaml: Deployment with host-level coredump collection
If you only want to deploy a single variant, either modify kustomization.yaml or apply individual manifests directly with oc apply -f deployment.yaml.
oc apply -k .
Expected Output (May vary, note warnings):
You might see warnings about PodSecurity policies (e.g., "would violate PodSecurity "restricted:latest"") if your cluster has strict default policies. These warnings indicate that certain requested capabilities (like SYS_PTRACE) or security contexts might be blocked. Despite warnings, the core objects should be created.
**Note:** The actual PVC name and resource names depend on your manifest configuration. The standard configuration uses `coredump-pvc` for the PersistentVolumeClaim.
namespace/dotnet-memory-leak-app created
serviceaccount/dotnet-app-sa created
role.rbac.authorization.k8s.io/dotnet-app-role created
clusterrole.rbac.authorization.k8s.io/dotnet-app-clusterrole created
rolebinding.rbac.authorization.k8s.io/dotnet-app-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/dotnet-app-clusterrolebinding created
service/dotnet-memory-leak-service created
limitrange/dotnet-limitrange created
persistentvolumeclaim/coredump-pvc created
Warning: would violate PodSecurity "restricted:latest": unrestricted capabilities (container "diagnostic-tools-sidecar" must set securityContext.capabilities.drop=["ALL"]; container "diagnostic-tools-sidecar" must not include "SYS_PTRACE" in securityContext.capabilities.add)
deployment.apps/dotnet-memory-leak-app created
route.route.openshift.io/dotnet-memory-leak-route created
securitycontextconstraints.security.openshift.io/dotnet-scc created
This section outlines how to use the application and leverage its diagnostic capabilities.
The application includes Swagger/OpenAPI documentation for interactive API exploration. This is only available when running in Development mode.
Accessing Swagger UI:
# Get the route hostname
export ROUTE_HOST=$(oc get route dotnet-memory-leak-route -n dotnet-memory-leak-app -o jsonpath='{.spec.host}')
# Access Swagger UI in your browser (only works if ASPNETCORE_ENVIRONMENT=Development)
http://$ROUTE_HOST/swagger
Note: For production deployments, Swagger is disabled for security reasons. Use the endpoints directly:
GET /: Basic health check - returns "DotNet Memory Leak App is running!"GET /healthz: Liveness probe - returns HTTP 200 OK with "Healthy" statusGET /readyz: Readiness probe - returns HTTP 200 OK with "Healthy" statusGET /triggerMemoryLeak: Initiates memory leak simulationGET /crash: Triggers segmentation fault for testing coredump captureGET /readme: Renders this README as HTML
To initiate the memory leak, access the /triggerMemoryLeak endpoint of your deployed application via its OpenShift Route.
# Get the route hostname
export ROUTE_HOST=$(oc get route dotnet-memory-leak-route -n dotnet-memory-leak-app -o jsonpath='{.spec.host}')
# Trigger the memory leak
curl http://$ROUTE_HOST/triggerMemoryLeak
The application will begin allocating memory in 1MB chunks, logging its progress.
As the memory leak progresses, you can observe the application's logs, which will show memory allocation updates and eventually OutOfMemoryException messages.
# Get the pod name
export POD_NAME=$(oc get pods -n dotnet-memory-leak-app -l app=dotnet-memory-leak-app -o jsonpath='{.items[0].metadata.name}')
# Stream application logs
oc logs "$POD_NAME" -n dotnet-memory-leak-app -f
# You will see output similar to:
info: Program[0]
Allocated 1516.00 MB this round, Total: 1516.00 MB
fail: Program[0]
OutOfMemoryException caught! Application is likely to crash soon.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at Program.<>c.<<<Main>$>b__0_0>d.MoveNext() in /app/Program.cs:line 37
The project offers multiple strategies for collecting crash dumps, suitable for different debugging scenarios and cluster security postures.
This is the primary and most reliable method for capturing the application's state during an OutOfMemory crash, especially in strict security environments. The .NET runtime automatically generates a full dump when the application crashes due to an unhandled exception.
Enhanced Features:
- Timestamped dump files: Dump files include process ID and timestamp for better tracking
- Programmatic coredump configuration: The application automatically configures
prctlandulimitsettings - Host-level coredump support: Optional host-level coredump collection via PersistentVolumes
- Automated cleanup: CronJob for cleaning up old dump files
Configuration:
- These variables are pre-configured in your deployment.yaml:
COMPlus_DbgEnableElfDumpOnCrash=1: Enables ELF crash dump generationCOMPlus_DbgCrashDumpType=4: Specifies a "full" dump (type 4 includes heap)COMPlus_DbgMiniDumpName=/app/dumps/core/host-dump.%e.%p.%t.dmp: Sets timestamped output path%e= executable name%p= process ID%t= timestamp
- The
/app/dumpsdirectory is backed by a PersistentVolumeClaim (coredump-pvc) to ensure persistence - Programmatic setup: The application automatically creates the dumps directory and configures coredump settings via
prctlsystem calls
Note: The base Containerfile sets a simpler pattern (/app/dumps/dump.dmp), but the deployment.yaml overrides this with the timestamped pattern above. Always check your deployment configuration for the actual dump naming convention in use.
Workflow:
- Trigger the memory leak
- Allow the application to run until it crashes (you'll see restarts in
oc get pods) - Once the pod crashes and restarts, a dump file with timestamp will be present in the
/app/dumpsvolume
Example Commands (after application crashes and restarts):
# Get the name of a running pod (it might be a new instance after restart)
export POD_NAME=$(oc get pods -n dotnet-memory-leak-app -l app=dotnet-memory-leak-app -o jsonpath='{.items[0].metadata.name}')
# Verify the dump file exists (look for timestamped files)
oc rsh "$POD_NAME" -c dotnet-app -- ls -l /app/dumps/
# Copy the dump file from the pod to your local machine for analysis
oc cp "$POD_NAME":/app/dumps/dump.*.dmp ./crash_dump.dmp -n dotnet-memory-leak-app
Host-Level Coredump Collection (Optional): For enhanced security and host-level dump collection:
# Deploy host-level coredump configuration
oc apply -f kubernetes/host-coredump/
# Check host-level dumps (if configured)
oc rsh "$POD_NAME" -c dotnet-app -- ls -l /var/crashdumps/
Optional: Analyze the dump locally using dotnet-dump
dotnet-dump analyze ./crash_dump.dmp
This method allows for interactive, real-time dump collection by running .NET diagnostic tools from a dedicated sidecar container within the same pod. It uses oc patch to temporarily modify the Deployment resource, which performs a controlled rollout of a new pod containing the application and a debug sidec
Step 1: Build the Correct Debug Image Create a Containerfile that starts from the correct .NET SDK version and installs the version-matched diagnostic tools. This image will be used for the on-demand sidecar.
FROM mcr.microsoft.com/dotnet/sdk:8.0
WORKDIR /app
# The user/group setup should match your environment's requirements
RUN chown 1001:0 /app
# Switch to root to install tools
USER root
# Install the .NET 8 versions of the diagnostic tools into a specific path
RUN mkdir -p /app/tools && chown 1001:0 /app/tools && \
dotnet tool install --tool-path /app/tools dotnet-dump --version "8.*" && \
dotnet tool install --tool-path /app/tools dotnet-gcdump --version "8.*" && \
dotnet tool install --tool-path /app/tools dotnet-trace --version "8.*" && \
dotnet tool install --tool-path /app/tools dotnet-counters --version "8.*"
# Switch back to the non-root user
USER 1001
Build this image and push it to your internal container registry (e.g., quay.io/rhn_support_arolivei/dotnet-debug:v1).
Step 2: Prepare Patch Files Create two JSON patch files. One to add the debugger and one to remove it.
add-sidecar.json:
[
{
"op": "add",
"path": "/spec/template/spec/containers/-",
"value": {
"name": "debugger",
"image": "quay.io/rhn_support_arolivei/dotnet-debug:v1",
"command": ["sleep", "infinity"],
"env": [
{
"name": "TMPDIR",
"value": "/app/dumps/tmp"
}
],
"volumeMounts": [
{
"mountPath": "/app/dumps",
"name": "dump-storage"
}
]
}
}
]
remove-sidecar.json:
[
{
"op": "remove",
"path": "/spec/template/spec/containers/1"
}
]
Step 3: Execute the Debugging Workflow Ensure your application is running using its base Deployment configuration. Patch the Deployment to add the sidecar. This triggers a rolling update.
oc patch deployment dotnet-memory-leak-app -n dotnet-memory-leak-app --type=json --patch-file add-sidecar.json
Wait for the new pod to be ready (it will show 2/2 containers).
oc get pods -n dotnet-memory-leak-app --watch
Exec into the debugger sidecar:
export POD_NAME=$(oc get pods -l app=dotnet-memory-leak-app -n dotnet-memory-leak-app -o jsonpath='{.items[0].metadata.name}')
oc exec -it "$POD_NAME" -n dotnet-memory-leak-app -c debugger -- /bin/bash
Collect the dump. Because you are in the same pod, you can see the application's process and use its PID.
# Inside the debugger shell, find the process ID
ps -ef
# Use the PID (e.g., 2) to collect the dump with the version-matched tool
/app/tools/dotnet-dump collect -p 2 -o /app/dumps/dump_SUCCESS.dmp
Remove the sidecar. After collecting the dump, patch the deployment again to remove the debug container and return to the original, hardened state.
oc patch deployment dotnet-memory-leak-app -n dotnet-memory-leak-app --type=json --patch-file remove-sidecar.json
This method allows you to dynamically inject a temporary container into an existing pod for on-demand debugging, without permanent changes to the deployment.yaml.
How it works:
kubectl debug creates a new, temporary container within an existing pod.
The --target flag ensures this ephemeral container joins the process namespace of your main application container.
You specify an image for the ephemeral container that contains the necessary diagnostic tools.
Pros:
- No Permanent Deployment Changes: Ideal for ad-hoc troubleshooting without altering your deployment.yaml.
- On-Demand Resources: The debug container only consumes resources when actively used.
Cons:
- Kubernetes Version Dependent: The kubectl debug
--targetcommand requires Kubernetes 1.25+ and enabled EphemeralContainers feature gates.
# 1. Get the pod name
export POD_NAME=$(kubectl get pods -n dotnet-memory-leak-app -l app=dotnet-memory-leak-app -o jsonpath='{.items[0].metadata.name}')
# 2. Launch an ephemeral debug container
# Use your application image (if it has tools and shell) or a full SDK image (recommended for debugging)
# NOTE: This command might fail if your cluster/client doesn't support --target,
# or if security policies block SYS_PTRACE.
kubectl debug -it "$POD_NAME" --image=quay.io/rhn_support_arolivei/dotnet-memory-leak-app:v1 --target=dotnet-app -- /bin/bash
# If the above fails or you need a richer toolset, try with the SDK image:
# kubectl debug -it "$POD_NAME" --image=registry.redhat.io/rhel8/dotnet-80:8.0 --target=dotnet-app -- /bin/bash
# 3. Inside the ephemeral debug container, find the main app's PID and collect a dump.
# (You might need to run: mount -t proc proc /proc first if ps -ef fails)
ps -ef
# Look for 'dotnet /app/DotNetMemoryLeakApp.dll'. Note its PID.
# 4. Collect a dump of the main application (replace <PID> with the actual PID)
TMPDIR=/proc/<PID>/root/tmp /app/tools/ dotnet-dump collect --process-id <PID> -o /app/dumps/ephemeral_collected_dump.dmp
# 5. Exit the debug container and copy the dump file
oc cp "$POD_NAME":/app/dumps/app_collected_dump.dmp ./app_collected_dump.dmp -n dotnet-memory-leak-app
Sample output
bash-4.4$ ps -ef
UID PID PPID C STIME TTY TIME CMD
1000150+ 1 0 0 12:46 ? 00:00:01 dotnet /app/DotNetMemoryLeakApp.dll
1000150+ 90 0 0 14:14 pts/0 00:00:00 /bin/bash
1000150+ 106 90 0 14:20 pts/0 00:00:00 ps -ef
bash-4.4$
bash-4.4$ TMPDIR=/proc/1/root/tmp /app/tools/dotnet-dump collect --process-id 1 -o /app/dumps/app_collected_dump.dmp
Writing full to /app/dumps/app_collected_dump.dmp
Complete
bash-4.4$
bash-4.4$ ls -la /app/dumps/
ls: cannot access '/app/dumps/': No such file or directory
bash-4.4$ ls -la /app/
total 0
drwxr-xr-x. 1 default root 19 Aug 22 16:31 .
dr-xr-xr-x. 1 root root 28 Sep 9 14:46 ..
drwxr-xr-x. 3 default root 103 Aug 22 16:31 tools
bash-4.4$ df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 52363264 12386196 39977068 24% /
tmpfs 65536 0 65536 0% /dev
shm 65536 0 65536 0% /dev/shm
tmpfs 1625532 76316 1549216 5% /etc/passwd
/dev/mapper/rhel-root 52363264 12386196 39977068 24% /etc/hosts
devtmpfs 4096 0 4096 0% /proc/keys
bash-4.4$ exit
[redhat@rhel96-microshift419-vm2 tmp]$ oc cp "$POD_NAME":/app/dumps/app_collected_dump.dmp ./app_collected_dump.dmp -n dotnet-memory-leak-app
Defaulted container "dotnet-app" out of: dotnet-app, debugger-xxlvc (ephem), debugger-sgcvg (ephem), debugger-xpn9g (ephem)
tar: Removing leading `/' from member names
[redhat@rhel96-microshift419-vm2 tmp]$ file app_collected_dump.dmp
app_collected_dump.dmp: ELF 64-bit LSB core file, x86-64, version 1 (GNU/Linux), SVR4-style, from 'dotnet', real uid: 1000150000, effective uid: 1000150000, real gid: 0, effective gid: 0, execfn: '/usr/bin/dotnet', platform: 'x86_64'
[redhat@rhel96-microshift419-vm2 tmp]$ du -m app_collected_dump.dmp
241 app_collected_dump.dmp
[redhat@rhel96-microshift419-vm2 tmp]$
[redhat@rhel96-microshift419-vm2 tmp]$ oc rsh dotnet-memory-leak-app-56457fbcdc-2rpf9
Defaulted container "dotnet-app" out of: dotnet-app, debugger-xxlvc (ephem), debugger-sgcvg (ephem), debugger-xpn9g (ephem), debugger-pvldq (ephem), debugger-shvx2 (ephem)
sh-4.4$ df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 52363264 12386868 39976396 24% /
tmpfs 65536 0 65536 0% /dev
shm 65536 0 65536 0% /dev/shm
tmpfs 1625532 76308 1549224 5% /etc/passwd
/dev/topolvm/7910c082-ecca-40b0-85e7-89a7cdf8728b 5177344 562484 4614860 11% /app/dumps
/dev/mapper/rhel-root 52363264 12386868 39976396 24% /etc/hosts
tmpfs 2097152 16 2097136 1% /run/secrets/kubernetes.io/serviceaccount
devtmpfs 4096 0 4096 0% /proc/keys
sh-4.4$ ls -l /app/dumps/
total 493328
-rw-------. 1 1000150000 1000150000 252518400 Sep 9 14:16 app_collected_dump.dmp
-rw-------. 1 1000150000 1000150000 252649472 Sep 9 14:25 app_collected_dump2.dmp
sh-4.4$
Important Note: Ephemeral containers with --target share the process namespace but NOT the mount namespace. The debug container cannot see /app/dumps directly, but dotnet-dump works because it controls the target process, which writes the dump to its own filesystem.
For live-process debugging, kubectl debug --target is required because it shares the PID namespace with the target container, allowing access to running processes. oc debug creates a separate pod and cannot access the original application's processes.
This method enhances Option D by adhering to strict security policies that forbid shells even in debug images. It uses a purpose-built, shell-less debug container with a compiled utility that automates the dump collection process.
How it works:
- A Go utility (
tools/pid-finder) is compiled into a static binary. - A multi-stage
Containerfile-debugbuilds a debug image that contains the.NETdiagnostic tools and this Go utility as itsENTRYPOINT. - When this debug container is launched, the Go utility executes automatically. It finds the target
.NETprocess, collects a full core dump, and then sleeps indefinitely. - This approach is fully automated, requires no interactive shell, and minimizes the attack surface of the debug image.
Workflow:
Step 1: Build the Secure Debug Image
Use the modified Containerfile-debug to build the image. This must be done from the root of the repository.
# Build the secure debug image
podman build -t quay.io/your-namespace/dotnet-secure-debug:v1 -f Containerfile-debug .
# Push the image to your container registry
podman push quay.io/your-namespace/dotnet-secure-debug:v1
Step 2: Launch the Ephemeral Debug Container
Use kubectl debug to attach the ephemeral container to your running application pod. The command is simpler because the container's entrypoint does all the work.
# Get the pod name
export POD_NAME=$(kubectl get pods -n dotnet-memory-leak-app -l app=dotnet-memory-leak-app -o jsonpath='{.items[0].metadata.name}')
# Launch the secure ephemeral debug container
# The --image should point to the one you just built.
kubectl debug -it "$POD_NAME" \
--image=quay.io/your-namespace/dotnet-secure-debug:v1 \
--share-processes \
--target=dotnet-app
You will see the output from the Go utility as it finds the process and collects the dump.
Step 3: Copy the Dump File
The dump is saved to /app/dumps/coredump.dmp inside the target container's filesystem (since the dotnet-dump command is executed by the target process). You can copy it out using kubectl cp.
# Copy the dump file from the application pod to your local machine
kubectl cp "$POD_NAME":/app/dumps/coredump.dmp ./coredump.dmp -n dotnet-memory-leak-app -c dotnet-app
This method provides a secure and non-interactive way to obtain diagnostics, making it ideal for production environments with strict security postures.
Sample output:
[redhat@rhel96-microshift419-vm2 DotNetBuggyApp]$ sudo podman load -i dotnet-secure-debug-v1.tar
[sudo] password for redhat:
Getting image source signatures
Copying blob bfaed8e0c4d1 done |
Copying blob 28de103bd9c3 skipped: already exists
Copying blob 85bbf55a0c9b skipped: already exists
Copying blob 37aef32bd2f2 skipped: already exists
Copying blob ec17d09b16f1 done |
Copying config c57643bf44 done |
Writing manifest to image destination
Loaded image: quay.io/rhn_support_arolivei/dotnet-secure-debug:v1
[redhat@rhel96-microshift419-vm2 DotNetBuggyApp]$ oc get pods
NAME READY STATUS RESTARTS AGE
dotnet-memory-leak-app-77b88ddf46-j4dn5 1/1 Running 0 6s
[redhat@rhel96-microshift419-vm2 DotNetBuggyApp]$ oc rsh dotnet-memory-leak-app-77b88ddf46-j4dn5
sh-4.4$ ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 16:09 ? 00:00:00 /usr/bin/pod
1000170+ 2 0 1 16:09 ? 00:00:00 dotnet /app/DotNetMemoryLeakApp.dll
1000170+ 22 0 0 16:09 pts/0 00:00:00 /bin/sh
1000170+ 24 22 0 16:09 pts/0 00:00:00 ps -ef
sh-4.4$ ls -l /proc/2/root/tmp
total 0
prwx------. 1 1000170000 1000170000 0 Sep 11 16:09 clr-debug-pipe-2-27034808-in
prwx------. 1 1000170000 1000170000 0 Sep 11 16:09 clr-debug-pipe-2-27034808-out
srw-------. 1 1000170000 1000170000 0 Sep 11 16:09 dotnet-diagnostic-2-27034808-socket
sh-4.4$ df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 52363264 15721224 36642040 31% /
tmpfs 65536 0 65536 0% /dev
shm 65536 0 65536 0% /dev/shm
tmpfs 1625532 76020 1549512 5% /etc/passwd
/dev/mapper/rhel-root 52363264 15721224 36642040 31% /tmp
/dev/topolvm/01b27068-1c48-46ea-92c6-49ba0ad97c40 5177344 314060 4863284 7% /app/dumps
tmpfs 2097152 16 2097136 1% /run/secrets/kubernetes.io/serviceaccount
devtmpfs 4096 0 4096 0% /proc/keys
sh-4.4$ ls -l /app/dumps/
total 0
sh-4.4$
[redhat@rhel96-microshift419-vm2 DotNetBuggyApp]$ export POD_NAME=$(kubectl get pods -n dotnet-memory-leak-app -l app=dotnet-memory-leak-app -o jsonpath='{.items[0].metadata.name}')
[redhat@rhel96-microshift419-vm2 DotNetBuggyApp]$ kubectl debug -it "$POD_NAME" --image=quay.io/rhn_support_arolivei/dotnet-secure-debug:v1 --target=dotnet-app
Targeting container "dotnet-app". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
--profile=legacy is deprecated and will be removed in the future. It is recommended to explicitly specify a profile, for example "--profile=general".
Defaulting debug container name to debugger-rlb4q.
If you don't see a command prompt, try pressing enter.
------------------------------------------------------------------------
Successfully triggered core dump generation in the application container.
The dump file is being written to '/app/dumps/coredump.dmp' inside the 'dotnet-app' container.
You can now copy the file from the application container.
Example: kubectl cp <pod-name>:/app/dumps/coredump.dmp ./coredump.dmp -c dotnet-app
This debug container will automatically exit in 10 seconds.
------------------------------------------------------------------------
Exiting debug container.
Session ended, the ephemeral container will not be restarted but may be reattached using 'kubectl attach dotnet-memory-leak-app-77b88ddf46-j4dn5 -c debugger-rlb4q -i -t' if it is still running
[redhat@rhel96-microshift419-vm2 DotNetBuggyApp]$ oc rsh dotnet-memory-leak-app-77b88ddf46-j4dn5
Defaulted container "dotnet-app" out of: dotnet-app, debugger-rlb4q (ephem)
sh-4.4$ ls -la /app/dumps/
total 236132
drwxrwsrwx. 2 root 1000170000 26 Sep 11 16:10 .
drwxr-xr-x. 1 root root 19 Sep 11 16:09 ..
-rw-------. 1 1000170000 1000170000 241799168 Sep 11 16:10 coredump.dmp
sh-4.4$
This method demonstrates how to run the application under a highly restrictive, non-root security context. It serves as a best-practice example for production environments where security is paramount. This approach uses a dedicated service account and a custom Security Context Constraint (SCC) to enforce strict security rules from the start.
Configuration:
This approach is defined in two files:
deployment-secure.yaml: A new deployment manifest that runs the pod with a locked-down security context.scc-and-rbac-secure.yaml: Contains the necessaryServiceAccountand a customSecurityContextConstraints(SCC) namedrestricted-v2.
Key security settings enforced by this configuration include:
runAsNonRoot: true: Ensures the container does not run as root.runAsUser: 1000/runAsGroup: 1000: Forces the container to run with a specific, non-privileged user and group ID.readOnlyRootFilesystem: true: Prevents any part of the container's root filesystem from being written to. Writable paths for dumps (/app/dumps) and temporary files (/tmp) are provided by volume mounts.capabilities: { drop: ["ALL"] }: Drops all Linux capabilities, reducing the process's potential privileges to the absolute minimum.seccompProfile: { type: RuntimeDefault }: Applies the default seccomp profile of the container runtime, blocking a wide range of potentially dangerous syscalls.
Workflow:
The new resources are included in the kustomization.yaml file. To deploy this hardened application alongside the default one, simply apply the kustomization.
Example Command:
# Apply all configurations, including the secure deployment
oc apply -k .
After the command succeeds, you will have two deployments running: the original dotnet-memory-leak-app and the new, hardened dotnet-memory-leak-app-secure. This allows you to compare their behavior and verify that the application still functions correctly under much stricter security constraints.
Sample output:
[redhat@rhel96-microshift419-vm2 kubernetes]$ kubectl get pods -n dotnet-memory-leak-app -l app=dotnet-memory-leak-app-secure
NAME READY STATUS RESTARTS AGE
dotnet-memory-leak-app-secure-86c78f8bf4-gmlls 1/1 Running 0 25s
[redhat@rhel96-microshift419-vm2 kubernetes]$ export POD_NAME=$(kubectl get pods -n dotnet-memory-leak-app -l app=dotnet-memory-leak-app-secure -o jsonpath='{.items[0].metadata.name}')
[redhat@rhel96-microshift419-vm2 kubernetes]$ kubectl debug -it "$POD_NAME" --image=quay.io/rhn_support_arolivei/dotnet-secure-debug:v1 --target=dotnet-app-secure
Targeting container "dotnet-app-secure". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
--profile=legacy is deprecated and will be removed in the future. It is recommended to explicitly specify a profile, for example "--profile=general".
Defaulting debug container name to debugger-5klct.
If you don't see a command prompt, try pressing enter.
Exiting debug container.
Session ended, the ephemeral container will not be restarted but may be reattached using 'kubectl attach dotnet-memory-leak-app-secure-86c78f8bf4-gmlls -c debugger-5klct -i -t' if it is still running
[redhat@rhel96-microshift419-vm2 kubernetes]$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
dotnet-memory-leak-dumps Bound pvc-89927e32-e3fe-4f26-8d61-e974e2d628d3 5Gi RWO topolvm-provisioner <unset> 91s
[redhat@rhel96-microshift419-vm2 kubernetes]$ oc rsh dotnet-memory-leak-app-secure-86c78f8bf4-gmlls
Defaulted container "dotnet-app-secure" out of: dotnet-app-secure, debugger-5klct (ephem)
sh-4.4$ ls -la /app/dumps/
total 235740
drwxrwsrwx. 3 root 1000 37 Sep 11 16:40 .
drwxr-xr-x. 1 root root 19 Sep 11 16:39 ..
-rw-------. 1 1000 1000 241397760 Sep 11 16:40 coredump.dmp
drwxrwsrwx. 2 1000 1000 6 Sep 11 16:39 tmp
sh-4.4$ date
Thu Sep 11 16:41:03 UTC 2025
sh-4.4$
[redhat@rhel96-microshift419-vm2 kubernetes]$ oc get pods dotnet-memory-leak-app-secure-6f7f4c4f49-thxdc -o yaml|egrep -A 10 "securityContext|share"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /app/dumps
--
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
stdin: true
targetContainerName: dotnet-app-secure
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
tty: true
--
securityContext:
fsGroup: 1000
runAsGroup: 1000
runAsNonRoot: true
runAsUser: 1000
seLinuxOptions:
level: s0:c13,c12
seccompProfile:
type: RuntimeDefault
serviceAccount: secure-app-sa
serviceAccountName: secure-app-sa
shareProcessNamespace: false
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
[redhat@rhel96-microshift419-vm2 kubernetes]$ export POD_NAME=$(kubectl get pods -n dotnet-memory-leak-app -l app=dotnet-memory-leak-app-secure -o jsonpath='{.items[0].metadata.name}')
[redhat@rhel96-microshift419-vm2 kubernetes]$ kubectl debug -it "$POD_NAME" --image=quay.io/rhn_support_arolivei/dotnet-secure-debug:v1 --target=dotnet-app-secure
Targeting container "dotnet-app-secure". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
--profile=legacy is deprecated and will be removed in the future. It is recommended to explicitly specify a profile, for example "--profile=general".
Defaulting debug container name to debugger-pkjlt.
If you don't see a command prompt, try pressing enter.
------------------------------------------------------------------------
Successfully triggered core dump generation in the application container.
The dump file is being written to '/app/dumps/coredump.dmp' inside the 'dotnet-app' container.
You can now copy the file from the application container.
Example: kubectl cp <pod-name>:/app/dumps/coredump.dmp ./coredump.dmp -c dotnet-app
This debug container will automatically exit in 10 seconds.
------------------------------------------------------------------------
Exiting debug container.
Session ended, the ephemeral container will not be restarted but may be reattached using 'kubectl attach dotnet-memory-leak-app-secure-6f7f4c4f49-7d55w -c debugger-pkjlt -i -t' if it is still running
[redhat@rhel96-microshift419-vm2 kubernetes]$
When using kubectl debug to attach ephemeral containers, be aware that performing this action repeatedly on the same pod instance can lead to instability.
Each debug session adds a new ephemeral container to the pod's specification and status. Over time, this history of terminated containers can clutter the pod object and may cause issues with the Kubelet and container runtime.
Symptoms of this instability include:
- The
kubectl debugcommand hanging and failing to provide a shell. - Standard commands like
oc logsfailing with errors such asunable to retrieve container logs. - The pod getting stuck in a
Terminatingstate when you try to delete it normally.
Best Practice / Solution: Treat the pod instance you are debugging as disposable. After one or two debug sessions, the recommended procedure is to delete the pod and allow its controller (Deployment, ReplicaSet, etc.) to create a fresh, clean instance.
# Delete the pod to get a clean instance for the next debug session
oc delete pod <your-pod-name>If the pod becomes stuck in the Terminating state, you may need to force its deletion as a last resort:
# Use this only if a normal delete fails
oc delete pod <your-pod-name> --force --grace-period=0Of course. Here is the rest of the README.md section, detailing the step-by-step instructions that follow the prerequisites we just defined.
After completing the prerequisites, follow these steps to generate the dump.
- SSH Access: You must have SSH access to the MicroShift node where the target pod is running.
- Root Privileges: You need
sudoorrootprivileges on the node. .NETDebugging Tools on the Host: Thecreatedumputility must be made available on the Base OS. Choose one of the two options below.
This is the most efficient method if the debug container image is already present on the node, as it doesn't require any external downloads.
-
Start a Temporary Helper Container: We'll use
podman(available on MicroShift nodes) to start a temporary container from your debug image in the background. This gives us access to its filesystem.# Replace with your actual debug image if different sudo podman run -d --name temp-debugger quay.io/rhn_support_arolivei/dotnet-debug:v1 sleep infinity -
Copy the
createdumpBinary to the Host: Find the path tocreatedumpinside the container (it's often in/usr/local/binor within the SDK path) and copy it to a standard location on the host.# Copy the utility from the container to the host's /usr/local/bin sudo podman cp temp-debugger:/usr/local/bin/createdump /usr/local/bin/createdump -
Clean Up: Remove the temporary container.
sudo podman rm -f temp-debugger
Use this method if you prefer to have the full SDK available on the host or cannot run a temporary container.
-
Download the SDK: On a machine with internet access, download the official .NET SDK binaries for Linux x64.
# Example for .NET 8 SDK wget https://download.visualstudio.microsoft.com/download/pr/25345091-6284-4443-8511-995f5434193c/a24ed1732055a4097e3a31c5b8b9560f/dotnet-sdk-8.0.401-linux-x64.tar.gz -
Transfer and Extract: Copy the
tar.gzfile to your MicroShift node (e.g., withscp) and extract it to a location like/opt/dotnet.# On the MicroShift node sudo mkdir -p /opt/dotnet sudo tar zxf dotnet-sdk-8.0.401-linux-x64.tar.gz -C /opt/dotnet -
Make
createdumpAccessible: The utility is now on the host, but it's deep inside the extracted folder. Create a symbolic link to it from/usr/local/binfor easy access.# The exact version folder (8.0.8) might differ. Adjust the path as needed. sudo ln -s /opt/dotnet/shared/Microsoft.NETCore.App/8.0.8/createdump /usr/local/bin/createdump
After completing either Option A or Option B, the createdump command will be available on your host, and you can proceed with the rest of the steps for generating the dump from the Base OS.
-
Find the Container's Host PID From the host's perspective, the container's main process is just a regular Linux process. Use
psto find its PID and export it as a variable for the next steps.# Find the PID of your dotnet process ps aux | grep DotNetMemoryLeakApp.dll # Example Output: # 1000180+ 2533 0.2 1.0 4038164 169476 ? Sl 14:00 0:01 dotnet /app/DotNetMemoryLeakApp.dll # Export the PID for use in subsequent commands (replace 2533 with your PID) export DOTNET_PID=2533
-
Enter the Container's Mount Namespace To ensure that file paths are correct (i.e.,
/app/dumpspoints to the container's volume), use thensenterutility. This command will give you a new shell session that sees the filesystem exactly as the container does.sudo nsenter -t $DOTNET_PID -m -- bashYour command prompt will change, indicating you are now operating within the container's mount namespace.
-
Generate the Memory Dump From inside the
nsentershell, execute thecreatedumpcommand. It targets the host PID you found earlier, but it writes the file to a path (/app/dumps/...) that is valid from the container's point of view.# This command is run INSIDE the nsenter shell createdump -u -o /app/dumps/coredump-from-host.dmp $DOTNET_PID
-
Exit and Copy the File First, leave the special
nsentershell to return to your normal SSH session on the host.# Inside the nsenter shell exit
Now, from your local machine (not the node's SSH session), use the standard
kubectl cpcommand to retrieve the dump file from the running pod using the Kubernetes API.# On your local machine export POD_NAME=$(kubectl get pods -n dotnet-memory-leak-app -l app=dotnet-memory-leak-app-secure -o jsonpath='{.items[0].metadata.name}') kubectl cp "$POD_NAME":/app/dumps/coredump-from-host.dmp ./coredump-from-host.dmp -c dotnet-app-secure
In a Kubernetes environment, managing compute resources like CPU and Memory is not just a best practice; it is critical for ensuring application performance and cluster stability. This is especially true for single-node deployments like MicroShift and self-contained air-gapped systems.
What are Resource Requests and Limits? When defining a Pod, you can specify resource requests and limits for each of its containers:
- Requests: This is the amount of CPU and Memory that Kubernetes guarantees for a container. The Kubernetes scheduler uses this value to decide where to place the Pod, ensuring it only runs on a node with enough available capacity.
- Limits: This is the maximum amount of CPU and Memory a container is allowed to use.
- If a container exceeds its Memory limit, it is terminated by the kernel (an "Out of Memory" or OOMKill).
- If a container exceeds its CPU limit, it is "throttled," meaning its CPU usage is artificially capped, which can degrade its performance.
YAML
# Example snippet for a container's spec
resources:
requests:
memory: "64Mi"
cpu: "250m" # 250 millicores (0.25 of a core)
limits:
memory: "128Mi"
cpu: "500m" # 500 millicores (0.5 of a core)
Why This is Critical for a MicroShift / Single-Node Cluster In a multi-node cluster, a single "runaway" application might crash one worker node, but the cluster and other applications remain operational. In a single-node cluster like MicroShift, the node is the "cluster".
- Preventing Node Starvation: If a single container without limits consumes all available Memory or CPU, it can starve the node's critical system processes, including the kubelet and the underlying operating system. This can cause the node to enter a NotReady state, effectively bringing down the entire cluster and making it unresponsive.
- Protecting the Control Plane: MicroShift runs its control plane components (like the API server) on the same single node. Enforcing limits on your applications ensures they cannot disrupt the resources needed by the control plane, thereby protecting the stability and availability of the cluster itself.
- Ensuring Quality of Service (QoS): By setting resource requests, you tell Kubernetes which Pods are more important. Pods with guaranteed resources (Guaranteed QoS class) are the last to be killed if the node runs out of memory, ensuring your critical applications survive.
Why This is Critical in an Air-Gapped Environment An air-gapped environment has a fixed, finite amount of hardware resources.
- Inability to Scale Out: Unlike a cloud environment where you can automatically provision more nodes in response to high load, an air-gapped system cannot be easily expanded. You must work within the physical constraints of your hardware.
- Enforcing Capacity Management: Limits are your primary tool for enforcing capacity management. They prevent any single application or team from consuming a disproportionate share of the fixed resources, which could cause a cascading failure of other essential services running in the same environment.
Putting it into Practice: The LimitRange Object Defining requests and limits for every Pod manually can be tedious. Kubernetes provides a policy object called LimitRange that you can apply to a namespace to enforce sane defaults and constraints.
A LimitRange can:
- Assign default request and limit values to containers that do not define their own.
- Enforce minimum and maximum values for CPU and Memory.
- Enforce a ratio between requests and limits.
Example limitrange.yaml: This LimitRange enforces that every container in the namespace will get default resources if not specified, and it prevents any single container from requesting too much.
apiVersion: v1
kind: LimitRange
metadata:
name: resource-limits-for-namespace
spec:
limits:
- type: Container
# Default resource request for any container created without one.
defaultRequest:
cpu: "100m"
memory: "64Mi"
# Default resource limit for any container created without one.
default:
cpu: "500m"
memory: "256Mi"
# Maximum resource limit any container in the namespace is allowed to have.
max:
cpu: "1" # 1 full core
memory: "1Gi"
# Minimum resource limit any container in the namespace is allowed to have.
min:
cpu: "50m"
memory: "32Mi"
By applying a LimitRange to your namespaces, you create a powerful safety net that significantly improves the stability and predictability of your cluster—a necessity for a production-grade, single-node system.
The application includes comprehensive health checks to ensure reliable operation:
# Test health endpoints
export ROUTE_HOST=$(oc get route dotnet-memory-leak-route -n dotnet-memory-leak-app -o jsonpath='{.spec.host}')
# Test liveness probe - should return HTTP 200 OK with "Healthy" status
curl -v http://$ROUTE_HOST/healthz
# Expected output: HTTP/1.1 200 OK
# Response body: {"status":"Healthy","totalDuration":"00:00:00.0001234"}
# Test readiness probe - should return HTTP 200 OK with "Healthy" status
curl -v http://$ROUTE_HOST/readyz
# Expected output: HTTP/1.1 200 OK
# Response body: {"status":"Healthy","totalDuration":"00:00:00.0001234"}
# Check pod status and probe results
oc get pods -n dotnet-memory-leak-app -o wide
oc describe pod <pod-name> -n dotnet-memory-leak-appUnderstanding Health Check Responses:
- HTTP 200 OK: Application is healthy and ready to serve traffic
- HTTP 503 Service Unavailable: Application is unhealthy (rarely occurs in this simple app)
- Connection refused: Pod is not running or network policy is blocking access
Verify that security controls are properly applied:
# Check pod security context
oc get pod <pod-name> -n dotnet-memory-leak-app -o yaml | grep -A 20 securityContext
# Verify network policy is applied
oc get networkpolicy -n dotnet-memory-leak-app
# Check RBAC permissions
oc auth can-i get pods --as=system:serviceaccount:dotnet-memory-leak-app:dotnet-app-sa -n dotnet-memory-leak-app
# Verify SCC binding
oc get scc dotnet-scc -o yaml# Trigger memory leak and monitor
curl http://$ROUTE_HOST/triggerMemoryLeak &
# Monitor memory usage
oc top pod -n dotnet-memory-leak-app
# Watch for OOM and restart
oc get pods -n dotnet-memory-leak-app -w
# Check for crash dumps after restart
oc rsh <pod-name> -n dotnet-memory-leak-app -- ls -la /app/dumps/# Test ingress from allowed sources (should work)
curl -f http://$ROUTE_HOST/healthz
# Test egress restrictions (should be limited to DNS and monitoring)
oc rsh <pod-name> -n dotnet-memory-leak-app -- nslookup kubernetes.default.svc.cluster.localThis application implements production-ready security practices:
This application implements a defense-in-depth security strategy with multiple layers of protection:
- Red Hat UBI RHEL9 base images: Uses exclusively Red Hat Universal Base Images for enterprise support and security compliance
- Non-root containers: Application runs as non-root user with OpenShift arbitrary UID support
- Read-only root filesystem: Container filesystem is read-only with writable volumes for dumps and temp files
- Minimal capabilities: All Linux capabilities dropped except those explicitly required
- Seccomp profiles: Runtime default seccomp profile for syscall filtering
- No debug tools in production: Diagnostic tools are only available in separate debug containers
- Network isolation: NetworkPolicy restricts ingress/egress traffic to only necessary communication
- Minimal RBAC: ServiceAccount has only necessary namespace-scoped permissions (no cluster-wide access)
- Security Context Constraints: Custom SCC provides only required SYS_PTRACE capability for debugging
- Resource limits: CPU and memory limits prevent resource exhaustion and node starvation
- Health checks: Liveness, readiness, and startup probes for reliable operation and failure detection
- Automount service account token disabled: Reduces attack surface by not automatically mounting service account tokens
- Process namespace isolation:
shareProcessNamespace: falseprevents process visibility between containers - Unique dump naming: Crash dumps include timestamps to prevent overwrites and enable tracking
Why these specific security settings?
runAsNonRoot: true: Prevents privilege escalation attacks and follows principle of least privilegereadOnlyRootFilesystem: true: Prevents malicious code from writing to container filesystem, forcing all writes to mounted volumescapabilities.drop: [ALL]: Removes all Linux capabilities, then explicitly adds only what's needed (SYS_PTRACE for debugging)seccompProfile: RuntimeDefault: Filters system calls to prevent exploitation of dangerous syscallsautomountServiceAccountToken: false: Reduces attack surface by not automatically providing cluster credentials- NetworkPolicy: Implements zero-trust networking by default-deny with explicit allow rules
Security vs. Functionality Balance:
- Debug capabilities are preserved through dedicated debug containers and SCC bindings
- Production image is minimal and secure by default
- Diagnostic tools are available on-demand without compromising production security
Deploying and debugging applications in OpenShift/Kubernetes, especially with advanced diagnostic tools, often involves navigating strict security policies.
Kubernetes environments, including OpenShift and MicroShift, utilize powerful security enforcement mechanisms to govern pod behavior and permissions. These are primarily implemented through policies such as Security Context Constraints (SCCs), which are specific to OpenShift, and the Kubernetes-native Pod Security Admission (PSA).
In a hardened cluster, it is common for a default, restrictive security policy to be applied at the namespace or cluster level. Such policies rigorously control the security-sensitive attributes a pod can request in its specification.
Consequently, if the securityContext defined in a deployment manifest includes settings that are disallowed by the active policy (for example, attempting to run as a specific user ID or requesting certain capabilities), the Kubernetes API server will reject the configuration. This typically results in an error during deployment, with messages such as Warning: would violate PodSecurity or Error creating: pods "..." is forbidden: violates PodSecurity "restricted".
Requirement: Tools like dotnet-dump collect need the CAP_SYS_PTRACE capability to attach to another process and inspect its memory. Challenge: Strict security policies often disallow or strip this capability from containers (e.g., drop: ALL is common in restricted policies). You might see errors like Invalid value: "SYS_PTRACE": capability may not be added. Solution: To enable SYS_PTRACE for interactive debugging, you typically need to: Ensure your securityContext in deployment.yaml add: - SYS_PTRACE (and doesn't drop: ALL). Bind your ServiceAccount to a more permissive SCC (e.g., privileged) or have a cluster administrator adjust the namespace's Pod Security Enforcement to baseline or eventually bind a custom SCC to the ServiceAccount.
Requirement: Containers often define a seccompProfile (e.g., RuntimeDefault) for enhanced security by filtering syscalls. Challenge: In very strict environments, even setting seccompProfile: type: RuntimeDefault might be forbidden, leading to errors like Forbidden: seccomp may not be set. Solution: If seccomp is blocked, you might need to remove the seccompProfile lines from your deployment.yaml's securityContext and rely solely on the privileged SCC to provide an unconfined or permissive seccomp profile.
Requirement: dotnet diagnostic tools use temporary directories (often /tmp/) for inter-process communication (IPC) when connecting to a target process. Challenge: If /tmp/ is not properly writable for the container's assigned user, or if TMPDIR environment variables are inconsistent between the diagnostic tool and the target application, connection issues can arise (e.g., "Please verify that /tmp/ is writable by the current user"). Solution: Explicitly set TMPDIR=/app/dumps/tmp for both the application and diagnostic containers in deployment.yaml. Ensure /app/dumps/tmp is created and made world-writable (chmod 777) at runtime via a command/args in your container definition, as volume mounts can overwrite built-in directories.
Challenge: If your application is actively consuming memory and approaching its resources.limits.memory, the operating system's OOM killer might terminate the process before the .NET runtime has a chance to fully write a crash dump, especially for full dumps. Solution: Temporarily increase the resources.limits.memory for your application container in deployment.yaml to provide a larger buffer, allowing more time for dump generation. Rely on the automatic OOM dumps (Option A), as they are designed to capture the state at the moment of crash.
Before considering the deployment successful, verify:
- Pod is running and ready (all probes passing)
- Health endpoints respond correctly (
/healthz,/readyz) - NetworkPolicy is applied and traffic is restricted
- RBAC permissions are minimal and namespace-scoped only
- SCC is bound and provides only necessary capabilities
- Resource limits are appropriate for your environment
- PVC is bound and accessible for dump storage
When debugging is needed:
- Assess the situation: Determine if you need live debugging or can wait for automatic dumps
- Choose the appropriate method:
- Automatic OOM dumps (Option B) for crash scenarios
- Ephemeral containers (Option D/E) for live debugging
- Sidecar injection (Option C) for extended debugging sessions
- Apply minimal privileges: Use the custom SCC that provides only SYS_PTRACE
- Clean up after debugging: Remove debug containers and restore original deployment
- Delete pod after multiple debug sessions: Prevent pod instability from accumulated ephemeral containers
If security concerns arise:
-
Immediate response:
# Isolate the pod by scaling down oc scale deployment dotnet-memory-leak-app --replicas=0 -n dotnet-memory-leak-app # Check for unauthorized access oc logs <pod-name> -n dotnet-memory-leak-app --previous
-
Investigation:
# Preserve crash dumps for analysis oc cp <pod-name>:/app/dumps/ ./dumps-backup/ -n dotnet-memory-leak-app # Check network policy violations oc get events -n dotnet-memory-leak-app --sort-by='.lastTimestamp'
-
Recovery:
# Restore from clean deployment oc apply -k kubernetes/ # Verify security posture oc get pod <pod-name> -n dotnet-memory-leak-app -o yaml | grep -A 10 securityContext
Regular maintenance tasks:
# Clean up old crash dumps (create a CronJob for this)
oc rsh <pod-name> -n dotnet-memory-leak-app -- find /app/dumps -name "*.dmp" -mtime +7 -delete
# Update image tags for security patches
oc set image deployment/dotnet-memory-leak-app dotnet-app=quay.io/your-namespace/dotnet-memory-leak-app:v2 -n dotnet-memory-leak-app
# Verify resource usage and adjust limits if needed
oc top pod -n dotnet-memory-leak-app
oc describe limitrange dotnet-limitrange -n dotnet-memory-leak-appMonitoring and alerting:
- Set up alerts for pod restarts (indicates potential OOM)
- Monitor PVC usage to prevent storage exhaustion
- Track network policy violations in cluster logs
- Alert on RBAC permission changes
For a deeper dive into the technologies and concepts explored in this project, refer to the following official documentation and resources:
- .NET Diagnostics
- .NET Diagnostics in Containers: https://learn.microsoft.com/en-us/dotnet/core/diagnostics/diagnostics-in-containers
- Debug Memory Leaks in .NET: https://learn.microsoft.com/en-us/dotnet/core/diagnostics/debug-memory-leak
dotnet-dumpglobal tool: https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dumpdotnet-traceglobal tool: https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-tracedotnet-countersglobal tool: https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-countersdotnet-gcdumpglobal tool: https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-gcdump
- OpenShift, MicroShift & Kubernetes Security
- Understanding Security Context Constraints (SCCs) in OpenShift: https://docs.openshift.com/container-platform/latest/authentication/managing-security-context-constraints.html
- Kubernetes Pod Security Standards: https://kubernetes.io/docs/concepts/security/pod-security-standards/
- Managing Pod Security Admission: https://kubernetes.io/docs/tasks/configure-pod-container/enforce-standards-admission-controller/
- Using seccomp in OpenShift: https://docs.openshift.com/container-platform/latest/security/seccomp-profiles.html
- OpenShift & Container Best Practices
- Red Hat Universal Base Images (UBI): https://access.redhat.com/containers
- Building Container Images in OpenShift: https://docs.openshift.com/container-platform/latest/builds/index.html
- Troubleshooting OpenShift Pods: https://docs.openshift.com/container-platform/latest/nodes/nodes-pods-debug-container-issues.html
Feel free to open issues or submit pull requests for any improvements or bug fixes.
This project is licensed under the Apache License 2.0. See the LICENSE file for details.