JVM OOM: When the Leak Is in a Library

When I troubleshoot a JVM OOM, I try not to assume my code is the leak. I recommend following memory evidence through JFR, heap dumps, and retained ownership until the object graph proves the source.

The important detail was the dependency path. Reactor Netty 1.1.8 pulled Netty 4.1.94.Final, and the macOS DNS native resolver path used netty-resolver-dns-native-macos. Netty issue 13480 documents a classloader leak in that area: io.netty.resolver.dns.macos.DnsResolver could be retained as a JNI global reference.

Symptoms

The first wrong move would have been increasing -Xmx. Heap was not the main problem. Metaspace and process RSS were. These were the sanitized numbers from the reproduction that matched production behavior:

Capture the evidence

I started with metrics, then captured runtime evidence. The dashboard had jvm.memory.used{area="nonheap",id="Metaspace"}, jvm.classes.loaded, GC pause count, process RSS, and container restarts. The pattern was a staircase: every reload increased Metaspace, and Full GC did not return it.

jcmd $PID JFR.start name=oom settings=profile duration=20m \
  filename=/tmp/oom.jfr path-to-gc-root=true

jcmd $PID GC.class_histogram > /tmp/classes-before.txt
jcmd $PID GC.heap_dump /tmp/before.hprof

# Run 30 to 50 plugin reloads, then capture again
jcmd $PID GC.class_histogram > /tmp/classes-after.txt
jcmd $PID GC.heap_dump /tmp/java_pid8763.hprof

JFR gave the timeline: class loading climbed during plugin reload, Metaspace thresholds were crossed, and Full GC pauses clustered near the end. The heap dumps gave ownership. That split matters. JFR told me when the JVM was hurting. Eclipse Memory Analyzer told me who was keeping memory alive.

Use MAT to prove the owner

When I talk about "Use MAT to prove the owner", I am looking for a measurable improvement in production code or runtime behavior.

In Eclipse Memory Analyzer, I opened java_pid8763.hprof, ran Leak Suspects, then used Class Loader Explorer and the Dominator Tree. The suspicious pattern was not one large business object. It was many copies of the same plugin classloader, each retaining classes that should have disappeared after validation.

The shortest path to GC roots was the decisive clue. The retained path went through io.netty.resolver.dns.macos.DnsResolver and ended at a JNI global reference. That changed the investigation. My code was creating classloaders, but it was not holding them. The external native resolver was.

Replicate and compare versions

The reproduction was small: create an isolated classloader, load the code path that initializes Reactor Netty DNS resolution, make one outbound call, close the client, drop all strong references, force GC, and repeat. Then compare versions.

Remediate without guessing

My recommendation in "Remediate without guessing" is to test JVM OOM on one representative service before standardizing it.

First, list the real Netty version. For Maven, use mvn dependency:tree -Dincludes=io.netty. For Gradle, use ./gradlew dependencyInsight --dependency netty-resolver-dns-native-macos. If the runtime resolves Netty below 4.1.98.Final and uses dynamic classloaders with the macOS native resolver, treat it as affected.

The fix was to move the Netty family to 4.1.98.Final or newer through the platform BOM, the Spring Boot dependency set, or an explicit Netty override. As a short term containment, I reduced plugin reload churn and removed the macOS native resolver from that worker where native DNS behavior was not required. The real remediation was the library upgrade, followed by the same JFR and MAT comparison to prove the retained classloaders disappeared.

The process to reuse

Do not begin with blame. Begin with memory area. If heap grows, inspect object ownership. If Metaspace grows, inspect classloaders. If direct memory grows, inspect native buffers and reference counted objects. Use JFR for the timeline, heap dumps for ownership, MAT for retained paths, and dependency comparison for proof.

A JVM OOM is solved when the same load test stops producing the memory cycle that caused it. In this case, the cycle was classloader creation, JNI global retention, Metaspace growth, Full GC with no recovery, then OOM. Once Netty was upgraded, the cycle broke.

Validation worker leak example

I use "Validation worker leak example" to keep JVM OOM grounded in a real system, because abstract patterns are too easy to agree with and too hard to operate.

The real production shape was a validation worker that loaded customer adapters with isolated classloaders. The application code dropped its references, but the native DNS resolver path kept classloaders reachable. The useful lesson is to prove the retaining owner before changing business code.

Classloader reproduction loop

I use "Classloader reproduction loop" to separate useful platform movement from release note excitement.

The reproduction loop should create, use, close, and drop each classloader:

for (int run = 0; run < 100; run++) {
  try (URLClassLoader loader = new URLClassLoader(pluginUrls, null)) {
    runValidationPlugin(loader);
  }
  System.gc();
  recordMetaspace(run);
}

The loop recreates plugin classloaders, runs work, forces cleanup pressure, and records metaspace. If metaspace keeps growing, the leak is outside normal request objects.