DASE

JCSE 18(4):214–223, 2024 · Hands-on demo

Expanding a PMD ruleset for mitigating Java deserialization vulnerabilities (CWE-502)

The paper turns four SEI CERT Java rules (SER01-J, SER03-J, SER07-J, SER12-J) into PMD XPath rules. Here those rules are run, exactly as printed, through real PMD 6.55 and PMD 7.7, alongside a revised PMD 7 ruleset. Both are scored on a labelled test corpus and on two real Java code bases. A read-only stream inspector shows the allowlist defence that SER12-J asks for. Paper DOI

CWE-502 in the NVD (paper Table 1)

Bars = CWE-502 entries per year (left axis); line = share of all NVD entries (%). 2024 counts run up to October.

What goes wrong

Serialization turns an object graph into bytes; deserialization rebuilds it. With ObjectInputStream.readObject(), the bytes choose which classes are instantiated, and those classes' readObject() hooks run on the way in. If an attacker controls the bytes and a suitable “gadget” class is on the classpath, rebuilding the object graph can execute code (RCE) or exhaust memory (DoS).

The same idea applies to JSON. In jackson-databind, default typing lets the JSON name the class to create. That is why 20 of the 25 CWE-502 CVEs the paper found in DoD open-source software are jackson-databind issues (Fig. 3).

Defences: do not deserialize untrusted data; if you must, allow only known classes (ObjectInputFilter, JEP 290/415, or resolveClass() look-ahead), keep libraries patched, and prefer data formats that bind to fixed types.

Fig. 2 · Serialized object lifecycle

Paper Fig. 2

Fig. 3 · CWE-502 CVEs in DDF by library

Paper Fig. 3

Fig. 1 · Vulnerability sharing systems

Paper Fig. 1

The four CERT rules the paper implements (Table 6)

RuleRequirementPMD priority
SER01-JDo not deviate from the proper signatures of serialization methods.1 (CERT P27)
SER03-JDo not serialize unencrypted sensitive data.4 (CERT P6)
SER07-JDo not use the default serialized form for classes with implementation-defined invariants (singletons need readResolve()).5 (CERT P4)
SER12-JPrevent deserialization of untrusted data (allowlist in resolveClass()).4 (CERT P9)
SER06-JDefensive copies of mutable components; excluded as not detectable by PMD.–
SEC58-JNo dangerous operations in deserialization methods; excluded (a recommendation).–

CWE-502 CVEs found in DDF and REDHAWK SDR (paper Tables 4–5)

Each case is a small Java file labelled with the CERT rule it violates (or none). The findings are real PMD output, not a simulation: the paper's rules under PMD 6.55.0 and PMD 7.7.0, and the revised rules under PMD 7.7.0. Highlighted lines: paper rule revised rule.

Labelled corpus: 23 files, 11 violations

TP = violation found, FP = false alarm, FN = missed violation, scored per file and rule. Caveat: the corpus and the revised rules were written by the same author, so the revised rules' perfect score here is expected and proves little. Use the two real code bases below for a fairer picture.

Alerts per rule on real code

Number of PMD findings. Commons Collections 3.2.1 is the library behind the 2015 gadget-chain disclosures; DDF is one of the two DoD projects analysed in the paper.

Precision and recall where ground truth exists

Held-out check on DDF: unfiltered deserialization entry points

Ground truth = every place in DDF main code that creates new ObjectInputStream(...) and calls readObject() with no filter (found by text search, so complete for this pattern). No jdk.serialFilter or setSerialFilter appears anywhere in the repository; one may still be set at deployment. Whether each entry point is exploitable depends on who controls the bytes.

How the revised rules were developed (and why that matters)

    The rules were tuned on Commons Collections, then frozen before DDF's findings were inspected. On DDF the frozen rules found 5 of the 7 entry points with no false alarms. The 2 misses are both streams declared as the ObjectInput interface, a pattern the rule had never seen. A paper that proposes rules should report exactly this kind of held-out result.

    Setup

    Tools
    PMD 6.55.0 (XPath 1.0 default), PMD 7.7.0, OpenJDK 21
    Paper rules
    Tables 7–10 verbatim; only the typo “MethodDeclarat or” joined
    PMD 7 run
    also with the rule class renamed to …rule.xpath.XPathRule
    CC 3.2.1
    273 files, revised rules with compiled classes on --aux-classpath
    DDF
    1,616 main-code files, no aux classpath (type resolution limited to JDK types)
    Reproduce
    eval/run.sh, rules/*.xml, corpus/ in this Space

    Paper rule vs revised rule

    As printed in the paper (PMD 6 syntax)

      
        
      Revised for PMD 7
        
      
        
    
        
    
        
      
        Why the paper's rules report nothing under PMD 7
        
      
        PMD 7.0.0 (March 2024) rewrote the Java AST. ClassOrInterfaceDeclaration became ClassDeclaration, MethodDeclarator disappeared, @Image on types became @SimpleName, @Private became @Visibility, and the rule class moved to net.sourceforge.pmd.lang.rule.xpath.XPathRule. With the old class name the ruleset does not load. With the new class name it loads, but every query silently matches nothing: no error, no warning, and a clean-looking report. Rules need regression tests (see corpus/) for exactly this reason.
        
      
        
    
        
  
        

  
  
        
    
        
      
        Paste a Java serialization stream (Base64 or hex). The page only reads the bytes: it lists the classes the stream would make ObjectInputStream instantiate, and checks them against a JEP 290-style filter, the allowlist that SER12-J asks for. Nothing is deserialized and no code runs. The samples are harmless objects produced with a real JDK.
        
    
        
    
        
      
        
        
        
        
        
        
        
        
        
      
        
      
        
        
        
        
        Choose a sample or paste a stream.
        
        
        
        
        
      
        
    
        
    
        First bytes
        
      
        Highlighted: magic AC ED and version 00 05. In Base64 these bytes become rO0AB…, which is how the paper's attack scenario (Table 11) spots serialized objects in HTTP traffic; the content type application/x-java-serialized-object is a third sign.
        
  
        

  
  
        
    
        
      
        Review points from today's perspective
        
      
          
        
        1. The rules are not evaluated. The paper presents the XPath queries but no detection results: no test cases, precision, recall or run on DDF/REDHAWK, the code bases that motivated it. Running them (tab ③) reveals problems that a small labelled test suite would have caught before publication.
          2. 
        
        2. The published queries target PMD 6 and fail silently on PMD 7. PMD 7.0.0 shipped in March 2024, before the paper was submitted. The PMD version should be stated, and rules should ship with regression tests, because an XPath rule that matches nothing looks the same as clean code.
          3. 
        
        3. Name and string matching cause both false alarms and misses.
          
            
            
          • SER01: misses inherited Serializable and java.io.Serializable; flags a protected readResolve(), which Java serialization allows.
            • 
            
          • SER07: keys on fields named INSTANCE/instance, so it flags non-serializable singletons and misses other names.
            • 
            
          • SER12: treats any variable whose name contains “whitelist” as a defence, even if unused, and does not recognise ObjectInputFilter (JEP 290, Java 9+).
            • 
          
          4. 
        
        4. Two query bugs. SER03 uses //ClassOrInterfaceBody (absolute path) instead of .//, so any public class in a file that mentions ByteArrayOutputStream is flagged; 21 of its 35 DDF alerts come from that alone. SER12's MethodCall[MethodName=…] mixes PMD 7 node names into PMD 6 syntax and never matches. Under PMD 6 it therefore fires on any class that merely mentions ObjectInputStream and has no “whitelist” variable or resolveClass(), including correct private readObject() hooks (41 of 42 alerts in Commons Collections).
          5. 
        
        5. The SER03 rule does not check SER03. SER03-J is about sensitive data, but the rule looks for NotSerializableException in classes that use output streams. Section V itself notes that the transient case was dropped. A field-name heuristic, as in the revised rule, is crude but at least targets the requirement.
          6. 
        
        6. The motivation and the rules do not line up. 80% of the CVEs analysed are jackson-databind polymorphic typing, and Log4Shell is a JNDI lookup. None of the four rules covers enableDefaultTyping()/activateDefaultTyping() or @JsonTypeInfo(use = Id.CLASS), and none would have caught Log4Shell. Dependency checking (SCA, e.g. OWASP Dependency-Check) addresses most of those CVEs more directly than coding rules.
          7. 
        
        7. Related tools and modern Java defences are not discussed. FindBugs is no longer maintained; its successor SpotBugs does support custom detectors, and the FindSecBugs plugin already has OBJECT_DESERIALIZATION and JACKSON_UNSAFE_DESERIALIZATION. Semgrep and CodeQL (java/unsafe-deserialization, taint-based) are natural baselines. JEP 290 serialization filters and JEP 415 context-specific filters (Java 17) are the platform answer to SER12-J.
          8. 
        
        8. Taint matters. A purely syntactic rule cannot tell whether the bytes are untrusted. In DDF, two entry points read bytes from internal stores (second-order risk), and Commons Collections' PrototypeFactory deserializes bytes it has just produced. Data-flow analysis (CodeQL, or PMD's Java rule API rather than XPath) is the next step.
          9. 
        
        9. Smaller points: the attack scenario (Table 11) is not traced to a rule that would have caught it, and as printed Table 9 contains a typo (“MethodDeclarat or”) that makes PMD 6 abort with an error.
          10. 
      
        
    
        
    
        
      
        
        
        Classroom exercises
        
        
          
          
        1. Run eval/run.sh, then add a case where the stream is declared as ObjectInput. Make the revised SER12 rule catch it without breaking the other 23 cases.
          2. 
          
        2. Write a PMD 7 rule for Jackson @JsonTypeInfo(use = JsonTypeInfo.Id.CLASS) and test it on a three-file corpus you label yourself.
          3. 
          
        3. In tab ⑤, write the tightest filter that still accepts the UserState sample. What happens to HashMap, and why does the JDK also check Map$Entry[]?
          4. 
          
        4. Pick one paper rule and one revised rule and argue which false-positive rate a weapon-system static-test checklist could tolerate.
          5. 
        
        
      
        
      
        
        
        What would make the study stronger
        
        
          
          
        • A labelled benchmark (CERT compliant/non-compliant pairs plus real CVE fix commits), with precision and recall per rule.
          • 
          
        • Runs on the motivating projects (DDF, REDHAWK), with alerts triaged by a second reviewer.
          • 
          
        • Comparison with SpotBugs + FindSecBugs, Semgrep and CodeQL on the same benchmark.
          • 
          
        • A Jackson rule and a dependency check, so the rules cover the CVEs that motivated them.
          • 
          
        • The ruleset and test corpus released, so the community can rerun them on new PMD versions.
          • 
        
        
      
        
    
        
  
        

  
        Research and teaching demo · All PMD findings are real outputs of PMD 6.55.0 / 7.7.0 recorded in this Space; the stream inspector parses bytes in your browser and never deserializes them. Paper figures © the authors, CC BY-NC 4.0.