Recognize JavaAgent -- get all classes loaded by the target process

Keywords: Java jvm JDK Weblogic

Author: Longofo @ know Chuangyu 404 Lab
Time: December 10, 2019
Original link: https://paper.seebug.org/1099/

Previously, a class was found in an application, but an error occurred during deserialization test. The error is not class not found, but other 0xxx errors. By searching this error, it is probably that the class was not loaded. Recently, I just saw the JavaAgent. After preliminary learning, I can intercept. I mainly use the Instrument Agent to enhance bytecode. I can do bytecode pegging, bTrace, Arthas and other operations. I can achieve more powerful functions by combining ASM, javassist, cglib framework. Java RASP is also based on Java agent. Record the basic concepts of JavaAgent while it is hot, and simply use JavaAgent to implement a test to obtain the loaded classes of the target process.

JVMTI and Java Instrument

Java Platform Debugger Architecture (JPDA) is a set of API s for debugging java code (from Wikipedia):

  • Java Debugger Interface (JDI) - a high-level java interface is defined. Developers can easily write remote debugging tools with JDI
  • Java Virtual Machine Tools Interface (JVMTI) - defines a native interface, which can check the status and control the operation of applications running in Java virtual machine
  • Java virtual machine debugging interface (JVMDI) - JVMDI is replaced by JVMTI in J2SE 5 and removed in Java SE 6
  • Java debug line protocol (JDWP) - defines the communication protocol between debug object (a Java application) and debugger process

JVMTI provides a set of "agent" program mechanism, which can support the third-party tool program to connect and access the JVM in the way of agent, and use the rich programming interface provided by JVMTI to complete many JVM related functions. JVMTI is event driven. Every time the JVM executes certain logic, it will call some event callback interfaces (if any), which can be used by developers to extend their own logic.

JVMTIAgent is a dynamic library which uses the exposed interface of JVMTI to provide the functions of agent on load, agent on attach and agent on unload. The Instrument Agent can be understood as a kind of JVMTIAgent dynamic library, which is also known as JPLISAgent(Java Programming Language Instrumentation Services Agent), and it is a special agent to support the plug-in service written in java language.

Instrumentation interface

The following interfaces are provided in [1] in the Java SE 8 API document (different versions may have different interfaces):

void addTransformer(ClassFileTransformer transformer, boolean canRetransform)//Register the ClassFileTransformer instance. Registering multiple instances will be called according to the registration order. After all classes are loaded, ClassFileTransformer instances will be called, which is equivalent to redefining them through the redefiniteclasses method. The Boolean parameter canRetransform determines whether the redefined class can be rolled back through the retransformClasses method.

void addTransformer(ClassFileTransformer transformer)//Equivalent to addTransformer(transformer, false), that is, the class redefined through ClassFileTransformer instance cannot be rolled back.

boolean removeTransformer(ClassFileTransformer transformer)//Remove (unregister) the ClassFileTransformer instance.

void retransformClasses(Class<?>... classes)//The method of the loaded class for re transformation. The re transformed class will be called back to the list of ClassFileTransformer for processing.

void appendToBootstrapClassLoaderSearch(JarFile jarfile)//Add a jar to Bootstrap Classpath, and other jars will be loaded first.

void appendToSystemClassLoaderSearch(JarFile jarfile)//Add a jar to Classpath for appclassload to load.

Class[] getAllLoadedClasses()//Get all classes that have been loaded.

Class[] getInitiatedClasses(ClassLoader loader)//Get all classes that have been initialized.

long getObjectSize(Object objectToSize)//Get the (byte) size of an object. Note that nested objects or property references in objects need to be calculated separately.

boolean isModifiableClass(Class<?> theClass)//Determine whether the corresponding class has been modified.

boolean isNativeMethodPrefixSupported()//Whether the prefix of the native method is supported.

boolean isRedefineClassesSupported()//Returns whether the current JVM configuration supports the redefinition of a class (modifying the bytecode of a class).

boolean isRetransformClassesSupported()//Returns whether the current JVM configuration supports class retranslation.

void redefineClasses(ClassDefinition... definitions)//Redefining a class means redefining a loaded class. The input parameter of ClassDefinition type includes the corresponding type class <? > object and the byte array corresponding to the bytecode file.

void setNativeMethodPrefix(ClassFileTransformer transformer, String prefix)//Set the prefix of some native methods, mainly for rule matching when finding native methods.
Redefiniteclasses and redefiniteclasses:

The redefinition function is introduced in Java SE 5, and the redefinition function is introduced in Java SE 6. One guess is to introduce the redefinition as a more general function, but the redefinition must be retained to achieve backward compatibility, and the redefinition operation is more convenient.

Two loading methods of Instrument Agent

As mentioned in the official API document [1], there are two ways to obtain the Instrumentation interface instance:

  1. When the JVM is started by specifying a proxy, the Instrumentation instance is passed to the premain method of the proxy class.
  2. The JVM provides a mechanism to start the agent at a certain time after startup. At this time, the Instrumentation instance will be passed to the agentmain method of the agent class code.

Premain corresponds to the Instrument Agent loading when VM starts, that is, agent on load. Agentmain corresponds to the Instrument Agent loading when VM runs, that is, agent on attach. The instrument agents loaded by the two loading forms pay attention to the same JVMTI event - ClassFileLoadHook event. This event is used for callback after reading the bytecode file. That is to say, the callback time of premain and agentmain is after reading the bytecode of the class file (or after loading the class), and then redefining or retranslating the bytecode. However, it is not modified The modified bytecode also needs to meet some requirements, and the limitations are explained at the end.

Difference between premain and agentmain:

The ultimate purpose of premain and agentmain is to call back the instance of Instrumentation and activate sun. Instrument. InstrumentationImpl ා transform() (InstrumentationImpl is the implementation class of Instrumentation) so as to call back the ClassFileTransformer registered in the Instrumentation to implement bytecode modification. In essence, there is no great difference in function. The difference between the two non essential functions is as follows:

  • Premain is introduced by JDK 1.5, and agentmain is introduced by JDK 1.6. After JDK 1.6, you can choose to use premain or agentmain.
  • premain needs to use the external agent jar package through the command line, that is - javaagent: agent jar package path; agentmain can directly attach to the target VM through the attach mechanism to load the agent, that is, under the agentmain mode, the program operating the attach and the program being proxied can be two completely different programs.
  • The class recalled to ClassFileTransformer by premain method is all classes loaded by virtual machine, which is determined by the order of loading by agent. In the logic of developer, it is: before all classes are loaded for the first time and enter the main() method of program, the premain method will be activated, and then all loaded classes will execute the callback in ClassFileTransformer list.
  • Because the agent main mode adopts the attach mechanism, the VM of the agent's target program may have been started a long time ago, of course, all its classes have been loaded. At this time, it is necessary to use the instrumentation ා retransformclasses (class <? >... Classes) to enable the corresponding classes to be re converted, so as to activate the re converted classes to execute the callback in the ClassFileTransformer list.
  • If the proxy Jar package of premain mode is updated, the server needs to be restarted. If the Jar package of agentmain mode is updated, the server needs to be reattached. But the reattachment of agentmain will also cause repeated bytecode insertion problems, but there are also Hotswap and DCE VM modes to avoid them.

You can also see some differences between them through the following tests.

premain loading mode

The preparation steps of premain are as follows:

1. Write the premain function, including one of the following two methods:

java
   public static void premain(String agentArgs, Instrumentation inst);
   public static void premain(String agentArgs);

If both methods are implemented, the priority with the Instrumentation parameter is higher and will be called first. agentArgs is the program parameter obtained from the premain function, which is passed in through the command line parameter

2. Define a MANIFEST.MF file, which must contain the premain class option. Generally, can redefine classes and can retransform classes options will also be added

3. jar the premain class and MANIFEST.MF file

4. Use the parameter - javaagent: jar package path to start the agent

The main loading process is as follows:

1. Create and initialize JPLISAgent
2. Parameters of manifest.mf file, and set some contents in JPLISAgent according to these parameters
3. Listen to VMInit event, and do the following after JVM initialization:
(1) create InstrumentationImpl object;
(2) listen to the ClassFileLoadHook event;
(3) call loadClassAndCallPremain method of InstrumentationImpl, in which the premain method of premain class specified in MANIFEST.MF of javaagent will be called

Here is a simple example (tested in JDK 1.8.0 µ):

PreMainAgent

package com.longofo;

import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.lang.instrument.Instrumentation;
import java.security.ProtectionDomain;

public class PreMainAgent {
    static {
        System.out.println("PreMainAgent class static block run...");
    }

    public static void premain(String agentArgs, Instrumentation inst) {
        System.out.println("PreMainAgent agentArgs : " + agentArgs);
        Class<?>[] cLasses = inst.getAllLoadedClasses();

        for (Class<?> cls : cLasses) {
            System.out.println("PreMainAgent get loaded class:" + cls.getName());
        }

        inst.addTransformer(new DefineTransformer(), true);
    }

    static class DefineTransformer implements ClassFileTransformer {

        @Override
        public byte[] transform(ClassLoader loader, String className, Class<?> classBeingRedefined, ProtectionDomain protectionDomain, byte[] classfileBuffer) throws IllegalClassFormatException {
            System.out.println("PreMainAgent transform Class:" + className);
            return classfileBuffer;
        }
    }
}

MANIFEST.MF: 

Manifest-Version: 1.0
Can-Redefine-Classes: true
Can-Retransform-Classes: true
Premain-Class: com.longofo.PreMainAgent

Testmain

package com.longofo;

public class TestMain {

    static {
        System.out.println("TestMain static block run...");
    }

    public static void main(String[] args) {
        System.out.println("TestMain main start...");
        try {
            for (int i = 0; i < 100; i++) {
                Thread.sleep(3000);
                System.out.println("TestMain main running...");
            }
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        System.out.println("TestMain main end...");
    }
}

Package the premanagent as a Jar package (you can package directly with idea, or you can package with maven plug-in). In idea, you can start it as follows:

The command line can be started with the form Java - javaagent: premanagent.jar path - jar TestMain/TestMain.jar

The results are as follows:

PreMainAgent class static block run...
PreMainAgent agentArgs : null
PreMainAgent get loaded class:com.longofo.PreMainAgent
PreMainAgent get loaded class:sun.reflect.DelegatingMethodAccessorImpl
PreMainAgent get loaded class:sun.reflect.NativeMethodAccessorImpl
PreMainAgent get loaded class:sun.instrument.InstrumentationImpl$1
PreMainAgent get loaded class:[Ljava.lang.reflect.Method;
...
...
PreMainAgent transform Class:sun/nio/cs/ThreadLocalCoders
PreMainAgent transform Class:sun/nio/cs/ThreadLocalCoders$1
PreMainAgent transform Class:sun/nio/cs/ThreadLocalCoders$Cache
PreMainAgent transform Class:sun/nio/cs/ThreadLocalCoders$2
...
...
PreMainAgent transform Class:java/lang/Class$MethodArray
PreMainAgent transform Class:java/net/DualStackPlainSocketImpl
PreMainAgent transform Class:java/lang/Void
TestMain static block run...
TestMain main start...
PreMainAgent transform Class:java/net/Inet6Address
PreMainAgent transform Class:java/net/Inet6Address$Inet6AddressHolder
PreMainAgent transform Class:java/net/SocksSocketImpl$3
...
...
PreMainAgent transform Class:java/util/LinkedHashMap$LinkedKeySet
PreMainAgent transform Class:sun/util/locale/provider/LocaleResources$ResourceReference
TestMain main running...
TestMain main running...
...
...
TestMain main running...
TestMain main end...
PreMainAgent transform Class:java/lang/Shutdown
PreMainAgent transform Class:java/lang/Shutdown$Lock

You can see that some necessary classes have been loaded before premanagent, that is, the premanagent get loaded class: xxx section. These classes have not been transformed. Then there are some classes that have been transformed before main. After main is started, there are also classes that have been transformed after main is finished. This can be compared with the result of agentmain.

agentmain loading mode

The steps of agentmain are as follows:

1. Write the agentmain function, including one of the following two methods:

 public static void agentmain(String agentArgs, Instrumentation inst);
   public static void agentmain(String agentArgs);

If both methods are implemented, the priority with the Instrumentation parameter is higher and will be called first. agentArgs is the program parameter obtained from the premain function, which is passed in through the command line parameter

2. Define a MANIFEST.MF file, which must contain the agent class option. Generally, can redefine classes and can retransform classes options will also be added

3. jar the classes of agentmain and the MANIFEST.MF file

4. The Agent is loaded directly through the attach tool. The program to execute the attach and the program to be proxied can be two completely different programs:

  // List all VM instances
   List<VirtualMachineDescriptor> list = VirtualMachine.list();
   // attach target VM
   VirtualMachine.attach(descriptor.id());
   // Target VM load Agent
   VirtualMachine#loadAgent("agent Jar path", "command parameter");

The loading process of agentmain is similar:

1. Create and initialize JPLISAgent
2. Analyze the parameters in MANIFEST.MF, and set some contents in JPLISAgent according to these parameters
3. Listen to VMInit event, and do the following after JVM initialization:
(1) create InstrumentationImpl object;
(2) listen to the ClassFileLoadHook event;
(3) call the loadClassAndCallAgentmain method of InstrumentationImpl, in which the agentmain method of the agent class specified in MANIFEST.MF of javaagent will be called.

Here is a simple example (tested on JDK 1.8.0 µ):

SufMainAgent

package com.longofo;

import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.lang.instrument.Instrumentation;
import java.security.ProtectionDomain;

public class SufMainAgent {
    static {
        System.out.println("SufMainAgent static block run...");
    }

    public static void agentmain(String agentArgs, Instrumentation instrumentation) {
        System.out.println("SufMainAgent agentArgs: " + agentArgs);

        Class<?>[] classes = instrumentation.getAllLoadedClasses();
        for (Class<?> cls : classes) {
            System.out.println("SufMainAgent get loaded class: " + cls.getName());
        }

        instrumentation.addTransformer(new DefineTransformer(), true);
    }

    static class DefineTransformer implements ClassFileTransformer {

        @Override
        public byte[] transform(ClassLoader loader, String className, Class<?> classBeingRedefined, ProtectionDomain protectionDomain, byte[] classfileBuffer) throws IllegalClassFormatException {
            System.out.println("SufMainAgent transform Class:" + className);
            return classfileBuffer;
        }
    }
}

MANIFEST.MF

Manifest-Version: 1.0
Can-Redefine-Classes: true
Can-Retransform-Classes: true
Agent-Class: com.longofo.SufMainAgent

TestSufMainAgent

package com.longofo;

import com.sun.tools.attach.*;

import java.io.IOException;
import java.util.List;

public class TestSufMainAgent {
    public static void main(String[] args) throws IOException, AgentLoadException, AgentInitializationException, AttachNotSupportedException {
        //Get all running virtual machines in the current system
        System.out.println("TestSufMainAgent start...");
        String option = args[0];
        List<VirtualMachineDescriptor> list = VirtualMachine.list();
        if (option.equals("list")) {
            for (VirtualMachineDescriptor vmd : list) {
                //If the name of the virtual machine is xxx, the virtual machine is the target virtual machine, get the pid of the virtual machine
                //Then load agent.jar and send it to the virtual machine
                System.out.println(vmd.displayName());
            }
        } else if (option.equals("attach")) {
            String jProcessName = args[1];
            String agentPath = args[2];
            for (VirtualMachineDescriptor vmd : list) {
                if (vmd.displayName().equals(jProcessName)) {
                    VirtualMachine virtualMachine = VirtualMachine.attach(vmd.id());
                    virtualMachine.loadAgent(agentPath);
                }
            }
        }
    }
}
Testmain

package com.longofo;

public class TestMain {

    static {
        System.out.println("TestMain static block run...");
    }

    public static void main(String[] args) {
        System.out.println("TestMain main start...");
        try {
            for (int i = 0; i < 100; i++) {
                Thread.sleep(3000);
                System.out.println("TestMain main running...");
            }
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        System.out.println("TestMain main end...");
    }
}

Package the sumainagent and testsumainagent as Jar packages (you can package directly with idea or with maven plug-in). First start Testmain, and then list the current Java programs:

Attach sumainagent to Testmain:

The results in Testmain are as follows:

TestMain static block run...
TestMain main start...
TestMain main running...
TestMain main running...
TestMain main running...
...
...
SufMainAgent static block run...
SufMainAgent agentArgs: null
SufMainAgent get loaded class: com.longofo.SufMainAgent
SufMainAgent get loaded class: com.longofo.TestMain
SufMainAgent get loaded class: com.intellij.rt.execution.application.AppMainV2$1
SufMainAgent get loaded class: com.intellij.rt.execution.application.AppMainV2
...
...
SufMainAgent get loaded class: java.lang.Throwable
SufMainAgent get loaded class: java.lang.System
...
...
TestMain main running...
TestMain main running...
...
...
TestMain main running...
TestMain main running...
TestMain main end...
SufMainAgent transform Class:java/lang/Shutdown
SufMainAgent transform Class:java/lang/Shutdown$Lock

Compared with the previous premain, we can see that the number of classes directly getloadedclasses in agentmain is more than that in premain, and the class of premain getloadedclasses + premain transform is basically the same as that of agentmain getloadedclasses (only for this test, if there are other communications in the program, it may be different). That is to say, if a class has not been loaded before, it will be transformed through the two settings, which can be seen from the final java/lang/Shutdown.

Test whether a class of Weblogic is loaded

Here, weblogic is used for testing, and agentmain is used for agent mode (tested on jdk1.6.0_):

WeblogicSufMainAgent

import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.lang.instrument.Instrumentation;
import java.security.ProtectionDomain;

public class WeblogicSufMainAgent {
    static {
        System.out.println("SufMainAgent static block run...");
    }

    public static void agentmain(String agentArgs, Instrumentation instrumentation) {
        System.out.println("SufMainAgent agentArgs: " + agentArgs);

        Class<?>[] classes = instrumentation.getAllLoadedClasses();
        for (Class<?> cls : classes) {
            System.out.println("SufMainAgent get loaded class: " + cls.getName());
        }

        instrumentation.addTransformer(new DefineTransformer(), true);
    }

    static class DefineTransformer implements ClassFileTransformer {

        @Override
        public byte[] transform(ClassLoader loader, String className, Class<?> classBeingRedefined, ProtectionDomain protectionDomain, byte[] classfileBuffer) throws IllegalClassFormatException {
            System.out.println("SufMainAgent transform Class:" + className);
            return classfileBuffer;
        }
    }
}

WeblogicTestSufMainAgent: 

import com.sun.tools.attach.*;

import java.io.IOException;
import java.util.List;

public class WeblogicTestSufMainAgent {
    public static void main(String[] args) throws IOException, AgentLoadException, AgentInitializationException, AttachNotSupportedException {
        //Get all running virtual machines in the current system
        System.out.println("TestSufMainAgent start...");
        String option = args[0];
        List<VirtualMachineDescriptor> list = VirtualMachine.list();
        if (option.equals("list")) {
            for (VirtualMachineDescriptor vmd : list) {
                //If the name of the virtual machine is xxx, the virtual machine is the target virtual machine, get the pid of the virtual machine
                //Then load agent.jar and send it to the virtual machine
                System.out.println(vmd.displayName());
            }
        } else if (option.equals("attach")) {
            String jProcessName = args[1];
            String agentPath = args[2];
            for (VirtualMachineDescriptor vmd : list) {
                if (vmd.displayName().equals(jProcessName)) {
                    VirtualMachine virtualMachine = VirtualMachine.attach(vmd.id());
                    virtualMachine.loadAgent(agentPath);
                }
            }
        }
    }
}

List running Java applications:

attach:

Weblogic output:

If a class has not been loaded before but can be found by Weblogic during the deserialization of Weblogic t3, the corresponding class will be transform ed by the Agent. However, some classes are in some Jar packages under the Weblogic directory, but Weblogic will not load them, and some special configurations are needed for Weblogic to find and load them.

Limitations of Instrumentation

In most cases, Instrumentation uses the function of bytecode Instrumentation, which is generally the function of class retranslation, but it has the following limitations:

  • When the bytecode is modified in two ways, premain and agentmain, it is after the Class file is loaded, that is to say, the parameter of Class type must be included. You cannot redefine a Class that does not exist by bytecode file and custom Class name. What we need to pay attention to here is the redefinition mentioned above. What we just said here is that we can't redefine a Class name, and the bytecode content can still be redefined and modified, but after the bytecode content is modified, we also need to meet the requirements of the second point.
  • In fact, class transformation eventually returns to class redefinition instrumentation ා retransformclasses(), which has the following limitations:

    1. The parents of the new class and the old class must be the same;
    2. The number of interfaces implemented by the new class and the old class should be the same, and they are the same interfaces;
    3. The accessors of new and old classes must be consistent. The number of fields and field names of new and old classes should be consistent;
    4. The methods of adding or deleting new and old classes must be modified by private static/final;
    5. The modified method body can be deleted.

In practice, we may encounter more than these limitations. If you encounter them, we can solve them. If you want to redefine a new class (the class name does not exist in the loaded class), you can consider the method based on class loader isolation: create a new custom class loader to define a new class through a new bytecode, but you can only call the new class through reflection.

Summary
This paper only describes some basic concepts related to Java agent, the purpose is to know that there is this thing, and then verify a problem encountered before. Some articles written by other tycoons are also used for reference [4] & [5]
In the process of writing this article, I read some ideas of vulnerability detection, such as a kind of PHP-RASP implementation [6], and used the technology of stain tracking, hook, syntax tree analysis, etc., as well as several articles about Java RASP organized by the big guys [2] & [3]. If I want to write a RASP based vulnerability detection / utilization tool, I can also draw on these ideas
The code is put on github. If you are interested, you can test it. Pay attention to the JDK version in pom.xml file. If there is an error in switching the JDK test, remember to modify the JDK version in pom.xml.

Reference resources
1.https://docs.oracle.com/javas...
2.https://paper.seebug.org/513/...
3.https://paper.seebug.org/1041...
4.http://www.throwable.club/201...
5.https://www.cnblogs.com/ricki...
6.https://c0d3p1ut0s.github.io/...

This article is published by Seebug Paper. If you need to reprint it, please indicate the source

Posted by CRichardson on Thu, 12 Dec 2019 22:15:23 -0800

Hot Keywords

©2024 Programmer Group