Jvm-Sandbox Source Analysis--Startup Analysis

Keywords: Java jvm Maven Permission denied

Preface

1. Work reason, use jvm-sandbox more, then do source code analysis, to know each other, personal ability is limited, if there are errors, welcome to correct.
2. About what jvm-sandbox is, how to install it, and how to move it Official Documents
3. Source analysis is based on jvm-sandbox Latest master code, tag-1.2.1.
4. The tentative plan starts with starting profiling, loading modules, refreshing modules, uninstalling modules, activating modules, etc. It covers the key jvm-sandbox processes through several articles.

start-up

attach mode start

sh sandbox/bin/sandbox.sh -p pid

A quick look at the startup script sandbox.sh

# the sandbox main function
function main() {

    check_permission

    while getopts "hp:vFfRu:a:A:d:m:I:P:ClSn:X" ARG
    do
        case ${ARG} in
            h) usage;exit;;
            p) TARGET_JVM_PID=${OPTARG};;
            v) OP_VERSION=1;;
            l) OP_MODULE_LIST=1;;
            R) OP_MODULE_RESET=1;;
            F) OP_MODULE_FORCE_FLUSH=1;;
            f) OP_MODULE_FLUSH=1;;
            u) OP_MODULE_UNLOAD=1;ARG_MODULE_UNLOAD=${OPTARG};;
            a) OP_MODULE_ACTIVE=1;ARG_MODULE_ACTIVE=${OPTARG};;
            A) OP_MODULE_FROZEN=1;ARG_MODULE_FROZEN=${OPTARG};;
            d) OP_DEBUG=1;ARG_DEBUG=${OPTARG};;
            m) OP_MODULE_DETAIL=1;ARG_MODULE_DETAIL=${OPTARG};;
            I) TARGET_SERVER_IP=${OPTARG};;
            P) TARGET_SERVER_PORT=${OPTARG};;
            C) OP_CONNECT_ONLY=1;;
            S) OP_SHUTDOWN=1;;
            n) OP_NAMESPACE=1;ARG_NAMESPACE=${OPTARG};;
            X) set -x;;
            ?) usage;exit_on_err 1;;
        esac
    done

    reset_for_env

    # reset IP
    [ -z ${TARGET_SERVER_IP} ] && TARGET_SERVER_IP="${DEFAULT_TARGET_SERVER_IP}";

    # reset PORT
    [ -z ${TARGET_SERVER_PORT} ] && TARGET_SERVER_PORT=0;

    # reset NAMESPACE
    [[ ${OP_NAMESPACE} ]] \
        && TARGET_NAMESPACE=${ARG_NAMESPACE}
    [[ -z ${TARGET_NAMESPACE} ]] \
        && TARGET_NAMESPACE=${DEFAULT_NAMESPACE}

    if [[ ${OP_CONNECT_ONLY} ]]; then
        [[ 0 -eq ${TARGET_SERVER_PORT} ]] \
            && exit_on_err 1 "server appoint PORT (-P) was missing"
        SANDBOX_SERVER_NETWORK="${TARGET_SERVER_IP};${TARGET_SERVER_PORT}"
    else
        # -p was missing
        [[ -z ${TARGET_JVM_PID} ]] \
            && exit_on_err 1 "PID (-p) was missing.";
        attach_jvm
    fi

    # -v show version
    [[ ! -z ${OP_VERSION} ]] \
        && sandbox_curl_with_exit "sandbox-info/version"

    # -l list loaded modules
    [[ ! -z ${OP_MODULE_LIST} ]] \
        && sandbox_curl_with_exit "sandbox-module-mgr/list"

    # -F force flush module
    [[ ! -z ${OP_MODULE_FORCE_FLUSH} ]] \
        && sandbox_curl_with_exit "sandbox-module-mgr/flush" "&force=true"

    # -f flush module
    [[ ! -z ${OP_MODULE_FLUSH} ]] \
        && sandbox_curl_with_exit "sandbox-module-mgr/flush" "&force=false"

    # -R reset sandbox
    [[ ! -z ${OP_MODULE_RESET} ]] \
        && sandbox_curl_with_exit "sandbox-module-mgr/reset"

    # -u unload module
    [[ ! -z ${OP_MODULE_UNLOAD} ]] \
        && sandbox_curl_with_exit "sandbox-module-mgr/unload" "&action=unload&ids=${ARG_MODULE_UNLOAD}"

    # -a active module
    [[ ! -z ${OP_MODULE_ACTIVE} ]] \
        && sandbox_curl_with_exit "sandbox-module-mgr/active" "&ids=${ARG_MODULE_ACTIVE}"

    # -A frozen module
    [[ ! -z ${OP_MODULE_FROZEN} ]] \
        && sandbox_curl_with_exit "sandbox-module-mgr/frozen" "&ids=${ARG_MODULE_FROZEN}"

    # -m module detail
    [[ ! -z ${OP_MODULE_DETAIL} ]] \
        && sandbox_curl_with_exit "sandbox-module-mgr/detail" "&id=${ARG_MODULE_DETAIL}"

    # -S shutdown
    [[ ! -z ${OP_SHUTDOWN} ]] \
        && sandbox_curl_with_exit "sandbox-control/shutdown"

    # -d debug
    if [[ ! -z ${OP_DEBUG} ]]; then
        sandbox_debug_curl "module/http/${ARG_DEBUG}"
        exit
    fi

    # default
    sandbox_curl "sandbox-info/version"
    exit

}

From the script source, we can see that when sandbox.sh is executed, the reset_for_env method is executed first
Key steps:
1. Use the default environment variable JAVA_HOME
2. Or set the sandbox environment variable through TARGET_JVM_PID lookup
3. Determine if the JVM version meets the requirements
4. If ${JAVA_HOME}/lib/tools.jar exists, add it to the end of the classpath through the -Xbootclasspath/a configuration in preparation for executing the attach_jvm method

reset_for_env()
{
    #Use the default environment variable JAVA_HOME
    # use the env JAVA_HOME for default
    [[ ! -z ${JAVA_HOME} ]] \
        && SANDBOX_JAVA_HOME="${JAVA_HOME}"

    # Or set the sandbox environment variable through TARGET_JVM_PID lookup
    # use the target JVM for SANDBOX_JAVA_HOME
    [[ -z ${SANDBOX_JAVA_HOME} ]] \
        && SANDBOX_JAVA_HOME="$(\
            ps aux\
            |grep ${TARGET_JVM_PID}\
            |grep java\
            |awk '{print $11}'\
            |xargs ls -l\
            |awk '{if($1~/^l/){print $11}else{print $9}}'\
            |sed 's/\/bin\/java//g'\
        )"

    [[ ! -x "${SANDBOX_JAVA_HOME}" ]] \
        && exit_on_err 1 "permission denied, ${SANDBOX_JAVA_HOME} is not accessible! please set JAVA_HOME"

    [[ ! -x "${SANDBOX_JAVA_HOME}/bin/java" ]] \
        && exit_on_err 1 "permission denied, ${SANDBOX_JAVA_HOME}/bin/java is not executable!"
    #Judging JVM version
    # check the jvm version, we need 6+
    local JAVA_VERSION=$("${SANDBOX_JAVA_HOME}/bin/java" -version 2>&1|awk -F '"' '/version/&&$2>"1.5"{print $2}')
    [[ -z ${JAVA_VERSION} ]] \
        && exit_on_err 1 "illegal java version: ${JAVA_VERSION}, please make sure target java process: ${TARGET_JVM_PID} run int JDK[6,11]"

    #If ${JAVA_HOME}/lib/tools.jar exists, add it to the end of the classpath through the -Xbootclasspath/a configuration in preparation for executing the attach_jvm method
    [[ -f "${SANDBOX_JAVA_HOME}"/lib/tools.jar ]] \
        && SANDBOX_JVM_OPS="${SANDBOX_JVM_OPS} -Xbootclasspath/a:${SANDBOX_JAVA_HOME}/lib/tools.jar"

}

Then execute the attach_jvm method
Key steps:
Launch sandbox-core.jar with java-jar command and pass parameter 1. TARGET_JVM_PID 2. sandbox-agent.jar 3. Launch data information to use

# attach sandbox to target JVM
# return : attach jvm local info
function attach_jvm() {

    # got an token
    local token=`date |head|cksum|sed 's/ //g'`

    # attach target jvm
    # Launch sandbox-core.jar with java-jar command and pass parameter 1. TARGET_JVM_PID 2. sandbox-agent.jar 3. Launch data information to use
    "${SANDBOX_JAVA_HOME}/bin/java" \
        ${SANDBOX_JVM_OPS} \
        -jar ${SANDBOX_LIB_DIR}/sandbox-core.jar \
        ${TARGET_JVM_PID} \
        "${SANDBOX_LIB_DIR}/sandbox-agent.jar" \
        "home=${SANDBOX_HOME_DIR};token=${token};server.ip=${TARGET_SERVER_IP};server.port=${TARGET_SERVER_PORT};namespace=${TARGET_NAMESPACE}" \
    || exit_on_err 1 "attach JVM ${TARGET_JVM_PID} fail."

    # get network from attach result
    SANDBOX_SERVER_NETWORK=$(grep ${token} ${SANDBOX_TOKEN_FILE}|grep ${TARGET_NAMESPACE}|tail -1|awk -F ";" '{print $3";"$4}');
    [[ -z ${SANDBOX_SERVER_NETWORK} ]]  \
        && exit_on_err 1 "attach JVM ${TARGET_JVM_PID} fail, attach lose response."

}

Now we're in the code analysis phase, let's look at the sandbox-core moudle

There is a plug-in configuration in the pom file as follows, and this main function is specified through mainClass, so we execute this function through the java-jar sandbox-core.jar command

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>attached</goal>
                        </goals>
                        <phase>package</phase>
                        <configuration>
                            <descriptorRefs>
                                <descriptorRef>jar-with-dependencies</descriptorRef>
                            </descriptorRefs>
                            <archive>
                                <manifest>
                                    <mainClass>com.alibaba.jvm.sandbox.core.CoreLauncher</mainClass>
                                </manifest>
                            </archive>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            

Let's look at the main method of this class CoreLauncher
Key steps:
1.attach pid
2.load sandbox-angent.jar

/**
     * Kernel Launcher
     *
     * @param args parameter
     *             [0] : PID
     *             [1] : agent.jar's value
     *             [2] : token
     */
    public static void main(String[] args) {
        try {

            // check args
            if (args.length != 3
                    || StringUtils.isBlank(args[0])
                    || StringUtils.isBlank(args[1])
                    || StringUtils.isBlank(args[2])) {
                throw new IllegalArgumentException("illegal args");
            }

            new CoreLauncher(args[0], args[1], args[2]);
        } catch (Throwable t) {
            t.printStackTrace(System.err);
            System.err.println("sandbox load jvm failed : " + getCauseMessage(t));
            System.exit(-1);
        }
    }
    
    public CoreLauncher(final String targetJvmPid,
                        final String agentJarPath,
                        final String token) throws Exception {

        // Load agent
        attachAgent(targetJvmPid, agentJarPath, token);

    }
    
    // Load Agent
    private void attachAgent(final String targetJvmPid,
                             final String agentJarPath,
                             final String cfg) throws Exception {

        VirtualMachine vmObj = null;
        try {
            //attach target pid
            vmObj = VirtualMachine.attach(targetJvmPid);
            if (vmObj != null) {
                //Loading sandbox-agent.jar through vm class
                vmObj.loadAgent(agentJarPath, cfg);
            }

        } finally {
            if (null != vmObj) {
                vmObj.detach();
            }
        }

    }

We can see that sandbox-agent.jar is loaded after attach pid
Next let's look at sandbox-agent.jar
Similar to sandbox-core.jar's pom file, the agent module also configures Premain-Class and Agent-Class parameters through the maven plug-in, both pointing to the AgentLauncher class

<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>attached</goal>
                        </goals>
                        <phase>package</phase>
                        <configuration>
                            <descriptorRefs>
                                <descriptorRef>jar-with-dependencies</descriptorRef>
                            </descriptorRefs>
                            <archive>
                                <manifestEntries>
                                    <Premain-Class>com.alibaba.jvm.sandbox.agent.AgentLauncher</Premain-Class>
                                    <Agent-Class>com.alibaba.jvm.sandbox.agent.AgentLauncher</Agent-Class>
                                    <Can-Redefine-Classes>true</Can-Redefine-Classes>
                                    <Can-Retransform-Classes>true</Can-Retransform-Classes>
                                </manifestEntries>
                            </archive>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

Next let's look at Agent Launcher
By attach pid, the agent main method in this class is called.
If you don't understand why configuring through the maven plug-in associates with a method in the specified class, you can refer to This article

 /**
     * Dynamic Loading
     *
     * @param featureString startup parameter
     *                      [namespace,token,ip,port,prop]
     * @param inst          inst
     */
    public static void agentmain(String featureString, Instrumentation inst) {
        LAUNCH_MODE = LAUNCH_MODE_ATTACH;
        final Map<String, String> featureMap = toFeatureMap(featureString);
        writeAttachResult(
                getNamespace(featureMap),
                getToken(featureMap),
                install(featureMap, inst)
        );
    }

java agent startup

Add in the application service startup script:

java -javaagent:/yourpath/sandbox/lib/sandbox-agent.jar

Starting via javaagent invokes the premain method in the AgentLauncher class

 /**
     * Start Loading
     *
     * @param featureString startup parameter
     *                      [namespace,prop]
     * @param inst          inst
     */
    public static void premain(String featureString, Instrumentation inst) {
        LAUNCH_MODE = LAUNCH_MODE_AGENT;
        install(toFeatureMap(featureString), inst);
    }

At this point, it is clear that whether it is started by attach pid or javaagent, the install method will eventually be executed.
Next let's see what this method does.

/**
     * Install jvm-sandbox in the current JVM
     *
     * @param featureMap Startup parameter configuration
     * @param inst       inst
     * @return Server IP:PORT
     */
    private static synchronized InetSocketAddress install(final Map<String, String> featureMap,
                                                          final Instrumentation inst) {

        final String namespace = getNamespace(featureMap);
        final String propertiesFilePath = getPropertiesFilePath(featureMap);
        final String coreFeatureString = toFeatureString(featureMap);

        try {

            // Inject Spy into BootstrapClassLoader
            inst.appendToBootstrapClassLoaderSearch(new JarFile(new File(
                    getSandboxSpyJarPath(getSandboxHome(featureMap))
                    // SANDBOX_SPY_JAR_PATH
            )));

            // Construct custom class loaders to minimize Sandbox's erosion of existing projects
            final ClassLoader sandboxClassLoader = loadOrDefineClassLoader(
                    namespace,
                    getSandboxCoreJarPath(getSandboxHome(featureMap))
                    // SANDBOX_CORE_JAR_PATH
            );

            // CoreConfigure Class Definition
            final Class<?> classOfConfigure = sandboxClassLoader.loadClass(CLASS_OF_CORE_CONFIGURE);

            // Deserialize to CoreConfigure class instance
            final Object objectOfCoreConfigure = classOfConfigure.getMethod("toConfigure", String.class, String.class)
                    .invoke(null, coreFeatureString, propertiesFilePath);

            // CoreServer Class Definition
            final Class<?> classOfProxyServer = sandboxClassLoader.loadClass(CLASS_OF_PROXY_CORE_SERVER);

            // Get CoreServer singletons
            final Object objectOfProxyServer = classOfProxyServer
                    .getMethod("getInstance")
                    .invoke(null);

            // CoreServer.isBind()
            final boolean isBind = (Boolean) classOfProxyServer.getMethod("isBind").invoke(objectOfProxyServer);


            // If not bound, an address needs to be bound
            if (!isBind) {
                try {
                    classOfProxyServer
                            .getMethod("bind", classOfConfigure, Instrumentation.class)
                            .invoke(objectOfProxyServer, objectOfCoreConfigure, inst);
                } catch (Throwable t) {
                    classOfProxyServer.getMethod("destroy").invoke(objectOfProxyServer);
                    throw t;
                }

            }

            // Return the address of the server binding
            return (InetSocketAddress) classOfProxyServer
                    .getMethod("getLocal")
                    .invoke(objectOfProxyServer);


        } catch (Throwable cause) {
            throw new RuntimeException("sandbox attach failed.", cause);
        }

    }

The comment on the source itself is already clear, and we'll continue to dig into the steps in subsequent articles. For now, just take a brief look at this part of the code, which actually goes back to the sandbox-core module and passes two parameters by calling the bind method of JettyCoreServer, the kernel boot configuration CoreConfiguration and Instrumentation, a jetty service is started, and subsequent command operations, such as loading, uninstalling, and so on, are performed through http requests.

private static final String CLASS_OF_PROXY_CORE_SERVER = "com.alibaba.jvm.sandbox.core.server.ProxyCoreServer";

...Omit Code...

// CoreServer Class Definition
            final Class<?> classOfProxyServer = sandboxClassLoader.loadClass(CLASS_OF_PROXY_CORE_SERVER);

            // Get CoreServer singletons
            final Object objectOfProxyServer = classOfProxyServer
                    .getMethod("getInstance")
                    .invoke(null);

            // CoreServer.isBind()
            final boolean isBind = (Boolean) classOfProxyServer.getMethod("isBind").invoke(objectOfProxyServer);


            // If not bound, an address needs to be bound
            if (!isBind) {
                try {
                    classOfProxyServer
                            .getMethod("bind", classOfConfigure, Instrumentation.class)
                            .invoke(objectOfProxyServer, objectOfCoreConfigure, inst);
                } catch (Throwable t) {
                    classOfProxyServer.getMethod("destroy").invoke(objectOfProxyServer);
                    throw t;
                }

            }
  
...Omit Code...

@Override
    public synchronized void bind(final CoreConfigure cfg, final Instrumentation inst) throws IOException {
        this.cfg = cfg;
        try {
            initializer.initProcess(new Initializer.Processor() {
                @Override
                public void process() throws Throwable {
                    //Initialize logback logging framework
                    LogbackUtils.init(
                            cfg.getNamespace(),
                            cfg.getCfgLibPath() + File.separator + "sandbox-logback.xml"
                    );
                    logger.info("initializing server. cfg={}", cfg);
                    jvmSandbox = new JvmSandbox(cfg, inst);//Create a sandbox object
                    initHttpServer();//Start httpServer
                    initJettyContextHandler();//Initialize the context of jetty
                    httpServer.start();
                }
            });

            // Initialize loading all modules
            try {
                jvmSandbox.getCoreModuleManager().reset();
            } catch (Throwable cause) {
                logger.warn("reset occur error when initializing.", cause);
            }

            final InetSocketAddress local = getLocal();
            logger.info("initialized server. actual bind to {}:{}",
                    local.getHostName(),
                    local.getPort()
            );

        } catch (Throwable cause) {

            // This throws into the target application layer, so leave an error message here
            logger.warn("initialize server failed.", cause);

            // Throw Outside to Target Application
            throw new IOException("server bind failed.", cause);
        }

        logger.info("{} bind success.", this);
    }

Posted by tomz0r on Sun, 01 Sep 2019 09:32:10 -0700