Hooking VMs is pretty cool, it is possible to look inside and see what exactly is happening in a Java application. Java provides an interface for hooking JavaVM, this interface is great if you want to develop a profiler, debugger or monitoring solution. There were two separate interfaces for profilers and debuggers, JVMPI (Java Virtual Machine Profiling Interface) and the JVMDI (Java Virtual Machine Debug Interface). Both of these interfaces are obsolete in the latest JDK and has been replaced with JVMTI (Java Virtual Machine Tool Interface).

JVMTI is an interface where you register callbacks for specific events. A library that contains these callbacks is called agent and it is written in a native language (ex. C/C++). This library will be loaded by JavaVM directly and there is no abstraction between JavaVM and library. Therefore the library will be sitting in the same process and address space as JavaVM. If something goes bad in the library it will crash whole JavaVM process.

JVMTI interface is supported on many Java implementation across different platforms. In my case I will be using Ubuntu 16.04 and OpenJDK. To start development of an agent you need to get headers. There are two options, copy headers files to your working directory or set the header location as a lookup path. On my setup the header files can be found in /usr/lib/jvm/java-8-openjdk-amd64/include/. Only three headers files are required to get started: jni.h, jni_md.h, jvmti.h

In jvmti.h header file you will find three interfaces:

1. Agent_OnLoad - Entry point in the agent library when agent is executed at the very start of application
2. Agent_OnAttach - Entry point in the agent library when agent is attached to a running Java application
3. Agent_OnUnload - Optional exit point, which will be called when agent is unloaded.

#include <string.h>
#include "jvmti.h"
#include "jni.h"

//Callback method
void VMInit(jvmtiEnv *jvmti_env, JNIEnv* jni_env, jthread thread)
{
    printf("VM Init\n");
}

//Call when agent is loaded by JavaVM
jint Agent_OnLoad(JavaVM *vm, char *options, void *reserved)
{
    printf("Agent Started\n");
    jvmtiEnv *environment;
    jvmtiEventCallbacks callbacks;
    jvmtiError error;
    jint result;

    //Get JVMTI environment
    result = (*vm)->GetEnv(vm, (void**)&environment, JVMTI_VERSION_1_2);
    if(result != 0){
        return JNI_ERR;
    }

    //Register callbacks
    (void)memset(&callbacks, 0, sizeof(callbacks));
    callbacks.VMInit = (void*)&VMInit;

    error = (*environment)->SetEventCallbacks(environment, &callbacks, (jint)sizeof(callbacks));
    if(error != JNI_OK){
        return JNI_ERR;
    }

    //Set notifications
    error = (*environment)->SetEventNotificationMode(environment, JVMTI_ENABLE,
        JVMTI_EVENT_VM_INIT, (jthread)NULL);
    if(error != JNI_OK){
        return JNI_ERR;
    }

    return JNI_OK;
}

//Call when agent is unloaded
void Agent_OnUnload(JavaVM *vm)
{
    printf("\nAgent Unloaded\n");
}

The code provides implementations for Agent_OnLoad, Agent_OnUnload and a basic callback. The Agent_Onload will be called when the library is being loaded by JavaVM. At that stage nothing else is started - no classes loaded, no objects create. At this stage you can start configuring your agent. In this example a VM Initialization callback is set and registered. This callback will be executed when VM is initialized, at this stage classes are loaded, threads started, etc. This stage is good to finish initialization of the agent. For now it will just print out message to verify that the callback is working.

It is straight forward to compile the agent

gcc -shared -o agent.so -fPIC agent.c

There are two ways to attach agent library to JavaVM:

1. -agentlib:<agent-library-name>=<options> Use this option if the agent library exists in the global system’s library directory
2. -agentpath:<path-to-agent>=<options> Use this option to provide path directly to the library

Options are optional, if they are provided, they will be available from the options parameter in Agent_OnLoad method. Provided options is just a string, you need to implement your own parser.

java -agentpath:./test.so program

The Java program is a very simple program that prints out “Program Started”. After executing the Java application with the agent attached, you should be seeing something like this.

Agent Started
VM Init
Program Started
Agent Unloaded

VMInit method does not require any capabilities, but many other callbacks or JVMTI functions will require configured capabilities before they can be used. Capabilities are used to control what features of the interface will be used by an agent. By default all capabilities are turned off in jvm environment, as having them all turned on will cause performance issues. Capabilities should be set in Agent_OnLoad method.

jvmtiCapabilities capabilities;
(void)memset(&capabilities,0, sizeof(capabilities));

capabilities.can_generate_method_entry_events       = 1;
capabilities.can_generate_method_exit_events        = 1;
capabilities.can_access_local_variables             = 1;
error = (*environment)->AddCapabilities(environment, &capabilities);

The code above will set capabilities so the agent can receive events when a method is being executed and when it is done. The last capability is required to access local variables.

After capabilities are set the agent must set references to the callback methods and enable notifications. In the agent, two new methods were are added MethodEntry and MethodExit. This new methods are referenced in jvmtiEventCallbacks struct.

callbacks.MethodEntry = (void*)&MethodEntry;
callbacks.MethodExit = (void*)&MethodExit;

error = (*jvmti_env)->SetEventNotificationMode(jvmti_env, JVMTI_ENABLE,
                   JVMTI_EVENT_METHOD_ENTRY, (jthread)NULL);
error = (*jvmti_env)->SetEventNotificationMode(jvmti_env, JVMTI_ENABLE,
                   JVMTI_EVENT_METHOD_EXIT, (jthread)NULL);

Example of MethodEntry method. The method will be called every time when any method in jvm is executed. In this case it will print information about the executed method - Name and parameters.

void MethodEntry(jvmtiEnv *jvmti_env, JNIEnv* jni_env, jthread thread,jmethodID method)
{
    jvmtiError error;
    char* name_p;
    char* signature_p;
    char* generic_p;

    error = (*jvmti_env)->GetMethodName(jvmti_env, method, &name_p, &signature_p, &generic_p);

    printf("Method Entry NAME:%s SIG:%s GEN:%s\n", name_p, signature_p, generic_p);
}

When compiling Java application use -g option. Then it will generate all debugging information, including local variables. By default, only line number and source file information is generated.