brief introduction
Nuwa is a popular open source implementation of Android hotpatch scheme, which is characterized by high success rate and simple implementation. Of course, there are many hot patch schemes, AndFix, Dexposed, Tinker and so on. The reason why we need to analyze Nuwa is that it represents a hot patch idea, through which we can see a lot of knowledge in this area, including further plug-in.
Nuwa working principle
The implementation of Nuwa is divided into Gradle plug-in and SDK. The plug-in part is responsible for compiling patch packs, and the SDK part is responsible for patching specific patches. Generally speaking, it seems that there are only two sentences. It is still difficult to realize them. Before the plug-in source code parsing, let's analyze the working principles of these two parts in detail, so as to have a technical understanding of Nuwa.
To generate patches, you first need to know which classes you have changed, such as we released version 2.8.1, and then changed classes on the code of 2.8.1: A, B and C, so these three classes should be combined into a patch package. Nuwa plugin is responsible for generating the patch package. He is a gradle plug-in. When the plug-in is applied, it first finds the task chain at the time of gradle compilation and then implements a custom task, which we call customTask. We insert customTask into the task before generating dex, then use the dependency of dexTask as the dependency of customTask, and then let dexTask depend on customTask. Why? To insert customTask into this location, we know by analyzing the compilation process that the task before dexTask compiles all classes into bytecode classes and then acts as input to dexTask. DexTask is responsible for compiling these classes into one or more DEX for subsequent generation of apk. Inserting into this location ensures that we get all classes before generating DEX so that we can analyze all classes and generate patch dex, a process called hook.
With the above hook as the foundation, we still need to do two things: 1. to all kinds of sockets, 2. to collect the changed classes and type them into dex packages.
Explanation 1: Why do we need to plug in? This involves the loading mechanism of android classes. Let's not expand on it. The simple understanding is that the replacement classes on android are not replaced by replacing classes. android will have a checking mechanism. It's not feasible to break the rules. The plugin is to bypass this checking mechanism in a flattering way, specifically by modifying byte codes, for each compiled cla. SS inserts an argument-free constructor and then lets the constructor reference a class in a separate DEX (this class is meaningless, just for cross-dex references).
Explanation 2: How to collect changed classes? In customTask, we generate hash for each class file that participates in compilation. The second time we perform this task, we compare the hash values of each class. If they are different, we think they have been modified. We collect these classes into folders, and then call the tool in build tools to generate dex.
The dex generated in Step 2 is our patch, which can be published to the server, downloaded to the user's mobile phone through some download mechanism, and then handed over to the sdk part to complete the real "patch" process.
SDK: SDK is an Android library, which needs to be typed in Apk. When the program runs properly, it calls its methods. It provides a core method: loadPatch(String path). It is responsible for loading incoming patches into memory. When the application is started, the DEX files in Apk will be loaded into memory one by one through ClassLoader, while maintaining a list sequentially, when the program runs. When you need to load a class, go to this list and look it up. Once you find it, you will use the specific class corresponding to dex. If you don't find it, you will report ClassNotFound error. The principle of loading patches is to insert our patch DEX into the beginning of the list by reflection, so that when you need to load a bug class, you will find it in the patch DEX first, so that the system will use the patch dex. Thermal repair can be achieved for the class. It should be noted that loadPatch must be invoked before the bug class is used. Once the bug class has been used, this repair will not work, only killing the process and restarting the application will take effect.
This time we will only analyze the Gradle plug-in part of the code, sdk code in the future has the opportunity to open another analysis.
Next, we start to analyze the implementation of Nuwa plugin in combination with engineering. For the sake of space, we only focus on the main process. Interested students can check the code and compare it. Project address
Project structure
code analysis
Implementing a plugin first implements the Plugin interface and rewrites the application function.
class NuwaPlugin implements Plugin<Project> {
HashSet<String> includePackage
HashSet<String> excludeClass
def debugOn
def patchList = []
def beforeDexTasks = []
private static final String NUWA_DIR = "NuwaDir"
private static final String NUWA_PATCHES = "nuwaPatches"
private static final String MAPPING_TXT = "mapping.txt"
private static final String HASH_TXT = "hash.txt"
private static final String DEBUG = "debug"
@Override
void apply(Project project) {
project.extensions.create("nuwa", NuwaExtension, project)
project.afterEvaluate {
def extension = project.extensions.findByName("nuwa") as NuwaExtension
includePackage = extension.includePackage
excludeClass = extension.excludeClass
debugOn = extension.debugOn
}
}
}apply is executed when a plug-in is declared by build.gradle, such as when an application plug-in is first declared using the build.gradle file of the plug-in module, the content of the application function in the plug-in is executed first when the build.gradle is executed.
apply plugin: 'com.android.application'
apply plugin: 'plugin.test'The application function is executed at the beginning: project.extensions.create("nuwa", "NuwaExtension", project). The purpose of this sentence is to create an extension based on the NuwaExtension class, and then you can declare properties in build.gradle according to the existing field of NuwaExtension.
class NuwaExtension {
HashSet<String> includePackage = []
HashSet<String> excludeClass = []
boolean debugOn = true
NuwaExtension(Project project) {
}
}Then you can declare in build.gradle:
includePackage = []
excludeClass = []
oldNuwaDir = "/Users/GaoGao/Demo/GradlePlugin/nuwa"
}The purpose of creating extensions is to facilitate our dynamic configuration.
Code execution is divided into two major branches: obfuscation and non-obfuscation, and we will only analyze non-obfuscation here.
def preDexTask =project.tasks.findByName("preDex${variant.name.capitalize()}")Find preDexTask, and if so, it turns on confusion. We don't have it here.
def dexTask = project.tasks.findByName("dex${variant.name.capitalize()}")Find dexTask, which is the key to task. Its superior task is responsible for compiling all classes. Its input is the class file of all classes (XXX.class).
// Create a task for patch, which is responsible for packaging class es that differ from each other into dex
def nuwaPatch = "nuwa${variant.name.capitalize()}Patch"
project.task(nuwaPatch) << {
if (patchDir) {
// Functions that are really responsible for packaging. Function implementations are analyzed below
NuwaAndroidUtils.dex(project, patchDir)
}
}
def nuwaPatchTask = project.tasks[nuwaPatch]
if(preDexTask) {
} else {
//Create a custom task that traverses all compiled classes and injects constructors into each class file. The constructor refers to a class in a separate dex, because this class is not in the current dex.
//So it prevents classes from being marked with ISPREVERIFIED
def nuwaJarBeforeDex = "nuwaJarBeforeDex${variant.name.capitalize()}"
//Create a custom task that traverses all compiled classes and injects constructors into each class file. The constructor refers to a class in a separate dex, because this class is not in the current dex.
//So it prevents classes from being marked with ISPREVERIFIED
Set<File> inputFiles = dexTask.inputs.files.files ≈
inputFiles.each { inputFile ->
// Here it gets all the compiled jar packages (more than one jar package, including all support jar packages and some dependent jar packages, as well as the jar packages typed out by the project source code,
// All in all, these jar s include all the classes in the apk.
def path = inputFile.absolutePath
if (path.endsWith(".jar")) {
// Really do class injection function, function implementation will be analyzed below.
NuwaProcessor.processJar(hashFile, inputFile, patchDir, hashMap, includePackage, excludeClass)
}
}
}
// Because project.task(nuwaJarBeforeDex) has created the task of nuwaJarBeforeDex in the previous step, we can get the real task object through the task as a system member variable.
def nuwaJarBeforeDexTask = project.tasks[nuwaJarBeforeDex]
// Make custom task dependent on dexTask dependencies
nuwaJarBeforeDexTask.dependsOn dexTask.taskDependencies.getDependencies(dexTask)
// Let dexTask depend on our custom task, which is equivalent to inserting our own task into the original task chain and doing our own thing without affecting the original process.
dexTask.dependsOn nuwaJarBeforeDexTask
// Let the task for patch depend on the task injected by class, so that we can execute the task manually in the console and type out the patch file.
nuwaPatchTask.dependsOn nuwaJarBeforeDexTask
}Well, that's the mainstream process. Here you may have a few questions about how class injection works, where to compare file differences, and where to type all changed files into patch es. Here are two key tool functions:
NuwaProcessor. ProceJar and NuwaAndroidUtils.dex. The former is responsible for class injection, while the latter is responsible for comparing and patch ing. The source code is as follows:
/**
Description of parameters:
hashFile: This compilation of all classes "class name: hash" storage file
jarFile: jar Packet, where this function is called, traverses all jar packages
patchDir: Changed files are stored uniformly in this directory
map: hash mapping of all classes compiled last time
includePackage: Additional specifies that only the classes under these packages need to be injected
excludeClass: Additionally specify classes that do not participate in injection
*/
public static processJar(File hashFile, File jarFile, File patchDir, Map map, HashSet<String> includePackage, HashSet<String> excludeClass) {
if (jarFile) {
// First, create the "same name. opt" file in the same level directory of the original jar, and type it into the opt file every time a class is injected.
// The opt file is actually a jar package. After all the classes are processed, the file suffix opt is changed to jar instead of jar.
def optJar = new File(jarFile.getParent(), jarFile.name + ".opt")
def file = new JarFile(jarFile);
Enumeration enumeration = file.entries();
// Creating an input opt file is actually a jar package
JarOutputStream jarOutputStream = new JarOutputStream(new FileOutputStream(optJar));
while (enumeration.hasMoreElements()) { // Traversing through each entry in the jar package
JarEntry jarEntry = (JarEntry) enumeration.nextElement();
String entryName = jarEntry.getName();
ZipEntry zipEntry = new ZipEntry(entryName);
InputStream inputStream = file.getInputStream(jarEntry);
jarOutputStream.putNextEntry(zipEntry);
if (shouldProcessClassInJar(entryName, includePackage, excludeClass)) { // Judge whether the class needs to be processed or not based on some rules and includePackage and excludeClass
def bytes = referHackWhenInit(inputStream); // Get the input stream of this class and call this function to complete bytecode injection
jarOutputStream.write(bytes); // Write the injected bytecode into the opt file
def hash = DigestUtils.shaHex(bytes) // Generate file hash
hashFile.append(NuwaMapUtils.format(entryName, hash)) take hash Values are written as key-value pairs to hash Documentation for next comparison
if (NuwaMapUtils.notSame(map, entryName, hash)) { // If this class is different from the hash generated last time in map, it is considered modified and copied to the folder that needs to be packaged eventually.
NuwaFileUtils.copyBytesToFile(bytes, NuwaFileUtils.touchFile(patchDir, entryName))
}
} else {
jarOutputStream.write(IOUtils.toByteArray(inputStream)); // If this class is not processed, it is written directly into the opt file
}
jarOutputStream.closeEntry();
}
jarOutputStream.close();
file.close();
if (jarFile.exists()) {
jarFile.delete()
}
optJar.renameTo(jarFile)
}
}/**
asm framework is used to modify the java bytecode file. It is very powerful. Interested students can search it. Similar frameworks include Javassist and BCEL. The actual action is to inject a nonparametric constructor into the class. The constructor refers to the "jiajixin/nuwa/Hack" class, which is in another dex. This DEX needs app. The lication entry is loaded.
This ensures that all classes have been pinched into memory before using this class, in order to prevent classes from being marked with ISPREVERIFIED, thus bypassing android's checking of classes and ensuring that patches are in effect.
*/
private static byte[] referHackWhenInit(InputStream inputStream) {
ClassReader cr = new ClassReader(inputStream);
ClassWriter cw = new ClassWriter(cr, 0);
ClassVisitor cv = new ClassVisitor(Opcodes.ASM4, cw) {
@Override
public MethodVisitor visitMethod(int access, String name, String desc,
String signature, String[] exceptions) {
MethodVisitor mv = super.visitMethod(access, name, desc, signature, exceptions);
mv = new MethodVisitor(Opcodes.ASM4, mv) {
@Override
void visitInsn(int opcode) {
if ("<init>".equals(name) && opcode == Opcodes.RETURN) {
super.visitLdcInsn(Type.getType("Lcn/jiajixin/nuwa/Hack;")); // Reference to a class in another dex
super.visitInsn(opcode);
}
}
return mv;
}
};
cr.accept(cv, 0);
return cw.toByteArray();
｝/**
NuwaAndroidUtils.dex
Packing operations are performed on classes copied to the patch folder in NuwaProcessor. ProceJar, where the command line in build-tools is used.
Description of parameters:
project: Engineering objects, from plug-ins
classDir: A folder containing classes that need to be packaged
*/
public static dex(Project project, File classDir) {
if (classDir.listFiles().size()) {
def sdkDir
Properties properties = new Properties()
File localProps = project.rootProject.file("local.properties")
if (localProps.exists()) {
properties.load(localProps.newDataInputStream())
sdkDir = properties.getProperty("sdk.dir")
} else {
sdkDir = System.getenv("ANDROID_HOME")
}
if (sdkDir) {
def cmdExt = Os.isFamily(Os.FAMILY_WINDOWS) ? '.bat' : ''
def stdout = new ByteArrayOutputStream()
project.exec {
commandLine "${sdkDir}/build-tools/${project.android.buildToolsVersion}/dx${cmdExt}",
'--dex',
"--output=${new File(classDir.getParent(), PATCH_NAME).absolutePath}",
"${classDir.absolutePath}"
standardOutput = stdout
}
def error = stdout.toString().trim()
if (error) {
println "dex error:" + error
}
} else {
throw new InvalidUserDataException('$ANDROID_HOME is not defined')
}
}
}Well, when we export the package, all classes in the generated apk are automatically injected. This time, we must save the folder where the generated hash file is located so that it can be compared after the next code change.
If a bug is found online, the code is cut back to the current version, and the command is executed to pass in the folder directory where the hash file was compiled last time, and a patch package (actually an dex) is generated, which contains only the classes we need to fix.
The order is as follows:
sh gradlew clean nuwaReleasePatch -P NuwaDir=/Users/GaoGao/nuwaAfter the class is downloaded by the client, the nuwa sdk section will be responsible for patching it.
OVER