Today, when testing phoenix's secondary index function, the following exception occurred when the index was created to write data:
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: Failed to build index for unexpected reason! at org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:206) at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:351) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1010) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$28.call(RegionCoprocessorHost.java:1007) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1007) at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.prepareMiniBatchOperations(HRegion.java:3487) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3896) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3854) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3785) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:908) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:836) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:799) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2551) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Caused by: java.lang.VerifyError: class org.apache.phoenix.hbase.index.covered.data.IndexMemStore$1 overrides final method compare.(Lorg/apache/hadoop/hbase/Cell;Lorg/apache/hadoop/hbase/Cell;)I at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.phoenix.hbase.index.covered.data.IndexMemStore.<init>(IndexMemStore.java:82) at org.apache.phoenix.hbase.index.covered.LocalTableState.<init>(LocalTableState.java:57) at org.apache.phoenix.hbase.index.covered.NonTxIndexBuilder.getIndexUpdate(NonTxIndexBuilder.java:52) at org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:90) at org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:503) at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:348)
It can be seen from the exception information that VerifyError was encountered when loading the class when the IndexMemStore class was initialized. Specifically, it is trying to override a method marked final. First look for the code of this class:
public class IndexMemStore implements KeyValueStore { private static final Log LOG = LogFactory.getLog(IndexMemStore.class); private IndexKeyValueSkipListSet kvset; private CellComparator comparator; public IndexMemStore() { this(new CellComparatorImpl(){ @Override public int compare(Cell a, Cell b) { return super.compare(a, b, true); } }); }
It covers the compare method of CellComparatorImpl. In hbase common module, we use phoenix version 5.0.0 and hbase version 2.0.0. Find the code:
@Override public int compare(Cell a, Cell b) { return compare(a, b, false); }
It can be seen that this method is not final. We are using CDH6.0.1, and the corresponding hbase is also version 2.0.0. It is different. Find the code:
@Override public final int compare(final Cell a, final Cell b) { return compare(a, b, false); }
Actually, there is a final modifier, so you can check the modification history of this class in the community. It is found that the above changes were made in 2.0.1, corresponding to jira: HBASE-20620 , you can see the change record:
@@ -57,21 +64,17 @@ public class CellComparatorImpl implements CellComparator { public static final CellComparatorImpl META_COMPARATOR = new MetaCellComparator(); @Override - public int compare(Cell a, Cell b) { + public final int compare(final Cell a, final Cell b) { return compare(a, b, false); }
It seems that CDH6.0.1 contains some patch es after HBase 2.0.0;
The solution is to modify hbase, remove the final modifier, and then package, stop the service, change the package, start the service, and try again;