Preface

This article is Translations between Protobuf and Json A follow-up is mainly to solve the problem of transformation between different ProtoBean s and POJOs in system hierarchy. Transformed Proobuf and Projo have the same name and type attributes (when Proto attribute type is Message, corresponding to Pojo's Object type attributes, they should have the same attributes).

Basic Thoughts on Transformation

The protobuf file used in the test is as follows:

StudentProto.proto

syntax = "proto3";

option java_package = "io.gitlab.donespeak.javatool.toolprotobuf.proto";

message Student {
    string name = 1;
    int32 age = 2;
    Student deskmate = 3;
}

DataTypeProto.proto

syntax = "proto3";

import "google/protobuf/any.proto";

option java_package = "io.gitlab.donespeak.javatool.toolprotobuf.proto";
package data.proto;

enum Color {
    NONE = 0;
    RED = 1;
    GREEN = 2;
    BLUE = 3;
}

message BaseData {
    double double_val = 1;
    float float_val = 2;
    int32 int32_val = 3;
    int64 int64_val = 4;
    uint32 uint32_val = 5;
    uint64 uint64_val = 6;
    sint32 sint32_val = 7;
    sint64 sint64_val = 8;
    fixed32 fixed32_val = 9;
    fixed64 fixed64_val = 10;
    sfixed32 sfixed32_val = 11;
    sfixed64 sfixed64_val = 12;
    bool bool_val = 13;
    string string_val = 14;
    bytes bytes_val = 15;

    Color enum_val = 16;

    repeated string re_str_val = 17;
    map<string, BaseData> map_val = 18;
}

Direct transformation

The attributes of the same name and the same category are directly copied by mapping method. This method is mainly realized by reflection mechanism.

[ A ] <--> [ B ]

The way of direct transformation can only be achieved through protobuf's reflection mechanism, which will be more difficult and is being tried. Another way is to try to copy attributes using Apache Common Bean Utils or Spring Bean Utils. Spring Bean Utils is used here for design, and the code is as follows:

public class ProtoPojoUtilWithBeanUtils {

    public static void toProto(Message.Builder destProtoBuilder, Object srcPojo) throws ProtoPojoConversionException {
        // Message s are immutable classes, no setter method, can only be setter through Builder
        try {
            BeanUtils.copyProperties(srcPojo, destProtoBuilder);
        } catch (Exception e) {
            throw new ProtoPojoConversionException(e.getMessage(), e);
        }
    }

    public static <PojoType> PojoType toPojo(Class<PojoType> destPojoKlass, Message srcMessage)
        throws ProtoPojoConversionException {
        try {
            PojoType destPojo = destPojoKlass.newInstance();
            BeanUtils.copyProperties(srcMessage, destPojo);
            return destPojo;
        } catch (Exception e) {
            throw new ProtoPojoConversionException(e.getMessage(), e);
        }
    }
}

This implementation is bound to be problematic for the following reasons

• ProtoBean does not allow null values, while Pojo does allow null values. From Pojo copy to Proto, there will inevitably be non-null anomalies.
• BeanUtils are matched by method name and getter/setter type, and nested types cannot be copied properly because of type mismatch
• Java generated by the Projo attribute of Map and List will add Map and List after the attribute name respectively. If you want to be able to copy, you need to specify Projo's attribute name according to this rule.
• Enum type mismatch can not be copied, if you want to be able to copy, you can try to use ProtoBean's Enum domain get**Value() method and name the Pojo attribute name accordingly.

In general, BeanUtils is not suitable for this task. Only later consideration can be given to using Protobuf's reflection for implementation. This is not the focus of this article. Let's move on to another implementation.

Indirect Conversion (Currency Conversion)

Conversion through a unified medium is like currency. For example, when renminbi is converted to yen, banks will first convert renminbi to dollars, then to yen, and vice versa.

[ A ] <--> [ C ] <--> [ B ]

Specifically, in the implementation, we can use platform-independent language-independent Json as intermediate medium C, first convert ProtoBean A into Json C, and then Json C into ProtoBean B object. This method will be explained in detail below.

code implementation

There are two tools for converting ProtoBean into Json, one is com.google.protobuf/protobuf-java-util, the other is com.googlecode.protobuf-java-format/protobuf-java-format. The performance and effect of the two tools need to be compared. We use com.google.protobuf/protobuf-java-util because JsonFormat in protobuf-java-format formats Map into a list of objects {"key":","value":"} while JsonFormat in protobuf-java-util can be serialized into an ideal key-value structure and conforms to the format of Pojo to json.

<!-- https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java-util -->
<dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java-util</artifactId>
    <version>3.7.1</version>
</dependency>

<!-- https://mvnrepository.com/artifact/com.googlecode.protobuf-java-format/protobuf-java-format -->
<dependency>
    <groupId>com.googlecode.protobuf-java-format</groupId>
    <artifactId>protobuf-java-format</artifactId>
    <version>1.4</version>
</dependency>

For the transformation of Pojo and Json, Gson is used here, because both Proobuf and Pojo are from Google's home.

The complete implementation is as follows: ProtoBeanUtils.jave

import java.io.IOException;

import com.google.gson.Gson;
import com.google.protobuf.Message;
import com.google.protobuf.util.JsonFormat;

/**
 * The getter and setter fields of the two transformed objects need to match perfectly.
 * In addition, for enum and bytes in ProtoBean, the following rules are followed when converting to POJO:
 * <ol>
 *     <li>enum -> String</li>
 *     <li>bytes -> base64 String</li>
 * </ol>
 * @author Yang Guanrong
 * @date 2019/08/18 23:44
 */
public class ProtoBeanUtils {

    /**
     * Converting ProtoBean objects into POJO objects
     *
     * @param destPojoClass Class types of target POJO objects
     * @param sourceMessage ProtoBean object instance with data
     * @param <PojoType> Class type paradigm for target POJO objects
     * @return
     * @throws IOException
     */
    public static <PojoType> PojoType toPojoBean(Class<PojoType> destPojoClass, Message sourceMessage)
        throws IOException {
        if (destPojoClass == null) {
            throw new IllegalArgumentException
                ("No destination pojo class specified");
        }
        if (sourceMessage == null) {
            throw new IllegalArgumentException("No source message specified");
        }
        String json = JsonFormat.printer().print(sourceMessage);
        return new Gson().fromJson(json, destPojoClass);
    }

    /**
     * Converting POJO objects into ProtoBean objects
     *
     * @param destBuilder Builder class for the target Message object
     * @param sourcePojoBean POJO objects with data
     * @return
     * @throws IOException
     */
    public static void toProtoBean(Message.Builder destBuilder, Object sourcePojoBean) throws IOException {
        if (destBuilder == null) {
            throw new IllegalArgumentException
                ("No destination message builder specified");
        }
        if (sourcePojoBean == null) {
            throw new IllegalArgumentException("No source pojo specified");
        }
        String json = new Gson().toJson(sourcePojoBean);
        JsonFormat.parser().merge(json, destBuilder);
    }
}

and Translations between Protobuf and Json Similarly, the above implementation cannot process Any-type data. You need to add your own TypeRegirstry to make the transformation.

A TypeRegistry is used to resolve Any messages in the JSON conversion. You must provide a TypeRegistry containing all message types used in Any message fields, or the JSON conversion will fail because data in Any message fields is unrecognizable. You don't need to supply a TypeRegistry if you don't use Any message fields.

Class JsonFormat.TypeRegistry @JavaDoc

The way to add TypeRegistry is as follows:

// https://codeburst.io/protocol-buffers-part-3-json-format-e1ca0af27774
final var typeRegistry = JsonFormat.TypeRegistry.newBuilder()
        .add(ProvisionVmCommand.getDescriptor())
        .build();
final var jsonParser = JsonFormat.parser()
        .usingTypeRegistry(typeRegistry);

final var envelopeBuilder = VmCommandEnvelope.newBuilder();
jsonParser.merge(json, envelopeBuilder);

test

A Pojo class BaseDataPojo.java that matches a Proo file

import lombok.*;

import java.util.List;
import java.util.Map;

/**
 * @author Yang Guanrong
 * @date 2019/09/03 20:46
 */
@Getter
@Setter
@ToString
@NoArgsConstructor
@AllArgsConstructor(access = AccessLevel.PRIVATE)
@Builder
public class BaseDataPojo {
    private double doubleVal;
    private float floatVal;
    private int int32Val;
    private long int64Val;
    private int uint32Val;
    private long uint64Val;
    private int sint32Val;
    private long sint64Val;
    private int fixed32Val;
    private long fixed64Val;
    private int sfixed32Val;
    private long sfixed64Val;
    private boolean boolVal;
    private String stringVal;
    private String bytesVal;

    private String enumVal;

    private List<String> reStrVal;
    private Map<String, BaseDataPojo> mapVal;
}

Test class ProtoBeanUtilsTest.java

package io.gitlab.donespeak.javatool.toolprotobuf.withjsonformat;

import static org.junit.Assert.*;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;

import org.junit.Test;

import com.google.common.io.BaseEncoding;
import com.google.protobuf.ByteString;

import io.gitlab.donespeak.javatool.toolprotobuf.bean.BaseDataPojo;
import io.gitlab.donespeak.javatool.toolprotobuf.proto.DataTypeProto;

/**
 * @author Yang Guanrong
 * @date 2019/09/04 14:05
 */
public class ProtoBeanUtilsTest {

    private DataTypeProto.BaseData getBaseDataProto() {
        DataTypeProto.BaseData baseData = DataTypeProto.BaseData.newBuilder()
            .setDoubleVal(100.123D)
            .setFloatVal(12.3F)
            .setInt32Val(32)
            .setInt64Val(64)
            .setUint32Val(132)
            .setUint64Val(164)
            .setSint32Val(232)
            .setSint64Val(264)
            .setFixed32Val(332)
            .setFixed64Val(364)
            .setSfixed32Val(432)
            .setSfixed64Val(464)
            .setBoolVal(true)
            .setStringVal("ssss..tring")
            .setBytesVal(ByteString.copyFromUtf8("itsbytes"))
            .setEnumVal(DataTypeProto.Color.BLUE)
            .addReStrVal("re-item-0")
            .addReIntVal(33)
            .putMapVal("m-key", DataTypeProto.BaseData.newBuilder()
                .setStringVal("base-data")
                .build())
            .build();

        return baseData;
    }

    public BaseDataPojo getBaseDataPojo() {
        Map<String, BaseDataPojo> map = new HashMap<>();
        map.put("m-key", BaseDataPojo.builder().stringVal("base-data").build());

        BaseDataPojo baseDataPojo = BaseDataPojo.builder()
            .doubleVal(100.123D)
            .floatVal(12.3F)
            .int32Val(32)
            .int64Val(64)
            .uint32Val(132)
            .uint64Val(164)
            .sint32Val(232)
            .sint64Val(264)
            .fixed32Val(332)
            .fixed64Val(364)
            .sfixed32Val(432)
            .sfixed64Val(464)
            .boolVal(true)
            .stringVal("ssss..tring")
            .bytesVal("itsbytes")
            .enumVal(DataTypeProto.Color.BLUE.toString())
            .reStrVal(Arrays.asList("re-item-0"))
            .reIntVal(new int[]{33})
            .mapVal(map)
            .build();

        return baseDataPojo;
    }

    @Test
    public void toPojoBean() throws IOException {
        DataTypeProto.BaseData baseDataProto = getBaseDataProto();
        BaseDataPojo baseDataPojo = ProtoBeanUtils.toPojoBean(BaseDataPojo.class, baseDataProto);

        // System.out.println(new GsonBuilder().setPrettyPrinting().create().toJson(baseDataPojo));

        asserEqualsVerify(baseDataPojo, baseDataProto);
    }

    @Test
    public void toProtoBean() throws IOException {
        BaseDataPojo baseDataPojo = getBaseDataPojo();

        DataTypeProto.BaseData.Builder builder = DataTypeProto.BaseData.newBuilder();
        ProtoBeanUtils.toProtoBean(builder, baseDataPojo);
        DataTypeProto.BaseData baseDataProto = builder.build();

        // System.out.println(new GsonBuilder().setPrettyPrinting().create().toJson(baseDataPojo));
        // Unable to use Gson to transform Messages (which contain nested structure and nested structure in Message), stack overflow
        // Because Protobuf has no null value
        // System.out.println(JsonFormat.printer().print(baseDataProto));

        asserEqualsVerify(baseDataPojo, baseDataProto);
    }

    private void asserEqualsVerify(BaseDataPojo baseDataPojo, DataTypeProto.BaseData baseDataProto) {
        assertTrue((baseDataPojo == null) == (!baseDataProto.isInitialized()));
        if(baseDataPojo == null) {
            return;
        }
        assertEquals(baseDataPojo.getDoubleVal(), baseDataProto.getDoubleVal(), 0.0000001D);
        assertEquals(baseDataPojo.getFloatVal(), baseDataProto.getFloatVal(), 0.00000001D);
        assertEquals(baseDataPojo.getInt32Val(), baseDataProto.getInt32Val());
        assertEquals(baseDataPojo.getInt64Val(), baseDataProto.getInt64Val());
        assertEquals(baseDataPojo.getUint32Val(), baseDataProto.getUint32Val());
        assertEquals(baseDataPojo.getUint64Val(), baseDataProto.getUint64Val());
        assertEquals(baseDataPojo.getSint32Val(), baseDataProto.getSint32Val());
        assertEquals(baseDataPojo.getSint64Val(), baseDataProto.getSint64Val());
        assertEquals(baseDataPojo.getFixed32Val(), baseDataProto.getFixed32Val());
        assertEquals(baseDataPojo.getInt64Val(), baseDataProto.getInt64Val());
        assertEquals(baseDataPojo.isBoolVal(), baseDataProto.getBoolVal());
        assertEquals(baseDataPojo.isBoolVal(), baseDataProto.getBoolVal());
        assertEquals(baseDataPojo.getStringVal(), baseDataProto.getStringVal());
        // ByteString to base64 Strings
        if(baseDataPojo.getBytesVal() == null) {
            // The default value is ""
            assertTrue(baseDataProto.getBytesVal().isEmpty());
        } else {
            assertEquals(baseDataPojo.getBytesVal(), BaseEncoding.base64().encode(baseDataProto.getBytesVal().toByteArray()));
        }
        // Enum to String
        if(baseDataPojo.getEnumVal() == null) {
            // The default value is 0
            assertEquals(DataTypeProto.Color.forNumber(0), baseDataProto.getEnumVal());
        } else {
            assertEquals(baseDataPojo.getEnumVal(), baseDataProto.getEnumVal().toString());
        }
        if(baseDataPojo.getReStrVal() == null) {
            // Default empty list
            assertEquals(0, baseDataProto.getReStrValList().size());
        } else {
            assertEquals(baseDataPojo.getReStrVal().size(), baseDataProto.getReStrValList().size());
            for(int i = 0; i < baseDataPojo.getReStrVal().size(); i ++) {
                assertEquals(baseDataPojo.getReStrVal().get(i), baseDataProto.getReStrValList().get(i));
            }
        }
        if(baseDataPojo.getReIntVal() == null) {
            // Default empty list
            assertEquals(0, baseDataProto.getReIntValList().size());
        } else {
            assertEquals(baseDataPojo.getReIntVal().length, baseDataProto.getReIntValList().size());
            for(int i = 0; i < baseDataPojo.getReIntVal().length; i ++) {
                int v1 = baseDataPojo.getReIntVal()[i];
                int v2 = baseDataProto.getReIntValList().get(i);
                assertEquals(v1, v2);
            }
        }

        if(baseDataPojo.getMapVal() == null) {
            // Default to empty collection
            assertEquals(0, baseDataProto.getMapValMap().size());
        } else {
            assertEquals(baseDataPojo.getMapVal().size(), baseDataProto.getMapValMap().size());
            for(Map.Entry<String, DataTypeProto.BaseData> entry: baseDataProto.getMapValMap().entrySet()) {
                asserEqualsVerify(baseDataPojo.getMapVal().get(entry.getKey()), entry.getValue());
            }
        }
    }

    @Test
    public void testDefaultValue() {
        DataTypeProto.BaseData baseData = DataTypeProto.BaseData.newBuilder()
            .setInt32Val(0)
            .setStringVal("")
            .addAllReStrVal(new ArrayList<>())
            .setBoolVal(false)
            .setDoubleVal(3.14D)
            .build();
        // Default values will not be output
        // double_val: 3.14
        System.out.println(baseData);
    }
}

The above tests can be completed and passed, especially the default values of class type attributes. There is no null value in Protobuf, so the default value for class type attributes will not be null. However, when mapped to Pojo, the default value of ProtoBean is converted to the default value of Pojo, that is, the default value of data type in Java.

Default Value List

This list is just a simple enumeration. If more detailed information is needed, it is recommended to see the official document of protobuf. Or there's an ingenious way to create a ProtoBean with all data types, such as DataTypeProto.BaseData, and then look at the parametric constructor in that class to see what the default value is.

...
private static final DataTypeProto.BaseData DEFAULT_INSTANCE;
static {
    DEFAULT_INSTANCE = new DataTypeProto.BaseData();
}
private BaseData() {
    stringVal_ = "";
    bytesVal_ = com.google.protobuf.ByteString.EMPTY;
    enumVal_ = 0;
    reStrVal_ = com.google.protobuf.LazyStringArrayList.EMPTY;
    reIntVal_ = emptyIntList();
}
public static iDataTypeProto.BaseData getDefaultInstance() {
    return DEFAULT_INSTANCE;
}
...

In particular, protobuf has no null value, can't set null value, and can't get null value.

For the Java data types supported by Protobuf, see: com. google. protobuf. Descriptors. Field Descriptor. JavaType

Reference and Recommended Reading

• Intertransformation between Protobuf and Json @DoneSpeak
• Protocol Buffers @Google Developers
• com.google.protobuf/protobuf-java-util @Github
• com.googlecode.protobuf-java-format/protobuf-java-format @Github
• Protocol Buffers, Part 3 — JSON Format
• Converting Protocol Buffers data to Json and back with Gson Type Adapters
• Any source @Github
• Any official document @Office