json parsing of Go: Marshal and Unmarshal

Keywords: Go JSON

brief introduction

json (JavaScript object notation) is a data exchange format, which is commonly used for front and rear data transmission. Either end converts the data into a json string, and the other end parses the string into a corresponding data structure, such as string type, strcut object, etc.

go language itself provides us with json toolkit "encoding/json".
For more usage methods, please refer to: https://studygolang.com/articles/6742

realization

Json Marshal: encodes data into json strings

Look at a simple example

type Stu struct {
    Name  string `json:"name"`
    Age   int
    HIgh  bool
    sex   string
    Class *Class `json:"class"`
}

type Class struct {
    Name  string
    Grade int
}

func main() {
    //Instantiate a data structure to generate json strings
    stu := Stu{
        Name: "Zhang San",
        Age:  18,
        HIgh: true,
        sex:  "male",
    }

    //Pointer variable
    cla := new(Class)
    cla.Name = "1 class"
    cla.Grade = 3
    stu.Class=cla

    //Error when Marshal failed= nil
    jsonStu, err := json.Marshal(stu)
    if err != nil {
        fmt.Println("generate json String error")
    }

    //jsonStu is a [] byte type, which is converted into a string type for easy viewing
    fmt.Println(string(jsonStu))
}

result:

{"name":"Zhang San","Age":18,"HIgh":true,"class":{"Name":"1 class","Grade":3}}

As can be seen from the results:

As long as it is an exportable member (variable initial capital), it can be converted to json. Because the member variable sex is not exportable, it cannot be converted to json.
If a variable is labeled with a json tag, such as json:"name" next to name, the converted json key will use the tag "name". Otherwise, take the variable name as the key, such as "Age" and "HIgh".
bool type can also be directly converted to json value. Channel, complex, and functions cannot be encoded as json strings. Of course, the circular data structure is also not good, which will cause marshal to fall into an endless loop.
Pointer variable, which is automatically converted to the value it points to when encoding, such as cla variable.
(of course, as like as two peas, the Stu struct member Class is the same as Class struct type. It is just the same pointer. It only saves the memory space faster.
Finally, emphasize that json is a pure string after it is encoded into a string.

The above member variables are all known types and can only receive the specified type. For example, the Name of string type can only assign the data of string type.
But sometimes, for the sake of generality or code simplicity, we want a type that can accept various types of data and encode it in json. This uses the interface {} type.

preface:

The interface {} type is actually an empty interface, that is, an interface without methods. Each type of go implements this interface. Therefore, any other type of data can be assigned to the interface {} type.

type Stu struct {
    Name  interface{} `json:"name"`
    Age   interface{}
    HIgh  interface{}
    sex   interface{}
    Class interface{} `json:"class"`
}

type Class struct {
    Name  string
    Grade int
}

func main() {
    //As in the previous example
    ......
}

result:

{"name":"Zhang San","Age":18,"HIgh":true,"class":{"Name":"1 class","Grade":3}}

It can be seen from the results that no matter string, int, bool or pointer type, they can be assigned to interface {} type, and they are encoded normally. The effect is the same as the previous example.

Supplement:
In the actual project, the data structure encoded into json string is often slice type. A slice of type [] StuRead is defined below

//Correct demonstration

//Method 1: declare only, do not allocate memory
var stus1 []*StuRead

//Method 2: allocate memory with initial value of 0
stus2 := make([]*StuRead,0)

//Error demonstration
//new() can only instantiate one struct object, and [] StuRead is a slice, not an object
stus := new([]StuRead)

stu1 := StuRead{Member assignment...}
stu2 := StuRead{Member assignment...}

//The slices created by modes 1 and 2 can successfully append data
//Method 2 is best to allocate 0 length, which will increase automatically when append. On the contrary, if the initial length is specified, it will not increase automatically when the length is not enough, resulting in data loss
stus1 := appen(stus1,stu1,stu2)
stus2 := appen(stus2,stu1,stu2)

//Successful encoding
json1,_ := json.Marshal(stus1)
json2,_ := json.Marshal(stus2)

When decoding, define the corresponding slice to accept

Json Unmarshal: decode json strings into corresponding data structures

We decode the above example

type StuRead struct {
    Name  interface{} `json:"name"`
    Age   interface{}
    HIgh  interface{}
    sex   interface{}
    Class interface{} `json:"class"`
    Test  interface{}
}

type Class struct {
    Name  string
    Grade int
}

func main() {
    //The "quotation mark" in the json character needs to be escaped by \, otherwise the compilation will make an error
    //The json string follows the above results, but the size of the key is modified and the sex data is added
    data:="{\"name\":\"Zhang San\",\"Age\":18,\"high\":true,\"sex\":\"male\",\"CLASS\":{\"naME\":\"1 class\",\"GradE\":3}}"
    str:=[]byte(data)

    //1. The first parameter of unmarshal is json string, and the second parameter is the data structure that accepts json parsing.
    //The second parameter must be a pointer, otherwise it cannot receive parsed data. For example, stu is still an empty object StuRead {}
    //2. You can directly stu:=new(StuRead). At this time, Stu itself is a pointer
    stu:=StuRead{}
    err:=json.Unmarshal(str,&stu)

    //If parsing fails, errors will be reported, such as incorrect json string format, missing "No.", missing} and so on.
    if err!=nil{
        fmt.Println(err)
    }

    fmt.Println(stu)
}

result:

{Zhang San 18 true <nil> map[naME:1 class GradE:3] <nil>}

Summary:

During json string parsing, a "receiver" is required to accept the parsed data, and the receiver must pass a pointer when Unmarshal. Otherwise, although the parsing does not report an error, the data cannot be assigned to the receiver. For example, StuRead {} reception is used here.
When parsing, the receiver can be defined by itself. The key in the json string automatically finds the matching item in the receiver for assignment. The matching rule is:
(1) First find the json tag that is the same as the key, and then assign it to the variable corresponding to the tag (such as Name).
(2) If there is no json tag, look for variables with the same variable name as key from top to bottom, such as Age. Or the variable name is the same as the key after ignoring the case. Such as HIgh, Class. The first match is assigned, and the subsequent matches are ignored. (provided that the variable must be exportable, i.e. capitalized).
Variables that cannot be exported cannot be parsed (for example, sex variables, although there is a k-v whose key is sex in the json string, its value is still nil after parsing, i.e. null value)
When there is an item in the receiver that cannot be matched in the json string, the parsing will automatically ignore the item and keep the original value. For example, for the variable Test, leave the null value nil.

You will find that the variable class does not seem to resolve to what we expect. Because the class at this time is a variable of interface {} type, and the value whose key is class in the json string is a composite structure, which is not a simple type of data that can be directly parsed (such as "Zhang San", 18, true, etc.). Therefore, when parsing, because the specific type of variable class is not specified, json automatically parses the data with value as composite structure into items of map[string]interface {} type. In other words, the struct Class object at this time has no relationship with the class variable in StuRead, so it has no relationship with this json parsing.

Let's take a look at the types of these interface {} variables after parsing

func main() {
    //It is consistent with the previous json parsing code
    ...
    fmt.Println(stu) //Print the variable type before json parsing
    err:=json.Unmarshal(str,&stu)
    fmt.Println("--------------json After analysis-----------")
    ... 
    fmt.Println(stu) //Print the variable type after json parsing    
}

//Use reflection to print variable types
func printType(stu *StuRead){
    nameType:=reflect.TypeOf(stu.Name)
    ageType:=reflect.TypeOf(stu.Age)
    highType:=reflect.TypeOf(stu.HIgh)
    sexType:=reflect.TypeOf(stu.sex)
    classType:=reflect.TypeOf(stu.Class)
    testType:=reflect.TypeOf(stu.Test)

    fmt.Println("nameType:",nameType)
    fmt.Println("ageType:",ageType)
    fmt.Println("highType:",highType)
    fmt.Println("sexType:",sexType)
    fmt.Println("classType:",classType)
    fmt.Println("testType:",testType)
}

result:

nameType: <nil>
ageType: <nil>
highType: <nil>
sexType: <nil>
classType: <nil>
testType: <nil>
--------------json After analysis-----------
nameType: string
ageType: float64
highType: bool
sexType: <nil>
classType: map[string]interface {}
testType: <nil>

It can be seen from the results

Before json parsing of interface {} type variables, the types printed are nil, but there is no specific type, which is the characteristic of empty interface (interface {} type).
After json parsing, the value in the json string, as long as it is "simple data", will be assigned according to the default type. For example, "Zhang San" is assigned to the string type into the Name variable, the number 18 corresponds to float64, and true corresponds to bool type.

"Simple data": refers to data that cannot be json parsed twice. For example, "name": "Zhang San" can only be json parsed once.
"Composite data": data like "CLASS \": {\ "naME \": \ "CLASS 1 \", \ "GradE \": 3} can be json parsed twice or even multiple times, because its value is also an independent json that can be parsed. That is, the key in the first parsing is the value of CLASS, and the key in the second parsing value is the value of naME and GradE

For "composite data", if the matched items in the receiver are declared as interface {} type, go will resolve to map[string]interface {} type by default. If we want to directly resolve to the struct Class object, we can define the item corresponding to the acceptor as the struct type. As follows:

type StuRead struct {
...
//Normal struct type
Class Class `json:"class"`
//Pointer type
Class *Class `json:"class"`
}

stu print results

Class Type:{Zhang San 18 true <nil> {1 Class 3} <nil>}
*Class Type:{Zhang San 18 true <nil> 0xc42008a0c0 <nil>}

It can be seen that when passing a Class pointer, the Class variable in stu stores a pointer. We can directly access the data through this pointer, such as stu.Class.Name/stu.Class.Grade

Type after Class variable resolution

classType: main.Class
classType: *main.Class

When parsing, what happens if there are two matching items in the receiver at the same time?
Test 1

type StuRead struct {
    NAme interface{}
    Name  interface{}
    NAMe interface{}    `json:"name"`
}

Result 1:

//When there is a matching json tag, its corresponding item is assigned.
//Remember: there can be no matching tags, but sometimes it's best to have only one
{<nil> <nil> Zhang San}

Test 2

type StuRead struct {
    NAme interface{}
    Name  interface{}
    NAMe interface{}    `json:"name"`
    NamE interface{}    `json:"name"`
}

Result 2

//When there are multiple matching json tags, the items corresponding to the tags will not be assigned.
//Ignore the label item and look from top to bottom for the assignment of the first item without label and matching
{Zhang San <nil> <nil> <nil>}

Test 3

type StuRead struct {
    NAme interface{}
    Name  interface{}
}

Result 3

//When there is no json tag, the first matching item will be assigned from top to bottom
{Zhang San <nil>}

Test 4

type StuRead struct {
    NAMe interface{}    `json:"name"`
    NamE interface{}    `json:"name"`
}

Result 4

//When there are multiple json tags with the same tag and there are no matches without tags, an error is reported
# command-line-arguments
src/test/b.go:48: stu.Name undefined (type *StuRead has no field or method Name, but does have NAMe)

It can be seen that the matching rules mentioned above are consistent.

If you don't want to specify the Class variable as a specific type and still want to keep the interface {} type, but you want the variable to be resolved to the struct Class object, what should you do?

This demand is likely to exist. For example, I met it

There are still some methods. We can define the variable as json.RawMessage type

type StuRead struct {
    Name  interface{}
    Age   interface{}
    HIgh  interface{}
    Class json.RawMessage `json:"class"` //Pay attention here
}

type Class struct {
    Name  string
    Grade int
}

func main() {
    data:="{\"name\":\"Zhang San\",\"Age\":18,\"high\":true,\"sex\":\"male\",\"CLASS\":{\"naME\":\"1 class\",\"GradE\":3}}"
    str:=[]byte(data)
    stu:=StuRead{}
    _:=json.Unmarshal(str,&stu)

    //Note here: secondary analysis!
    cla:=new(Class)
    json.Unmarshal(stu.Class,cla)

    fmt.Println("stu:",stu)
    fmt.Println("string(stu.Class):",string(stu.Class))
    fmt.Println("class:",cla)
    printType(&stu) //The previous examples of function implementation are
}

result

stu: {Zhang San 18 true [123 34 110 97 77 69 34 58 34 49 231 143 173 34 44 34 71 114 97 100 69 34 58 51 125]}
string(stu.Class): {"naME":"1 class","GradE":3}
class: &{1 Class 3}
nameType: string
ageType: float64
highType: bool
classType: json.RawMessage

It can be seen from the results

In the receiver, the variable declared as json.RawMessage type still retains the original value of JSON when JSON is parsed, that is, it is not automatically parsed as map[string]interface {}. For example, the parsed value of variable Class is: {"naME": "1 shift", "GradE": 3}
It can also be seen from the printed type that the type of variable class is json.RawMessage when JSON is parsed for the first time. At this point, we can perform secondary JSON parsing on the variable because its value is still an independent and resolvable complete JSON string. We just need to define a new receiver, such as json.Unmarshal(stu.Class,cla)

Posted by ch3m1st on Fri, 17 Sep 2021 18:52:54 -0700

Programmer Group

json parsing of Go: Marshal and Unmarshal

brief introduction

realization

Json Marshal: encodes data into json strings

Json Unmarshal: decode json strings into corresponding data structures

Hot Keywords