9.4 Go data format

9.4 Go data format

1.1. Msgpack format

Official msgpack official website sentence summary:.. It's like JSON but fast and small simple terms, it is similar to the data format and json, but in the memory of the digital, multi-byte characters, arrays, and so do a lot of optimization, reducing useless character, binary format, but also to ensure that no extra character of the increase in storage space.

msgpack example code

package main

import (
    "fmt"
    "github.com/vmihailenco/msgpack"
    "io/ioutil"
    "math/rand"
)

type Person struct {
    Name string
    Age  int
    Sex  string
}

//写入json
func writeJson(filename string) (err error) {
    var persons []*Person //切片，只能放入Person地址
    //生成10个结构体对象
    for i := 0; i < 100; i++ {
        p := &Person{
            Name: fmt.Sprintf("name%d", i),
            Age:  rand.Intn(100),
            Sex:  "Man",
        }
        //循环讲结构体放入切片
        persons = append(persons, p)
    }
    //序列化切片
    data, err := msgpack.Marshal(persons)
    if err != nil {
        fmt.Printf("=marshal failed, err:%v\n", err)
        return
    }
    //写入序列化后的数据，到filename中
    err = ioutil.WriteFile(filename, data, 0755)
    if err != nil {
        fmt.Printf("write file failed, err:%v\n", err)
        return
    }

    return
}

//读取json数据，传入文件参数
func readJson(filename string) (err error) {
    var persons []*Person
    data, err := ioutil.ReadFile(filename)
    if err != nil {
        return
    }

    err = msgpack.Unmarshal(data, &persons)
    if err != nil {
        return
    }

    for _, v := range persons {
        fmt.Printf("%#v\n", v)
    }
    return
}

func main() {
    filename := "./person.dat"
    err := writeJson(filename)
    if err != nil {
        fmt.Printf("write json failed, err:%v\n", err)
        return
    }

    err = readJson(filename)
    if err != nil {
        fmt.Printf("read json failed, err:%v\n", err)
        return
    }
}

json Code

package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "math/rand"
)

type Person struct {
    Name string
    Age  int
    Sex  string
}

//写入json
func writeJson(filename string) (err error) {
    var persons []*Person //切片，只能放入Person地址
    //生成10个结构体对象
    for i := 0; i < 100; i++ {
        p := &Person{
            Name: fmt.Sprintf("name%d", i),
            Age:  rand.Intn(100),
            Sex:  "Man",
        }
        //循环讲结构体放入切片
        persons = append(persons, p)
    }
    //序列化切片
    data, err := json.Marshal(persons)
    if err != nil {
        fmt.Printf("=marshal failed, err:%v\n", err)
        return
    }
    //写入序列化后的数据，到filename中
    err = ioutil.WriteFile(filename, data, 0755)
    if err != nil {
        fmt.Printf("write file failed, err:%v\n", err)
        return
    }

    return
}

//读取json数据，传入文件参数
func readJson(filename string) (err error) {
    var persons []*Person
    data, err := ioutil.ReadFile(filename)
    if err != nil {
        return
    }

    err = json.Unmarshal(data, &persons)
    if err != nil {
        return
    }

    for _, v := range persons {
        fmt.Printf("%#v\n", v)
    }
    return
}

func main() {
    filename := "./person.txt"
    err := writeJson(filename)
    if err != nil {
        fmt.Printf("write json failed, err:%v\n", err)
        return
    }

    err = readJson(filename)
    if err != nil {
        fmt.Printf("read json failed, err:%v\n", err)
        return
    }
}

Json package and msgpack packet protocol, as the difference between:

API的不同
json.Marshal    
json.Unmarshal

msgpack.Marshal
msgpack.Unmarshal

msgpack占用传输更快
yugoMBP:gobook yuchao$ ls -lh
total 32
-rw-r--r--  1 yuchao  staff   1.3K Mar 10 18:16 main.go
-rwxr-xr-x  1 yuchao  staff   3.3K Mar 10 18:16 person.dat
-rwxr-xr-x  1 yuchao  staff   3.8K Mar 10 18:16 person.txt

1.2. Protobuf Data Interchange Format

What is protobuf

Protobuf（Protocol Buffer）是google 的一种数据交换的格式，它独立于语言，独立于平台。google 提供了多种语言的实现：java、c#、c++、go 和 python，每一种实现都包含了相应语言的编译器以及库文件。由于它是一种二进制的格式，比使用 xml 进行数据交换快许多。可以把它用于分布式应用之间的数据通信或者异构环境下的数据交换。作为一种效率和兼容性都很优秀的二进制数据传输格式，可以用于诸如网络传输、配置文件、数据存储等诸多领域。

What is protoc

protoc是protobuf文件（.proto）的编译器，可以借助这个工具把 .proto 文件转译成各种编程语言对应的源码，包含数据类型定义、调用接口等。

By looking protoc source (see github library can know), the design protoc protobuf and the decoupling of the different languages, the underlying With c ++ protobuf structure capable of storing, and then generates the source code through different languages as a plug. Protoc compilation process can be divided into two simple steps (as shown above): 1) Analysis .proto file, translates protobuf native data structure stored in memory; 2) pass protobuf associated data structures corresponding to plug compiled language, by the plug-in charge of rendering a language-specific template output according to the received protobuf native structure?.

---

Why protobuf

最近的项目中，一直使用Json做数据传输。Json用起来的确很方便。但相对于protobuf数据量更大些。做一个移动端应用，为用户省点流量还是很有必要的。正好也可以学习一下protobuf的使用

跟Json相比protobuf性能更高，更加规范

编解码速度快，数据体积小

使用统一的规范，不用再担心大小写不同导致解析失败等蛋疼的问题了

但也失去了一些便利性

改动协议字段，需要重新生成文件。

数据没有可读性

Introduction:

Google's data interchange format

Binary

Automatic code generation

Universe fastest format

开发流程：
IDL编写
生成指定语言的代码
序列化-反序列化

1.3. Installation protoc

download link:

https://github.com/protocolbuffers/protobuf/releases

osx：https://github.com/protocolbuffers/protobuf/releases/download/v3.7.0/protoc-3.7.0-osx-x86_64.zip

windows 64位：https://github.com/protocolbuffers/protobuf/releases/download/v3.7.0/protoc-3.7.0-win64.zip

After the download is complete, the configuration of the PATH environment variable protoc

PATH="添加protoc的环境变量"

1.4 Download golang source plug-protoc-gen-go

安装golang代码插件，如果下载出现问题，删除文件夹github.com/golang/protobuf重新下载
go get -u github.com/golang/protobuf/protoc-gen-go
go install github.com/golang/protobuf/protoc-gen-go

Add protoc-gen-go command to the PATH variable

进入protobuf的文件夹，创建address文件夹
mkdir address
//生成go文件
protoc --go_out=./address/ ./person.proto

Applications

package main

import (
    "fmt"
    "io/ioutil"

    "./address"
    "github.com/golang/protobuf/proto"
)

func writeProto(filename string) (err error) {
    var contactBook address.ContactBook
    for i := 0; i < 64; i++ {
        p := &address.Person{
            Id:   int32(i),
            Name: fmt.Sprintf("武%d", i),
        }

        phone := &address.Phone{
            Type:   address.PhoneType_HOME,
            Number: "15210858000",
        }

        p.Phones = append(p.Phones, phone)
        contactBook.Persons = append(contactBook.Persons, p)
    }
    //序列化protobuf数据
    data, err := proto.Marshal(&contactBook)
    if err != nil {
        fmt.Printf("marshal proto buf failed, err:%v\n", err)
        return
    }

    err = ioutil.WriteFile(filename, data, 0755)
    if err != nil {
        fmt.Printf("write file failed, err:%v\n", err)
        return
    }
    return
}

func readProto(filename string) (err error) {
    var contactBook address.ContactBook
    data, err := ioutil.ReadFile(filename)
    if err != nil {
        return
    }
    err = proto.Unmarshal(data, &contactBook)
    if err != nil {
        return
    }

    fmt.Printf("proto:%#v\n", contactBook)
    return
}

func main() {
    filename := "./contactbook.dat"
    err := writeProto(filename)
    if err != nil {
        fmt.Printf("write proto failed, err:%v\n", err)
        return
    }
    err = readProto(filename)
    if err != nil {
        fmt.Printf("read proto failed, err:%v\n", err)
        return
    }
}