XXX_* type in generated *.pb.go file

Question

I'm working on a tutorial about gRPC. When I generated the .pb.go file, I'm getting some XXX_* type in my struct.

This is my consignment.proto file:

syntax = "proto3";

package go.micro.srv.consignment; 

service ShippingService {
    rpc CreateConsignment(Consignment) returns (Response) {}
}

message Consignment {
    string id = 1;
    string description = 2;
    int32 weight = 3;
    repeated Container containers = 4;
    string vessel_id = 5;
}

message Container {
    string id = 1;
    string customer_id = 2;
    string origin = 3;
    string user_id = 4;
}

message Response {
    bool created = 1;
    Consignment consignment = 2;
}

This is the struct in the .pb.go file. Can someone tell me why are there 3 XXX types in my struct ? Shouldn't the struct be reflecting what I'm defining in my proto ?

type Consignment struct {
    Id                   string       `protobuf:"bytes,1,opt,name=id" json:"id,omitempty"`
    Description          string       `protobuf:"bytes,2,opt,name=description" json:"description,omitempty"`
    Weight               int32        `protobuf:"varint,3,opt,name=weight" json:"weight,omitempty"`
    Containers           []*Container `protobuf:"bytes,4,rep,name=containers" json:"containers,omitempty"`
    VesselId             string       `protobuf:"bytes,5,opt,name=vessel_id,json=vesselId" json:"vessel_id,omitempty"`
    XXX_NoUnkeyedLiteral struct{}     `json:"-"`
    XXX_unrecognized     []byte       `json:"-"`
    XXX_sizecache        int32        `json:"-"`
}

Answer 1

The XXX_ types are used by the Protobuf library to store unknown fields. When you decode a proto, there may be additional fields in the serialized data that the library doesn't know what to do with. This can happen, for instance, when the reader and writer of the data are using different copies of the proto file. This is a feature to help with backwards compatibility between clients and serves built at different times.

Additionally, the XXX fields allow you to expose extensions , which were part of Proto2. They were removed in Proto3 in favor of Any , but the library still needs to support them.

As for what you should do with these? I would just leave them alone, and don't reference them. You don't need to set them, and you don't need to read them. The Go protobuf library will handle them for you.

Answer 2

If you want to specifically exclude them when generating, you can add this to your file

import "github.com/gogo/protobuf/gogoproto/gogo.proto";
option (gogoproto.goproto_unkeyed_all) = false;
option (gogoproto.goproto_unrecognized_all) = false;
option (gogoproto.goproto_sizecache_all) = false;

edit: Styling

Answer 3

The official protobuffer generator for golang still includes the XXX_ fields and the issue is still open and decisions are pending on wether to remove it ot not but you can use the gogo third party generator. I am using it for a long time without a problem. Just use the gogofaster_out

https://github.com/gogo/protobuf
Here are the section that discussed about exclude the XXX_ fields

More Speed and more generated code

Fields without pointers cause less time in the garbage collector. More code generation results in more convenient methods.

Other binaries are also included:

protoc-gen-gogofast (same as gofast, but imports gogoprotobuf)
protoc-gen-gogofaster (same as gogofast, without XXX_unrecognized, less pointer fields)
protoc-gen-gogoslick (same as gogofaster, but with generated string, gostring and equal methods)

Installing any of these binaries is easy. Simply run:

go get github.com/gogo/protobuf/proto
go get github.com/gogo/protobuf/{binary}
go get github.com/gogo/protobuf/gogoproto

After installing the lib use

protoc --gogofaster_out=plugins=grpc