In contrast to protobuf2 there is no way in protobuf3 to mark some fields as optional and some other fields as required. Instead, any field might be omitted leading this field to be set to its default zero-value. I believe there were many good reasons for such a design decision. However, while this behavior might be superior to the proto2's explicit distinction between required and optional fields, it also has some unfortunate implications.

From proto3 Language Guide:

When a message is parsed, if the encoded message does not contain a particular singular element, the corresponding field in the parsed object is set to the default value for that field

• For strings, the default value is the empty string.
• For bytes, the default value is empty bytes.
• For bools, the default value is false.
• For numeric types, the default value is zero.
• For enums, the default value is the first defined enum value, which must be 0.
• For message fields, the field is not set. Its exact value is language-dependent. See the generated code guide for details.

Note that for scalar message fields, once a message is parsed there's no way of telling whether a field was explicitly set to the default value (for example whether a boolean was set to false) or just not set at all: you should bear this in mind when defining your message types. For example, don't have a boolean that switches on some behaviour when set to false if you don't want that behaviour to also happen by default. Also note that if a scalar message field is set to its default, the value will not be serialized on the wire.

Sometimes, the absence of field is not the same as the field in its default value. Or there might be no default value at all. Example:

// person.proto

syntax = "proto3";
package main;

message Person {
    int32 age = 1;  // in full years
};

Compile it:

protoc --go_out=. person.proto

Test program:

// main.go

package main
import "fmt"

func main() {
    john := Person{}
    fmt.Println(john.Age)
}

Run it:

go run person.pb.go main.go
> 0

For a kid less than a year old, a corresponding Person object will have age equal to 0. However, if a person's profile was filled only partially, we simply might not know the age of the person. But if we set the age of such person to its default value (which is 0 for numeric types), we would automatically make the person a child. While a redesigning our domain model and refactoring the age to dateOfBirth could be a better solution for this particular problem, it would be great to have a way to address such situations on the protocol level too.

So, it means that we need a strategy, how to deal with truly optional fields. Luckily, protobuf3 offers a solution - wrapper types for scalar value types from the Protocol Buffers Well-Known Types namespace. The idea is really simple. Instead of having an integer (or float, or boolean, etc.) field in the message, we use Int32Value field which is a wrapper around int32 scalar value type.

// person.proto

syntax = "proto3";
package main;

import "google/protobuf/wrappers.proto";

message Person {
    google.protobuf.Int32Value age = 2;
};

Wrappers make scalar types composite, enabling some behavior customization. Implementation for different languages may vary, but generated Go code will look as follows:

// person.bp.go

import (
    wrappers "github.com/golang/protobuf/ptypes/wrappers"
)

type Person struct {
    Age *wrappers.Int32Value `protobuf:"bytes,2,opt,name=age,proto3" json:"age,omitempty"`
    // ...
}

// ...

After construction of a Person object, we will simply have nil for the age field as it's a default zero-value for pointers in Go. But now we also can distinguish between an absence of a value for a field and its zero-value. Check the snippet:

// main.go

package main
import "fmt"
import "github.com/golang/protobuf/ptypes/wrappers"

func main() {
    john := Person{}
    fmt.Println(john.Age)

    alice := Person{Age: &wrappers.Int32Value{Value: 0}}
    fmt.Printf("%#v\n", alice.Age)
}

Run it:

go run person.pb.go main.go
> <nil>
> &wrappers.Int32Value{Value:0, ...}

Thus, if you don't know the age of a person, just skip the field. It will be set to nil and after decoding on the receiver side, the absence of the value will be noticeable. But if you have a value, even though it might be equal to the type's default value (i.e. 0), the fact that you specified it will be also transparent. And the protobuf compiler is smart enough to deal with this convention for all the supported backends. The output above shows that for Go wrappers are basically structs with the field Value of the corresponding scalar type.

Bonus: JSON interoperability

Protobuf is striving to achieve maximal interoperability between different programming languages as well as between heterogeneous systems. It provides a first class support for JSON encoding of the messages. Specifically for Go, one can notice the generated JSON annotations in the .pb.go files (see an example above). However, the default Go encoding/json package doesn't work well when it comes to wrapper types:

// main.go

package main

import "encoding/json"
import "fmt"
import "github.com/golang/protobuf/ptypes/wrappers"

func main() {
    alice := Person{Age: &wrappers.Int32Value{Value: 0}}
    bytes, _ := json.Marshal(alice)
    fmt.Println(string(bytes))

    bob := Person{Age: &wrappers.Int32Value{Value: 42}}
    bytes, _ = json.Marshal(bob)
    fmt.Println(string(bytes))
}

Run it:

go run person.pb.go main.go
> {"age":{}}
> {"age":{"value":42}}

The produced JSON above doesn't look valid since we have an object instead of an integer field. To solve this problem one needs to use jsonpb package provided by Go protocol buffer backend:

// main.go

package main

import "fmt"
import "github.com/golang/protobuf/jsonpb"
import "github.com/golang/protobuf/ptypes/wrappers"

func main() {
    alice := Person{Age: &wrappers.Int32Value{Value: 0}}
    bob := Person{Age: &wrappers.Int32Value{Value: 42}}

    m := jsonpb.Marshaler{}
    a, _ := m.MarshalToString(&alice)
    fmt.Println(a)

    b, _ := m.MarshalToString(&bob)
    fmt.Println(b)
}

Run it:

go run person.pb.go main.go
> {"age":0}
> {"age":42}

This package is aware of the wrappers purpose and can generate a valid JSON for wrapped scalar fields.

Make code not war!

Level up your server-side game — join 20,000 engineers getting insightful learning materials straight to their inbox: