Saved searches

Use saved searches to filter your results more quickly

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

heetch / avro Public

Avro codec and code generation for Go

License

Notifications You must be signed in to change notification settings

heetch/avro

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Folders and files

Latest commit

History

Repository files navigation

Avro - Go-idiomatic encoding and decoding of Avro data

This package provides both a code generator that generates Go data structures from Avro schemas and a mapping between native Go data types and Avro schemas. The API is modelled after that of Go's standard library encoding/json package. The documentation can be found here. It also provides support for encoding and decoding messages using an Avro schema registry - see github.com/heetch/avro/avroregistry.

How are Avro schemas represented as Go datatypes?

• "int" is represented as int
• "long" is represented as int64
• "float" is represented as float32
• "double" is represented as float64
• "string" is represented as string
• "boolean" is represented as bool
• "bytes" is represented as []byte
• "null" is represented as the Go value nil
• is represented as []T
• is represented as map[string]T
• is represented a Go int type with String , MarshalText and UnmarshalText methods so it will encode as a string when used in JSON.
• will encode as a Go [123]byte type named F
• ["null", T] encodes as *T
• [T, "null"] encodes as *T
• [T₁, T₂, . ] (a union) encodes as interface<> that should hold only the types for T₁ , T₂ , etc.
• encodes as a Go struct type named R with corresponding fields.
• is represented as time.Time type
• is represented as github.com/google/uuid.UUID type.
• is represented as time.Duration type.

If a definition has a go.package annotation the type from that package will be used instead of generating a Go type. The type must be compatible with the Avro schema (it may contain extra fields, but all fields in common must be compatible).

If a definition has a go.name annotation the associated string will be used for the generated Go type name.

Comparison with other Go Avro packages

github.com/linkedin/goavro/v2, is oriented towards dynamic processing of Avro data. It does not provide an idiomatic way to marshal/unmarshal Avro data into Go struct values. It does, however, provide good support for encoding and decoding with the standard Avro JSON format, which this package does not.

github.com/actgardner/gogen-avro was the original inspiration for this package. It generates Go code for Avro schemas. It uses a neat VM-based schema for encoding and decoding (and is also used by this package under the hood), but the generated Go data structures are awkward to use and don't reflect the data structures that people would idiomatically define in Go.

For example, in gogen-avro the Avro type ["null", "int"] (either null or an integer) is represented as a struct containing three members, and an associated enum type:

type UnionNullIntTypeEnum int const ( UnionNullIntTypeEnumNull UnionNullIntTypeEnum = 0 UnionNullIntTypeEnumInt UnionNullIntTypeEnum = 1 ) type UnionNullInt struct < Null *types.NullVal Int int32 UnionType UnionNullIntTypeEnum >

With heetch/avro , the above type is simply represented as a *int , a representation likely to be familiar to most Go users.

Integration testing

A github.com/heetch/avro/avroregistrytest package is provided to run integration test against a real schema registry.

import "github.com/heetch/avro/avroregistrytest" type X struct < A int > avroregistrytest.Register(context.Background(), t, A<>, "test-topic")

This code snippet register an avro type for X struct for test-topic in the schema registry defined by KAFKA_REGISTRY_ADDR environment variable that must set to host:port form.