Add support for editions

Editions are the future of protobuf syntax.

Currently, when you set edition = "2023" at the top of a .proto file, and run it through prost-build using protoc_arg("--experimental_editions"), it will panic on this line because the value of FileDescriptorProto.syntax is "editions". That's pretty much the extent of my investigation. I just figured I'd file this bug since support for editions doesn't seem to be tracked elsewhere.

This may sound like it's adding complexity to Prost, but it looks like it should actually simplify Prost once Prototiller is released, since that will effectively normalize all the different syntaxes and editions into a canonical representation using features, and Prost will only have to support that canonical representation in the long term. It would be a somewhat heavy refactor up front, though.

I agree, editions are important. However, due to backwards compatibility of protoc itself, we need to keep supporting proto2 and proto3 for some time.

Are you willing to do the work to add this support?

That makes sense. I won't have bandwidth for a couple months, but I'm down to contribute, especially after Prototiller is released.

Up to spec support for editions requires at least the following:

Unknown fields, as discussed in Add support for unknown fields #2.
Open and closed enumeration behavior. I've started on this in OpenEnum: an enum to represent protobuf's enumeration field values #1079.
Field presence disciplines.

Drive by chiming in from someone on the Protobuf team at Google here.

Unfortunately open sourcing the prototiller tool is held up so we still don't have a date for that (sorry!), but in nearly all cases manually upgrading a file from Proto3 to Editions 2023 files shouldn't be too hard so there shouldn't be a very strong reason to wait for it.

In general, the vision of Edition2023 is simply that it can reunify the syntaxes Proto2 and Proto3 and set the groundwork for safe syntax evolution (the name is actually inspired by Rust Editions, with the same idea of always supporting interop between editions and tooling to safely upgrade files to the next edition). There's not really anything extra in Edition 2023 that wasn't already there in Proto2 and Proto3. The ideal is that any Proto2 or Proto3 file should be able to move to Edition 2023 and have it be perfectly behavior preserving way (basically: start with Proto3, change it to Edition-2023, and the right options at the top of the file the the Prost codegen should be exactly the same as it was).

Ideally Editions 2023 support should only involve identifying any conditionals in the code generation which check exactly "is syntax == Proto2 or Proto3?" to decide things like "should this field be in an Option<>?" and decide in a more generalized way to also support Edition 2023 (which supports all proto2 and proto3 behavior under one syntax, and so a check on syntax can't decide those things anymore).

There's some argument against doing any extra conformance-to-spec work for things like open-enums specifically related to Prost advertising support of Edition2023 inputs, since ideally users should be able to move their Proto3 file to Edition2023 files and still get the same Prost gencode if they want. This does have oddities that if your implementation has preexisting non-conformant behavior on open-vs-closed enums, if you 'fix' that behavior only on Edition2023 it will make it a breaking change for users to upgrade their .proto files, so the strategy we recommend is more "make Edition2023 behavior whatever the corresponding preexisting proto2/proto3 behavior was, then separately consider fixing nonconformant behavior of things like open enums on both Proto3 and Edition2023 together a broader "conformance to intended spec" topic.

I glanced at the Prost generator code, I only immediately see 2 conditionals against .syntax in code_generator.rs that would need to be fixed for it to handle Edition2023 inputs:
https://github.com/tokio-rs/prost/blob/master/prost-build/src/code_generator.rs#L1037

The main work that would need to happen for Prost to support Edition 2023 is for those 2 conditionals to honor Feature Resolution: this relates to the behavior where under Editions the idea is that you can set if fields should be treated as optional or not by default at any of the file > message > nested message > individual field level scopes and it applies downward within the scope. For every feature, we also publish the syntax-dependent defaults including for Proto2/Proto3, so the intended design is that you can have your code generator operate only use this feature resolution notion and you'll have support for all possible syntaxes (including easily extend to future ones if the file-level default of a given feature might change in eg Protobuf Edition 2025).

Protoc plugins written in C++ get this feature-scoping/cascading behavior for free since we implemented it in the descriptor.h that we publish, but code generators written in other languages that have done their own bootstrap on descriptor.proto (like Prost has) directly need to implement that. It generally amounts to a DFS to propagate the feature values downward in scopes, but it could e.g. a decision on a field could on demand traverse upwards until you find the narrowest scope that decides what the optionality of that field should be. Once you have that "feature cascading" behavior in place, I believe Prost supporting Edition 2023 is only like a ~3 line delta.

There's more info about Feature Resolution here:
https://protobuf.dev/editions/implementation/#:~:text=specific%20test%20cases-,Feature%20Resolution,-Editions%20use%20lexical&text=specific%20test%20cases-,Feature%20Resolution,-Editions%20use%20lexical

Field presence is going to be challenging. Starting from edition 2023, all fields have explicit presence (equivalent to declaring the fields optional in proto3) by default. There is pushback even about message-typed fields being represented with an Option, as seen in #763 and a number of duplicate issues.

I see two ways to deal with this:

Open structs: Maintain the current approach: keep all struct fields public, represent explicitly present fields with Option, alleviate the pain with getter and setter methods for scalar fields.
Opaque structs: Follow what Google did with the Go bindings and change the code generator to produce opaque structs, with private fields and public accessor methods. In-memory representation for field presence can be optimized with presence flags.

Advantages:

A simple and straightforward representation. The rich and standard Option API can be used to handle the potentially missing value in a way most appropriate to the application.
Memory layout of message structs can be more compact. The generated structs may be more suitable as application data domain types (however, this is not generally achievable due to validity constraints out of scope for Protobuf).

Disadvantages:

As explicit presence becomes the norm, the convenience of destructuring diminishes: you have to match for the None case on each field of interest.
It's impossible to destructure or use struct initializer syntax at all. The latter is not Protobuf-compliant, though; forward-compatible code should use builders.

@esrauchg

@esrauchg where could I follow the situation? Personally, generating the types without Option when it is required is extremely valuable, so I would like to be follow the situation around the topic

Can you clarify what information would be helpful in this context?

Just to preface, Google is not proscriptive about open source protobuf implementations: we have our wire format spec and associated conformance test, and we have our protoc-plugin support. Open source implementations can be interoperable while implementing many details as they see appropriate. Whats right for Prost and whats right for the implementations that Google maintains is often not the same in other cases.

This is just my own take, but I would probably say Prost should probably just use Option<> and then non-option getters. Any Google implementation would actually always be opaque API but our usecases are a bit different.

Theres a few topics that I can already elaborate on some of the topics here:

1) Motivations on Edition 2023 choosing Explicit Presence (the "optional" keyword behavior from proto3) by default

We unfortunately don't have much published retrospective about this, but we do intend to publish something more official than a github comment on this topic in the future.

The tldr is that 'implicit presence' (proto3 fields that aren't marked optional) broadly didn't turn out to have the performance benefits that were expected when it was designed in the early 2010s. We've also found presence to be quite valuable in general, and actually a performance benefit (not regression) to maintain the has-bits when they are implemented as a bitmask.

And we have found downsides of "it is very confusing re: 0 is a legal value or not on an implicit-presence field"; for example Google Cloud APIs generally use Proto3 and use annotations like (google.api.field_behavior) = OPTIONAL,: on implicit presence fields that means "zero/empty-string is not a legal value for this" and on fields marked with the optional keyword that instead means "must be assigned to any value".

2) Explicit presence isn't really "nullable"

We touch on this a little bit in documentation here

Explicit presence is trying to track whether a field was affirmatively assigned or not, which is really an adjacent topic from traditional nullability.

In general explicit-presence are not intended to be understood as "nullable" though. Its a bit subtle but we would say philosophically that optional fields in general should be treated as non-nullable in gencode surface area. Basically we do expect that nearly all readers of optional fields will still read that field without caring if the has bit was set or not, so if optional fields are always exposed as Option<> that is a a big overhead for that very normal usecase.

This also applies to proto3; in our philosophy even there when a field is explicitly marked optional, that generally does not mean "readers of this should care about the has-bit on every read". It's just a lesser pain point in Prost in Proto3 today, since since most fields happen to not be marked optional.

3) Open structs?

That leads to another large topic here is "open structs" though: as open structs it becomes very difficult to maintain the has-bits over non-nullable fields.

That one is an oddball topic in this context: broadly we have found open structs aren't great at implementing Protobuf in general for many reasons, by precluding us from doing optimizations at scale, but also serious semantic problems like "you can't add a field to a preexisting message without it being a breaking change" which would be horribly problematic for Googles normal usage of messages.

Of the official implementations only Go was implemented that way and now supports an opaque API instead (which we discussed at https://go.dev/blog/protobuf-opaque). And then if open structs aren't "good" anyway, then the difficulty of open structs in implementing optional fields as non-nullable is not actually a major topic.

But Rust as a language introduced some pretty unique that may takes some of the sharp edge problems off of open structs, including the ability to mark structs #[non_exhaustive] combined with ..Default::default() which no other major language had. Though... Prost doesn't mark gencode as #[non_exhaustive] either right now (which is a "adding a field is a breaking change" problem but it also e.g. causes more problems for Prost to start to maintain unknown-fields).

And Rust also has some nice features that make it so open structs are more beneficial in a way that is burdensome to maintain over getters, including destructuring behaviors (enabling destructuring over accessors is quite burdensome).

Broadly, Google's official implementations still wouldn't pursue an open structs design even with those benefits though, but it may be very reasonable that Prost does given Prosts goals and target audience.

@esrauchg

@esrauchg I am working on an event sourcing WASM-based stack; right now, we would like to use Protobuffer as the data types that we deal in in our callbacks:

pub fn initial_state() -> State {
    State {
        id: None,
        status: Status::Paused,
    }
}

pub fn is_terminal(_state: &State) -> bool {
    false
}

pub fn decide(
    state: &State,
    command: &Command,
) -> Result<Decision<State, Command, Event, Error>, Error> {

}

pub fn evolve(state: &State, event: &Event) -> State {}

State, Event, Command are Protobuffer messages; right now, the only suboptimal situation we found is that we do not get types that are precise enough (related to #1286) every message is a Option type even thou in the protobuffer definition is required.

We would benefit a ton if such generated types are structurally precise, otherwise the code is full of Some types, leading to a lot of verbose code, and most important, sometimes incorrect if the developers aren't paying attention.

For extra context, and being unorthodox here. The App-level data types vs. serialization data types distinction doesn't matter in our case, unless protobuffer break changes for a given version, we are trusting Google that it would not happens, we are OK combining the serialization.

I am waiting for the official Google Protobuf package to move everything to it by the way, we want to take advantage of the official packages here, especially for Rust, Go and JS/TypeScript if possible

Ah; I think officially Protobuf is just not really optimal fit for wasm/FFI bridge style usecases; Protobuf is focused on ensuring behavior in the face of skewed schemas between sender/receiver so that you can safely change schema over time in distributed system; by contrast FFI you can know for sure there's no schema skew. Generally any dedicated FFI technologies will suit you better for that, as well as some other serialization technologies like capnproto or flatbuffers.

This is getting a bit further the weeds about a different topic that this Prost Editions support topic; if there's any further questions that the Protobuf team can help with (both from Prost contributors or any other users), please feel free open an issue on the Protobuf github and we are happy to discuss further there, or

@esrauchg

Thank you @esrauchg for the detailed writeup and some Google lore that I haven't seen widely shared previously.

In general explicit-presence are not intended to be understood as "nullable" though. Its a bit subtle but we would say philosophically that optional fields in general should be treated as non-nullable in gencode surface area. Basically we do expect that nearly all readers of optional fields will still read that field without caring if the has bit was set or not, so if optional fields are always exposed as Option<> that is a a big overhead for that very normal usecase.

This also applies to proto3; in our philosophy even there when a field is explicitly marked optional, that generally does not mean "readers of this should care about the has-bit on every read". It's just a lesser pain point in Prost in Proto3 today, since since most fields happen to not be marked optional.

For optional scalar fields, prost-build generates getter methods that behave as recommended by the Protobuf guide: return the default value if the field is not set. This is also where custom default values are implemented for such fields, since the field of the open struct is an Option with no information on the fallback value.

For message-typed fields, this becomes more tricky. First off, philosophically, it's not certain that Default::default() is a good value to return for a missing message field. I suppose a discerning application developer would not neglect to check the has-flag, but there is no speed bump/code review wart for that hot-headed code jockey and the wrong code just compiles too... Secondly, this is where Rust's ownership and mutability semantics come into play. You probably don't want to return every struct type by value from getter methods, as structs can be big and cloning can be wasteful. You can't borrow something that only exists in Some variant if your containing struct reference is not mutable, which would be a cumbersome signature for a getter. So the remaining solution is to make the fields non-optional also in the struct, initialize them with the default value, and track presence with has-flags. In addition to necessarily making the struct opaque, this would introduce a performance tradeoff: even though all default values in Prost's data model are non-allocating, default-initializing a large message field is more costly than plugging it with None when it's wrapped in an Option.

But Rust as a language introduced some pretty unique that may takes some of the sharp edge problems off of open structs, including the ability to mark structs #[non_exhaustive] combined with ..Default::default() which no other major language had. Though... Prost doesn't mark gencode as #[non_exhaustive] either right now (which is a "adding a field is a breaking change" problem but it also e.g. causes more problems for Prost to start to maintain unknown-fields).

This is a major non-compliance issue I personally have with Prost; while it's possible to annotate all generated structs with #[non_exhaustive], to make this gencode usable you'd need to also generate builders or something similar. NB: ..Default::default() update syntax does not work with non-exhaustive structs.

And Rust also has some nice features that make it so open structs are more beneficial in a way that is burdensome to maintain over getters, including destructuring behaviors (enabling destructuring over accessors is quite burdensome).

Completely agree here. One neat trick with destructuring is being able to handle the happy path and missing-field errors in a match expression, with the compiler making sure you did not miss a case:

struct Foo {
    bar: Option<String>,
    baz: Option<i32>,
}

fn handle_foo(foo: Foo) {
    match foo {
        Foo {
            bar: Some(b),
            baz: Some(z),
            ..
        } => {
            println!("Bar: {}, Baz: {}", b, z);
        }
        Foo { baz: None, .. } => {
            eprintln!("Baz is missing");
        }
        Foo { bar: None, .. } => {
            eprintln!("Bar is missing");
        }
    }
}

~~However,~~ if Protobuf writers proliferate optional scalars where a missing field legitimately means the default value, ~~our open-struct approach will make such matching cumbersome.~~ this is workable too: the general rule is "match required fields, use getters for defaultable ones".

NB: ..Default::default() update syntax does not work with non-exhaustive structs.

Maybe I'm misunderstanding but what I was thinking of does work, here's an example in the Rust playground

This is also where custom default values are implemented for such fields, since the field of the open struct is an Option with no information on the fallback value.

Makes sense to me :)

You can't borrow something that only exists in Some variant if your containing struct reference is not mutable

It may be that Prost has reasons to choose to go a different way with this, but our typical semantics would amount to this:

message Parent { Child c = 1; }
message Child {}

Being handled like this, and just hand out some &'static of a const in the getter when it is unset:

#[derive(Default)]
struct Parent {
    c: Option<Child>
}

#[derive(Default)]
struct Child {}

const CHILD_DEFAULT: Child = Default::default();

impl Parent {
    fn c(&self) -> &Child {
        self.c.as_ref().unwrap_or(&CHILD_DEFAULT)
    }

   fn c_mut(&mut self) -> &mut Child {
       self.c.get_or_insert_default()
   }
}

NB: ..Default::default() update syntax does not work with non-exhaustive structs.

Maybe I'm misunderstanding but what I was thinking of does work, here's an example in the Rust playground

That only works within the same crate, not with exported APIs.

const CHILD_DEFAULT: Child = Default::default();
impl Parent {
    fn c(&self) -> &Child {
        self.c.as_ref().unwrap_or(&CHILD_DEFAULT)
    }
}

Not as simple as that: Default::default is not const, and you can't generate a const initializer in all cases without getting into weeds with LazyLock or similar, because e.g. HashMap cannot be const-initialized without sacrifices in security.

While this seems feasible, I'm not sure it's an architectural choice that Prost should adopt, being a minimal-cost Protobuf marshalling/unmarshalling implementation. Maybe with open structs, the getters could be considered amenities with an acceptable potential for synchronization blockage? The developers would need to be educated about this. I'm assuming code generated for no_std can only use const-initializable types, so at least that is not a concern.

Any chance we can get #1318 merged? Lack of support for editions prevents adoption of newer protos. The mentioned PR works on our forks, but we would like to use upstream.

I prepared a new PR as #1318 seemed to be stale. I added #1347 for review with an example. It's still not 100% editions compatible as some conformance tests have been skipped in the failing_tests.txt but I think it might be a step in the right direction and it does take into account the recent discussions and feedback about editions (e.g. Options). Hope this this helps and I'm happy to work more on this PR to get it ready, though I'm a little concerned that this PR is a significant/potentially breaking change that respects some attention. I must say that I do prefer this native Rust implementation over the Official Protobuf crate (currently in RC stage). Seems like this is the right approach, compared to wrapping C++ libs. Though its arguably a well maintained/common/potentially more secure approach.

Failed attempt to switch rust proto files to editions

Would it be possible to emulate optional with features/metadata? E.g.

message Foo {
    int32 id = 1 [(prost.optional) = false];
    string name = 1 [(prost.optional) = true];
}

which would generate

struct Foo {
    id: i32,
    name: Option<String>,
}

antigravity-codes 0.1.10: wrap the Antigravity localharness runtime

Navigation Menu

Repository navigation

Add support for editions #1031

1) Motivations on Edition 2023 choosing Explicit Presence (the "optional" keyword behavior from proto3) by default

2) Explicit presence isn't really "nullable"

3) Open structs?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add support for editions #1031

Description

Activity

caspermeijn commented on Apr 19, 2024

ouillie commented on Apr 22, 2024

mzabaluev commented on Nov 17, 2024

esrauchg commented on Mar 6, 2025

mzabaluev commented on Jun 2, 2025

yordis commented on Jun 2, 2025

esrauchg commented on Jun 2, 2025

1) Motivations on Edition 2023 choosing Explicit Presence (the "optional" keyword behavior from proto3) by default

2) Explicit presence isn't really "nullable"

3) Open structs?

yordis commented on Jun 2, 2025

esrauchg commented on Jun 2, 2025

mzabaluev commented on Jun 2, 2025

esrauchg commented on Jun 2, 2025

mzabaluev commented on Jun 2, 2025

mateuszlitwin commented on Oct 1, 2025

abarke commented on Oct 10, 2025

ensc commented on Aug 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions