Support for Unknown Fields (Rebase of #574) #1340

blizzardfinnegan · 2025-09-24T21:44:59Z

For more information see Issue #2

This PR is largely similar to #574 , rebased to main, and with edits suggested by the maintainer on said PR implemented to the best of my ability.

Does not compile, incomplete.

- Rename UnknownFieldSet to UnknownFieldList, as requested by maintainer caspermeijn - Change fixed32 and fixed64 representation to be consistent with C++ implementation, as requested by caspermeijn - start resolving build-time errors - Currently failing with reexported crate issues

- TODO: What endianness is protobuf?

This section should be added back (and/or modified to fit new standards), but commenting out for now to reduce points of failure to resolve creating bad Copy derives

Instead of checking against empty, which can fail differently. Attempting to force fields with unknown to not have a derived Copy.

While this is a change made by rustfmt/clippy, it's outside the scope of this PR, so it's been removed

Seems like local clippy check runs in rust edition 2021, CI runs in 2024

…ields_support

blizzardfinnegan · 2025-10-08T16:06:35Z

@caspermeijn
If there are any objections or additional comments on this, I would love to know. This is an important feature, and one the company I work with is dependent on.

Unfortunately, Github doesn't have a clear way to request reviews on a PR for someone outside the org and/or not a direct developer on the project, otherwise I would have done so. Apologies for the ping if that isn't the right channel.

alebar42 · 2025-10-09T14:12:24Z

@blizzardfinnegan AFAIK you could remove or comment out the ignored tests here in order to verify the implementation: https://github.com/blizzardfinnegan/prost/blob/feature/unknown_fields_support/conformance/failing_tests.txt

blizzardfinnegan · 2025-10-09T15:05:25Z

@alebar42 Thank you for the correction and clarification!
I made the relevant changes, and pushed them to my branch. ~~It doesn't seem to have passed through to the PR yet? Might be a CI issue...~~ 🤷

Either way, the tests do in fact pass ~~(locally, at least)~~.

Edit: Unclear what was the issue, but said changes are now in the PR, which is the important part.

blizzardfinnegan · 2025-11-05T21:19:01Z

@caspermeijn
Any chance there's some movement on this?

aidandj · 2025-11-14T00:31:13Z

Ran into an issue using pbjson_build with this. Because when you generate unknown fields, pbjson_build doesn't know what to do with them and it errors:

error[E0063]: missing field `unknown_fields` in initializer of `google::protobuf::generated_code_info::Annotation`
    --> /out/pbjson/google.protobuf.serde.rs:3929:20
     |
3929 |                 Ok(super::tonic::google::protobuf::generated_code_info::Annotation {

caspermeijn

Thank you for taking this forward. It is a large PR, so it takes some time for me to respond.

I miss user facing documentation. Could you add a section to the README that describes this new functionality?

prost-build/src/config.rs

prost-build/src/context.rs

prost-derive/src/field/mod.rs

caspermeijn · 2025-11-14T15:09:54Z

prost-derive/src/field/unknown.rs

+
+        for attr in attrs {
+            if word_attr("unknown_fields", attr) {
+                set_bool(&mut unknown, "duplicate message attribute")?;


This error message is incorrect

Why is that? If the same attribute (unknown_fields) is found twice, that should be a duplicate attribute. I do think that it should probably have different behaviour though; it should probably just pull all the unknown fields in, I would assume.

Sorry for the miscommunication, I do agree the error should be here.

The error text doesn't match the error situation. I think it should be something like: "multiple unknown_fields in one message". Do you have an example of what the complete error message that the users sees is?

I can definitely make the error message something more descriptive, sure! I don't have an example off-hand though, unfortunately.

caspermeijn · 2025-11-14T15:17:49Z

prost-derive/src/lib.rs

    });
+    let merge_fallback = match fields.iter().find(|&(_, f)| f.is_unknown()) {
+        Some((field_ident, field)) => {
+            let merge = field.merge(&prost_path, quote!(value));


Why is this in seperate section? The fields are already sorted with the unknown fields as last, so this can be in the first loop, right?

It seems like it's in a separate iterator call so it can check if there are any unknown fields, and if there aren't, add the skip_field call at the end of the iteration. There's definitely a better way to do it though.

It would be great (but not required) if you could find a better way. My suggestion would be to do a single loop for all field and then always do the skip. If unknown fields is enabled, then the skip branch is never triggered. I don't know whether the compiler accepts a double catch all

Makes sense to me!

prost/src/unknown.rs

tests/src/decode_error.rs

conformance/failing_tests.txt

blizzardfinnegan · 2025-11-18T14:39:44Z

Thank you for the review! I pretty much did as clean of a direct uplift as I could with the original MR, without changing too much. I'll go through and add docs and resolve the issues you pointed out.

Apologies for the repeated pings. It's definitely a bear of a PR, I wasn't sure if you had seen it or not.

- Remove Option wrapper on unknown fields - Actually utilise custom iterator instead of nested for loops - use Iterator.sum() instead of manual counter - use Buf.get_u64_le() and Buf.get_u32_le() instead of manual copies - Update conformance to not require failing_tests.txt - remove empty failing_tests.txt

- Add user-facing documentation for `prost` and `prost-build` for enabling unknown fields, along with limitations - Modify `prost_build::Config::include_unknown_fields` to have a default value for the field in which unknown fields are stored. (defaults to `unknown_fields`)

blizzardfinnegan · 2025-11-18T17:36:32Z

CI is currently failing, for reasons unclear to me. All CI failures are due to this error:

no function or associated item named `new` found for struct `error::DecodeError` in the current scope

Which is not accurate. It's only failing in CI (this failure does not occur if you run the same commands locally, afaict), and only on the files I've changed.

blizzardfinnegan · 2025-11-19T16:51:04Z

Ran into an issue using pbjson_build with this. Because when you generate unknown fields, pbjson_build doesn't know what to do with them

I think that might be a separate uplift for the pbjson to support unknown fields as currently written in this PR. There might be some changes to make in the prost implementation to help with that change, but I think getting initial support in to prost is more important than making sure the entire ecosystem works all at once.

If this PR goes through, I would highly recommend opening an issue in pbjson to that effect. I'm not familiar with the project personally, and the devs there would have a better sense of what, if anything, would need to change in prost to get their project to work properly, and could stack a patch on top of this PR.

aidandj · 2025-11-19T16:59:58Z

The change was quite minor, and it worked for me, but I wanted to call out how much of a breaking change this is. And how it is kind of awkward to have to initialize the unknown field field during struct creation.

In all other languages, unknown fields are accessed after the fact, but the user doesn't need to think about them unless needed. I think this is partially an issue with rust not allowing an inherent default value (like other languages) and prost allowing struct literal creation.

In the issue I suggested moving to a constructor with a private unknown fields or something, but those are also breaking changes.

blizzardfinnegan · 2025-11-19T18:29:24Z

Part of the issue is also the fact that, the PR as originally written and as reimplemented here, is almost entirely for round trip consistency, and largely doesn't expose the unknown fields. You get an iterator, so it's not impossible to check based on tags, but it's not intuitive or ergonomic, especially since other structs just directly expose the internal fields. This PR also doesn't fully interpret fields beyond the bare minimum to parse the raw message, so there would be more logic necessary to fully interpret contained information.

I like the suggestion of introducing constructors, I think that would help a lot in terms of new stuff being written. That being said, I think you could build a transition plan to introduce constructors, and still allow but deprecate manually creating struct literals, and then loop back around to disallowing struct construction directly later on.

aidandj · 2025-11-19T18:40:43Z

That's fair. I'm new to the rust world and was mostly hoping there was some magic that could be used to include something in a struct, but not have it be in a struct literal. As my use case (and most unknown fields use cases) are just making sure they don't get dropped in transit. I'm not actually doing anything with the unknown fields.

blizzardfinnegan · 2025-11-19T20:00:10Z

There is the ..Default::default() call, which sets any not already set values to their default values, but you do have to know that that call is necessary. I've added that to the documentation in the README, but it's not an obvious solution if you aren't already aware of the function.

aidandj · 2025-11-19T20:04:11Z

Yes, that's what I used in my pbjson update, but either way, it's a breaking change if you want to support unknown Fields

aidandj · 2025-11-19T20:04:17Z

Yes, that's what I used in my pbjson update, but either way, it's a breaking change if you want to support unknown Fields

caspermeijn · 2025-11-21T15:22:09Z

CI is currently failing, for reasons unclear to me. All CI failures are due to this error:
no function or associated item named `new` found for struct `error::DecodeError` in the current scope

I recently merge a change to DecodeError: #1330

I think you can reproduce this problem if you rebase your branch to the latest main

caspermeijn · 2025-11-21T15:41:50Z

Yes, that's what I used in my pbjson update, but either way, it's a breaking change if you want to support unknown Fields

No, this PR is not a breaking change. You have to explicitly enable the unknown fields, so existing users are not broken. Only the users that choose for unknown fields are broken.

aidandj · 2025-11-21T15:45:15Z

Right. I was saying that enabling unknown fields is a breaking change, and in all other languages it is just implicitly supported. I don't think there is a current way to make this not the case, but I originally was starting a discussion to see if one could be teased out.

caspermeijn · 2025-11-21T15:57:49Z

prost-derive/src/field/unknown.rs

+
+        for attr in attrs {
+            if word_attr("unknown_fields", attr) {
+                set_bool(&mut unknown, "duplicate message attribute")?;


Sorry for the miscommunication, I do agree the error should be here.

The error text doesn't match the error situation. I think it should be something like: "multiple unknown_fields in one message". Do you have an example of what the complete error message that the users sees is?

caspermeijn · 2025-11-21T16:05:03Z

prost-derive/src/lib.rs

    });
+    let merge_fallback = match fields.iter().find(|&(_, f)| f.is_unknown()) {
+        Some((field_ident, field)) => {
+            let merge = field.merge(&prost_path, quote!(value));


It would be great (but not required) if you could find a better way. My suggestion would be to do a single loop for all field and then always do the skip. If unknown fields is enabled, then the skip branch is never triggered. I don't know whether the compiler accepts a double catch all

caspermeijn · 2025-11-21T16:07:32Z

prost/src/unknown.rs

+
+    /// Gets an iterator over the fields contained in this set.
+    pub fn iter(&self) -> impl Iterator<Item = (u32, &UnknownField)> {
+        UnknownFieldIter {


Is it possible to replace the custom iterator by some kind of Iterator::flatten() call? Please let me know if you don't understand what I mean with that.

I tried tinkering with it for a while, but since the second iterator is inside a tuple, I wasn't having much luck. I'll have another go at it this coming Tuesday, see if I can manage something better.

aidandj · 2025-11-21T23:22:38Z

Random thought. Protobuf naming guide recommends not using underscores as the initial character. Although it's not enforced, suggesting or using a default struct field name with a underscore prefix would be much more unlikely to have naming conflicts.

https://protobuf.dev/programming-guides/style/

Blizzard Finnegan added 18 commits September 22, 2025 11:14

Blind copy of changes in PR tokio-rs#574

af54e00

Does not compile, incomplete.

continue fixed32/fixed64 tweaks

f0101e1

- TODO: What endianness is protobuf?

Clarify endianness of unknown fields

d0dac13

Resolve minor testing issues

7b7c95e

Rename derive attribute

7d727f8

Rename test messages to be more clear

7bbaabc

Temporarily comment out section

fe9c170

This section should be added back (and/or modified to fit new standards), but commenting out for now to reduce points of failure to resolve creating bad Copy derives

Modify unknown fields to be Option

89dedaa

Instead of checking against empty, which can fail differently. Attempting to force fields with unknown to not have a derived Copy.

Update code-generator to support Option unknown

cf6dd55

Update name of unknown field in derive builds

387f369

Fix bad derive Copy error

c4d52ba

Messages should be passed to config individually

dd7d163

Finalise commits for first pass

2bb62e5

rustfmt and clippy

1fcbce8

Undo edit to otherwise untouched file

257f260

While this is a change made by rustfmt/clippy, it's outside the scope of this PR, so it's been removed

Resolve CI/clippy issue

5f24936

Seems like local clippy check runs in rust edition 2021, CI runs in 2024

Merge remote-tracking branch 'upstream/master' into feature/unknown_f…

3064aef

…ields_support

blizzardfinnegan changed the title ~~Support for Unknown Fields~~ Support for Unknown Fields (Rebase of #574) Oct 8, 2025

blizzardfinnegan and others added 2 commits October 9, 2025 08:23

Merge branch 'tokio-rs:master' into feature/unknown_fields_support

2c28f1b

Confirm Protobuf unknown field tests pass

81dc2f6

abarke mentioned this pull request Oct 10, 2025

feat: protobuf edition 2023 support #1347

Open

caspermeijn requested changes Nov 14, 2025

View reviewed changes

aidandj mentioned this pull request Nov 20, 2025

Aidan/support unknown fields influxdata/pbjson#151

Open

caspermeijn reviewed Nov 21, 2025

View reviewed changes

blizzardfinnegan and others added 3 commits November 21, 2025 12:32

Merge branch 'tokio-rs:master' into feature/unknown_fields_support

20ca085

Resolve DecodeError CI error msg

0a75b29

rustfmt && clippy

59e57f1

Support for Unknown Fields (Rebase of #574) #1340

Are you sure you want to change the base?

Support for Unknown Fields (Rebase of #574) #1340

Uh oh!

Conversation

blizzardfinnegan commented Sep 24, 2025

Uh oh!

blizzardfinnegan commented Oct 8, 2025

Uh oh!

alebar42 commented Oct 9, 2025

Uh oh!

blizzardfinnegan commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blizzardfinnegan commented Nov 5, 2025

Uh oh!

aidandj commented Nov 14, 2025

Uh oh!

caspermeijn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blizzardfinnegan commented Nov 18, 2025

Uh oh!

blizzardfinnegan commented Nov 18, 2025

Uh oh!

blizzardfinnegan commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aidandj commented Nov 19, 2025

Uh oh!

blizzardfinnegan commented Nov 19, 2025

Uh oh!

aidandj commented Nov 19, 2025

Uh oh!

blizzardfinnegan commented Nov 19, 2025

Uh oh!

aidandj commented Nov 19, 2025

Uh oh!

aidandj commented Nov 19, 2025

Uh oh!

caspermeijn commented Nov 21, 2025

Uh oh!

caspermeijn commented Nov 21, 2025

Uh oh!

aidandj commented Nov 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aidandj commented Nov 21, 2025

Uh oh!

blizzardfinnegan commented Oct 9, 2025 •

edited

Loading

blizzardfinnegan commented Nov 19, 2025 •

edited

Loading