WIP Add a 'foreign' interface to prost-derive to support scalar encodings defined outside of prost

[bickfordb]

Motivation

Modern webapps make heavy use of values like UUIDs and DateTimes. These are effectively numeric/string/byte scalers. It's important to make these easy to use for productivity. These are missing support in Prost and can be represented as Message structs, but this is awkward to use with libraries like SeaORM and require unnecessary conversions between protobuf-ish structs and SeaORM structs.

1. Due to the rules of the Rust trait system where traits can only be added in the type's package or the trait's package, 3rd party application libraries can't implement prost::Message for types likeuuid::Uuid or chrono::DateTime.
2. UUIDs and DateTimes have many different possible encodings for different purposes. For instance, a UUID is essentially a 128 bit integer which can be encoded most compactly in 16 bytes, but is often encoded as a 36 byte string for presentation layers.
3. This adds an extension mechanism where the encoding definition can be defined and specified via a module instead of via implementing the Message trait

Example

my-web-app/src/entity/account.rs

use prost_derive::Message;
use proto::account;
use sea_orm::entity::prelude::*;
use serde::{Deserialize, Serialize};
use uuid::Uuid;

pub mod codec {
    pub mod uuid {
       // Encode UUIDs as protobuf strings:
        pub mod simple {
            use prost::encoding::string;
            use uuid::Uuid;
            pub fn default() -> Uuid {
                Uuid::nil()
            }
            pub fn encode(tag: u32, value: &Uuid, buf: &mut impl ::prost::bytes::BufMut) {
                let s = value.to_string();
                string::encode(tag, &s, buf)
            }

            pub fn encoded_len(tag: u32, value: &Uuid) -> usize {
                let s = value.to_string();
                string::encoded_len(tag, &s)
            }
            pub fn clear(value: &mut Uuid) {
                *value = Uuid::nil()
            }
            pub fn merge(
                wire_type: ::prost::encoding::WireType,
                value: &mut Uuid,
                buf: &mut impl ::prost::bytes::Buf,
                ctx: ::prost::encoding::DecodeContext,
            ) -> Result<(), prost::DecodeError> {
                let mut s: String = "".to_string();
                string::merge(wire_type, &mut s, buf, ctx)?;
                *value = Uuid::parse_str(&s)
                    .map_err(|_| prost::DecodeError::new("failed to decode uuid string"))?;
                Ok(())
            }
        }
    }
}

#[derive(Message, Clone, Debug, PartialEq, Eq, DeriveEntityModel, Deserialize, Serialize)]
#[sea_orm(table_name = "account")]
pub struct Model {
    #[sea_orm(primary_key, auto_increment = false)]
    #[prost(::codec::uuid::simple, tag = 1)]
    pub id: Uuid,
    #[sea_orm(column_type = "Text")]
    #[prost(string, tag=2)]
    pub name: String,
}

my-web-app/src/account.proto

service Accounts {
    rpc SignUp(SignUpRequest) returns (SignUpResponse) {}
}

message SignUpRequest {
 string name = 1;
 string password = 2;
}

message SignUpResponse {
    Account acccount = 1;
}

message Account { 
   string id = 1 ;
   string name = 2;
}

[caspermeijn]

I think this is a very interesting idea. I had always thought a trait would be required to make something like this work. Prost had an encoding module, the is doc hidden. It implements the encoding for all the types, and you want to reuse that structure for foreign types.

I think we should first clean up and document the encoding module. After that, it would be possible to allow foreign types.

[caspermeijn]

One drawback of the modules is the lack of generics. Therefore, you can't make one encoding module that works for every boxed type.

[Sculas]

I'm also encountering this. I have a Uuid type that's just a newtype around uuid::Uuid, but it also contains some validation functions and other things. I can wrap this with #[sqlx(transparent)] and then it just works, but that's not the case with prost.

Being able to specify my own scalar codec is extremely helpful in cases like these, because right now I need to implement prost::Message myself for each newtype wrapper, and that is then exacerbated by the fact that BytesAdapter is sealed (why?!), which suddenly makes a newtype wrapper go from 2 lines to 46 lines, which as a cherry on top now contains a useless message header for something that should've been a scalar type to begin with.

For my use case, this would also need an extern_path_with_codec function, since that's what I use at the moment for my messages to reference the correct Uuid newtype (since that's defined in Rust, not Protobuf). Happy to PR that if that's okay.

[Sculas]

Also, since the foreign type needs to start with ::, how do you handle cases where a codec is at the root of the crate? (e.g. crate::codec::uuid) This won't work since ::crate tries to look for the crate named crate. It should be fairly simple to fix, just trim the leading colon if the first path element is crate (no actual crate with that name exists, and it's probably reserved anyway).

---

One drawback of the modules is the lack of generics. Therefore, you can't make one encoding module that works for every boxed type.

Doesn't type inference handle that? e.g.

pub fn encode<T: prost::Message>(tag: u32, value: &Box<T>, buf: &mut impl prost::bytes::BufMut) {
    prost::encoding::message::encode(tag, value, buf);
}

pub fn test() {
    let mut buf = Vec::with_capacity(1024);
    
    let box_a = Box::new(1u32);
    let box_b = Box::new("hello".to_string());
    let box_c = Box::new(TestMessage { test: "world".to_string() });
    
    encode(1u32, &box_a, &mut buf);
    encode(2u32, &box_b, &mut buf);
    encode(3u32, &box_c, &mut buf);
}