Project Structure best practices with multiple services

[tfmertz]

We have a large application that has 9 different gRPC services and we are currently trying to transfer them into a micro services architecture. Before, all of these proto files were in a single proto/ directory:

service/
  proto/
    foo.proto
    foo.pb.go
    bar.proto
    bar.pb.go
...

Now, we are splitting each service out to run in its own kubernetes pod:

services/
  foo/
    proto/
      foo.proto
      foo.pb.go
  bar/
    proto/
      bar.proto
      bar.pb.go
...

We are running into issues where our foo service returns a message that bundles a message from the bar service, but the foo service knows nothing about any of the bar service's messages.

I have tried importing a bar_shared.proto into the foo.proto file to give it the required message but namespacing issues abound. I have run into issue #67. This makes me think that I'm using the wrong approach.

I wanted to get a reality check on our structure, and if there is a better way to accomplish what we are attempting to do. Should we create a master proto/ folder like before and have everything in the same namespace and include all the compiled pb files in each service?

Or is it bad practice to include the bar service messages inside of a foo service message?

Thanks!

[cybrcodr]

There is nothing wrong with splitting up the proto definitions of the services. That would be how I would do it as well. Do define them in separate proto packages as well.

As for the shared definitions, you can define them in a different proto file somewhere else perhaps under services/shared/. Do also define them in one or more separate proto packages and the services protos can import and reference them. Do make sure that when you generate the Go code for these to use equivalent Go packages (i.e. one to one with proto packages) as well and hence there shouldn't be any collisions. I'd suggest using the go_package option inside the proto definitions to provide the Go full package path. But that will also depend on your build environment if these are automatically generated or not.

Hope that helps.

[tfmertz]

Thanks for the information! I spent some time trying to get the separate proto definitions per service working, but in the end we just went with the one proto package approach. Our plan is to make the proto package its own repo and pull it into each service and compile on build.

This means that each service would have access to every other service's methods and messages. It might come back to bite us in the end, but it makes development a lot easier because all requests / messages are in the same namespace and don't need to be mapped. Plus, if a service needs to talk with another service, it already has access to the other services requests and replies to make the call.

One downside is that we need to be careful about naming because all requests and replies are in the same namespace, so we can't have a GetRequest for each services, we need a GetUserRequest and a GetRoleRequest, etc. It's less sightly, but in the end very explicit.

Another complication is how we will navigate updating proto definitions and having them available for each service, but we'll figure that out when we get there.

[davidhenrygao]

Hi, I'm a newer to golang protobuf.
I have a question:

/proto
    --/common
       --shared.proto
    --/test
      --test.proto

how can I import shared.proto in test.proto?
My %GOPATH% is D:\golang
And dir proto is in E:\golang\src\github.com\xxx\myproject\proto
shared.proto define package common
and test.proto define package test

Thanks!

[davidhenrygao]

I found this question solved at here.
Duplicate of #139