Skip to content

Use buf for proto generation #286

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Oct 16, 2023
Merged

Use buf for proto generation #286

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
34ab77f
Switch docs and scaffolding to buf:
tomhjp Oct 11, 2023
6817bef
buf generate
tomhjp Oct 12, 2023
12f868f
Fix build: embed unimplemented servers for forwards compatibility
tomhjp Oct 11, 2023
2619194
Revert "Fix build: embed unimplemented servers for forwards compatibi…
tomhjp Oct 12, 2023
4853f39
Don't require embedded unimplemented structs
tomhjp Oct 12, 2023
e340950
Use BSR for all proto generation plugins
tomhjp Oct 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions buf.gen.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: MPL-2.0

version: v1
plugins:
- plugin: go
out: .
opt:
- paths=source_relative
- plugin: go-grpc
out: .
opt:
- paths=source_relative

7 changes: 7 additions & 0 deletions buf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: MPL-2.0

version: v1
build:
excludes:
- examples/
4 changes: 1 addition & 3 deletions examples/bidirectional/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,6 @@ If you update the protocol buffers file, you can regenerate the file
using the following command from this directory. You do not need to run
this if you're just trying the example.

For Go:

```sh
$ protoc -I proto/ proto/kv.proto --go_out=plugins=grpc:proto/
$ buf generate
```
14 changes: 14 additions & 0 deletions examples/bidirectional/buf.gen.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: MPL-2.0

version: v1
plugins:
- plugin: go
out: .
opt:
- paths=source_relative
- plugin: go-grpc
out: .
opt:
- paths=source_relative

4 changes: 4 additions & 0 deletions examples/bidirectional/buf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: MPL-2.0

version: v1
1 change: 1 addition & 0 deletions examples/bidirectional/proto/kv.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

syntax = "proto3";
package proto;
option go_package = "./proto";
Copy link

@johanbrandhorst johanbrandhorst Oct 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are somewhat unusual. I might suggest using the "managed mode" feature of Buf's generation instead: https://buf.build/docs/generate/managed-mode. That would mean you can drop this option in all the files and buf would automatically set it to something sensible. It'll almost certainly look something like this:

managed:
  enabled: true
  go_package_prefix:
    default: github.com/hashicorp/go-plugin/examples/bidirectional/proto # go.mod path + proto output path

Copy link
Contributor Author

@tomhjp tomhjp Oct 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to do this but keep the slightly odd grpctest package name the same? I know it's only a test package, but it is technically part of the public API so I'd like to be quite strict about avoiding breaking changes, just to make the safety of the change easy to evaluate.


message GetRequest {
string key = 1;
Expand Down
7 changes: 3 additions & 4 deletions examples/grpc/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
*.pyc
kv
kv-*
kv_*
!kv_*.py
/kv
/kv-*
/kv_*
13 changes: 4 additions & 9 deletions examples/grpc/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,9 @@ $ export KV_PLUGIN="./kv-go-netrpc"
This plugin is written in Python:

```
$ python -m venv plugin-python/.venv
$ source plugin-python/.venv/bin/activate
$ pip install -r plugin-python/requirements.txt
$ export KV_PLUGIN="python plugin-python/plugin.py"
```

Expand All @@ -58,14 +61,6 @@ If you update the protocol buffers file, you can regenerate the file
using the following command from this directory. You do not need to run
this if you're just trying the example.

For Go:

```sh
$ protoc -I proto/ proto/kv.proto --go_out=plugins=grpc:proto/
```

For Python:

```sh
$ python -m grpc_tools.protoc -I ./proto/ --python_out=./plugin-python/ --grpc_python_out=./plugin-python/ ./proto/kv.proto
$ buf generate
```
17 changes: 17 additions & 0 deletions examples/grpc/buf.gen.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: MPL-2.0

version: v1
plugins:
- plugin: go
out: .
opt:
- paths=source_relative
- plugin: go-grpc
out: .
opt:
- paths=source_relative
- plugin: python
out: plugin-python
- plugin: buf.build/grpc/python:v1.58.1
Copy link

@johanbrandhorst johanbrandhorst Oct 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So easy 😍. If you want, we could use remote plugins the same way for the go and go-grpc plugins and drop those tools from this repo. Usually the argument against it is that we don't want to depend on the BSR but since we're doing this for python anyway, why not?

Copy link
Contributor Author

@tomhjp tomhjp Oct 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think that would be a nice win to make it easy to use consistent plugin versions too. Added in e340950.

out: plugin-python
4 changes: 4 additions & 0 deletions examples/grpc/buf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: MPL-2.0

version: v1
1 change: 1 addition & 0 deletions examples/grpc/plugin-python/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/.venv
4 changes: 2 additions & 2 deletions examples/grpc/plugin-python/plugin.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@

import grpc

import kv_pb2
import kv_pb2_grpc
from proto import kv_pb2
from proto import kv_pb2_grpc

from grpc_health.v1.health import HealthServicer
from grpc_health.v1 import health_pb2, health_pb2_grpc
Expand Down
0 examples/grpc/plugin-python/kv_pb2.py → examples/grpc/plugin-python/proto/kv_pb2.py
View file
Open in desktop
File renamed without changes.
0 examples/grpc/plugin-python/kv_pb2_grpc.py → ...s/grpc/plugin-python/proto/kv_pb2_grpc.py
View file
Open in desktop
File renamed without changes.
3 changes: 3 additions & 0 deletions examples/grpc/plugin-python/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
grpcio==1.59.0
grpcio-health-checking==1.59.0
protobuf==4.24.4
1 change: 1 addition & 0 deletions examples/grpc/proto/kv.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

syntax = "proto3";
package proto;
option go_package = "./proto";

message GetRequest {
string key = 1;
Expand Down
6 changes: 0 additions & 6 deletions internal/plugin/gen.go
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion internal/plugin/grpc_broker.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

syntax = "proto3";
package plugin;
option go_package = "plugin";
option go_package = "./plugin";

message ConnInfo {
uint32 service_id = 1;
Expand Down
2 changes: 1 addition & 1 deletion internal/plugin/grpc_controller.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

syntax = "proto3";
package plugin;
option go_package = "plugin";
option go_package = "./plugin";

message Empty {
}
Expand Down
2 changes: 1 addition & 1 deletion internal/plugin/grpc_stdio.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

syntax = "proto3";
package plugin;
option go_package = "plugin";
option go_package = "./plugin";

import "google/protobuf/empty.proto";

Expand Down
6 changes: 0 additions & 6 deletions test/grpc/gen.go
View file
Open in desktop

This file was deleted.

2 changes: 2 additions & 0 deletions test/grpc/test.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ syntax = "proto3";

package grpctest;

option go_package = "./grpctest";

import "google/protobuf/empty.proto";

message TestRequest {
Expand Down