Request for help - Typescript, Protocol Buffers and the Browser

[Brahmasmi]

Landed up here from microsoft/TypeScript#9509 (comment).

Could the clutz team please document an end to end workflow for using protocol buffers on the browser side using typescript within Angular?

Based on @evmar 's comment referenced earlier, there is a workflow to achieve the above within Google. But, just like many others on that thread have discovered, it is not very obvious and/or feasible to folks who may not be very proficient with Closure Compiler and/or Protocol Buffers.

Thanks.

[Brahmasmi]

Well, further searching landed me back on protocolbuffers/protobuf#420.

@lhecker is correct in that thread - there is significant growth in Angular usage, and with gRPC-web, more and more folks will land up here.

@rkirov - Please consider this to be the narrowly scoped request for documentation, as mentioned by you in protocolbuffers/protobuf#420.

[rkirov]

Thanks, I think this is fair narrowing down:

The caveats are:

1. when you say protocol buffers for browser, we can only support docs for the google implementation https://github.com/google/protobuf/tree/master/js vs the competing ones like https://github.com/dcodeIO/ProtoBuf.js/. I don't know why the google one doesn't have a high-level tutorial here https://developers.google.com/protocol-buffers/docs/tutorials, but we will not be able to provide much info on using protoc, beyond what is needed to get some output that is usable by clutz.

2. any integration with a build system is beyond the scope. At some point bazel will have enough functionality to do that, but we are not ready with TypeScript for bazel yet.

[Brahmasmi]

Thanks Rado.

1. I understand that the docs would support the Google protobuf implementation. That was also my intention with this request. I am not sure what implementation gRPC-web is using, but I would presume it to be using the official Google implementation. As regards to the absence of a Javascript tutorial, I have filed an issue to track that on the protobuf repo https://github.com/google/protobuf/issues/3021.

2. I hope when bazel + Typescript is good to go, we may be able to revisit integration with a build system. In case you want me to open a separate request for tracking that, please let me know.

[Brahmasmi]

@rkirov Any luck/progress on this request for documentation? It would help a lot.

Thanks.

[Brahmasmi]

@rkirov friendly ping a.k.a. polite reminder.

[whymarrh]

@Brahmasmi this seems to be possible with the current state of the tools. I've managed to get TypeScript definitions for protocol buffer schemas that I've created using Clutz (via the CLI) and a bit of figuring-out. (The repository I managed to get this working in is open-source, so you can take a look at it if you want: LakeMaps/schemas.)

The gist* of the solution is:

1. Produce JavaScript output using protoc

   • The google/protobuf JS source code has setup instructions. You'll want Closure imports, not CommonJS.

2. Produce TypeScript definitions for the JS outputted by protoc

   • The command is a bit involved, but trial-and-error led me to:

     clutz/build/install/clutz/bin/clutz protoc-output.js -o clutz-output.d.ts \
         --externs \
             closure-compiler/externs/es3.js \
             closure-compiler/externs/es5.js \
             closure-compiler/externs/es6.js \
             closure-library/closure/goog/base.js \
             closure-library/closure/goog/array/array.js \
             closure-library/closure/goog/asserts/asserts.js \
             closure-library/closure/goog/crypt/base64.js \
             closure-library/closure/goog/debug/error.js \
             closure-library/closure/goog/dom/nodetype.js \
             closure-library/closure/goog/labs/useragent/browser.js \
             closure-library/closure/goog/labs/useragent/engine.js \
             closure-library/closure/goog/labs/useragent/platform.js \
             closure-library/closure/goog/labs/useragent/util.js \
             closure-library/closure/goog/object/object.js \
             closure-library/closure/goog/reflect/reflect.js \
             closure-library/closure/goog/string/string.js \
             closure-library/closure/goog/useragent/product.js \
             closure-library/closure/goog/useragent/useragent.js \
             protobuf/js/map.js \
             protobuf/js/message.js \
             protobuf/js/binary/arith.js \
             protobuf/js/binary/constants.js \
             protobuf/js/binary/decoder.js \
             protobuf/js/binary/encoder.js \
             protobuf/js/binary/reader.js \
             protobuf/js/binary/utils.js \
             protobuf/js/binary/writer.js
     

     (You'll need to have the Closure Compiler, Closure Library, and the protobuf source code lying around in the CWD for those paths to resolve.)

3. Clutz's outputted clutz-output.d.ts file needed a bit of touching up for me to get it have the correct package name, so I created a new .d.ts file with a new module declaration that exported the private clutz stuff:

    declare module 'desired-package-name' {
        export = ಠ_ಠ.clutz.proto;
    }
   

   And of course where you're referencing contents of other files you'll need to add those comments, so the file looks more like:

    /// <reference path="closure.lib.d.ts" />
    /// <reference path="clutz-output.d.ts" />
    declare module 'desired-package-name' {
        export = ಠ_ಠ.clutz.proto;
    }
   

* I needed a few extra things to get everything to work just right, like adding --skipEmitRegExp to the Clutz invocation. I make no guarantees that these instructions are complete or that they work—I can only note that I managed to get definitions for the few files I had.

I don't know how you want to involve Angular here, but getting TypeScript definitions should work with any TypeScript project including Angular.

Edit: please don't misconstrue the Contributor note on this comment as anything official, I'm not part of the project team.

[Brahmasmi]

Thank you @whymarrh for the very detailed steps. As a rookie, I cannot thank you enough for the very relevant guidance - Closure over CommonJS, --skipEmitRegExp et al.

I will follow these steps and report back on how well it works.

[rkirov]

Sorry for the radio silence here. Basically I wrote up something like #501 (comment) and decided that the experience is so bad for someone unfamiliar with Closure compiler that it is not worth it.

I rather recommend using something like https://github.com/dcodeIO/ProtoBuf.js/ and skipping clutz althogether.