python: use relative imports in generated modules

[little-dude]

I have a package foo that looks like this:

.
├── data
│   ├── a.proto
│   └── b.proto
└── generated
    ├── a_pb2.py
    ├── b_pb2.py
    └── __init__.py

# a.proto
package foo;

# b.proto
import "a.proto";

package foo;

Generate the code: protoc -I ./data --python_out=generated data/a.proto data/b.proto.
Here is the failure:

Python 3.5.1 (default, Mar  3 2016, 09:29:07) 
[GCC 5.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from generated import b_pb2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/corentih/repro/generated/b_pb2.py", line 16, in <module>
    import a_pb2
ImportError: No module named 'a_pb2'

This is beacuse the generated code looks like this:

import a_pb2

If the import was relative it would actually work:

from . import a_pb2

[goldenbull]

I have the exactly same problem, hope to be fixed

[little-dude]

@goldenbull I submitted a fix, let's see if it makes it through. I'm just not sure: are there cases where we don't want relative imports?

[goldenbull]

@little-dude how about if a_pb2.py is generated into a different folder as b_pb2.py?

[little-dude]

Could you provide a small example of what you're thinking about, so that I try it with my change?

[goldenbull]

.
├── proto
│   ├── a.proto
│   └── b.proto
├── pkg_a
│   ├── a_pb2.py
│   └── __init__.py
└── pkg_b
     ├── b_pb2.py
     └── __init__.py

maybe this is not a good case, I don't have enough knowledge about how protobuf/python/etc. deal with the importing

[little-dude]

I don't think this is actually possible because the generated modules follow the hierarchy of the proto files.
However we could imagine that we have the following:

.
└── data
    ├── a.proto
    ├── b.proto
    └── sub
        ├── c.proto
        └── sub
             └── d.proto

with the following:

# a.proto
package foo;
import "b.proto";
import "sub/c.proto";
import "sub/sub/d.proto";

# b.proto
package foo;
import "sub/c.proto";
import "sub/sub/d.proto";

# sub/c.proto
package foo;
import "sub/d.proto";

# sub/sub/d.proto
package foo;

We generate the code with:

protoc -I data -I data/sub -I data/sub/sub --python_out=generated data/a.proto data/b.proto data/sub/c.proto data/sub/sub/d.proto

which generated the following:

.
└── generated
    ├── a_pb2.py
    ├── b_pb2.py
    └── sub
        ├── c_pb2.py
        └── sub
            └── d_pb2.py

But this is a more complex case than what I am trying to fix.

Edit: I'm not even sure this is a valid case but here is the error I'm getting with the master branch (4c6259b):

In [1]: from generated import a_pb2
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-f28bccc761b6> in <module>()
----> 1 from generated import a_pb2

/home/corentih/repro/generated/a_pb2.py in <module>()
     14 
     15 
---> 16 import b_pb2 as b__pb2
     17 from sub import c_pb2 as sub_dot_c__pb2
     18 from sub.sub import d_pb2 as sub_dot_sub_dot_d__pb2

/home/corentih/repro/generated/b_pb2.py in <module>()
     14 
     15 
---> 16 from sub import c_pb2 as sub_dot_c__pb2
     17 from sub.sub import d_pb2 as sub_dot_sub_dot_d__pb2
     18 

/home/corentih/repro/generated/sub/c_pb2.py in <module>()
     14 
     15 
---> 16 from sub import d_pb2 as sub_dot_d__pb2
     17 
     18 

/home/corentih/repro/generated/sub/sub/d_pb2.py in <module>()
     20   package='foo',
     21   syntax='proto2',
---> 22   serialized_pb=_b('\n\x0fsub/sub/d.proto\x12\x03\x66oo')
     23 )
     24 _sym_db.RegisterFileDescriptor(DESCRIPTOR)

TypeError: __init__() got an unexpected keyword argument 'syntax'

[little-dude]

@haberman is there any chance for this to be fixed before next release? It's quite limiting for python 3.

[drats]

I have exactly the same problem (using protobuf v3beta3), the generated imports do not conform to PEP 328 (finalized in 2004), restated in the Python docs: https://docs.python.org/3/tutorial/modules.html#intra-package-references - the 12 yr-old specification is enforced in Python3, so generated protobufs are unusable without further modification.

[asgoel]

any updates on this?