Use of importlib_metadata breaks jsonschema usage in AWS Lambda

[BAllbeeEplexity]

While trying to use this library (through python-jsonschema-objects, to generate classes) for a project running in an AWS Lambda, the attempt to retrieve the package __version__ using

__version__ = metadata.version("jsonschema")

raises a PackageNotFoundError — importlib_metadata cannot be found, despite it being installed.

I have worked around this by simply replacing that dunder-value with the actual version-value retrieved ('3.2.0' in this case), but thought it might be worth bringing to your attention, particularly since the __version__ should be a constant for any given release, and there's no obvious benefit to calculating it vs. using any of several other options that aren't beholden to yet another library. ;-)

[Julian]

Hi. Sorry you're having trouble.

This is a duplicate of #584 (and relatedly #614).

For which my position is still essentially "these are unsupported environments at the minute due to lack of CI, and more specifically that this particular issue is a bug in Lambda".

Happy to hear further thoughts, ideally in #584, but yeah I don't see this changing unfortunately.

[BAllbeeEplexity]

Well, since I have a relatively simple workaround, I won't pursue too much further, then, but I'd disagree that it's the same issue, or even an unsupported environment, if only because all the requisite packages are present (see screencap). Yes, it takes building a deployment ZIP, but I can take that same zip, drop it on any machine that the individual libraries run on, and it works just fine.

The fact that changing that one line, hardcoding the version-number instead of using a library that may not work under Lambda, allows at least the basic functionality to work, would indicate that the problem's not in this library directly, but in using importlib_metadata to retrieve the package version from…wherever.

Thank you for the quick reply, at any rate. :-)

[Julian]

importlib_metadata retrieves installed package metadata. Normal Python deployments install packages. If you don't install a package, you can't retrieve its metadata. Lambda seems to have decided not to install packages from what users report. That's not supported. Other packages besides jsonschema will break (for other reasons with the same root cause. In your zip example, for reasons like compiling a package with C extension for the wrong platform). And yeah just any environment that doesn't run in CI is not supported. You'll note your issue is specific to Lambda, so it's certainly different :). Surely you can understand not supporting environments that cannot be guaranteed to continue to function because they're not being tested.

…

On Tue, Nov 19, 2019, 17:24 BAllbeeEplexity ***@***.***> wrote: Well, since I have a relatively simple workaround, I won't pursue too much further, then, but I'd disagree that it's the same issue, or even an unsupported environment, if only because all the requisite packages are present (see screencap). Yes, it takes building a deployment ZIP, but I can take that same zip, drop it on any machine that the individual libraries run on, and it works just fine. The fact that changing that one line, hardcoding the version-number instead of using a library that may not work under Lambda, allows at least the basic functionality to work, would indicate that the problem's not in this library directly, but in using importlib_metadata to retrieve the package version from…wherever. Thank you for the quick reply, at any rate. :-) [image: Screenshot from 2019-11-19 15-19-38] <https://user-images.githubusercontent.com/26661644/69191679-2f419280-0ae0-11ea-9e16-f4bc2c037584.png> — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#628>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACQQXQZDNPYJHSWHDPS2UDQURRRJANCNFSM4JPJRIIQ> .

[BAllbeeEplexity]

Surely you can understand not supporting environments that cannot be guaranteed to continue to function because they're not being tested.

Sure. Don't get me wrong, I don't disagree, philosophically, with that decision — I don't have enough insight into the behind-the-scenes efforts and processes to have more than a vague and perhaps unfounded opinion, and one that's driven by immediate need at that. :-)

Since I've got a workaround — basically just copying the source-of-truth version from the METADATA file into the code in place of the only call to the real problem package that I can find anywhere in jsonschema — which takes care of it, I'm not too concerned, at least for now. I'm intrigued (and slightly amused) that the decision for the where that source-of-truth version comes from is the only thing preventing this package from being usable in Lambda, though. :-D

Again, thanks!

[Julian]

Cool! Glad your workaround is working, happy to hear that certainly!

Hopefully anyone else running into this finds it, so definitely appreciate filing the issue.

[BAllbeeEplexity]

So far, at least. :-) No worries!

I may write it up in more detail, and could share it here if there'd be any interest.

[freekwiekmeijer]

In case the AWS Lambda is deployed using Serverless framework, there is a very simple workaround: http://biercoff.com/how-to-fix-package-not-found-error-importlib-metadata/

I reproduced the problem (Serverless deployment of python code with jsonschema 3.2.0 to AWS Lambda). The validated the workaround (custom.pythonRequirements.slim: true -> false). Works indeed.

[AlexThomas90210]

@BAllbeeEplexity Can you share your fix? I have having the same issue in a lambda function.

@freekwiekmeijer solution is not working for me, maybe because I am putting jsonschema inside a lambda layer

[BAllbeeEplexity]

I have two that I can share. :-)

The first is quicker and easier, but is not a stable fix — It'd have to be re-applied any time there was an update to this package. It basically boils down to commenting out the last line in jsonschema.__init__.py like so:

# __version__ = metadata.version("jsonschema")

The better solution in my case may or may not work for you, but might be adaptable. I'm (for now manually) building a Lambda deployment-package (a ZIP file) after each dev-test cycle, and manually deploying it in the AWS Lambda console. I've attached a stripped-down copy of the (kludgy? Brute-force?) script I'm using to do that, which basically:

• Reads the requirements.txt file,
• Finds the code- and metadata-directories and files in the pipenv environment for the project for each package in requirements.txt (dealing with known packages where those directory-names don't match the package-name — e.g.,
  • attrs => attr,
  • python-jsonschema-objects => python_jsonschema_objects)
• Adds them to the zip-archive

Attaching a screen-cap of the relevant example (with customer-sensitive information blacked out, sorry, ;-) ). The various "S.json" items in that screen-cap are JSON schema definitions that the code uses.

Ultimately, as far as I can tell, as long as you get the code- and metadata-directories both in the Lambda package, it will work. I presume, though I haven't tested it, that a variation of the same process would work in a Lambda Layer.

Attachments:
generic-lambda-package-builder.txt (GitHub won't allow attachment of Python files to comments? Wow. O.o )
example-deployment-package