mip: package installation using GitLab URLs for repos in subgroups

[andyrids]

This is more of an awareness piece and something to trigger a discussion. There is also the possibility that I am wrong and the installation process is working as intended.

I'm currently developing MicroPython packages on GitLab, which sit inside a 'Libraries' subgroup, which is inside a parent 'MicroPython IoT Projects' group.

Due to the way that the mip _rewrite_url function parses GitLab and GitHub URLs, the assumption is made that the first component after splitting the URL, is the org and the second is the repository slug, which doesn't account for subgroups.

micropython-lib/micropython/mip/mip/__init__.py

Line 63 in 96bd01e

def _rewrite_url(url, branch=None):

As a result, I had to change the way I reference my packages and package extension urls and deps in the package.json files. My network-utils 'package.json' has to have the following structure:

{
    "urls": [
        ["network_utils/__init__.py", "https://gitlab.com/micropython-iot-projects/libraries/micropython-network-utils/-/raw/HEAD/network-utils/network_utils/__init__.py"]
    ],
    "deps": [
        ["logging", "latest"],
        ["github:josverl/micropython-stubs/mip/typing.mpy", "main"],
        ["github:josverl/micropython-stubs/mip/typing_extensions.mpy", "main"]
    ],
    "version": "0.0.1"
}

In order to install network-utils with mpremote, I would have to use the following command:

mpremote mip install https://gitlab.com/micropython-iot-projects/libraries/micropython-network-utils/-/raw/HEAD/network-utils/package.json

The command below will not work, resulting in a 403 error:

mpremote mip install gitlab:micropython-iot-projects/libraries/micropython-network-utils/network-utils/package.json

The network-utils-mqtt extension package, which requires network-utils, had to have its package.json written like so:

{
    "urls": [
        ["network_utils/mqtt.py", "https://gitlab.com/micropython-iot-projects/libraries/micropython-network-utils/-/raw/HEAD/network-utils-mqtt/network_utils/mqtt.py"]
    ],
    "deps": [
        ["https://gitlab.com/micropython-iot-projects/libraries/micropython-network-utils/-/raw/HEAD/network-utils/",  "develop"]
    ],
    "version": "0.0.1"
}

The URL for the deps, which is the network-utils package this one extends, had to be rewritten so that the package.json could be found correctly.

[agatti]

Ow. This makes things much more complicated than expected.

If the manifest file format is set in stone, the location of the user/project/path splits in the gitlab: identifier can be obtained by using the REST API offered by gitlab (see https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/api/openapi/openapi_v3.yaml) [1].

If altering the manifest format is an option, then extra fields can be used to provide more accurate information (eg. ["https://gitlab.com/micropython-iot-projects/libraries/micropython-network-utils/-/raw/HEAD/network-utils/", "develop"] in your example becomes {"provider": "gitlab", "user": "micropython-iot-project", "repo": "libraries/micropython-network-utils", "name": "micropython-network-utils", "ref": "develop"}), and thus it lets the URL rewriter fill in the same fields as before but with the right values this time.

...or the mip installer could be split into mip, mip-gitlab, mip-<provider>, and so on if there's some special handling required, but maybe that's a bit too much, isn't it?

[1]

The /api/v4/projects/{id} endpoint could be used to check if a project exists by recursively adding path components to the query until either we run out of components or we get a successful response. Eg. case of gitlab:user/group/subgroup1/subgroup2/project@develop, on a 404 on user/group, it'd call the API endpoint with user/group/subgroup1 and if it still gets a 404 it will try user/group/subgroup1/subgroup2, etc. The main issues with this approach, complexity aside, are increased code size and that probably access patterns like that might be seen as suspicious by the server and possibly trigger an IP block/bot verification.

[Josverl]

@andyrids , I actually ran into this with your library as I happened to pick it for an automated test of the different MIP sources.
I initially tried just copied part of the url to construct the Uri for MIP. And only after failing started to read the documentation 😎 where you used the Uri you mention in the OP.

I'm not sure what the seize penalty is of adding just-enough-logic to solve this, withouth changing the package format.

Alternatives if the size is too much are:

• document it, with examples
• create a hosted function/lambda to do the complex lookup, return a http redirect and cache it

[agatti]

Assuming manifest format stays the same, the size penalty for the extra lookups it probably isn't too much for mpremote but it might be on the order of two/three hundreds of bytes minimum for mip. After all we don't care about what the response looks like but whether the query completes with a 200 or not [1], so that avoids having to bring in the json module too.

I'm not the right person to judge the development effort for adding a lambda to do this sort of task, but going that route you add an extra point of failure to the whole lot - which is probably not an issue in this case, but worth keeping in mind - and still require changes to mip anyway.

Still, I'll have a go at adding the extended manifest format to mip and see how much space it takes. Stay tuned.

[1] Unless they operate like a company I used to work for, whose web services often returned "200 Fatal Error" :)

[Josverl]

I don`t know gitlab that well - so below may be overly complex.
A partial solution/workaround seems to be to get GitLab to create redirects for the repo/project by transferring it to a top level group, and then back to wherever you want.
Gitlab is kind enough to generate HTTP redirects on the repo level, which also work for transfers between groups / subgroups.

You will need a unique top-group though

So after transferring it back and forth, you can mip install gitlab:/jv_sample/jv_subgroup/parrot
using

• import mip; mip.install("gitlab:jv_sample/parrot")
• mpremote mip install gitlab:jv_sample/parrot ( Not tested )

The same does not seem to work in dependencies though :-(

• import mip; mip.install("gitlab:jv_sample/petshop") fails as it has a dependency on above parrot
  And I have not found a way to make that work yet - other than coding the full URI

petshop package

{
  "urls": [
    ["petshop.py", "petshop.py"]
  ],
    "deps": [
        ["logging", "latest"],
        ["gitlab:josverl/jv_sample/parrot", "main"],
        ["github:josverl/micropython-stubs/mip/typing_mpy.json", "main"],
    ],
    "working"= [
        ["https://gitlab.com/jv_sample/jv_subgroup/parrot/-/raw/main/package.json?ref_type=heads", ""]
    ],
    "version": "0.1"
}

offtopic : ** note the .../typing_mpy.json that also pulls in the 2nd level dependencies