Heads up: pathlib.VirtualPath may be coming soon

[barneygale]

I've logged a CPython PR that adds a private pathlib._VirtualPath class:

python/cpython#106337

If/when that lands, it's not much more work to drop the underscore and introduce pathlib.VirtualPath to the world. This would be Python 3.13 at the earliest.

Would it be suitable for use in pathy? It would be great to hear your feedback!

[justindujardin]

Any public API that Pathy could hang off of would be very useful. Please feel free to update this issue when there is a build version that I can use.

[barneygale]

Hello - I've published CPython's private PathBase class in a PyPI package: https://pypi.org/project/pathlib-abc/0.1.0/. No docs yet but I should have them ready soon.

If the PyPI package succeeds and matures, I'm hopeful we can make PathBase public in Python itself. Let me know if you have any problems/questions? Thanks!

[justindujardin]

I was about to resort to doing something similar to this package but baked into Pathy, so you saved me a lot of time and duplication. It's especially nice knowing this might feed back into Python and that this package could serve as a fallback after that.

I will spend some time today looking into integrating this with Pathy and ask questions as they arise.

Thanks for publishing a package that contains this functionality 🙇

[justindujardin]

@barneygale, So far, so good; there are some things I have to refactor to account for the pathlib_abc library not providing a concrete file system implementation, but I think it will ultimately simplify things.

The main question that's come up is whether you plan to add type hints or would be willing to accept a PR to add them. I haven't delved deep into why, but mypy/Pylance both dislike the pathlib_abc module here and there, warning about Unknown base class types and improper subclass signatures. I can add type ignores or reimplement the properties to add signatures, but that feels kind of icky.

Here are some examples of what I mean

PathBase.root is partially known: str | bytes | Unknown

Overridden methods have an implicit return type that conflicts with the concrete implementation expectations

The typing bits aside, there are no blockers so far; I should be able to open a PR soon.

[justindujardin]

I'm adding support for __lt__ to support sorted, and I notice the pathlib class has "_cparts" cached property that's used in the < operator:

    @property
    def _cparts(self):
        # Cached casefolded parts, for hashing and comparison
        try:
            return self._cached_cparts
        except AttributeError:
            self._cached_cparts = self._flavour.casefold_parts(self._parts)
            return self._cached_cparts

Can you tell me anything about the significance of this cache? I'm all for caches when needed, but that you've left it out of the ABC library makes me wonder how much it contributes? Also, I see that you've cached most properties in the ABC classes, so I wonder if there's maybe a simpler solution you had in mind with comparable performance using the existing cached properties?

[barneygale]

It's only important in pathlib because Windows paths need to perform case-insensitive comparisons, and even then I'm not sure if it has a huge effect!

FWIW I've just removed most/all caching in the ABCs. PyPI release coming up...

[justindujardin]

One troubling thing I found is that Pathy will no longer inherit from pathlib.Path while using this library. This comes with some negative side-effects, like I noted in another issue, where a serialization library that spaCy uses has a function to coerce strings to Path objects if they're not already that type:

def force_path(location, require_exists=True):
    if not isinstance(location, Path):
        location = Path(location)
    if require_exists and not location.exists():
        raise ValueError(f"Can't read file: {location}")
    return location

Pathy objects used to make it through unscathed, but now they get corrupted and transformed into Path objects that don't work.

Pathy isn't so widely used that it will be a significant problem, but it's something to keep in mind. Do you see any workarounds I might be missing that could keep this compatibility?

[barneygale]

That sounds like a bug in spacy, judging by this bit of their CONTRIBUTING.md:

Code that interacts with the file-system should accept objects that follow the
pathlib.Path API, without assuming that the object inherits from pathlib.Path.
If the function is user-facing and takes a path as an argument, it should check
whether the path is provided as a string. Strings should be converted to
pathlib.Path objects.

(edit: sorry, just read your comment on the issue and realised you're ahead of me!)

They might be interested in depending on pathlib_abc - I'll reach out.

[justindujardin]

Yeah, it was just an example of the type of side-effect that could happen in other libraries. They have a more "correct" implementation in spaCy directly, i.e.

def ensure_path(path: Any) -> Any:
    """Ensure string is converted to a Path.

    path (Any): Anything. If string, it's converted to Path.
    RETURNS: Path or original argument.
    """
    if isinstance(path, str):
        return Path(path)
    else:
        return path

So this is probably a bug in srsly and how it interacts with spaCy.

[barneygale]

With pathlib_abc that method could be implemented as follows:

from pathlib_abc import PathBase
from pathlib import Path

def ensure_path(path: PathBase | os.PathLike) -> PathBase:
    if not isinstance(path, PathBase):
        path = Path(path)
    return path

But srsly seems to be a low-dependencies affair, so it could instead adopt spaCy's string check behaviour.

[barneygale]

In general I don't think it's correct to inherit from pathlib.Path unless your object represents a local filesystem path. That's sort-of the whole point of Path.

[justindujardin]

In general I don't think it's correct to inherit from pathlib.Path unless your object represents a local filesystem path.

I agree with you on that. I always felt awkward using internals and inheriting behavior that didn't apply to my use-case. Still, sometimes we work with solutions that aren't ideal because they're easily grasped.

Moving to pathlib_abc feels much cleaner. It will be nice not to refactor Pathy again next year and the year after with each new Python release because the pathlib internals were refactored. 😅