JP2Grok: support native Grok decompression of vsicurl files

[boxerab]

What does this PR do?

@rouault will enjoy this minimal PR : allow JP2Grok driver to use native Grok decompression for vsicurl files

AI tool usage

• AI supported my development of this PR.

Tasklist

• Make sure code is correctly formatted (cf pre-commit configuration)
• Add test case(s)
• Review
• Adjust for comments

[rouault]

This needs to be properly rebased on top of latest master to solve merge conflict

I do believe that the network integration is fragile, and could cause headaches to users that use a GDAL networking option (or S3 authentication scheme) that isn't translated into Grok. I would certainly have strong reservations if Grok was a default driver.

[rouault]

Aaron, I'm giving up. I'll stop reviewing your PRs. I just can't psychology deal with being at the receiving end of LLM generated PRs. I don't know how to explain it, but it makes me mentally very uncomfortable, and it is exhausting. If another maintainer feels up for the task, fine to them.

[aaron-boxer]

Aaron, I'm giving up. I'll stop reviewing your PRs. I just can't psychology deal with being at the receiving end of LLM generated PRs. I don't know how to explain it, but it makes me mentally very uncomfortable, and it is exhausting. If another maintainer feels up for the task, fine to them.

No problem, I understand. I will not be making any more AI assisted PRs for this driver.
How about we wait a few weeks and then merge this one as it is already written with passing tests ?

[boxerab]

Hello again @rouault , hope all is well. Three weeks have passed since opening this relatively small PR - half of the changes are unit tests, all passing. The feature is a useful one IMHO as it allows Grok to efficiently handle more network files, and a conservative approach is taken for different auth options - those unsupported by Grok will trigger GDAL VSILFILE handling via callback.

Please consider accepting this one. 🙏

[rouault]

I'm not convinced that letting Grok handle I/O is the way to go. It will ultimately lead to issues similar to #14508 where there's a discrepancy between the library re-implementation of the GDAL protocol.
Maybe the thing to explore would be to identify what would be missing in GDAL VSIVirtualHandle , if anything is missing, and improve the brige with Grok I/O layer

[boxerab]

I'm not convinced that letting Grok handle I/O is the way to go. It will ultimately lead to issues similar to #14508 where there's a discrepancy between the library re-implementation of the GDAL protocol. Maybe the thing to explore would be to identify what would be missing in GDAL VSIVirtualHandle , if anything is missing, and improve the brige with Grok I/O layer

agree, we don't want to have more issues like #14508.

The way IO currently works is that GDAL manages authentication, and then passes the url and any auth information like custom headers to Grok, which then reads the file. In the case of network files, it uses libcurl directly in a way that is customized for J2K file format, making use of the main header and TLM markers if present to very efficiently pull the data, using range requests for tiles. It's not re-implementing the GDAL protocol, which is more general.

Replicating the Grok https fetcher in GDAL would be a lot of work and would only be suitable for one format : J2K. Not sure this is worth it, IMHO.

[rouault]

if present to very efficiently pull the data, using range requests for tiles

That's vague. Concretely, what type of calls is missing in VSIVirtualHandle to implement efficient reading from grok ?