The submodule federated-query has vanished

[gruuya]

What language are you using?

Rust

What version are you using?

0.3.2-alpha.2

What database are you using?

PostgreSQL, MySQL and SQLite

What dataframe are you using?

N/A

Can you describe your bug?

The federated-query submodule has been deleted or made private (the GH repo 404s: https://github.com/sfu-db/federated-query).

This causes the dependents of connector-x to fail during builds (see below).

What are the steps to reproduce the behavior?

N/A

Database setup if the error only happens on specific data or data type

N/A

Example query / code

N/A

What is the error?

From the failed job:

Updating git repository `[https://github.com/splitgraph/arrow-rs`](https://github.com/splitgraph/arrow-rs%60)
    Updating git submodule `[https://github.com/apache/parquet-testing.git`](https://github.com/apache/parquet-testing.git%60)
    Updating git submodule `[https://github.com/apache/arrow-testing`](https://github.com/apache/arrow-testing%60)
    Updating crates.io index
    Updating git repository `[https://github.com/splitgraph/convergence`](https://github.com/splitgraph/convergence%60)
    Updating git repository `[https://github.com/delta-io/delta-rs`](https://github.com/delta-io/delta-rs%60)
    Updating git repository `[https://github.com/splitgraph/connector-x`](https://github.com/splitgraph/connector-x%60)
    Updating git submodule `[https://github.com/sfu-db/federated-query.git`](https://github.com/sfu-db/federated-query.git%60)
error: failed to get `connectorx` as a dependency of package `datafusion-remote-tables v0.1.0 (/home/runner/work/seafowl/seafowl/datafusion_remote_tables)`

Caused by:
  failed to load source for dependency `connectorx`

Caused by:
  Unable to update https://github.com/splitgraph/connector-x?rev=df8f50b3f53606717407c6677c7c0c2cbcc7f6ce#df8f50b3

Caused by:
  failed to update submodule `federated-query`

Caused by:
  failed to fetch submodule `federated-query` from https://github.com/sfu-db/federated-query.git

Caused by:
  failed to authenticate when downloading repository

  * attempted to find username/password via git's `credential.helper` support, but failed

  if the git CLI succeeds then `net.git-fetch-with-cli` may help here
  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

Caused by:
  failed to acquire username/password from local configuration

(The connector-x fork there contains only the Datafusion 19 upgrade and nothing else.)

[wangxiaoying]

Hi @gruuya , may I ask how do you build connectorx? You don't need the federated-query submodule to build connectorx since we have its compiled jar file here.

[gruuya]

Oh sure, I did a poor job of explaining our setup.

Basically, one of Seafowl's features, called remote tables, is built on connector-x.

This is facilitated through a crate called datafusion-remote-tables, which belongs to the Seafowl repo workspace. It is pointed at our fork of connector-x, which we only use to upgrade to the new Datafusion version every couple of weeks.

That git dependency however poses a problem now that federated-query is inaccessible, since cargo will attempt to checkout the submodules too. You can check it out for yourself; if you git clone seafowl repo and try to build it you should see the same output as above. (not anymore, as I've went with the first solution below in our connector-x fork so the build now works)

I think there are a couple of easy solutions:

• Add update=none to .gitmodules for federated-query, and cargo should skip it.
• If federated-query has actually been deleted, or is being included via the jar file, then perhaps the submodule can be deleted entirely?

[wangxiaoying]

@gruuya , thanks for the explanation, it makes sense. I just removed the submodule from this repo. Please let me know if I missed anything.

[gruuya]

Great, that will work!

Note, that there are still a couple of obsolete references to federated-query in .gitignore and Justfile, but that won't pose a problem with builds so I'm closing this. Thank you!