🚸 Always import the entire public API & enable switching instances in the same Python session

[falexwolf]

Behavior if no instance is configured:

>>> import lamindb as ln
! not connected, call: ln.connect('account/name')
>>> ln.track()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/lamindb/core/_context.py", line 355, in _track
    instance_settings = ln_setup.settings.instance
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_settings.py", line 152, in instance
    self._instance_settings = load_instance_settings()
                              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/sub/lamindb-setup/lamindb_setup/core/_settings_load.py", line 42, in load_instance_settings
    raise CurrentInstanceNotConfigured
lamindb_setup.errors.CurrentInstanceNotConfigured: No instance is connected! Call
- CLI:     lamin connect / lamin init
- Python:  ln.connect()  / ln.setup.init()
- R:       ln$connect()  / ln$setup$init()

No solution yet for when the user attempts queries -- challenge is that we need to pass a valid db connection string to Django and then Django would need to be intercepted later

>>> ln.Artifact.df() Traceback (most recent call last): File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/django/db/backends/utils.py", line 105, in _execute return self.cursor.execute(sql, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/django/db/backends/sqlite3/base.py", line 354, in execute return super().execute(query, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sqlite3.OperationalError: no such table: lamindb_artifact

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "", line 1, in
File "/Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/lamindb/models/sqlrecord.py", line 505, in df
return query_set[:limit].df(include=include, features=features)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/falexwolf/repos/laminhub/rest-hub/sub/lamindb/lamindb/models/query_set.py", line 650, in df
df = pd.DataFrame(queryset.values(*field_names, *list(annotate_kwargs.keys())))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/pandas/core/frame.py", line 843, in init
data = list(data)
^^^^^^^^^^
File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/django/db/models/query.py", line 381, in iter
self._fetch_all()
File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/django/db/models/query.py", line 1909, in _fetch_all
self._result_cache = list(self._iterable_class(self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/django/db/models/query.py", line 213, in iter
for row in compiler.results_iter(
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1531, in results_iter
results = self.execute_sql(
^^^^^^^^^^^^^^^^^
File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1580, in execute_sql
cursor.execute(sql, params)
File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/django/db/backends/utils.py", line 79, in execute
return self._execute_with_wrappers(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/django/db/backends/utils.py", line 100, in _execute
with self.db.wrap_database_errors:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/django/db/utils.py", line 91, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/django/db/backends/utils.py", line 105, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/django/db/backends/sqlite3/base.py", line 354, in execute
return super().execute(query, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.OperationalError: no such table: lamindb_artifact

Behavior upon trying to toggle auto-connect:

% lamin settings set auto-connect false
! setting auto_connect to `False` no longer has an effect and the setting will likely be removed in the future; since lamindb 1.7, auto_connect `True` no longer clashes with connecting in a Python session

Credit goes to @Koncopd through the below change that enables switching instances in the same python session:

• ✨ Allow re-connecting through re-configuring Django in the same Python process lamindb-setup#1046

The other important change that enabled removing the gated models import is here:

• 🚸 Set up Django on every import of lamindb via a "mock instance" lamindb-setup#1063

This enables a few simplifications in the CLI:

• ♻️ Remove auto-connect switches lamin-cli#138

Supersedes:

• 🚸 Lazy import proxy for lamindb.models #2849
• 🚸 Always import the entire user API #2850

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.58%. Comparing base (8265549) to head (b2fd95b).
Report is 51 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2851      +/-   ##
==========================================
- Coverage   91.84%   91.58%   -0.27%     
==========================================
  Files          69       67       -2     
  Lines       10807     9448    -1359     
==========================================
- Hits         9926     8653    -1273     
+ Misses        881      795      -86     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[falexwolf]

Ok, now tests are working.

[github-actions]

Deployment URL: https://1623a0f9.lamindb.pages.dev

[Koncopd]

Why is it commented out?

[falexwolf]

There indeed was an issue here that I hadn't yet caught. Now dealt with here: laminlabs/lamindb-setup@7523b97

This is now maybe the remaining issue: some modules that might import lamindb somewhere. But we'll gather some experience for this and I think it's less bad than the constant errors you obtain when you try to normally develop with lamindb without all these changes of this PR.