✨ Enable executing notebooks via `jupyter nbconvert --execute`

[falexwolf]

According to ChatGPT4 and Stackoverflow, this is the most established way of executing notebooks.

I attempted to use it in our lamin-cli tests by adding these lines in out test_save_notebooks.py:

    # try re-running - it fails
    result = subprocess.run(
        f"jupyter nbconvert --to html --execute {notebook_path}",
        shell=True,
        capture_output=True,
        env=env,
    )
    print(result.stdout.decode())
    print(result.stderr.decode())
    assert result.returncode == 1
    assert (
        "Call ln.track() and copy/paste the output into the notebook"
        in result.stderr.decode()
    )

It fails:

[NbConvertApp] Converting notebook /Users/falexwolf/repos/laminapp-ui/rest-app/sub/lamindb/sub/lamin-cli/tests/notebooks/with-title-and-initialized-consecutive.ipynb to html
/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
Traceback (most recent call last):
  File "/Users/falexwolf/miniconda3/envs/py310/bin/jupyter-nbconvert", line 8, in <module>
    sys.exit(main())
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/jupyter_core/application.py", line 282, in launch_instance
    super().launch_instance(argv=argv, **kwargs)
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/traitlets/config/application.py", line 1046, in launch_instance
    app.start()
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/nbconvertapp.py", line 410, in start
    self.convert_notebooks()
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/nbconvertapp.py", line 585, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/nbconvertapp.py", line 551, in convert_single_notebook
    output, resources = self.export_single_notebook(
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/nbconvertapp.py", line 477, in export_single_notebook
    output, resources = self.exporter.from_filename(
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/exporters/templateexporter.py", line 389, in from_filename
    return super().from_filename(filename, resources, **kw)  # type:ignore
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/exporters/exporter.py", line 201, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/exporters/templateexporter.py", line 395, in from_file
    return super().from_file(file_stream, resources, **kw)  # type:ignore
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/exporters/exporter.py", line 220, in from_file
    return self.from_notebook_node(
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/exporters/html.py", line 260, in from_notebook_node
    html, resources = super().from_notebook_node(nb, resources, **kw)
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/exporters/templateexporter.py", line 411, in from_notebook_node
    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/exporters/exporter.py", line 154, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/exporters/exporter.py", line 352, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/preprocessors/base.py", line 48, in __call__
    return self.preprocess(nb, resources)
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/preprocessors/execute.py", line 100, in preprocess
    self.preprocess_cell(cell, resources, index)
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbconvert/preprocessors/execute.py", line 121, in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/jupyter_core/utils/__init__.py", line 168, in wrapped
    return loop.run_until_complete(inner)
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbclient/client.py", line 1058, in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
  File "/Users/falexwolf/miniconda3/envs/py310/lib/python3.10/site-packages/nbclient/client.py", line 914, in _check_raise_for_error
    raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
ln.settings.transform.stem_uid = "hlsFXswrJjtt"
ln.settings.transform.version = "1"
ln.track()
------------------

----- stdout -----
❗ Can not find the notebook in any server session or by using other methods.
------------------

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[2], line 3
      1 ln.settings.transform.stem_uid = "hlsFXswrJjtt"
      2 ln.settings.transform.version = "1"
----> 3 ln.track()

File ~/repos/laminapp-ui/rest-app/sub/lamindb/lamindb/core/_run_context.py:245, in run_context._track(cls, transform, new_run, path)
    241 transform = Transform.filter(
    242     uid__startswith=stem_uid, version=version
    243 ).one_or_none()
    244 if is_run_from_ipython:
--> 245     key, name = cls._track_notebook(path=path)
    246     transform_type = TransformType.notebook
    247     transform_ref = None

File ~/repos/laminapp-ui/rest-app/sub/lamindb/lamindb/core/_run_context.py:351, in run_context._track_notebook(cls, path)
    344 @classmethod
    345 def _track_notebook(
    346     cls,
    347     *,
    348     path: str | None,
    349 ):
    350     if path is None:
--> 351         path = get_notebook_path()
    352     key = Path(path).stem
    353     if isinstance(path, (Path, PurePath)):

File ~/repos/laminapp-ui/rest-app/sub/lamindb/lamindb/core/_run_context.py:119, in get_notebook_path()
    117     raise RuntimeError(msg_path_failed) from None
    118 if path is None:
--> 119     raise RuntimeError(msg_path_failed) from None
    120 return path

RuntimeError: failed to infer notebook path.

This is the PR: laminlabs/lamin-cli#50

Not urgent, but something for the future, @Koncopd.

[falexwolf]

It gives me the following traceback.

Traceback

(py312) falexwolf@mbpalex adhoc % jupyter nbconvert --to notebook --execute test-nbconvert.ipynb
[NbConvertApp] Converting notebook test-nbconvert.ipynb to notebook
Traceback (most recent call last):
  File "/Users/falexwolf/miniconda3/envs/py312/bin/jupyter-nbconvert", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/jupyter_core/application.py", line 283, in launch_instance
    super().launch_instance(argv=argv, **kwargs)
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/traitlets/config/application.py", line 1075, in launch_instance
    app.start()
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbconvert/nbconvertapp.py", line 420, in start
    self.convert_notebooks()
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbconvert/nbconvertapp.py", line 597, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbconvert/nbconvertapp.py", line 563, in convert_single_notebook
    output, resources = self.export_single_notebook(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbconvert/nbconvertapp.py", line 487, in export_single_notebook
    output, resources = self.exporter.from_filename(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbconvert/exporters/exporter.py", line 201, in from_filename
    return self.from_file(f, resources=resources, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbconvert/exporters/exporter.py", line 220, in from_file
    return self.from_notebook_node(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbconvert/exporters/notebook.py", line 36, in from_notebook_node
    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbconvert/exporters/exporter.py", line 154, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbconvert/exporters/exporter.py", line 353, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbconvert/preprocessors/base.py", line 48, in __call__
    return self.preprocess(nb, resources)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbconvert/preprocessors/execute.py", line 103, in preprocess
    self.preprocess_cell(cell, resources, index)
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbconvert/preprocessors/execute.py", line 124, in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/jupyter_core/utils/__init__.py", line 165, in wrapped
    return loop.run_until_complete(inner)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/asyncio/base_events.py", line 686, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbclient/client.py", line 1062, in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbclient/client.py", line 918, in _check_raise_for_error
    raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
import lamindb as ln

ln.track()
------------------

----- stdout -----
→ connected lamindb: falexwolf/docsbuild
----- stdout -----
! Can not find the notebook in any server session or by using other methods.
----- stdout -----
! Consider installing ipylab (pip install ipylab) if you use jupyter lab.
------------------

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[1], line 3
      1 import lamindb as ln
----> 3 ln.track()

File ~/repos/laminhub/rest-hub/sub/lamindb/lamindb/core/_context.py:345, in Context.track(self, transform, project, space, params, new_run, path)
    343 description = None
    344 if is_run_from_ipython:
--> 345     self._path, description = self._track_notebook(path_str=path)
    346     transform_type = "notebook"
    347     transform_ref = None

File ~/repos/laminhub/rest-hub/sub/lamindb/lamindb/core/_context.py:475, in Context._track_notebook(self, path_str)
    469 def _track_notebook(
    470     self,
    471     *,
    472     path_str: str | None,
    473 ) -> tuple[Path, str | None]:
    474     if path_str is None:
--> 475         path = get_notebook_path()
    476     else:
    477         path = Path(path_str)

File ~/repos/laminhub/rest-hub/sub/lamindb/lamindb/core/_context.py:68, in get_notebook_path()
     66     raise RuntimeError(msg_path_failed) from None
     67 if path is None:
---> 68     raise RuntimeError(msg_path_failed) from None
     69 return Path(path)

RuntimeError: failed to infer notebook path.
fix: pass `path` to `ln.track()`

[falexwolf]

Also the following fails with the same traceback.

jupyter execute test.ipynb

Traceback

% jupyter execute test-nbconvert.ipynb 
[NbClientApp] Executing test-nbconvert.ipynb
[NbClientApp] Executing notebook with kernel: 
Traceback (most recent call last):
  File "/Users/falexwolf/miniconda3/envs/py312/bin/jupyter-execute", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/jupyter_core/application.py", line 283, in launch_instance
    super().launch_instance(argv=argv, **kwargs)
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/traitlets/config/application.py", line 1074, in launch_instance
    app.initialize(argv)
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/traitlets/config/application.py", line 118, in inner
    return method(app, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbclient/cli.py", line 161, in initialize
    self.run_notebook(path)
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbclient/cli.py", line 215, in run_notebook
    client.execute()
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/jupyter_core/utils/__init__.py", line 165, in wrapped
    return loop.run_until_complete(inner)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/asyncio/base_events.py", line 686, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbclient/client.py", line 709, in async_execute
    await self.async_execute_cell(
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbclient/client.py", line 1062, in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
  File "/Users/falexwolf/miniconda3/envs/py312/lib/python3.12/site-packages/nbclient/client.py", line 918, in _check_raise_for_error
    raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
import lamindb as ln

ln.track()
------------------

----- stdout -----
→ connected lamindb: falexwolf/docsbuild
----- stdout -----
! Can not find the notebook in any server session or by using other methods.
----- stdout -----
! Consider installing ipylab (pip install ipylab) if you use jupyter lab.
------------------

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[1], line 3
      1 import lamindb as ln
----> 3 ln.track()

File ~/repos/laminhub/rest-hub/sub/lamindb/lamindb/core/_context.py:345, in Context.track(self, transform, project, space, params, new_run, path)
    343 description = None
    344 if is_run_from_ipython:
--> 345     self._path, description = self._track_notebook(path_str=path)
    346     transform_type = "notebook"
    347     transform_ref = None

File ~/repos/laminhub/rest-hub/sub/lamindb/lamindb/core/_context.py:475, in Context._track_notebook(self, path_str)
    469 def _track_notebook(
    470     self,
    471     *,
    472     path_str: str | None,
    473 ) -> tuple[Path, str | None]:
    474     if path_str is None:
--> 475         path = get_notebook_path()
    476     else:
    477         path = Path(path_str)

File ~/repos/laminhub/rest-hub/sub/lamindb/lamindb/core/_context.py:68, in get_notebook_path()
     66     raise RuntimeError(msg_path_failed) from None
     67 if path is None:
---> 68     raise RuntimeError(msg_path_failed) from None
     69 return Path(path)

RuntimeError: failed to infer notebook path.
fix: pass `path` to `ln.track()`

[falexwolf]

Ok, Gemini and also @Koncopd before recommended to go through an env variable.

I don't like the env variable solution because it requires an awkward call from the user.

And if multiple notebooks are run in parallel in the same environment, things will break down.

So, instead, I'm following Gemini's second suggestion of inspecting the args of the parent process.

• 🚸 Support inferring notebook path when executed through nbconvert nbproject#288

I'll now proceed to adding tests within lamin-cli.