nbconvert 6.0.0 causing build failures due to TemplateNotFound error

[vEpiphyte]

When executing nbconvert to convert a jupyter notebook into RST format with a custom template that extends rst.tpl, I receive an exception for jinja2.exceptions.TemplateNotFound: display_priority.j2. We are using a custom template to extend the rst.tpl file in order to surpress output blocks in the RST output which have been marked with the hide_code jupyter extension. Since we run notebooks as part of our unittests to ensure our documentation is up-to-date, this is causing build failures for our project (vertexproject/synapse). This can be seen in vertexproject/synapse#1876 and https://app.circleci.com/pipelines/github/vertexproject/synapse/4660/workflows/dcb1e54f-fc51-49fd-9134-e88ef80c5c7d/jobs/19312

This is the custom template file

{%- extends 'rst.tpl' -%}

{% block input_group -%}
{%- if cell.metadata.hideCode or nb.metadata.hide_input -%}
{%- else -%}
    {{ super() }}
{%- endif -%}
{% endblock input_group %}

{% block output_group -%}
{%- if cell.metadata.hideOutput -%}
{%- else -%}
    {{ super() }}
{%- endif -%}
{% endblock output_group %}

{#{% block output_area_prompt %}#}
{#{%- if cell.metadata.hide_input or nb.metadata.hide_input -%}#}
{#    <div class="prompt"> </div>#}
{#{%- else -%}#}
{#    {{ super() }}#}
{#{%- endif -%}#}
{#{% endblock output_area_prompt %}#}

We're not currently constraining the version of the nbconvert library installed, instead relying on the packages installed by installing the jupyter library.

I do not have a minimal test case example, but it is reproducible.

# Make a temporary directory to test with
cd /tmp
mkdir jtest
cd jtest
# Make and activate a venv (using pyenv in this example
pyenv virtualenv --copies 3.7.7 jtest377
pyenv shell jtest377
python -m pip install -U pip wheel
# Check out the affected project and install the build requirements
git clone https://github.com/vertexproject/synapse.git
cd synapse
python -m pip install -U -r requirements_doc.txt 
# Execute nbconvert directly. normally this is done in bulk with scripts/doctests.py
jupyter nbconvert --debug --execute --stdout --to rst --template /home/epiphyte/git/synapse/docs/vertex.tpl /home/epiphyte/git/synapse/docs/synapse/httpapi.ipynb

This eventually fails with the following error

[NbConvertApp] content: {'execution_state': 'idle'}                                                                              
[NbConvertApp] Skipping non-executing cell 4        
[NbConvertApp] Skipping non-executing cell 5                                                                       
[NbConvertApp] Applying preprocessor: coalesce_streams
[NbConvertApp] Applying preprocessor: HighlightMagicsPreprocessor                      
[NbConvertApp] Applying preprocessor: ExtractOutputPreprocessor
[NbConvertApp] Attempting to load template vertex.tpl                                                                                                                                                              
[NbConvertApp]     template_paths: /home/epiphyte/.local/share/jupyter/nbconvert/templates:/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates:/usr/local/share/jupyter/nbconvert/templates:/
usr/share/jupyter/nbconvert/templates:/home/epiphyte/git/synapse/docs:/home/epiphyte/.local/share/jupyter:/home/epiphyte/.local/share/jupyter/nbconvert/templates:/home/epiphyte/.local/share/jupyter/nbconvert/tem
plates/compatibility:/home/epiphyte/.pyenv/versions/jtest377/share/jupyter:/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates:/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconver
t/templates/compatibility:/usr/local/share/jupyter:/usr/local/share/jupyter/nbconvert/templates:/usr/local/share/jupyter/nbconvert/templates/compatibility:/usr/share/jupyter:/usr/share/jupyter/nbconvert/template
s:/usr/share/jupyter/nbconvert/templates/compatibility      
[NbConvertApp] Template paths:
        /home/epiphyte/.local/share/jupyter/nbconvert/templates
        /home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates
        /usr/local/share/jupyter/nbconvert/templates
        /usr/share/jupyter/nbconvert/templates
        /home/epiphyte/git/synapse/docs
        /home/epiphyte/.local/share/jupyter
        /home/epiphyte/.local/share/jupyter/nbconvert/templates
        /home/epiphyte/.local/share/jupyter/nbconvert/templates/compatibility
        /home/epiphyte/.pyenv/versions/jtest377/share/jupyter
        /home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates
        /home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates/compatibility
        /usr/local/share/jupyter
        /usr/local/share/jupyter/nbconvert/templates
        /usr/local/share/jupyter/nbconvert/templates/compatibility
        /usr/share/jupyter
        /usr/share/jupyter/nbconvert/templates
        /usr/share/jupyter/nbconvert/templates/compatibility
Traceback (most recent call last):
  File "/home/epiphyte/.pyenv/versions/jtest377/bin/jupyter-nbconvert", line 8, in <module>
    sys.exit(main())
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/jupyter_core/application.py", line 270, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/traitlets/config/application.py", line 837, in launch_instance
    app.start()
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 345, in start
    self.convert_notebooks()
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 519, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 484, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 413, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 182, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 200, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 382, in from_notebook_node
    output = self.template.render(nb=nb_copy, resources=resources)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/home/epiphyte/git/synapse/docs/vertex.tpl", line 5, in top-level template code
    {%- extends 'rst.tpl' -%}
  File "/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates/compatibility/rst.tpl", line 2, in top-level template code
    {%- extends 'rst/index.rst.j2' -%}
  File "/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates/rst/index.rst.j2", line 1, in top-level template code
    {%- extends 'display_priority.j2' -%}
jinja2.exceptions.TemplateNotFound: display_priority.j2

Executing the conversion without the --template argument provided does work to convert the notebook, but includes the information I want to drop.

I did try modifying the index.rst.j2 file to load 'base/display_priotiy.j2 and that did successfully execute my builds, but I don't know if that change has other ramifications for nbconvert at a larger scale or not.

(jtest377) epiphyte@vertex05:/tmp/jtest/synapse$ head -n 2 /home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates/rst/index.rst.j2 
{%- extends 'base/display_priority.j2' -%}

This is using nbconvert 6.0.1. I have also confirmed that 6.0.0 also fails. 5.6.1 worked fine.

The following is a dump of libraries from python -m pip freeze the venv used above

aiohttp==3.6.2
alabaster==0.7.12
apipkg==1.5
argon2-cffi==20.1.0
async-generator==1.10
async-timeout==3.0.1
attrs==20.2.0
autopep8==1.5.4
Babel==2.8.0
backcall==0.2.0
bleach==3.1.5
bump2version==1.0.0
certifi==2020.6.20
cffi==1.14.2
chardet==3.0.4
coverage==5.2.1
cryptography==3.1
decorator==4.4.2
defusedxml==0.6.0
docutils==0.16
entrypoints==0.3
execnet==1.7.1
fastjsonschema==2.14.5
hide-code==0.5.2
idna==2.10
imagesize==1.2.0
importlib-metadata==1.7.0
ipykernel==5.3.4
ipython==7.18.1
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.17.2
Jinja2==2.11.2
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.7
jupyter-console==6.2.0
jupyter-core==4.6.3
jupyterlab-pygments==0.1.1
lark-parser==0.9.0
lmdb==0.99
MarkupSafe==1.1.1
mistune==0.8.4
more-itertools==8.5.0
msgpack==0.6.2
multidict==4.7.6
nbclient==0.5.0
nbconvert==6.0.1
nbformat==5.0.7
nbstripout==0.3.9
nest-asyncio==1.4.0
notebook==6.1.4
packaging==20.4
pandocfilters==1.4.2
parso==0.7.1
pdfkit==0.6.1
pexpect==4.8.0
pickleshare==0.7.5
pluggy==0.13.1
prometheus-client==0.8.0
prompt-toolkit==3.0.7
ptyprocess==0.6.0
py==1.9.0
pycodestyle==2.6.0
pycparser==2.20
Pygments==2.6.1
pyOpenSSL==19.1.0
pyparsing==2.4.7
pyrsistent==0.16.0
pytest==5.4.3
pytest-cov==2.10.1
pytest-forked==1.3.0
pytest-xdist==1.34.0
python-dateutil==2.8.1
pytz==2020.1
PyYAML==5.3.1
pyzmq==19.0.2
qtconsole==4.7.7
QtPy==1.9.0
regex==2020.7.14
requests==2.24.0
Send2Trash==1.5.0
six==1.15.0
snowballstemmer==2.0.0
Sphinx==1.8.5
sphinx-rtd-theme==0.5.0
sphinxcontrib-serializinghtml==1.1.4
sphinxcontrib-websupport==1.2.4
terminado==0.8.3
testpath==0.4.4
toml==0.10.1
tornado==6.0.4
traitlets==5.0.4
typing-extensions==3.7.4.3
urllib3==1.25.10
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.5.1
xxhash==1.4.4
yarl==1.5.1
zipp==3.1.0

[SylvainCorlay]

Hey @vEpiphyte thanks for reporting!

The way custom templates are done is one of the main changes in nbconvert 6.0, and the main reason for the major version bump indicating backward incompatibility. You can check out the documentation here: https://nbconvert.readthedocs.io/en/latest/customizing.html There is also a tutorial in the works on how to create more sophisticated templates.

[anpaz]

I'm seeing the same problem on a Mac.
After installing transitively nbconvert 6.0.1 from installing jupyter with:
pip install --user pytest jupyter
when I try to use jupyter nbconvert Notebook.ipynb --execute --stdout --to html
I get a ValueError: No template sub-directory with name 'lab' found in the following paths:
looking into the templates paths, like /usr/local/share/jupyter/nbconvert/templates/html/; they are empty. From the documentation seems to me there should be some directories in it with files for the templates.
How are the templates supposed to get there? Are they part of another package that is missing, or are they not getting copied during installation?

[SylvainCorlay]

@anpaz-msft thanks for reporting. Your issue is unrelated to the original one (which was expected behavior). Yours appear to be a legitimate bug with --user installes of nbconvert. I am closing this one and opening another related to user installs.

[SylvainCorlay]

@anpaz-msft actually, I am not reproducing your issue.

Could you please report the output to

which jupyter
jupyter --paths

and the content of your ~/.local/share/jupyter/nbconvert/templates directory?

[vEpiphyte]

@SylvainCorlay Thanks for following up. I look forward to seeing the tutorial about migrating old templates.

[LuisVictoria]

I'm seeing the same problem on a Mac.
After installing transitively nbconvert 6.0.1 from installing jupyter with:
pip install --user pytest jupyter
when I try to use jupyter nbconvert Notebook.ipynb --execute --stdout --to html
I get a ValueError: No template sub-directory with name 'lab' found in the following paths:
looking into the templates paths, like /usr/local/share/jupyter/nbconvert/templates/html/; they are empty. From the documentation seems to me there should be some directories in it with files for the templates.
How are the templates supposed to get there? Are they part of another package that is missing, or are they not getting copied during installation?

I had the same problem on Windows 10, and after reading @SylvainCorlay reply, I unistall nbconvert and then reinstalled it without the flag --user and it worked. Every template appers and I was able to convert my notebook.

I think @SylvainCorlay is correct, something goes wrong when using the --user.

My directory ...\share\jupyter\nbconvert\templateswas empty before my ''fix''.

[MSeal]

Tutorials, docs updates, and some fixes for a few issues related to this have all been resolved. I'm going to close this issue as I believe it's addressed