Issue combining a pre-processing filter with a document format.

[zachcp]

Description

Hi Team Quarto,

Thanks again for your fast responses over the past few days. I am working to get a filter working as a sort of pre-processor for quarto-live. What I am trying to do is:

1. Get codeBlock with tag molviewspec-live
2. Use lua to apply a template. Take the contents of the codeblock and sunstitute it into a template.
3. return markdown or pandoc.
4. process normally with quarto live.

I am able to do 1/2 but am stuck on 3/4. Basically, I am not sure of the order of operations (affects 4) or the right pandoc type to return (needed by 3). My example repo is here, my header yaml is below, and my lua code is below.

Any suggestions on how to troubleshoot this issue would be highly appreciated.

zach cp

YAML Header

---
title: "About"
filters:
  - molviewspec-live
format: live-html
---

Lua Filter FN

function CodeBlock(el)
    function apply_template(template, template_vars)
        local file = io.open(quarto.utils.resolve_path("templates/" .. template), "r")
        assert(file)
        local content = file:read("*a")
        for k, v in pairs(template_vars) do
            content = string.gsub(content, "{{" .. k .. "}}", v)
        end
        return content
    end

    if el.classes and el.classes:includes("molviewspec-live") then
        local content = el.text
        local updatedmarkdown = apply_template("two_column.qmd", { content = content })
        -- return pandoc.RawBlock("markdown", updatedmarkdown)
        -- return pandoc.RawBlock("html", updatedmarkdown)
        local parsed = pandoc.read(updatedmarkdown, "markdown")
        return parsed.blocks
    end
    return el
end

HTML Output

You can see my markdown is being inlined into HTML without further processing. I don't know if thats an Order-of-operations issue or a pandoc issue (or soemthing else).

[cscheid]

Ah, I have a guess to what's happening.

Unfortunately, Pandoc doesn't support the following syntax:

```{pyiodide}
...
```

Quarto implements that by taking over part of the frontend parsing. If you need to parse Markdown into Pandoc's AST with all of the syntax that Quarto supports, you'll need to use our parsing instead of pandoc.read. It's not currently exposed as a public API, but the following call isn't going anywhere in the near-to-medium term future:

quarto._quarto.utils.string_to_blocks(updatedmarkdown)

I'm not positive that's the cause, but the fact that the HTML starts breaking at around the ```{pyiodide} block makes me pretty suspicious.

[zachcp]

So thats progress. If I use quarto._quarto.utils.string_to_blocks(updatedmarkdown), the code blocks seem to get parsed however the code blocks themselves are not processed. In this case I would expect the leftside to be the quarto-live black and the right side to be the OJS block associated with it.

[zachcp]

I'll sleep on it. If I want to target quarto-live, I will need to emit the quarto-live code block for reinterpretation. I don't really see a way around that. But maybe theres a simpler template format that will work better......

[cscheid]

"the code blocks seem to get parsed however the code blocks themselves are not processed"

I apologize, I think I don't understand very well what you're trying to do, so let me instead describe how Quarto works. Quarto operates in the following approximate order:

• include shortcodes are resolved
• engines execute code cells in markdown and produce annotated markdown
• OJS cells and diagram cells (dot and mermaid) are handled
• Lua filters are executed

If your filter creates pyiodide code cells, then it needs to be executed before the pyiodide filter that detects those code cells, and the structure you generate needs to precisely match the expectation of the quartolive package.

If you allow me to be frank, I think you're trying to do something pretty complicated and would be better served with first building smaller extensions that will help you better understand the details of Quarto's execution.

Instead of trying to do all of these things at once, I would start by attempting to create a very basic filter that adds a new hardcoded pyiodide code cell. When you get that to work, then extend it to try and load text from disk, etc.

In addition, I unfortunately don't think it's currently possible to create new OJS cells in Lua. This is for similar reasons as creating code cells in Lua not being possible: Lua filters are typically meant to post-process a document (as in they can take existing OJS cells and annotate them), and so engines need to run before Lua filters.

[zachcp]

Good morning! Thank you @cscheid , that definitely helps my understanding of the quarto system. And I agree that precompilation to target quarto-live is too complicated/brittle. I think I will resume my original approach of building my app in JS and using a simple filter to embed it in DIV in the doc. I had been enticed by quarto-live but for now I don't think this is the best approach.

On a separate note - your description of the Quarto compilation would have been very helpful to me when I was trying to grok the system. Maybe if would be useful as an addition to the Extensions docs? If you think so, I would be happy to submit a PR.

[cscheid]

On a separate note - your description of the Quarto compilation would have been very helpful to me when I was trying to grok the system. Maybe if would be useful as an addition to the Extensions docs? If you think so, I would be happy to submit a PR.

There's a whole book being written right this second about it. We appreciate your pacience!

[mcanouil]

Maybe you can change your strategy and use divs as in the panelize Quarto extension.

[zachcp]

Thank you @mcanouil . I'll give it a look but I think I am going to go back to my original plan of self-contained JS app that can be mounted into a DIV.