Command-line options --css and --include-in-header override corresponding metadata fields instead of accumulating

[Oblomov]

Hello,

I just noticed that using the command-line -c (respectively, -H) option overrides any css: (respectively, header-includes:) keys in the YAML metadata sections.

This is a bit surprising, since the options themselves are otherwise cumulative (one can specify multiple -c and -H options, and they get all used), and makes it much harder to have both common and file-specific CSS and header includes (in this sense, this bug is related to #3115).

I believe that, to avoid violating the principle of least surprise, the -c and -H options should be cumulative with the file metatada (and specifically added after the internal ones). OTOH, such a change in behavior would break backwards compatibility, so maybe a way to specify additional CSS and header-includes from the command line should be given?

(If the replacing behavior is preserved, it would also be a nice idea to document it in the manual.)

[ickc]

There's some sort of discussion about this in #3138. @jgm mentioned it is simple to make the change, but since it breaks backward compatibility (some people might rely on this to override YAML's metadata), it requires a discussion in pandoc-discuss: Discussion needed—How should pandoc handle when meatada in YAML collide with command line option - Google Groups.

[cagix]

@jgm @ickc are there any news to this topic? the linked discussion appears to be out of date?

It would be very helpful to implement this merging behavior. If there are concerns about backward compatibility, may be a new option could enable the old behaviour?

[dhimmel]

+1 to making --include-in-header append to header-includes rather than overwriting it.

The current behavior is leading to a poor user experience in the wild. See manubot/rootstock#275: the pandoc-eqnos filter we use now prints the following stderr:

pandoc-eqnos: Wrote the following blocks to header-includes.  If you
use pandoc's --include-in-header option then you will need to manually
include these yourself.

    <!-- pandoc-eqnos: equation style -->
    <style>
      .eqnos { display: inline-block; position: relative; width: 100%; }
      .eqnos br { display: none; }
      .eqnos-number { position: absolute; right: 0em; top: 50%; line-height: 0; }
    </style>

[jgm]

See #5881. I think the changes I'm proposing there will also have the effect that include-in-header will append rather than overwriting.

[jgm]

The issue here is a subtle one: when pandoc populates variables for a template, metadata values are used only if the corresponding variable doesn't have a value set on the command line. And the effect of --include-in-header is to set the variable. That's why it overrides.

One solution in your filter would be to check the metadata. If there is a header-includes field already, then you can append your content to this instead of setting a variable.

[dhimmel]

metadata values are used only if the corresponding variable doesn't have a value set on the command line. And the effect of --include-in-header is to set the variable. That's why it overrides.

I see. So it is out of the question to change metadata values such that they are not discarded, but added to, when command line options are also specified? Currently --include-in-header is acting more like --replace-header-includes.

ec043e0 seems like it will give a way accumulate --include-in-header command line options with preexisting include-in-header fields in default files. Is the whole purpose of defaults to have an alternative to metadata where command line options don't discard existing values?

If there is a header-includes field already, then you can append your content to this instead of setting a variable.

My earlier example was in relation to @tomduck's pandoc-eqnos package, so tagging him in case this is useful. I think the issue is that the filter is unaware whether the user specifies --include-in-header and therefore is unable to know whether the header-includes metadata it sets will be discarded. If defaults can be accessed by a filter perhaps this would enable a way for the filter to add a --include-in-header to defaults as a workaround?

How does a defaults.include-in-header interact with metadata.header-includes?