Semantic comments

[stasm]

Having a way to semantically describe a message would benefit tooling. It would allow tools to better inform the user what they can do in the translation, and give hints and suggestions.

Perhaps we could consider using something similar JSDoc. In particular the @param tag: http://usejsdoc.org/tags-param.html. JSDoc conveniently allows to specify the type, the description and the default value, which could be used by tools to display an example of a formatted translation.

# @param {number} [$num = 4] Number of new messages
new-messages = { $num ->
   *[one] You have 1 new message.
    [other] You have { $num } new messages.
}

Would it make sense to make this meta-information first-class? Rust differentiates between regular comments (//) and doc comments (///). We could do something similar by making the @ sigil special:

@param {number} [$num = 4] Number of new messages
new-messages = { $num ->
   *[one] You have 1 new message.
    [other] You have { $num } new messages.
}

This is possibly related to #7.

[stasm]

Some more thoughts on how this relates to #7: we could introduce our custom @-tags for language-specific meta information which would be ignored by tools like compare-locales:

@meta masculine
brand-name = Firefox

[stasm]

We could also introduce versioning for messages . This would allow making small changes to the original copy without having to change the identifier. The default and implicit revision would be revision 0:

accept = Accept the Terms and Conditions.

and some time later:

@rev 1
accept = Please accept the Terms and Conditions.

[stasm]

Some more thoughts on how this relates to #7: we could introduce our custom @-tags for language-specific meta information which would be ignored by tools like compare-locales

In #7 (comment) @Pike suggested that we separate semantic comments and grammatical data due to their having different owners.

[mathjazz]

I love that. One possible use case is to define maximum string length (we already use it in .lang files, e.g. for translating promotional tweets).

[gunchleoc]

We could also introduce versioning for messages . This would allow making small changes to the original copy without having to change the identifier.

I'll bet you 100€ that some developer will forget to version a string and that we'll end up with significant changes in the source language that will be missing from the target languages - while you might still control this in QA somehow for Mozilla's core products, you can't count on every extension developer to be aware of the problem. The only way that works reliably in my book is the way gettext handles this: If the string content has changed, it will become fuzzy. It is annoying for localizers if strings become invalid due to a typo having been fixed in the source language, but it's the lesser of 2 evils.

[stasm]

In #59 @zbraniecki asked about the possibility of using semantic comments for tags. I'm pasting my reply below to keep the discussion in one place.

My understanding of the scope of semantic comments is that they would be a place to put extra data available to tools rather than the runtime. In fact they wouldn't be parsed on runtime at all. This would make it impossible to use them for tags.

[zbraniecki]

Even before we move forward with this, can we get a consensus on how we'd like such comments to look like so that we can start writing such comments even without them being "semantic" yet?

I'm trying to decide between:

# Description
#
# Variables
#   $num (String) - Description

and:

# Description
#
#   $num {String}: Description

or sth in between? Thoughts?

[stasm]

Even before we move forward with this, can we get a consensus on how we'd like such comments to look like (…)?

Isn't this the exact goal of this issue? :)

Some prior art:

JSDoc:

#  @param {number} $num - The $num value.

Some Python projects use this style:

#  :param $num: The $num value.
#  :type $num: number

I like simple ideas of the form:

# $num (Number) - Description

But without a @param or :param how would we introduce other information like max-length or revision? Could we just use @ for data not related to arguments?

# $num (Number) Description
# @max-length 140

[flodolo]

If the goal is to parse comments, and extract information about parameters, I feel like we should enforce @param. If it's only about documenting the parameters, that your last example looks good to me.

[zbraniecki]

One good use of semantic comments would be to instruct the localization tool like Pontoon on what context the string will be used in.

The particular case is where a message like this:

key = Click on <a>my</a> link to paste <a> into the textbox

The latter <a> should be displayed so we probably want to use <a>at least and without knowing if the key goes to alert() or to DOM it's impossible for the tool/localizer to know how to encode it.

Semantic comments could make it trivial:

# @env: html
key = Click on <a>my</a> link to paste <a> into the textbox

and to prevent having to place it in front of every string, we could use group comments and resource comments to annotate the whole file.

[zbraniecki]

Semantic Comments v1 proposal

(updated: April 5 2018)

Description

Semantic comments is the concept which brings basic computer readable structure to comments. The idea is to design a set of patterns that can be codified which enable algorithmic interpretation of a comment.
Such models allow comments to be parsed and data from them used in tooling.

The core design goal is to develop rules that are easy to naively interpret and memorize by humans with minimal overhead, while at the same time allowing computers to assign meaning.

Semantic comments may serve several high level roles:

1. They can unify the way meta information is being stored and presented improving readability of the comments and reducing cognitive load on the reader.
2. They can help tools present contextual information from the developer to localizers in a well organized manner.
3. They can help tools interpret the translation and provide additional checks and UI options.

In principle, the nature of the data stored in the comments is limited - runtime parsers should be able to skip comments without parsing them and failure to retrieve information from the comment should not result in any serious reduction in usability of the system.

Experience from other programming languages shows that some form of semantic comments are helpful in most languages from JavaScript, Python, C++ to Rust and CSS.

Below is my initial proposal for the first version of Semantic Comments.

Title Line

It would be useful to be able to capture a short description of the section for UI tools to use when operating on long lists of strings.

A great example of such use case is the current preferences.ftl with hundreds of messages clustered into sections.

My proposal is to identify the title line of any comment as fitting into one of two conditions:

1. Being the only non-empty line of the comment
2. Being the first non-empty line of the comment with a blank line right below it.

That means that the following two are titles lines:

## Privacy Section - Site Data

## Privacy Section - Site Data
##
## This sections will contain several messages
## that should be translated by a lawyer if possible.

And the result in Pontoon, for example, may look like this

Meta-infrormation

Meta information by the definition should be an open ended system. It means that while we can specify the syntax around it and define a set of values that are defined and known, this system should also be open to be extended in the future with new keys. For that reason, I believe that a key-value param system would work well.

The initial uses of meta information may provide details like: in which context the message is being used? Communication style to use for such message. Are there any legal requirements associated with it (branding policy etc.), what UI toolkit it will use etc, string version, etc.

Another use case is to instruct the tooling about any soft-limitations imposed on the translation. For example we may want to instruct the localization tool that a given file/group/message should remain simple - no formatters, no variants etc.
Such limitation may result from some downdumb conversion that will happen on the file later in the release cycle, but a semantic comment may be useful early on to help CATs.

JSDoc has a nice system of block and inline tags. Copying it, it may look like this:

### Preferences
###
### @license LGPL
### @toolkit HTML


## General Section - Site Data


# @context Window Title
sitedata-exceptions-title =
    .label = Exceptions
    .accesskey = E


# @context Menu button
sitedata-exceptions =
    .label = Exceptions
    .accesskey = E

# @policy "Firefox" should be treated as a brand and kept in English,
#           while "Home" and "(Default)" can be localized.
home-mode-choice-default =
    .label = Firefox Home (Default)

Since the system is open ended, the only first step is to define the syntax for it. I'd like it to be:

@name value
@name value {type}
@name value {type} - description

with all three being optional.

Variables

Variables could either be a particular type of block tags, or a separate thing:

# @arg $name {string} - Name of a search engine.
search-keyword-warning-engine = You have chosen a keyword that is currently in use by “{ $name }”. Please select another.

or:

# $name {string} - Name of a search engine.
search-keyword-warning-engine = You have chosen a keyword that is currently in use by “{ $name }”. Please select another.

I'm not very opinionated here, and we could start with the former and maybe one day add the latter as a convenience mechanism ("($.*)" becomes a "@arg $1").

Syntax coloring / validation

The last item I'd like us to consider is syntax coloring and augumentation.

There are three areas where we may end up placing a syntax from another programming language:

1. In the DOM Overlay
2. As an argument (style: width: 15em)
3. In a comment

For the DOM Overlay, I believe that block tag @toolkit HTML on file/section/message comment should be sufficient for any form of syntax highlighting to kick in.

For arguments it's a bit more tricky, and I thought we could annotate it like this:

# This string is currently used only in Firefox 60 and will be removed when not
# needed for x-channel. See bug 1445686 for details.
#
# @toolkit.style CSS
search-input =
    .title = My Window Title
    .style = width: 15.4em

to allow us to specify that the .style attribute is in CSS.

Finally, in the comment, I really like the RST way:

# This message will be displayed inside a :html:`<span/>` with :css:`font-weight: bold`.
#
# @arg $name {string} - Name of a search engine.
search-keyword-warning-engine = You have chosen a keyword that is currently in use by “{ $name }”. Please select another.

This could be introduced gradually - we could now specify "`" as the sygil for code only, and let tooling autoguess the syntax highlighting for it, and one day extend it with :XXX: prefix to allow for specifying the language if needed.

The total outcome might look like this