Change to use micromark

This is a giant change for remark.
It replaces the 5+ year old internals with a new low-level parser:
<https://github.com/micromark/micromark>
The old internals have served billions of users well over the years, but
markdown has changed over that time.
micromark comes with 100% CommonMark (and GFM as an extension) compliance,
and (WIP) docs on parsing rules for how to tokenize markdown with a state
machine: <https://github.com/micromark/common-markup-state-machine>.
micromark, and micromark in remark, is a good base for the future.

`remark-parse` now defers its work to [`micromark`][micromark] and
[`mdast-util-from-markdown`][from-markdown].
`micromark` is a new, small, complete, and CommonMark compliant low-level
markdown parser.
`from-markdown` turns its tokens into the previously (and still) used syntax
tree: [mdast][].
Extensions to `remark-parse` work differently: they’re a two-part act.
See for example [`micromark-extension-footnote`][micromark-footnote] and
[`mdast-util-footnote`][from-markdown-footnote].

* change: `commonmark` is no longer an option — it’s the default
* move: `gfm` is no longer an option — moved to `remark-gfm`
* remove: `pedantic` is no longer an option — this legacy and buggy flavor of
  markdown is no longer widely used
* remove: `blocks` is no longer an options — it’s no longer suggested to
  change the internal list of HTML “block” tag names

remark-stringify now defers its work to [`mdast-util-to-markdown`][to-markdown].
It’s a new and better serializer with powerful features to ensure serialized
markdown represents the syntax tree (mdast), no matter what plugins do.
Extensions to it work differently: see for example
[`mdast-util-footnote`][to-markdown-footnote].

* change: `commonmark` is no longer an option, it’s the default
* change: `emphasis` now defaults to `*`
* change: `bullet` now defaults to `*`
* move: `gfm` is no longer an option — moved to `remark-gfm`
* move: `tableCellPadding` — moved to `remark-gfm`
* move: `tablePipeAlign` — moved to `remark-gfm`
* move: `stringLength` — moved to `remark-gfm`
* remove: `pedantic` is no longer an option — this legacy and buggy flavor of
  markdown is no longer widely used
* remove: `entities` is no longer an option — with CommonMark there is almost
  never a need to use character references, as character escapes are preferred
* new: `quote` — you can now prefer single quotes (`'`) over double quotes
  (`"`) in titles

All of these are for CommonMark compatibility.
Most of them are inconsequential.

* **notable**: references (as in, links `[text][id]` and images `![alt][id]`)
  are no longer present as such in the syntax tree if they don’t have a
  corresponding definition (`[id]: example.com`).
  The reason for this is that CommonMark requires `[text *emphasis
  start][undefined] emphasis end*` to be emphasis.
* **notable**: it is no longer possible to use two blank lines between two
  lists or a list and indented code.
  CommonMark prohibits it.
  For a solution, use an empty comment to end lists (`<!---->`)
* inconsequential: whitespace at the start and end of lines in paragraphs is
  now ignored
* inconsequential: `<mailto:foobarbaz>` are now correctly parsed, and the
  scheme is part of the tree
* inconsequential: indented code can now follow a block quote w/o blank line
* inconsequential: trailing indented blank lines after indented code are no
  longer part of that code
* inconsequential: character references and escapes are no longer present as
  separate text nodes
* inconsequential: character references which HTML allows but CommonMark
  doesn’t, such as `&copy` w/o the semicolon, are no longer recognized
* inconsequential: the `indent` field is no longer available on `position`
* fix: multiline setext headings
* fix: lazy lists
* fix: attention (emphasis, strong)
* fix: tabs
* fix: empty alt on images is now present as an empty string
* …plus a ton of other minor previous differences from CommonMark

* get folks to use this and report problems!

* make `remark-gfm`
* start making next branches for plugins
* get types into {from,to}-markdown and use them here

Closes GH-218.
Closes GH-306.
Closes GH-315.
Closes GH-324.
Closes GH-398.
Closes GH-402.
Closes GH-407.
Closes GH-439.
Closes GH-450.
Closes GH-459.
Closes GH-493.
Closes GH-494.
Closes GH-497.
Closes GH-504.
Closes GH-517.
Closes GH-521.
Closes GH-523.

Closes remarkjs/remark-lint#111.

[micromark]: https://github.com/micromark/micromark

[from-markdown]: https://github.com/syntax-tree/mdast-util-from-markdown

[to-markdown]: https://github.com/syntax-tree/mdast-util-to-markdown

[micromark-footnote]: https://github.com/micromark/micromark-extension-footnote/blob/main/index.js

[to-markdown-footnote]: https://github.com/syntax-tree/mdast-util-footnote/blob/main/to-markdown.js

[from-markdown-footnote]: https://github.com/syntax-tree/mdast-util-footnote/blob/main/from-markdown.js

[mdast]: https://github.com/syntax-tree/mdast

Author: wooorm

doc/getting-started.md

@@ -24,8 +24,6 @@ and written:
 # Alpha #
 Bravo charlie **delta** __echo__.
 - Foxtrot
-*  Golf
-+ Hotel
 ```
 
 Yields:
@@ -35,14 +33,12 @@ Yields:
 
 Bravo charlie **delta** **echo**.
 
--   Foxtrot
--   Golf
--   Hotel
+*   Foxtrot
 ```
 
 But, much more can be done, [through plugins][plugins].
 
-## Command-line
+## Command line
 
 **remark**’s CLI is a simple way to process Markdown files from the
 command line.  Its interface is provided by [**unified-args**][unified-args].
@@ -108,9 +104,9 @@ The `--save-dev` option stores the dependencies in our `package.json`:
    "name": "my-package",
    "version": "1.0.0",
 +  "devDependencies": {
-+    "remark-cli": "^6.0.0",
-+    "remark-html": "^8.0.0",
-+    "remark-preset-lint-markdown-style-guide": "^2.0.0"
++    "remark-cli": "^8.0.0",
++    "remark-html": "^12.0.0",
++    "remark-preset-lint-markdown-style-guide": "^3.0.0"
 +  },
    "scripts": {
      "test": "node test.js"
@@ -126,9 +122,9 @@ configuration:
    "name": "my-package",
    "version": "1.0.0",
    "devDependencies": {
-     "remark-cli": "^6.0.0",
-     "remark-html": "^8.0.0",
-     "remark-preset-lint-markdown-style-guide": "^2.0.0"
+     "remark-cli": "^8.0.0",
+     "remark-html": "^12.0.0",
+     "remark-preset-lint-markdown-style-guide": "^3.0.0"
    },
    "scripts": {
 -    "test": "node test.js"

doc/plugins.md

@@ -287,11 +287,10 @@ virtual files.
 
 ## Using plugins
 
-To use a plugin programmatically, invoke the [`use()`][unified-use]
-function.
+To use a plugin programmatically, call the [`use()`][unified-use] function.
 
-To use plugin with `remark-cli`, pass a [`--use` flag][unified-args-use]
-or specify it in a [configuration file][config-file-use].
+To use plugin with `remark-cli`, pass a [`--use` flag][unified-args-use] or
+specify it in a [configuration file][config-file-use].
 
 ## Creating plugins
 

license

@@ -1,6 +1,6 @@
 (The MIT License)
 
-Copyright (c) 2014-2016 Titus Wormer <[email protected]>
+Copyright (c) 2014-2020 Titus Wormer <[email protected]>
 Copyright (c) 2011-2014, Christopher Jeffrey (https://github.com/chjj/)
 
 Permission is hereby granted, free of charge, to any person obtaining a copy

log.txt

@@ -10,39 +10,32 @@
     "url": "https://opencollective.com/unified"
   },
   "devDependencies": {
-    "bail": "^1.0.0",
     "browserify": "^16.0.0",
     "camelcase": "^6.0.0",
-    "clone": "^2.0.0",
     "dtslint": "^4.0.0",
     "execa": "^4.0.0",
     "lerna": "^3.0.0",
     "mdast-util-assert": "^3.0.0",
-    "mdast-util-compact": "^2.0.0",
-    "mdast-zone": "^4.0.0",
+    "mdast-util-gfm": "^0.1.0",
+    "micromark-extension-gfm": "^0.3.0",
     "nyc": "^15.0.0",
     "prettier": "^2.0.0",
     "remark-preset-wooorm": "^7.0.0",
     "tape": "^5.0.0",
     "tinyify": "^3.0.0",
     "typescript": "^4.0.0",
     "unified": "^9.1.0",
-    "unist-builder": "^2.0.0",
     "unist-util-remove-position": "^3.0.0",
-    "unist-util-visit": "^2.0.0",
-    "vfile": "^4.0.0",
-    "wcwidth": "^1.0.0",
     "xo": "^0.33.0"
   },
   "scripts": {
     "postinstall": "lerna bootstrap --no-ci",
-    "format": "packages/remark-cli/cli.js . -qfo && prettier . --write && xo --fix",
+    "format": "prettier . -w --loglevel warn && xo --fix",
     "build-bundle": "browserify packages/remark -s remark > remark.js",
     "build-mangle": "browserify packages/remark -s remark -p tinyify > remark.min.js",
     "build": "npm run build-bundle && npm run build-mangle",
     "test-api": "tape \"packages/*/test.js\" \"test/index.js\"",
-    "test-api-extensive": "TEST_EXTENDED=true npm run test-api",
-    "test-coverage": "nyc --reporter lcov tape \"test/index.js\" \"packages/*/test.js\"",
+    "test-coverage": "nyc --reporter lcov tape \"packages/*/test.js\" \"test/index.js\"",
     "test-types": "dtslint packages/remark-parse/types && dtslint packages/remark-stringify/types && dtslint packages/remark/types",
     "test": "npm run format && npm run build && npm run test-coverage && npm run test-types"
   },
@@ -89,7 +82,6 @@
   },
   "remarkConfig": {
     "plugins": [
-      "./script/list-of-methods",
       "preset-wooorm",
       [
         "toc",

package.json

@@ -17,53 +17,6 @@ Command line interface for [**remark**][remark].
 *   Uses configuration from [`remarkConfig` fields in `package.json`
     files][config-file]
 
-## Sponsors
-
-Support this effort and give back by sponsoring on [OpenCollective][collective]!
-
-<!--lint ignore no-html-->
-
-<table>
-<tr valign="middle">
-<td width="20%" align="center" colspan="2">
-  <a href="https://www.gatsbyjs.org">Gatsby</a> 🥇<br><br>
-  <a href="https://www.gatsbyjs.org"><img src="https://avatars1.githubusercontent.com/u/12551863?s=256&v=4" width="128"></a>
-</td>
-<td width="20%" align="center" colspan="2">
-  <a href="https://vercel.com">Vercel</a> 🥇<br><br>
-  <a href="https://vercel.com"><img src="https://avatars1.githubusercontent.com/u/14985020?s=256&v=4" width="128"></a>
-</td>
-<td width="20%" align="center" colspan="2">
-  <a href="https://www.netlify.com">Netlify</a><br><br>
-  <!--OC has a sharper image-->
-  <a href="https://www.netlify.com"><img src="https://images.opencollective.com/netlify/4087de2/logo/256.png" width="128"></a>
-</td>
-<td width="10%" align="center">
-  <a href="https://www.holloway.com">Holloway</a><br><br>
-  <a href="https://www.holloway.com"><img src="https://avatars1.githubusercontent.com/u/35904294?s=128&v=4" width="64"></a>
-</td>
-<td width="10%" align="center">
-  <a href="https://themeisle.com">ThemeIsle</a><br><br>
-  <a href="https://themeisle.com"><img src="https://avatars1.githubusercontent.com/u/58979018?s=128&v=4" width="64"></a>
-</td>
-<td width="10%" align="center">
-  <a href="https://boostio.co">BoostIO</a><br><br>
-  <a href="https://boostio.co"><img src="https://avatars1.githubusercontent.com/u/13612118?s=128&v=4" width="64"></a>
-</td>
-<td width="10%" align="center">
-  <a href="https://expo.io">Expo</a><br><br>
-  <a href="https://expo.io"><img src="https://avatars1.githubusercontent.com/u/12504344?s=128&v=4" width="64"></a>
-</td>
-</tr>
-<tr valign="middle">
-<td width="100%" align="center" colspan="10">
-  <br>
-  <a href="https://opencollective.com/unified"><strong>You?</strong></a>
-  <br><br>
-</td>
-</tr>
-</table>
-
 ## Install
 
 [npm][]:
@@ -154,6 +107,53 @@ This project has a [code of conduct][coc].
 By interacting with this repository, organization, or community you agree to
 abide by its terms.
 
+## Sponsor
+
+Support this effort and give back by sponsoring on [OpenCollective][collective]!
+
+<!--lint ignore no-html-->
+
+<table>
+<tr valign="middle">
+<td width="20%" align="center" colspan="2">
+  <a href="https://www.gatsbyjs.org">Gatsby</a> 🥇<br><br>
+  <a href="https://www.gatsbyjs.org"><img src="https://avatars1.githubusercontent.com/u/12551863?s=256&v=4" width="128"></a>
+</td>
+<td width="20%" align="center" colspan="2">
+  <a href="https://vercel.com">Vercel</a> 🥇<br><br>
+  <a href="https://vercel.com"><img src="https://avatars1.githubusercontent.com/u/14985020?s=256&v=4" width="128"></a>
+</td>
+<td width="20%" align="center" colspan="2">
+  <a href="https://www.netlify.com">Netlify</a><br><br>
+  <!--OC has a sharper image-->
+  <a href="https://www.netlify.com"><img src="https://images.opencollective.com/netlify/4087de2/logo/256.png" width="128"></a>
+</td>
+<td width="10%" align="center">
+  <a href="https://www.holloway.com">Holloway</a><br><br>
+  <a href="https://www.holloway.com"><img src="https://avatars1.githubusercontent.com/u/35904294?s=128&v=4" width="64"></a>
+</td>
+<td width="10%" align="center">
+  <a href="https://themeisle.com">ThemeIsle</a><br><br>
+  <a href="https://themeisle.com"><img src="https://avatars1.githubusercontent.com/u/58979018?s=128&v=4" width="64"></a>
+</td>
+<td width="10%" align="center">
+  <a href="https://boostio.co">BoostIO</a><br><br>
+  <a href="https://boostio.co"><img src="https://avatars1.githubusercontent.com/u/13612118?s=128&v=4" width="64"></a>
+</td>
+<td width="10%" align="center">
+  <a href="https://expo.io">Expo</a><br><br>
+  <a href="https://expo.io"><img src="https://avatars1.githubusercontent.com/u/12504344?s=128&v=4" width="64"></a>
+</td>
+</tr>
+<tr valign="middle">
+<td width="100%" align="center" colspan="10">
+  <br>
+  <a href="https://opencollective.com/unified"><strong>You?</strong></a>
+  <br><br>
+</td>
+</tr>
+</table>
+
 ## License
 
 [MIT][license] © [Titus Wormer][author]

packages/remark-cli/readme.md

@@ -1,17 +1,21 @@
 'use strict'
 
-var unherit = require('unherit')
-var xtend = require('xtend')
-var Parser = require('./lib/parser.js')
-
 module.exports = parse
-parse.Parser = Parser
+
+var fromMarkdown = require('mdast-util-from-markdown')
 
 function parse(options) {
-  var settings = this.data('settings')
-  var Local = unherit(Parser)
+  var self = this
 
-  Local.prototype.options = xtend(Local.prototype.options, settings, options)
+  this.Parser = parse
 
-  this.Parser = Local
+  function parse(doc) {
+    var settings = Object.assign({}, self.data('settings'), options)
+    // Note: these options are not in the readme.
+    // The goal is for them to be set by plugins on `data` instead of being
+    // passed by users.
+    settings.extensions = settings.micromarkExtensions
+    settings.mdastExtensions = settings.fromMarkdownExtensions
+    return fromMarkdown(doc, settings)
+  }
 }