with_toc_data with Japanese

[NathanHazout]

I have a markdown file in Japanese.

A header might look like:

## アプリケーション・ストリングの翻訳

The generated HTML looks like:
<h2 id=">アプリケーション・ストリングの翻訳</h2>

As you can see, it's not just that it leaves it empty, it generates invalid HTML (missing closing quote), breaking the entire page.

PS. This is happening inside a Jekyll 3 project

[eliot-akira]

I was just now experiencing the same issue using Jekyll, and came here to investigate.

### ヒューマンリーダブル

...becomes...

<h3 id=">ヒューマンリーダブル</h3>

Trial and error showed that even one Japanese character will produce this issue.

Possibly related to this commit and the regular expression used to strip inline markup from headers: 8ec8275

[KitaitiMakoto]

I also encountered the same problem. Minimal code to reproduce the problem is here:

# coding: utf-8                                                                                                                                
require 'redcarpet'

puts Redcarpet::VERSION

renderer = Redcarpet::Render::HTML.new(with_toc_data: true)
markdown = Redcarpet::Markdown.new(renderer, {})
puts markdown.render('# 見出し')

$ ruby broken-html.rb
3.3.4
<h1 id=">見出し</h1>

As an aside, "見出し" means heading(s).

[NathanHazout]

According to last comment, it means it is purely redcarpet and not Jekyll-related.
@robin850 any ideas?

[IdanAdar]

Any idea if this issue is being tracked by the redcarpet team?

[scvthedefect]

The heading id is generated by its content. For content do not match /^\w$/ will cause this bug, id is incomplete and not closed in the right way.

I use heading tags explicitly (<h1></h1>) to avoid this but don't feel good.

[NathanHazout]

@robin850 @vmg or anyone at redcarpet - any news with this?
Can someone review the attached Pull Request see if it's good?
We want to go live soon with our project and this is a problem...

[snkrheadz]

@nasht00 hi, this is my solution for now.

class MyRender < Redcarpet::Render::HTML
  def header(text, header_level)
    %Q{<h#{header_level} id="#{text.downcase.gsub(" ", "-")}">#{text}</h#{header_level}>}.html_safe
  end
end

[NathanHazout]

@akinrt I tried your fix.
It works is most cases, however it breaks if the header contains a link.

For example:

### [Setting up Your Development Environment](../setting-up-your-development-environment/)

Generates:

<h3 id="<a-href="../setting-up-your-development-environment/">setting-up-your-development-environment</a>"><a href="../setting-up-your-development-environment/">Setting up Your Development Environment</a></h3>

Which is obviously no good. It used to work before including this fix.