Code spans should take precedence over link brackets #816

Summary

Redcarpet currently resolves link labels before code spans. When a link label contains a code span, a ] inside the code span closes the label. The CommonMark spec specifies the opposite precedence: "code spans, autolinks, and raw HTML tags bind more tightly than the brackets in link text."

Getting this precedence wrong changes the destination of a link. The following Markdown is spec-correct CommonMark for a link to https://two.example whose visible text is inline code:

[`text](https://one.example)`](https://two.example)

Redcarpet 3.6.1 renders a link to https://one.example instead:

<p><a href="https://one.example">`text</a>`](https://two.example)</p>

cmark, the CommonMark reference implementation, renders the intended link:

<p><a href="https://two.example"><code>text](https://one.example)</code></a></p>

Test cases

These are the tests I intend to include in a pull request, written against test/markdown_test.rb. The expected outputs are what cmark 0.31.2 produces.

1. A code span containing ]( inside a link label. The label is the code span and the destination is the trailing URL.

   def test_code_spans_take_precedence_over_link_brackets
     output   = render("[`text](https://one.example)`](https://two.example)")
     expected = %(<p><a href="https://two.example"><code>text](https://one.example)</code></a></p>)
   
     assert_equal expected, output
   end

   Failure: test_code_spans_take_precedence_over_link_brackets(MarkdownTest)
   <"<p><a href=\"https://two.example\"><code>text](https://one.example)</code></a></p>"> expected but was
   <"<p><a href=\"https://one.example\">`text</a>`](https://two.example)</p>">
   

2. A code span that swallows the only ]. There is no closing bracket left, so the input is not a link. This is example 537 in the spec.

   def test_bracket_with_code_span_but_no_closing_bracket_is_not_a_link
     output   = render("[foo`](/uri)`")
     expected = "<p>[foo<code>](/uri)</code></p>"
   
     assert_equal expected, output
   end

   Failure: test_bracket_with_code_span_but_no_closing_bracket_is_not_a_link(MarkdownTest)
   <"<p>[foo<code>](/uri)</code></p>"> expected but was
   <"<p><a href=\"/uri\">foo`</a>`</p>">
   

3. The same as case 1 with multi-backtick delimiters, since a span opened by a run of N backticks closes only on a run of N backticks.

   def test_multi_backtick_code_spans_take_precedence_over_link_brackets
     output   = render("[``text](https://one.example)``](https://two.example)")
     expected = %(<p><a href="https://two.example"><code>text](https://one.example)</code></a></p>)
   
     assert_equal expected, output
   end

   Failure: test_multi_backtick_code_spans_take_precedence_over_link_brackets(MarkdownTest)
   <"<p><a href=\"https://two.example\"><code>text](https://one.example)</code></a></p>"> expected but was
   <"<p><a href=\"https://one.example\">``text</a>``](https://two.example)</p>">
   

4. An unclosed backtick inside a label. A backtick with no matching closing run is literal text and must not prevent the ] from closing the label. Redcarpet already handles this correctly, and this test passes today; it pins the boundary of the rule.

   def test_unclosed_backtick_in_link_label_stays_literal
     output   = render("[foo`bar](/uri)")
     expected = "<p><a href=\"/uri\">foo`bar</a></p>"
   
     assert_equal expected, output
   end

Out of scope

An escaped backtick inside a label, as in [a\`b](/uri), does not open a code span in either cmark or Redcarpet, so it is unaffected by the precedence rule. Redcarpet's rendering of the escape itself already differs from cmark on master for unrelated reasons, so that case is deliberately excluded here.

Fix

One possible fix is at #817