[Bug][Monarch] A token in a string is still considered as a token rather than a string

[Beishao Cao (BeishaoCao-db)]

Reproducible in vscode.dev or in VS Code Desktop?

• Not reproducible in vscode.dev or VS Code Desktop

Reproducible in the monaco editor playground?

• Reproducible in the monaco editor playground

Monaco Editor Playground Code

// Create your own language definition here
// You can safely look at other samples without losing modifications.
// Modifications are not saved on browser refresh/close though -- copy often!
return {
  // Set defaultToken to invalid to see what you do not tokenize yet
  // defaultToken: 'invalid',

  keywords: [
    'abstract', 'continue', 'for', 'new', 'switch', 'assert', 'goto', 'do',
    'if', 'private', 'this', 'break', 'protected', 'throw', 'else', 'public',
    'enum', 'return', 'catch', 'try', 'interface', 'static', 'class',
    'finally', 'const', 'super', 'while', 'true', 'false'
  ],

  typeKeywords: [
    'boolean', 'double', 'byte', 'int', 'short', 'char', 'void', 'long', 'float'
  ],

  operators: [
    '=', '>', '<', '!', '~', '?', ':', '==', '<=', '>=', '!=',
    '&&', '||', '++', '--', '+', '-', '*', '/', '&', '|', '^', '%',
    '<<', '>>', '>>>', '+=', '-=', '*=', '/=', '&=', '|=', '^=',
    '%=', '<<=', '>>=', '>>>='
  ],

  // we include these common regular expressions
  symbols:  /[=><!~?:&|+\-*\/\^%]+/,

  // C# style strings
  escapes: /\\(?:[abfnrtv\\"']|x[0-9A-Fa-f]{1,4}|u[0-9A-Fa-f]{4}|U[0-9A-Fa-f]{8})/,

  // The main tokenizer for our languages
  tokenizer: {
    root: [
      [
        /LANGUAGE\s+PYTHON\s+AS\s+\$\$/, // Note: does not handle new lines
        { token: 'keyword', bracket: '@open', next: '@python_block', nextEmbedded: 'python' },
      ],
      // identifiers and keywords
      [/[a-z_$][\w$]*/, { cases: { '@typeKeywords': 'keyword',
                                   '@keywords': 'keyword',
                                   '@default': 'identifier' } }],
      [/[A-Z][\w\$]*/, 'type.identifier' ],  // to show class names nicely

      [/<style\s*>/,   { token: 'keyword', bracket: '@open'
                   , next: '@css_block', nextEmbedded: 'text/css' }],

      [/<\/style\s*>/, { token: 'keyword', bracket: '@close' }],
      // whitespace
      { include: '@whitespace' },

      // delimiters and operators
      [/[{}()\[\]]/, '@brackets'],
      [/[<>](?!@symbols)/, '@brackets'],
      [/@symbols/, { cases: { '@operators': 'operator',
                              '@default'  : '' } } ],

      // @ annotations.
      // As an example, we emit a debugging log message on these tokens.
      // Note: message are supressed during the first load -- change some lines to see them.
      [/@\s*[a-zA-Z_\$][\w\$]*/, { token: 'annotation', log: 'annotation token: $0' }],

      // numbers
      [/\d*\.\d+([eE][\-+]?\d+)?/, 'number.float'],
      [/0[xX][0-9a-fA-F]+/, 'number.hex'],
      [/\d+/, 'number'],

      // delimiter: after number because of .\d floats
      [/[;,.]/, 'delimiter'],

      // strings
      [/"([^"\\]|\\.)*$/, 'string.invalid' ],  // non-teminated string
      [/"/,  { token: 'string.quote', bracket: '@open', next: '@string' } ],

      // characters
      [/'[^\\']'/, 'string'],
      [/(')(@escapes)(')/, ['string','string.escape','string']],
      [/'/, 'string.invalid']
    ],

    python_block: [[/\$\$/, { token: 'keyword', next: '@popall', nextEmbedded: '@pop' }],
    [/"/, 'string', '@string' ]],

    css_block: [
      [/[^"<]+/, ''],
      [/<\/style\s*>/, { token: '@rematch', next: '@pop', nextEmbedded: '@pop' }],
      [/"/, 'string', '@string' ],
      [/</, '']
    ],

    comment: [
      [/[^\/*]+/, 'comment' ],
      [/\/\*/,    'comment', '@push' ],    // nested comment
      ["\\*/",    'comment', '@pop'  ],
      [/[\/*]/,   'comment' ]
    ],

    string: [
      [/[^\\"]+/,  'string'],
      [/@escapes/, 'string.escape'],
      [/\\./,      'string.escape.invalid'],
      [/"/,        { token: 'string.quote', bracket: '@close', next: '@pop' } ]
    ],

    whitespace: [
      [/[ \t\r\n]+/, 'white'],
      [/\/\*/,       'comment', '@comment' ],
      [/\/\/.*$/,    'comment'],
    ],
  },
};

Reproduction Steps

1. Using the same code in monarch playground https://microsoft.github.io/monaco-editor/monarch.html
2. add one token in tokenizer.root:
   [ /LANGUAGE\s+PYTHON\s+AS\s+\$\$/, { token: 'keyword', bracket: '@open', next: '@python_block', nextEmbedded: 'python' }, ],
   and one python block:
   python_block: [ [/\$\$/, { token: 'keyword', next: '@popall', nextEmbedded: '@pop' }], [/"/, 'string', '@string' ]],
3. Then we enter LANGUAGE PYTHON AS $$ "$$" $$, then we will see that the python_block exit even we in a string state, which is contradictory to the doc of monarch (https://microsoft.github.io/monaco-editor/monarch.html#nextEmbedded)

Actual (Problematic) Behavior

Expected Behavior

$$ in doublequote should be considered as a string rather than exit token

Additional Context

If this is a designed behavior, can you guys update the docs about monarch? If not, please fix this bug.

[Henning Dieterichs (hediet)]

$$ in doublequote should be considered as a string rather than exit token

Why do you think it should work like that?

[stefnotch]

Henning Dieterichs (@hediet) I ran into the same thing today. The Monaco documentation at https://microsoft.github.io/monaco-editor/monarch.html says

Also inside the CSS we use the string state to tokenize CSS strings such that we do not stop the CSS block when we find </style> inside a string.

I interpreted that just like Beishao Cao (@BeishaoCao-db) . When the [/"/, 'string', '@string' ] matches, then we're temporarily inside the @string state and thus won't exit the Python language.

[Beishao Cao (BeishaoCao-db)]

$$ in doublequote should be considered as a string rather than exit token

Why do you think it should work like that?

Hi Henning Dieterichs (@hediet) , thanks for your reply. As shown in the code, since we are already in the string substate, I think the $$ should be consider as a part in the string rather than an exit token.