Generating languages for Monarch / Monaco Editor

[kevinkhill]

What do you think about using a defined language in Chev and generating an object that would work when plugged into here...???

I swapped the editor on my playground from CodeMirror to Monaco and was wondering how hard it would be to make something similar to the d.ts generator but to generate an object with the structure that would seamlessly work to highlight (and even semantically check????) my language? 🤯 ????

Just wondering if this is far fetched / contrived or if maybe an easy task.....

[kevinkhill]

Example of the structure to avoid clicking a link ^_^

This is the javascript language definition

// Create your own language definition here
// You can safely look at other samples without losing modifications.
// Modifications are not saved on browser refresh/close though -- copy often!
return {
  // Set defaultToken to invalid to see what you do not tokenize yet
  // defaultToken: 'invalid',

  keywords: [
    'abstract', 'continue', 'for', 'new', 'switch', 'assert', 'goto', 'do',
    'if', 'private', 'this', 'break', 'protected', 'throw', 'else', 'public',
    'enum', 'return', 'catch', 'try', 'interface', 'static', 'class',
    'finally', 'const', 'super', 'while', 'true', 'false'
  ],

  typeKeywords: [
    'boolean', 'double', 'byte', 'int', 'short', 'char', 'void', 'long', 'float'
  ],

  operators: [
    '=', '>', '<', '!', '~', '?', ':', '==', '<=', '>=', '!=',
    '&&', '||', '++', '--', '+', '-', '*', '/', '&', '|', '^', '%',
    '<<', '>>', '>>>', '+=', '-=', '*=', '/=', '&=', '|=', '^=',
    '%=', '<<=', '>>=', '>>>='
  ],

  // we include these common regular expressions
  symbols:  /[=><!~?:&|+\-*\/\^%]+/,

  // C# style strings
  escapes: /\\(?:[abfnrtv\\"']|x[0-9A-Fa-f]{1,4}|u[0-9A-Fa-f]{4}|U[0-9A-Fa-f]{8})/,

  // The main tokenizer for our languages
  tokenizer: {
    root: [
      // identifiers and keywords
      [/[a-z_$][\w$]*/, { cases: { '@typeKeywords': 'keyword',
                                   '@keywords': 'keyword',
                                   '@default': 'identifier' } }],
      [/[A-Z][\w\$]*/, 'type.identifier' ],  // to show class names nicely

      // whitespace
      { include: '@whitespace' },

      // delimiters and operators
      [/[{}()\[\]]/, '@brackets'],
      [/[<>](?!@symbols)/, '@brackets'],
      [/@symbols/, { cases: { '@operators': 'operator',
                              '@default'  : '' } } ],

      // @ annotations.
      // As an example, we emit a debugging log message on these tokens.
      // Note: message are supressed during the first load -- change some lines to see them.
      [/@\s*[a-zA-Z_\$][\w\$]*/, { token: 'annotation', log: 'annotation token: $0' }],

      // numbers
      [/\d*\.\d+([eE][\-+]?\d+)?/, 'number.float'],
      [/0[xX][0-9a-fA-F]+/, 'number.hex'],
      [/\d+/, 'number'],

      // delimiter: after number because of .\d floats
      [/[;,.]/, 'delimiter'],

      // strings
      [/"([^"\\]|\\.)*$/, 'string.invalid' ],  // non-teminated string
      [/"/,  { token: 'string.quote', bracket: '@open', next: '@string' } ],

      // characters
      [/'[^\\']'/, 'string'],
      [/(')(@escapes)(')/, ['string','string.escape','string']],
      [/'/, 'string.invalid']
    ],

    comment: [
      [/[^\/*]+/, 'comment' ],
      [/\/\*/,    'comment', '@push' ],    // nested comment
      ["\\*/",    'comment', '@pop'  ],
      [/[\/*]/,   'comment' ]
    ],

    string: [
      [/[^\\"]+/,  'string'],
      [/@escapes/, 'string.escape'],
      [/\\./,      'string.escape.invalid'],
      [/"/,        { token: 'string.quote', bracket: '@close', next: '@pop' } ]
    ],

    whitespace: [
      [/[ \t\r\n]+/, 'white'],
      [/\/\*/,       'comment', '@comment' ],
      [/\/\/.*$/,    'comment'],
    ],
  },
};

[kevinkhill]

Dig dig dig..... THEY ALREADY HAVE A TYPE FOR IT 😁

[medihack]

Sounds like a nice idea, but I think that would be an immense task. I am in the middle of implementing a language service with Monaco integration and the parser (of course written with Chevrotain) is just a little part of it. Even for just generating the Monarch file a lot of metadata is missing that would be needed to be attached to the parser somehow. In the end, I guess you would be attaching so much metadata to your parser that you could have written the Monarch configuration itself (and that's just the Monarch part). When adding validation and auto-completion there is additional stuff that is very much dependent on your language. And then there is also that web worker stuff for the Monaco integration.

As a reference, if you want to build a language service and Monaco integration yourself I can recommend the below resources. It's the easiest I found, but uses a handwritten parser:
https://github.com/microsoft/vscode-css-languageservice
https://github.com/microsoft/monaco-editor/tree/main/src/basic-languages/css
https://github.com/microsoft/monaco-editor/tree/main/src/language/css

Another good example of a language service that uses Chevrotain is https://github.com/SAP/xml-tools.

[kevinkhill]

attached to the parser somehow

I was envisioning the other way, like how TypeScript files are generated form the parser. Make a package that can take our parser and squeeze out the regexes and token names

[bd82]

After a short look at this "Monarch" it seems it is mostly about the syntax highlights, meaning the lexer part of Chevrotain.
It also seems like there is additional metadata that is not included in a Chevrotain lexer as @medihack mentioned.

[kevinkhill]

I have started plodding down the path, just to see where it takes me.... I was able to copy an example from their site and made a little wrapper for customizing....

[kevinkhill]

[bd82]

For writing language servers you may want to have a look at Langium

• https://github.com/langium/langium

Incidentally it is using Chevrotain under the hood 😄