Suggestion: make public method instead of protected

[jlguenego]

RULE, SUBRULE, MANY, OPTIONS, CONSUME...

All these methods are protected.

I am currently implementing ASN1 rules, and they are a lot. So I would like to split my file in many. I had to wrap all these method in public one in order to do it...

So I would suggest all of these method to be public.

Thanks.

[bd82]

This may be a good point.

Splitting up a large grammar into many files in a good practice, e.g:

• https://github.com/jhipster/prettier-java/tree/main/packages/java-parser/src/productions

However there are dozens of these utility methods which would make auto-complete suggestions on a parser instance very crowded.

TypeScript is just a design time tool, there is no runtime meaning if something is private, protected or public.
Could you provide a small example of a grammar you are trying to split into multiple files?
Perhaps some missing typescript cast/type assertion would be able to workaround the problem.

[jlguenego]

You can see my open source project : https://github.com/jlguenego/asn.1
The part where I am using chevrotain is in the following directory : https://github.com/jlguenego/asn.1/tree/master/src/analysis
I want to parse ASN1 files and DER messages.
ASN1 spec are big : https://www.itu.int/rec/T-REC-X.680

So that's why I would like to split the CstParser class in many part.

[bd82]

Perhaps instead of passing the parser instance as an argument
Try to pass it as the this context, using call or apply.

• Relevant docs: https://www.typescriptlang.org/docs/handbook/functions.html#this-parameters

> function bar () { return this.x +1 }
undefined
> const thisContext = {x : 2}
undefined
> bar.call(thisContext)

// 
export function initConstrainedTypeRules(this: ASN1CstParser) {
   this.MANY3(/*...*/):
}

[jlguenego]

Great ! Yes your suggestion works. And I think this is acceptable. Thank you for this. Up to you to close this issue.

[mtiller]

I'm looking into this solution. It is still a bit awkward, but it seems like it will definitely help with the organization of the code. Another alternative to consider would be to create an interface with all those methods and have a way to export (from a CstParser) a proxy of some kind with all those methods defined on it. Make the proxy creation method protected, but what it returns wouldn't be. This would allow you to export rules by passing the proxy to functions that defined rules.

Just a thought.

[mtiller]

Just to follow up. I tried out the approach I mentioned previously. It worked pretty well. What I did was first define the interface:

export interface RuleProxy {
  RULE: CstParser["RULE"];
  CONSUME: CstParser["CONSUME"];
  CONSUME1: CstParser["CONSUME1"];
  CONSUME2: CstParser["CONSUME2"];
  CONSUME3: CstParser["CONSUME3"];
  CONSUME4: CstParser["CONSUME4"];
  CONSUME5: CstParser["CONSUME5"];
  CONSUME6: CstParser["CONSUME6"];
  CONSUME7: CstParser["CONSUME7"];
  CONSUME8: CstParser["CONSUME8"];
  CONSUME9: CstParser["CONSUME9"];
  OR: CstParser["OR"];
  OR1: CstParser["OR1"];
  OR2: CstParser["OR2"];
  OR3: CstParser["OR3"];
  OR4: CstParser["OR4"];
  OR5: CstParser["OR5"];
  OR6: CstParser["OR6"];
  OR7: CstParser["OR7"];
  OR8: CstParser["OR8"];
  OR9: CstParser["OR9"];
  AT_LEAST_ONE: CstParser["AT_LEAST_ONE"];
  AT_LEAST_ONE_SEP: CstParser["AT_LEAST_ONE_SEP"];
  MANY: CstParser["MANY"];
  MANY_SEP: CstParser["MANY_SEP"];
  SUBRULE: CstParser["SUBRULE"];
}

I then defined an ExternalRule type:

export type ExternalRule = (
  proxy: RuleProxy,
  rules: MyParser
) => ReturnType<CstParser["RULE"]>;

Then I can write an external rule like:

export const importClause: ExternalRule = (proxy, rules) => {
  return proxy.RULE("importStmt", () => {
    proxy.CONSUME(baseTokens.import);
    proxy.CONSUME1(baseTokens.identifier);
    proxy.CONSUME(baseTokens.from);
    proxy.SUBRULE(rules.packageName);
    proxy.CONSUME(baseTokens.semi);
  });
};

I kind of like this approach because there is a clear distinction between methods that are used to build rules (since they reference proxy) and existing rules associated with my parser (since they reference rules).

Finally, my actual MyParser class is much simpler because it is just full of lines like this (for all the rules that need to be public, i.e., the top rule and any subrules referenced in other rules):

export class MyParser extends CstParser {
  constructor() {
    super(baseTokens);
    this.performSelfAnalysis();
  }

  protected get proxy(): RuleProxy {
    return this as any; // Cast just to shed the "protected" part, better than writing proxy functions for each field
  }

  public top = externalTopRule(this.proxy, this);
  public common1 = someCommonRule(this.proxy, this);
  ...
}

All the details of the rules can then be stored in other files and imported. No having to use function, .bind or .apply or anything.

Another thing I found interesting about this is that if you have a subrule that is not shared, you can actually define it inside the rule that uses it, e.g.,

export const commonRule: ExternalRule = (proxy) => {
    const nested = proxy.RULE("nested", () => { ... });
    return proxy.RULE("common", () => {
        ...
        this.MANY(nested);
        ...
    });
}

In other words, you can encapsulate such "utility" rules in such a way that they are not really seen unless you dig into the details of how the rule that invokes them is defined which means you can keep them out of the parser definition entirely.

It might be useful to have the RuleProxy interface defined in chevrotain itself (there is nothing specific to my application there). You could even create a parameterized ExternalRule type and include that as well:

export type ExternalRule<P> = (
  proxy: RuleProxy,
  rules: P // Any class that contains rules...doesn't even need to be the parser class itself
) => ReturnType<CstParser["RULE"]>;

[bd82]

Hello again @mtiller

As Chevrotain Grammars are just plain JavaScript there are multiple opinionated ways to achieve splitting up a grammar into multiple files. I don't think I am interested in providing "native" support for the specific approach you have demonstrated as I'm trying to reduce the library's scope. However, this discussion does show the need for an example
with a multiple files grammar.

Feedback on your suggested pattern:

• How do we decide which methods RuleProxy needs to expose?
  • for example what about LA() method that is used in GATES?
• You may be able to define the Proxy's interface more easily using TypeScript Pick mapped type
  • https://www.typescriptlang.org/docs/handbook/advanced-types.html#mapped-types
• Won't you lose auto-complete (in IDE) for other SUBRULE names when accessing them via the proxy?

[jlguenego]

export function initConstrainedTypeRules(this: ASN1CstParser) {
   this.MANY3(/*...*/):
}

Using the this parameter...

...is a way to seperate the code in multiple files. However, it creates circular dependancies, which may be hard for webpack or other tools do tree shaking. Personnaly I am still hoping theses method would be public one day because instead of inheritance, we could use encapsulation.

On my free time, I did my own syntax analysis library in Typescript (for educational purpose). May be you can have a look at it to see how I see things about declaring a Context Free Grammar. I tried to stick how Aho Ullman books and CS143 present things.

https://github.com/jlguenego/syntax-analysis

[bd82]

I am not opposed to making the methods public to improve compatibility with some implementation patterns.
But if I make these methods public it could damage the experience in more common use cases.

e.g: if the methods are public, they would be offered as auto-complete suggestions on an instance of CstParser.
Instead of only offering the new methods (grammar rules) defined by the end users.

One way to make the methods public at the end user level is to use patch-package to modify the d.ts file included with Chevrotain.

• https://www.npmjs.com/package/patch-package

Would this help?

[mbett7]

Using CstParser instead of ASN1CstParser for the this parameter would avoid circular dependencies. Or possibly the type-only imports introduced in typescript 3.8.

Alternatively extract an interface for the parser. Could then add a mixin involving that interface to avoid duplicate declaration of rules in the concrete parser.

I'm skeptical there is a use case where making them public instead of protected solves a problem that can't be done with creative typing.