Ability to rewrite LLM input

[Ziaunys]

Is there currently any way to rewrite the LLM input using this extension? I see that with the OutputGuardrails you can do this by using successWith(). However, there doesn't appear to be a similar API for InputGuardrails. Is there another way to do this outside of just sanitizing the input before passing it to the LLM? It's fine if I have to do that.

The problem I'm running into is that if I don't sanitize the inputs myself then users can pass in {} characters which get interpolated by Qute templates. I'd like to avoid this.

[geoand]

cc @cescoffier

[cescoffier]

@mariofusco added this a few months ago. It should be possible to write an input guardrail that modify the user input.

[Ziaunys]

Oh, excellent. I happened to be looking at an older version. Thank you! Exactly what I wanted. 🙏

[Ziaunys]

I just updated the extension and tried using successWith(String rewrittenInput) on the validate(String userMessage) method, however, it looks like the Qute template evaluation happens before the guardrails run which results in a template exception thrown if there is any user input like:

Hello {bar} This will throw an exception

Exception:

: io.quarkus.qute.TemplateException: Rendering error: Key "bar" not found in the template data map with keys [current_date, response_schema, memoryId, current_time, current_date_time] in expression {bar}
        at io.quarkus.qute.TemplateException$Builder.build(TemplateException.java:169)
        at io.quarkus.qute.EvaluatorImpl.propertyNotFound(EvaluatorImpl.java:234)
        at io.quarkus.qute.EvaluatorImpl.resolve(EvaluatorImpl.java:204)
        at io.quarkus.qute.EvaluatorImpl.resolveReference(EvaluatorImpl.java:131)
        at io.quarkus.qute.EvaluatorImpl.evaluate(EvaluatorImpl.java:85)
        at io.quarkus.qute.ResolutionContextImpl.evaluate(ResolutionContextImpl.java:29)
        at io.quarkus.qute.ExpressionNode.resolve(ExpressionNode.java:36)
        at io.quarkus.qute.Results.resolveWith(Results.java:107)
        at io.quarkus.qute.Results.resolveAndProcess(Results.java:57)
        at io.quarkus.qute.SectionNode$SectionResolutionContextImpl.execute(SectionNode.java:238)
        at io.quarkus.qute.SectionHelper$SectionResolutionContext.execute(SectionHelper.java:66)
        at io.quarkus.qute.Parser$1.resolve(Parser.java:1331)
        at io.quarkus.qute.SectionNode.resolve(SectionNode.java:53)
        at io.quarkus.qute.SectionNode.resolve(SectionNode.java:58)
        at io.quarkus.qute.TemplateImpl$TemplateInstanceImpl.renderData(TemplateImpl.java:272)
        at io.quarkus.qute.TemplateImpl$TemplateInstanceImpl.renderAsyncNoTimeout(TemplateImpl.java:239)
        at io.quarkus.qute.TemplateImpl$TemplateInstanceImpl.render(TemplateImpl.java:164)
        at io.quarkiverse.langchain4j.QuarkusPromptTemplateFactory$QuteTemplate.render(QuarkusPromptTemplateFactory.java:69)
        at dev.langchain4j.model.input.PromptTemplate.apply(PromptTemplate.java:102)
        at io.quarkiverse.langchain4j.runtime.aiservice.AiServiceMethodImplementationSupport.prepareUserMessage(AiServiceMethodImplementationSupport.java:675)
        at io.quarkiverse.langchain4j.runtime.aiservice.AiServiceMethodImplementationSupport.doImplement(AiServiceMethodImplementationSupport.java:162)
        at io.quarkiverse.langchain4j.runtime.aiservice.AiServiceMethodImplementationSupport.implement(AiServiceMethodImplementationSupport.java:138)
        at io.quarkiverse.langchain4j.runtime.aiservice.MethodImplementationSupportProducer$1$1.apply(MethodImplementationSupportProducer.java:31)
        at io.quarkiverse.langchain4j.runtime.aiservice.MethodImplementationSupportProducer$1$1.apply(MethodImplementationSupportProducer.java:28)
        at io.quarkiverse.langchain4j.runtime.aiservice.SpanWrapper.wrap(SpanWrapper.java:32)
        at io.quarkiverse.langchain4j.runtime.aiservice.MethodImplementationSupportProducer$1$2.apply(MethodImplementationSupportProducer.java:40)
        at io.quarkiverse.langchain4j.runtime.aiservice.MethodImplementationSupportProducer$1$2.apply(MethodImplementationSupportProducer.java:37)
        at io.quarkiverse.langchain4j.runtime.aiservice.MethodImplementationSupportProducer$1.implement(MethodImplementationSupportProducer.java:46)
        at io.smallrye.context.impl.wrappers.SlowContextualRunnable.run(SlowContextualRunnable.java:19)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1848)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:627)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
        at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1575)

[Ziaunys]

Another update. I was able to work around this issue by setting the following properties:

quarkus.qute.strict-rendering=false
quarkus.qute.property-not-found-strategy=output-original

This is maybe not the best solution but at least an exception won't be thrown and I can handle the input in guardrails.

[Ziaunys]

I created a simple project with a basic assistant and a test case to demonstrate the issue.
Test case:
https://github.com/Ziaunys/langchain4j-quarkus-sandbox/blob/main/src/test/java/dev/ziaunys/TestAssistant.java
Assistant interface:
https://github.com/Ziaunys/langchain4j-quarkus-sandbox/blob/main/src/main/java/dev/ziaunys/Assistant.java

[geoand]

Are you sure you've added all the code to the sample? Asking because I can neither reproduce the problem nor do I see any guardrails

[Ziaunys]

I didn't initially add a guardrail because I was just trying to demonstrate that the exception gets thrown when you pass in {foo} before the guardrail is even evaluated. I have now added the guardrail as well just to show it never gets called.
See:
https://github.com/Ziaunys/langchain4j-quarkus-sandbox/blob/main/src/main/java/dev/ziaunys/UnreachableGuardrail.java

Does the test fail for you? It should pass if you reproduce the issue since I'm using assertThrows() to ensure that TemplateException is thrown.

[geoand]

Adding {foo} to a template that nevers gets passed any value for foo is expected to fail.

[Ziaunys]

I guess I just didn't expect to be able to pass in template strings to the .chat() which means I as a user of the API have to care about the implementation details of Qute templates and sanitize the String beforehand. If that is the answer then that is fine. I just wasn't sure if this was expected behavior.