Towards a new annotation-based model

[cescoffier]

Hello,

This topic has been discussed many times (briefly) in several issues (#237, #210, #42, #41, #10). This discussion aims to see if another annotation-based model is possible and what it would look like.

Current approach

Currently, declarative AI services are annotated with @RegisterAiService. In this annotation, we configure the memory, retriever, tools, moderation model, and chat model... Then, in the application.properties, we configure the LLM provider used by this service.

This design is simple and avoids having many annotations, but it also has a few limits:

• when you have multiple chat models, the selection requires a supplier. It is flexible but convoluted.
• tools and document retrievers are global to all methods of the AI service. This will likely be a more significant concern with the advanced RAG support from langchain4j.
• the system message must be repeated on each method

In this discussion, I would like to discuss a more sophisticated model.

The overall idea

The main ideas are:

• ease the selection of the LLM
• allows retriever and tools declaration on methods

A simpler @RegisterAiService

We first need to simplify the @RegisterAiService annotation to achieve these goals. The retriever and tools attributes will be deprecated and planned for removal.

AI qualifier or new RegisterAiService attribute

The second step is to allow the selection of the LLM to be more easily (without the supplier). There are several ways to do this, such as adding a "name" attribute or a dedicated qualifier (annotation):

@RegisterAiService(llmName="my-llm")
// or
@RegisterAiService
@LLM("my-llm")

This will enable multiple instance configurations of each LLM:

# Default
quarkus.langchain4j.openai.timeout=60s
quarkus.langchain4j.openai.my-llm.timeout=60s

Open Question: Should it be quarkus.langchain4j.<name>.<provider>.attribute=value, or quarkus.langchain4j.<provider>.<name>.attribute=value.

IMPORTANT: An AI service is bound to a single LLM in this proposal. Multi-model AI Service is out of scope. The workaround is to use multiple interfaces. One issue with multi-models is handling the "memory," as the included messages are specific to the LLM.

SystemMessage on the class

The system message (when supported) allows explaining the "role" or "scope" to the LLM. A system message requires a memory of at least 3 (system + user + response).

Currently, we need to configure the system message on each method of the AI service interface. It would be helpful to configure it once. It would also:

• create a natural boundary to the methods from the AiService, like a bounded context
• avoid sending multiple times the system message when we call several methods (and use the memory)
• guarantee that the system message is the same for all the methods

Tools and Retriever on method

Unlike the system message, tools and retrievers need a bit more flexibility. Adding documents to every call might not be helpful and can be misleading for the LLM. It can also be better to restrict the method allowed to access tools.

So, the idea would be to define two new annotations:

• DocumentRetriever ContentRetriever - this annotation can select the document retriever, but it might need to be changed with the advanced RAG idea emerging in langchain4j
• Toolbox - the list of classes containing methods annotated with @Tool (note: declaring a class that does not contain @Tool should be considered illegal).

The tools and document retriever would be added when the method is invoked instead of all the calls.

Example

@RegisterAiService
@SystemMessage("""
        You are MovieMuse, an AI answering questions about the top 100 movies from IMDB.
        Your response must be polite, use the same language as the question, and be relevant to the question.

        Introduce yourself with: "Hello, I'm MovieMuse, how can I help you?"
        """)
@Singleton 
public interface MovieMuse {

    @ContentRetriever((RetrieverExample.class) // Edited: was DocumentRetriever initially
    String chat(@MemoryId Object session, @UserMessage String question);
}

@RegisterAiService
@SystemMessage("You are a professional poet")
public interface MyAiService {

    @UserMessage("Write a poem about {topic}. The poem should be {lines} lines long. Then send this poem by email.")
    @Toolbox(EmailService.class)
    String writeAndSendAPoem(String topic, int lines);
    
    @UserMessage("Write a poem about {topic}. The poem should be {lines} lines long.")
    String writePoem(String topic, int lines);

Concerns

This proposal has a few drawbacks:

• it complexifies the code as it requires more checks, but the resulting model is more flexible.
• some parts are unclear - typically, everything related to RAG needs to work with the new advanced RAG idea
• it would require the creation of annotations specific to Quarkus

[geoand]

I think this make a lot of sense. The @SystemMessage one should be easy and the start is here.

The rest is totally doable and I do think it's the proper way to provide more control. My question really is, should we do it now or wait until this is really needed?

[geoand]

@langchain4j are you okay with the implied changes in the langchain4j annotations?

[geoand]

Yeah @Toolbox.

I just saw that I already made the change to @SystemMessage :)

[geoand]

I was about to start implementing @ToolBox in LangChain4j, but then realized that it won't be tremendously useful as users would likely still need to provide a description for the function (and perhaps even configure the name) - therefore they would still need to use @Tool

[geoand]

Then I must be missing something, as @RegisterAiService currently has tools where a user passes the classes that contain methods with @Tool

[geoand]

@langchain4j have people asked for something like that? Why do they need that kind of control?

[geoand]

Very interesting

[cescoffier]

As far as I remember (aka trying to decypher my handwritten notes...), the idea was to remove tools from the @RegistewrAiService but declare them per method. As we cannot reuse the @Tool annotation (declaring a tool), I went with @Toolbox.

[geoand]

Ah okay, so similar high level idea as @langchain4j