[Bug]: Rag does not tokenise PDF upload

[dcolley]

What happened?

Unable to load a PDF file as message context or in Attached Files

Steps to Reproduce

1. Add lmstudio endpoint to librechat.yml

endpoints:
  custom:
    - name: 'lmstudio'
      apiKey: 'lmstudio'
      baseURL: 'http://192.168.1.107:1234/v1/'
      models:
        default:
          [
            'Meta-Llama-3-8B-Instruct-GGUF',
          ]
        fetch: false
      titleConvo: true
      titleModel: 'Meta-Llama-3-8B-Instruct-GGUF'
      modelDisplayLabel: 'lmstudio'
      temperature: 0.5
      top_p: 5
      stream: true

2. uncomment RAG API in docker-compose.override.yml

# USE RAG API IMAGE WITH LOCAL EMBEDDINGS SUPPORT
  rag_api:
    image: ghcr.io/danny-avila/librechat-rag-api-dev:latest
    environment:
      - RAG_OPENAI_BASEURL=http://192.168.1.107:1234/v1
      - RAG_OPENAI_API_KEY=lmstudio

3. launch

docker compose up

4. Start a new chat, load a simple PDF file, check the RAG_API docker logs

2024-06-12 17:51:21 2024-06-12 16:51:21,306 - root - INFO - Request POST http://rag_api:8000/embed - 200
2024-06-12 17:52:47 2024-06-12 16:52:47,102 - httpx - INFO - HTTP Request: POST http://192.168.1.107:1234/v1/embeddings "HTTP/1.1 400 Bad Request"
2024-06-12 17:52:47 2024-06-12 16:52:47,102 - root - ERROR - Error code: 400 - {'error': "'input' field must be a string or an array of strings"}
2024-06-12 17:52:47 2024-06-12 16:52:47,103 - root - INFO - Request POST http://rag_api:8000/embed - 200

What browsers are you seeing the problem on?

Chrome

Relevant log output

rag_api       | 2024-06-12 17:02:36,870 - httpx - INFO - HTTP Request: POST http://192.168.1.107:1234/v1/embeddings "HTTP/1.1 400 Bad Request"
rag_api       | 2024-06-12 17:02:36,872 - root - ERROR - Error code: 400 - {'error': "'input' field must be a string or an array of strings"}
rag_api       | 2024-06-12 17:02:36,873 - root - INFO - Request POST http://rag_api:8000/embed - 200
LibreChat     | 2024-06-12 17:02:36 error: Error embedding file File embedding failed.
LibreChat     | 2024-06-12 17:02:36 error: [/files] Error processing file: File embedding failed.


2024-06-12 17:51:21 2024-06-12 16:51:21,306 - root - INFO - Request POST http://rag_api:8000/embed - 200
2024-06-12 17:52:47 2024-06-12 16:52:47,102 - httpx - INFO - HTTP Request: POST http://192.168.1.107:1234/v1/embeddings "HTTP/1.1 400 Bad Request"
2024-06-12 17:52:47 2024-06-12 16:52:47,102 - root - ERROR - Error code: 400 - {'error': "'input' field must be a string or an array of strings"}
2024-06-12 17:52:47 2024-06-12 16:52:47,103 - root - INFO - Request POST http://rag_api:8000/embed - 200

Screenshots

Error processing file

Code of Conduct

• I agree to follow this project's Code of Conduct

[dcolley]

The local lmstudio server is able to answer queries from code:

import { OpenAI, OpenAIEmbeddings } from "@langchain/openai";
import { GithubRepoLoader } from "langchain/document_loaders/web/github";
import { HNSWLib } from "@langchain/community/vectorstores/hnswlib";
import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import { RetrievalQAChain } from "langchain/chains";
import { Document } from "langchain/document";
import * as fs from "fs";

import { Chroma } from "@langchain/community/vectorstores/chroma";

const fields = {
  openAIApiKey: 'lmstudio', // process.env.OPENAI_API_KEY,
  temperature: 0.1,
}
const config = {
  //apiKey: 'lmstudio', // process.env.OPENAI_API_KEY,
  // baseURL: "http://192.168.1.107:1234/v1",
  baseURL: "https://abd9-31-22-13-147.ngrok-free.app/v1",  // proxy in order to debug payload...
  modelName: "lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf",
}
const model = new OpenAI(fields, config) as any;
const embeddings = new OpenAIEmbeddings(fields, config);

export const run = async () => {
  console.log("Running");

  try {

    // create vector store from chroma
    const vectorStore = new Chroma(embeddings, { url: 'http://localhost:8000', collectionName: 'polkadot-sdk' });

    // Create a chain that uses the OpenAI LLM and HNSWLib vector store.
    const chain = RetrievalQAChain.fromLLM(
      model,
      vectorStore.asRetriever(5),
      {
        returnSourceDocuments: true,
      }
    );

    const followUp = await chain.call({
      temperature: 0.1,
      query: 'hi, tell me about the polkadot sdk',
    });
    console.log({ followUp });
    console.log(JSON.stringify(followUp.sourceDocuments, null, 2));

  } catch (error) {
    console.error("An error occurred:", error);
  }
};

run();

in ngrok I see this request:

POST /v1/embeddings HTTP/1.1
Host: abd9-31-22-13-147.ngrok-free.app
User-Agent: OpenAI/JS 4.50.0
Content-Length: 145
Accept: application/json
Accept-Encoding: gzip,deflate
Authorization: Bearer lmstudio
Content-Type: application/json
X-Forwarded-For: 31.22.13.147
X-Forwarded-Host: abd9-31-22-13-147.ngrok-free.app
X-Forwarded-Proto: https
X-Stainless-Arch: arm64
X-Stainless-Lang: js
X-Stainless-Os: MacOS
X-Stainless-Package-Version: 4.50.0
X-Stainless-Runtime: node
X-Stainless-Runtime-Version: v18.17.0

{
  "model": "text-embedding-ada-002",
  "input": "hi, tell me about the polkadot sdk"
}

[dcolley]

When i attempt to load a PDF I see this in ngrok

POST /v1/embeddings HTTP/1.1
Host: abd9-31-22-13-147.ngrok-free.app
User-Agent: OpenAI/Python 1.30.1
Content-Length: 99
Accept: application/json
Accept-Encoding: gzip, deflate
Authorization: Bearer lmstudio
Content-Type: application/json
X-Forwarded-For: 31.22.13.147
X-Forwarded-Host: abd9-31-22-13-147.ngrok-free.app
X-Forwarded-Proto: https
X-Stainless-Arch: arm64
X-Stainless-Async: false
X-Stainless-Lang: python
X-Stainless-Os: Linux
X-Stainless-Package-Version: 1.30.1
X-Stainless-Runtime: CPython
X-Stainless-Runtime-Version: 3.10.14

{"input": [[4919, 499, 29385, 1082, 30]], "model": "nomic-embed-text", "encoding_format": "base64"}

[larspohlmann]

I ran into the same problem. @dcolley : Did you find a solution for that?

[larspohlmann]

After some digging, I found a workaround:

In the rag-api container edit the file: config.py

Add the parameter "check_embedding_ctx_length=False" to the OpenAIEmbeddings() call, it will then look like this:

return OpenAIEmbeddings( check_embedding_ctx_length=False, model=model, api_key=RAG_OPENAI_API_KEY, openai_api_base=RAG_OPENAI_BASEURL, openai_proxy=RAG_OPENAI_PROXY, )

[larspohlmann]

I added a feature request here: danny-avila/rag_api#115