Tweaking embedding and inference settings for even better performance.

[ww2283]

I spinned up LLocalSearch with the default nomic-embed-text:v1.5 and hermes-2-Pro-Mistral-7B models for preliminary testing on a CPU with four threads and an 8GB vRAM card. The outcomes significantly surpassed my expectations. I feel a substantial potential for further enhancement, particularly with the employment of larger models. I have a Threadripper workstation with two Ada6000 cards, prompting me to contemplate which embedding and inference models would optimally enhance performance? Additionally, I am curious about the possibility of adjusting the context length for the search results input into the inference model, especially if it can accommodate much longer contexts, such as 32k or more. I greatly appreciate the effort invested in this project and eagerly anticipate future developments.

[nilsherzig]

Check out the latest release (new video in readme) I just pushed a settings window, which should cover the things you've mentioned in your message :)

Edit: Appreciate the kind words :)

[ww2283]

Much appreciate the new improvement.
Some updates on my usage front: I tried with 2x Ada6000 loading the WizardLM-2 8x22B at q_4 and the result is stunning. The speed is fast, per usage it is noticeably faster than using ChatGPT.

Is there a plan for automate the thread saving, like Ollama is doing?

[nilsherzig]

hahah i could not resit to test it out myself

automated thread saving? like "recent chats"? yea i have that ready on my own build, but i have to fix a couple of small bugs before publishing it