Enhancing LLM Performance with Tool Calling: A Practical Implementation Approach

Abstract

Language models (LLMs) struggle with handling real-time, personalized data efficiently. Traditional approaches, such as direct context injection and retrieval-augmented generation (RAG), have limitations when dealing with dynamic, user-specific information.

This paper presents an optimized tool-calling architecture, referred to as the Advisor Chain, which enhances LLM performance by dynamically selecting and processing relevant data before final inference. Our implementation significantly reduces token usage, improves response accuracy, and minimizes latency through structured query decomposition and controlled execution of external tools.

---

1. Introduction

1.1 The Challenge of Context Overload in LLMs

The traditional method of injecting all available data into an LLM prompt presents significant inefficiencies:

• Context bloat: Including unnecessary past data increases token usage and decreases retrieval accuracy.
• RAG limitations: Standard vector search does not work well for real-time, user-personalized data.
• SQL query inefficiencies: While structured databases are useful, they are limited in handling semantic reasoning beyond keyword matching.

1.2 Why Tool Calling?

Tool calling provides a structured mechanism for dynamic data retrieval. Instead of forcing LLMs to process all available information at once, tool calling allows them to:

1. Retrieve only the necessary data at inference time.
2. Reduce prompt size, lowering costs and improving accuracy.
3. Apply reasoning layers before generating a response, enhancing contextual relevance.

---

2. Implementation Strategy

There are multiple ways to implement tool calling within an LLM architecture:

1. Single-chain integration (directly embedded in the model prompt).
2. Separate chain execution (tool calling handled independently before LLM inference).
3. Agent-based systems (adaptive tool use with multiple iterations).

Our implementation uses option (2) - Separate Chain Execution, as it offers:

• Better control over tool execution.
• Easier debugging and maintainability.
• More predictable behavior, avoiding infinite loops found in agent-based methods.

2.1 The Advisor Chain Architecture

We designed an intermediate processing layer, the Advisor Chain, which intercepts user queries, determines the best external tools to call, and structures responses before passing them to the LLM.

Key advantages of this approach:

• Ensures tool execution consistency (unlike agent models, which may skip tools).
• Minimizes redundant processing, optimizing speed and cost.
• Provides flexibility in integrating various data sources (real-time, historical, structured, or unstructured).

---

3. Key Performance Metrics and Results

3.1 Reduction in Token Usage

Our implementation led to a significant decrease in token consumption per query:

• Baseline (direct context injection): ~40,000 tokens per query.
• Tool-calling implementation: ~3,000 tokens per query.

This resulted in a 90% reduction in token usage, lowering inference costs and improving response speed.

3.2 Accuracy Improvement

By enabling the model to selectively reason about relevant data, we observed:

• Higher precision in responses.
• Better alignment with real-time user needs.

3.3 Latency Optimization

Although tool calling introduces additional processing steps, we mitigated latency through:

• Optimized prompts (minimizing token output).
• Leveraging prompt caching (significantly reducing redundant computation).
• Asynchronous tool execution (parallel processing of multiple calls when possible).

---

4. Best Practices for Tool Calling

4.1 Ensuring Tools Are Called

To enforce proper tool execution, we:

• Used strict prompting to explicitly instruct the LLM on when and how to invoke tools.
• Selected tool-friendly models (e.g., openai/gpt-4o-mini with strict_model = True and tools_use = "required").

4.2 Improving Tool Selection Accuracy

Ensuring the correct tools were invoked required:

• Explicitly defining function arguments and constraints.
• Providing structured selection guidelines, such as:

- Start with the most specific processor that matches query intent.
- Add complementary processors only if they provide essential additional data.
- Prefer time-bounded processors (e.g., 30-day history) over unlimited ranges.
- For user attributes or profile info, always use the latest version.

Incorporating an optional “planning” argument, allowing the model to pre-select the best tool set before execution.

4.3 Latency Reduction Techniques

To maintain system efficiency: • Optimized tool output to be concise. • Used prompt caching to leverage previous results. • Made “planning” calls optional to reduce expensive reasoning steps when unnecessary.

5. Applications and Future Enhancements

The Advisor Chain framework extends beyond tool calling: 1. Intent classification – Dynamically route queries to appropriate models or workflows. 2. Automated feedback collection – Analyze user interactions to refine model behavior (e.g., detecting frustration signals like “I don’t like this answer”). 3. Hybrid data integration – Incorporate structured (SQL) and unstructured (RAG) data sources seamlessly. 4. External API calls – Extend capabilities to shopping recommendations, product matching, or real-time analytics. 5. Device control – Enable LLMs to execute commands directly on hardware (e.g., adjusting skincare device settings).

6. Conclusion

Tool calling provides a structured, efficient way to optimize LLM queries, improving both accuracy and cost-effectiveness. Our Advisor Chain implementation ensures: • Selective data retrieval, reducing token costs. • Improved reasoning quality, leading to more precise responses. • Scalable, flexible architecture adaptable to multiple AI applications.

By adopting these techniques, AI-driven systems can handle real-time, personalized data more effectively, unlocking new capabilities in intelligent automation and decision-making.

7. References

1.	Multi-Needle in a Haystack: Context Retrieval Challenges
2.	Lost in the Middle: How Language Models Use Long Contexts
3.	OpenAI API Documentation: Tool Calling and Function Execution