Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce Utils.TokenUsageCollector for token usage tracking #251

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

nallwhy
Copy link
Contributor

@nallwhy nallwhy commented Feb 8, 2025

This PR introduces LangChain.Utils.TokenUsageCollector, a utility module designed to simplify token usage tracking.

Motivation:

Currently, while the on_llm_token_usage callback provides valuable token usage information, there isn't a readily available, best-practice approach for aggregating this data, especially in scenarios involving multiple LLM interactions within a single task. This PR aims to address this by providing a reusable and easy-to-use solution.

Benefits:

  • Simplified Token Tracking: Makes it easier for developers to track token usage across multiple LLM calls.
  • Inspiration for Callback Usage: Offers a practical example of how to effectively utilize the on_llm_token_usage callback.

I believe this module can be a valuable addition to LangChain, providing a more user-friendly experience for managing token usage. I'm open to feedback and suggestions for further improvements.

@nallwhy nallwhy marked this pull request as draft February 10, 2025 00:10
@nallwhy
Copy link
Contributor Author

nallwhy commented Feb 10, 2025

@brainlid
The get_callback/1 function will likely error out when used with ChatBumblebee because ChatBumblebee doesn't have a model attribute in the callback data. I'm looking for suggestions on how best to handle this case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant