Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compute better scoring for node performance #232

Closed
2 tasks
jorgeantonio21 opened this issue Feb 14, 2025 · 0 comments · Fixed by #252
Closed
2 tasks

Compute better scoring for node performance #232

jorgeantonio21 opened this issue Feb 14, 2025 · 0 comments · Fixed by #252
Assignees

Comments

@jorgeantonio21
Copy link
Contributor

Following issue atoma-network/atoma-node#410, we need to improve the node performance scoring computation, through by incorporating these more fine grained collected metrics. Ideally:

  • Node specs are used to compute scores. These include KV cache allocations, thermal power values, cpu/gpu usage, bandwidth capacity.

These scores are used by the proxy to select the best nodes available to newly purchased Stacks (ahead of time). Then the proxy can use metrics around usage and performance, such as:

  • Number of allocated KV blocks, node processed requests, throughput and latency (ITL and TTFT) across the board, to pick the most available node to process a new request (real time).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants