Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding CPU/GPU distribution to the logs and reports #11

Open
dan-and opened this issue Jun 11, 2024 · 3 comments
Open

Adding CPU/GPU distribution to the logs and reports #11

dan-and opened this issue Jun 11, 2024 · 3 comments
Assignees
Labels
enhancement New feature or request

Comments

@dan-and
Copy link
Contributor

dan-and commented Jun 11, 2024

While checking your result statistics on https://llm.aidatatools.com/ I always missed a notification if the model was completely loaded into a GPU or if it runs in a mixed environment.

Implementing such a check could be a low-hanging fruit, as ollama keeps the last model running after closing the request at
run_benchmark.py:
75: result = subprocess.run([ollamabin, 'run', model_name, one_prompt['prompt'],'--verbose'], capture_output=True, text=True, check=True, encoding='utf-8')

If you add another call subprocess.run([ollamabin, 'ps'], capture_output=True, text=True, check=True, encoding='utf-8')
you can still gather the utilization.

e.g.:
`
$ ollama ps
NAME ID SIZE PROCESSOR UNTIL
qwen2:1.5b f6daf2b25194 1.8 GB 100% GPU 4 minutes from now

$ ollama ps
NAME ID SIZE PROCESSOR UNTIL
llama3:70b 786f3184aec0 41 GB 79%/21% CPU/GPU 4 minutes from now
`

Based on the ollama documentation, it will be possible to have several models loaded at the same time. So you need to expect that in future, ollama ps will report several rows of models. Adding a filter to the model_name of the ollama ps output should be future-proof .

At the end: it would be great to see that usage/distribution on your results pages.

@chuangtc chuangtc self-assigned this Jun 11, 2024
@chuangtc chuangtc added the enhancement New feature or request label Jun 11, 2024
@nuffin

This comment was marked as duplicate.

@chuangtc

This comment was marked as duplicate.

@nuffin

This comment was marked as duplicate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants