Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Submit batch evaluation run on PR #61

Open
jjgriff93 opened this issue Feb 4, 2025 · 0 comments
Open

Submit batch evaluation run on PR #61

jjgriff93 opened this issue Feb 4, 2025 · 0 comments

Comments

@jjgriff93
Copy link
Collaborator

https://learn.microsoft.com/en-us/azure/ai-studio/how-to/flow-bulk-test-evaluation

What is the implication of dynamic guideline configurations being implemented in #19?

We might need to version guidelines (agents) in the db and have them as inputs to the flow so we can specify versions as variants when running eval. Or we have a separate non-dynamic flow node (similar to the concept of the streaming flag) that we call with dynamic=disabled that ignores fetching guidelines from Cosmos and uses local files (jinja templates) instead. This could also be used in Test and Prod environment to keep fixed prompts defined in files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant