Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using a ResponseFormat of "json_object" causes errors local OpenAI-like models using LM Studio #55

Open
therealjohn opened this issue Jan 17, 2025 · 0 comments
Assignees
Labels
Investigating Issue is being investigated triage Initial state for our team to determine nessessary action

Comments

@therealjohn
Copy link
Member

Version

main

Describe the bug

Image

I need to use "json_schema" here per the documentation: https://lmstudio.ai/docs/advanced/structured-output

I do not know why they made it different than OpenAI hosted models.

To Reproduce
Use the WeatherBot sample with a local model hosted with LM Studio.

this._agent =
                new()
                {
                    Instructions = AgentInstructions,
                    Name = AgentName,
                    Kernel = this._kernel,
                    Arguments = new KernelArguments(new OpenAIPromptExecutionSettings() 
                    { 
                        FunctionChoiceBehavior = FunctionChoiceBehavior.Auto(), 
                        ResponseFormat = "json_object" // Wrong format for this, but its what the property suggests 
                    }),
                };

Define some JSON schema in LM Studio:

Image

Use local endpoints:
AIServices:LocalModel:ModelName = qwen2.5-7b-instruct
AIServices:LocalModel:Endpoint = http://127.0.0.1:1234/v1/

builder.Services.AddOpenAIChatCompletion(
        modelId: builder.Configuration.GetSection("AIServices:LocalModel").GetValue<string>("ModelName"),
        endpoint: new Uri(builder.Configuration.GetSection("AIServices:LocalModel").GetValue<string>("Endpoint")));

Additional context
Maybe this is more a feature request to consider local model usage for OpenAI-like models that aren't OpenAI hosted models.

@therealjohn therealjohn added the triage Initial state for our team to determine nessessary action label Jan 17, 2025
@MattB-msft MattB-msft added the Investigating Issue is being investigated label Feb 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Investigating Issue is being investigated triage Initial state for our team to determine nessessary action
Projects
None yet
Development

No branches or pull requests

3 participants