Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Render Audio Player if Tool Result resource mime type is audio. #99

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

evalstate
Copy link
Contributor

@evalstate evalstate commented Dec 1, 2024

Show the browser audio player if resource mime type is audio.

Motivation and Context

Makes testing audio tool results integration simpler.

How Has This Been Tested?

This has been tested with an MCP Server calling Parler/TTS.

...
      const audioBase64 = await generateSpeech(text);
      return {
        content: [{
          type: "resource",
          resource: {
            uri: "audio:///" + crypto.randomUUID(),
            mimeType: "audio/wav",
            blob: audioBase64
          }
        }]
      };
    }
...

async function generateSpeech(text: string): Promise<string> {
  const client = await Client.connect("parler-tts/parler_tts");
  const result = await client.predict("/gen_tts", {
    text,
    description: "Laura's voice is monotone yet slightly fast in delivery, with a very close recording that almost has no background noise.",
    use_large: false
  });
  
  // Fetch the audio file
  const response = await fetch(result.data[0].url);
  const arrayBuffer = await response.arrayBuffer();
  
  // Convert to base64
  const base64 = Buffer.from(arrayBuffer).toString('base64');
  
  return base64;
}



Breaking Changes

No - but the Client Reference Implementation (Claude Desktop does not support Audio content).

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Checklist

  • I have read the MCP Documentation
  • My code follows the repository's style guidelines
  • New and existing tests pass locally
  • I have added appropriate error handling
  • I have added or updated documentation as needed

Copy link
Member

@ashwin-ant ashwin-ant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! Just one small suggestion.

client/src/components/ToolsTab.tsx Outdated Show resolved Hide resolved
@evalstate evalstate force-pushed the feature/audio-rendering branch from 068d213 to 66b1b73 Compare December 1, 2024 16:37
@evalstate evalstate requested a review from ashwin-ant December 3, 2024 17:12
@ashwin-ant
Copy link
Member

Hey @evalstate, apologies for the delay here. After some consideration, we'd like to avoid having the inspector showcase any features that aren't in the official spec. So this particular PR will need to wait until the spec discussion you started gets to a resolution.

@evalstate
Copy link
Contributor Author

evalstate commented Dec 3, 2024

Hi Ashwin -

This PR renders the audio component if the mime type of the Resource is audio (so in-spec). It's very convenient for testing.

The deleted commit handled the change to the Schema - and was accidentally committed to the branch. Ideally changes to the spec could be coordinated with updates to SDK and Inspector for testing. I fat-fingered this one - hence the delete of that commit .

Best.

@jerome3o-anthropic jerome3o-anthropic requested review from jerome3o-anthropic and removed request for ashwin-ant January 28, 2025 07:07
Copy link

@jerome3o-anthropic jerome3o-anthropic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me, thank you for this (sorry taking a long time to get to this!)

I don't have state on the previous iterations, but the diff looks good to me as is

@jerome3o-anthropic jerome3o-anthropic dismissed ashwin-ant’s stale review January 28, 2025 07:17

The relevant code has been removed, so the requested changes are no longer relevant

@jerome3o-anthropic
Copy link

I think you just need to just run npx prettier --write . on thisto have it merged

Checking formatting...
[warn] client/src/components/ToolsTab.tsx
[warn] Code style issues found in the above file. Run Prettier with --write to fix.

inspector on  feature/audio-rendering [$⇡] is 📦 v0.3.0 via  v22.11.0
❯ npx prettier --write client/src/components/ToolsTab.tsx
client/src/components/ToolsTab.tsx 109ms
@evalstate
Copy link
Contributor Author

Done!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants