Puppeteer / URL Server Bot Detection and Permissions?

[pauls-ai2]

Read the blog post and started looking at the code (but haven't had time to run it). I'm looking into options for allowing MCP to hit various websites which have typically had bot protections enabled. It would be smart to start thinking about how to allow and/or throttle requests from various MCP based tools...maybe even include a permissioning scheme to the protocol.

Some initial thoughts might be to include a specific User-Agent header, maybe a set of x-mcp-* headers, or maybe some handshake that grants a bearer token. Personally, I really like the idea of a bearer token, because it allows for something like OAuth, where an MCP server can get access to private data as granted by a user.

Do you all have any thoughts on where bot detection and/or permissions might go in the future?

[jspahrsummers]

We just added an example URL fetching server which has some of this configurability via User-Agent. I'm curious if that's sort of along the lines you're thinking about?

(cc @jackadamson, who worked on it)

[jackadamson]

Just to clarify, are you asking about how site owners can specifically block / rate-limit requests from MCP servers?

Or more generally, how authentication / authorization might work between an MCP server and a remote web server?

[pauls-ai2]

I was thinking more as a site owner that is currently blocking bots, how can I allow MCP servers to access my content. Moreover, how can I allows an MCP server to get access to privileged data on behalf of the prompter. I love this idea of making agent systems generally more useful, but know there is a practical side I have to consider as a site owner.

[jspahrsummers]

Ah, I see. I don't think there's anything generic we can do here, since MCP servers can be literally any code (they don't even have to be built using our SDK), and can make network requests in any way under the sun.

[jackadamson]

If the site has an API, then usually they will be exempt from anti-bot rules (since they are consumed by automation, and are usually authenticated). As such, you could use a setup similar to the example github MCP server which is configured with a credential (the GitHub personal access token) via environment variables.

This has the benefits that:

• Your credential isn't sent to the model (very important)
• The MCP server can authenticate to GitHub as you

[jackadamson]

If you specifically want to use a browser for most of the actions, then this becomes the problem of "how to authenticate a browser given a static credential", which could be solved in many ways, including using oauth2 flows, or even having puppeteer set a cookie based on some action taken by the MCP server.