edge-gateway

IPFS edge gateway

The IPFS edge gateway for nft.storage is not "another gateway", but a caching layer for NFTs that sits on top of existing IPFS public gateways.

Getting started

One time set up of your cloudflare worker subdomain for dev:

• pnpm install - Install the project dependencies from the monorepo root directory.

• Sign up to Cloudflare and log in with your default browser.

• npm i @cloudflare/wrangler -g - Install the Cloudflare wrangler CLI

• wrangler login - Authenticate your wrangler cli; it'll open your browser.

• Copy your cloudflare account id from wrangler whoami

• Update wrangler.toml with a new env. Set your env name to be the value of whoami on your system you can use pnpm start to run the worker in dev mode for you.

  wrangler.toml

  [env.bobbytables]
  workers_dev = true
  account_id = "<what does the `wrangler whoami` say>"

• Add secrets

    wrangler secret put SENTRY_DSN --env $(whoami) # Get from Sentry (not required for dev)
    wrangler secret put LOGTAIL_TOKEN --env $(whoami) # Get from Logtail

• Add KV namespaces

  wrangler kv:namespace create DENYLIST --preview --env USER
  # Outputs something like: `{ binding = "DENYLIST", preview_id = "7e441603d1bc4d5a87f6cecb959018e4" }`
  # but you need to put `{ binding = "DENYLIST", preview_id = "7e441603d1bc4d5a87f6cecb959018e4", id = "7e441603d1bc4d5a87f6cecb959018e4" }` inside the `kv_namespaces`.
  # for production: wrangler kv:namespace create DENYLIST --env production

• pnpm run publish - Publish the worker under your env. An alias for wrangler publish --env $(whoami)

• pnpm start - Run the worker in dev mode. An alias for wrangler dev --env $(whoami)

You only need to pnpm start for subsequent runs. PR your env config to the wrangler.toml to celebrate 🎉

High level architecture

nftstorage.link is serverless code running across the globe to provide exceptional performance, reliability, and scale. It is powered by Cloudflare workers running as close as possible to end users.

Thanks to IPFS immutable nature, a CDN cache is an excellent fit for content retrieval as a given request URL will always return the same response. Accordingly, as a first IPFS resolution layer, nftstorage.link leverages Cloudflare Cache API to look up for content previously cached in Cloudflare CDN (based on geolocation of the end user).

In the event of content not being already cached, a race with multiple IPFS gateways is performed. As soon as one gateway successfully responds, its response is forwarded to the user and added to Cloudflare Cache.

Zooming in on the actual edge gateway:

Notes:

• Cloudflare Cache is limited to 512 MB size objects.
• Using a CDN Cache also enables us to implement intelligent caching strategies to warm the cache for content with high likelihood of being requested soon.

Usage

nft.storage Gateway provides IPFS path style resolutions https://nftstorage.link/ipfs/{cid} as follows:

> curl https://nftstorage.link/ipfs/bafkreidyeivj7adnnac6ljvzj2e3rd5xdw3revw4da7mx2ckrstapoupoq
Hello nft.storage! 😎
> curl https://nftstorage.link/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX
...

In practice, when nft.storage Gateway receives a IPFS path style request, it will redirect to a subdomain style resolution maintaining compliance with the same-origin policy. The canonical form of access https://{CID}.ipfs.nftstorage.link/{optional path to resource} causes the browser to interpret each returned file as being from a different origin.

> curl https://bafkreidyeivj7adnnac6ljvzj2e3rd5xdw3revw4da7mx2ckrstapoupoq.ipfs.nftstorage.link
Hello nft.storage! 😎

Please note that subdomain resolution is only supported with CIDv1 in case-insensitive encoding such as Base32 or Base36. When using IPFS path resolution, the requested CID will be converted before the redirect.

Rate limiting

nft.storage Gateway is currently rate limited at 200 requests per minute to a given IP Address. In the event of a rate limit, the IP will be blocked for 30 seconds.

Deny List

We rely on badbits denylist together wtth our own denylist to prevent serving malicious content to the nftstorage.link users.

When new malicious content is discovered, it should be reported to badbits denylist given it is shared among multiple gateways. When the reported CIDs are added into badbits, we just need to force our denylist sync workflow to run manually.

As a workaround, or to block content only relevant for nftstorage.link we can simply use our denylist as described in our CLI documentation.

Persistence

Several metrics per gateway are persisted to track the performance of each public gateway over time. Moreover, the list of gateways that have previously fetched successfully a given CID are also persisted.