Openpose Editor for ControlNet in Stable Diffusion WebUI

This extension is specifically build to be integrated into Stable Diffusion WebUI's ControlNet extension.

Installation

On UI restart, the extension will try to download the compiled Vue app from Github. Check whether stable-diffusion-webui\extensions\sd-webui-openpose-editor\dist exists and has content in it.

Some users in China have reported having issue downloading dist with the autoupdate script. In such situtations, the user has 2 following options to get dist manually:

Option1: Build the application

Make sure you have nodeJS environment ready and follow Development section. Run npm run build to compile the application.

Option2: Download the compiled application

You can download the compiled application(dist.zip) from the release page. Unzip the package in the repository root and make sure hte unziped directory is named dist.

Usage

The openpose editor core is build with Vue3. The gradio extension script is a thin wrapper that mounts the Vue3 Application on /openpose_editor_index.

The user can directly access the editor at localhost:7680/openpose_editor_index if desired, but the main entry point is invoking the editor in the ControlNet extension. In ControlNet extension, select any openpose preprocessor, and hit the run preprocessor button. A preprocessor result preview will be genereated. Click Edit button at the bottom right corner of the generated image will bring up the openpose editor in a modal. After the edit, clicking the Send pose to ControlNet button will send back the pose to controlnet.

Features

1. Support for face/hand used in controlnet.
   • The extension recognizes the face/hand objects in the controlnet preprocess results.
   • The user can add face/hand if the preprocessor result misses them. It can be done by either
     • Add Default hand (Face is not supported as face has too many keypoints (70 keypoints), which makes adjust them manually really hard.)
     • Add the object by uploading a pose JSON. The corresponding object of the first person will be used.
2. Visibility toggle
   • If a keypoint is not recognized by ControlNet preprocessor, it will have (-1, -1) as coordinates. Such invalid keypoints will be set as invisible in the editor.
   • If the user sets a keypoint as invisible and send the pose back to controlnet, the limb segments that the keypoint connects will not be rendered. Effectively this is how you remove a limb segment in the editor.
3. Group toggle
   • If you don't want to accidentally select and modify the keypoint of an canvas object (hand/face/body). You can group them.
   • Note: If you group the object, you will not be able to rotate/scale the group. The only operation the group accepts is move. This is an issue with fabric.js. If anyone find a solution, I am willing to adjust the implementation. In order to rotate/scale multiple keypoints, currently the user has to select keypoints and rotate/scale the selection.

Here is a demo showing the difference between group and selection:

Development

Recommended IDE Setup

VSCode + Volar (and disable Vetur) + TypeScript Vue Plugin (Volar).

Type Support for .vue Imports in TS

TypeScript cannot handle type information for .vue imports by default, so we replace the tsc CLI with vue-tsc for type checking. In editors, we need TypeScript Vue Plugin (Volar) to make the TypeScript language service aware of .vue types.

If the standalone TypeScript plugin doesn't feel fast enough to you, Volar has also implemented a Take Over Mode that is more performant. You can enable it by the following steps:

1. Disable the built-in TypeScript Extension
   1. Run Extensions: Show Built-in Extensions from VSCode's command palette
   2. Find TypeScript and JavaScript Language Features, right click and select Disable (Workspace)
2. Reload the VSCode window by running Developer: Reload Window from the command palette.

Customize configuration

See Vite Configuration Reference.

Project Setup

npm install

Compile and Hot-Reload for Development

npm run dev

Type-Check, Compile and Minify for Production

npm run build

Run Unit Tests with Vitest

npm run test:unit

Lint with ESLint

npm run lint