Add XenClean and XenBootFix instructions

These tools are already usable regardless of the driver's signing state.

Signed-off-by: Tu Dinh <ngoc-tu.dinh@vates.tech>

docs/troubleshooting/windows-pv-tools.md

@@ -1,4 +1,4 @@
-# Windows Agent / PV-Tools
+# Windows PV Tools
 
 Common issues with Windows PV tools.
 
@@ -21,19 +21,8 @@ If despite running the Windows tools installer, there's no devices visible in th
 ### Solutions
 
 #### Leftovers from old Citrix XenServer Client Tools.
-1. remove any xen*.* files from `C:\Windows\system32` like
-    * xenbus_coinst_7_2_0_51.dll
-    * xenvbd_coinst_7_2_0_40.dll
-    * xenbus_monitor_8_2_1_5.exe
-    * and similiar `xen*_coinst` and `xen*_monitor` files
-2. remove any leftover `XenServer` devices from device manager, also display hidden `XenServer` devices and remove them!
-    * To show hidden devices in Device Manager: `View -> Show Hidden Devices`
 
-#### There was an issue with the installing of the drivers certificate, so the drivers did not load silently
-
-Resolved with version 8.2.2.200-RC1 and newer.
-
-***
+See the XenClean guides below for instructions.
 
 ## Network PV drivers aren't working.
 
@@ -48,7 +37,7 @@ If the tools are installed, while XCP-ng Center says that I/O is optimized, but
 * Clean your system from `Citrix Client Tools` _AND_ `XCP-ng Client Tools` to create a clean state.
 * Then install the Client Tools from scratch.
 
-[This Guide](../../vms#%EF%B8%8F-guest-tools) may help you through the process.
+See the XenClean guides below for instructions.
 
 ## Not all PV drivers are correctly installed
 
@@ -80,6 +69,76 @@ Consult your motherboard manual for details; for example, on Dell systems with i
 ![](../../static/img/performance-setting.png)
 </div>
 
+## Completely removing existing Xen PV drivers with XenClean
+
+XenClean is an utility for cleanly removing Xen PV drivers and management agents of the following products:
+
+* XCP-ng Windows PV Tools, versions 8.2 and 9.1
+* XenServer VM Tools for Windows, versions 9.3 and 9.4
+* Other Xen drivers
+
+It is included in the installation package of XCP-ng Windows PV Tools 9.1 and above.
+[See the newest releases here.](https://github.com/xcp-ng/win-pv-drivers/releases)
+
+To use XenClean, simply run the `Invoke-XenClean.ps1` script **as Administrator**. Your system will automatically reboot.
+
+**Note**: You should always use the latest version of XenClean regardless of your Windows PV driver version. The same applies to XenBootFix.
+
+**Note**: You should disable the "Manage Citrix PV drivers via Windows Update" option on your VM before running XenClean. Otherwise, Windows may reinstall PV drivers from Windows Update after rebooting.
+
+**Note**: If you downloaded XenClean from the internet, you may need to unblock the script file before running it. This can be done by right-clicking the file, then choosing **Properties** - **Unblock** - **OK**.
+
+**Tip**: XenClean leaves its log files at `%TEMP%\xenclean-<time>`. Please provide these logs in case of uninstallation failure.
+
+## Windows fails to boot (hanging at boot or BSOD with Stop code `INACCESSIBLE_BOOT_DEVICE`)
+
+In some situations (failed uninstallation, major Windows version upgrades), Xen PV drivers (whether Citrix or XCP-ng) may cause Windows to fail to start (hanging at boot, BSOD with Stop code `INACCESSIBLE_BOOT_DEVICE`).
+The XenBootFix utility included with XCP-ng Windows PV Tools 9.1 and above helps you disable any active Xen PV drivers and get your system to a bootable state before running XenClean.
+
+**Note**: The utility only runs in Windows Preinstallation Environment (PE) or Windows Recovery Environment (RE). It will not run from Safe Mode.
+
+Below is a procedure for using XenBootFix to recover a non-booting VM:
+
+1. Boot into Windows PE or Windows RE in command line-only mode. There are a few ways to accomplish this:
+  * If your Windows installation BSODs on boot multiple times, it will automatically boot into Windows RE. Choose **Troubleshoot** - **Command Prompt**.
+  * When running Windows Server, press **F8** before Windows starts, then choose **Repair Your Computer**. Choose **Troubleshoot** - **Command Prompt**.
+  * Boot your VM using a Windows Setup or Windows PE CD image. If you don't see a command line, press **Shift+F10**.
+2. Identify your Windows installation drive letter.
+  * Use the `dir` command to list files in a given drive letter. For example: `dir C:\` (the backslash is required)
+  * In some cases, your Windows partition should already be mounted. Try the first few letters (`C:`, `D:`, `E:`).
+  * If you cannot find your Windows drive letter, you may need to assign a new drive letter with Diskpart.
+    * Type `diskpart` at the command line. The command prompt should change to `DISKPART>`
+    * Type `list vol` then make a note of your Windows partition and its drive letter (if any).
+    * If your Windows partition does not have a drive letter, type `sel vol N` where `N` is the volume number shown in Diskpart, then type `assign letter=W`. Your Windows partition will be assigned the drive letter `W:`.
+    * Finally, at the `DISKPART>` prompt, type `exit` to exit Diskpart.
+3. Obtain XenBootFix.
+  * If you're using XCP-ng Windows PV Tools 9.1 or later, it is located at `W:\Program Files\XCP-ng\Windows PV Drivers\XenBootFix\XenBootFix.exe` where `W:` is your Windows drive letter.
+  * If you have PowerShell, run the following command: `Invoke-WebRequest https://<download path of XenBootFix.exe> -OutFile XenBootFix.exe`
+  * Failing all that, you can create a new ISO image containing XenBootFix using WinCDEmu, ImgBurn or a similar tool, then attach it to your VM.
+  * **Note**: If using Windows PE, do not remove its CD image when it's running. You may encounter unexpected errors otherwise.
+4. Run the command `<path to XenBootFix.exe> W:\Windows` where `W:` is your Windows drive letter.
+  * **Note**: Make sure the drive letter belongs to your actual Windows installation and not Windows PE/RE. By default, Windows PE/RE use the drive letter **X:**.
+5. Type `exit` to close Command Prompt. If using Windows RE, choose **Continue** to boot into Windows. Windows should now start normally.
+6. **You must immediately run XenClean from within Windows to remove the remaining Xen drivers**. See instructions above.
+
+## Connecting to guests using serial console
+
+In some situations, you might want to expose a VM's serial console to network for troubleshooting purposes (e.g. Windows kernel debugging).
+You can use the following procedure:
+
+* Shut down the VM.
+* On your host, run the following command: `xe vm-param-set uuid=<uuid> platform:hvm_serial=tcp::<port>,server,nodelay,nowait` where `<uuid>` is the VM's UUID and `<port>` is the TCP port on the host that exposes the VM's serial port, e.g. 7001.
+* You need to open the selected port on the host's firewall: `/etc/xapi.d/plugins/firewall-port open 7001`
+  * Note that any host running the VM will need this command.
+* Connecting to port 7001 on the host will connect to your VM's serial port. You can use Telnet, PuTTY or any similar tools.
+* To configure kernel debugging on Windows, disable Secure Boot and BitLocker then run the following commands within the VM:
+  ```
+  bcdedit /debug on
+  bcdedit /dbgsettings serial debugport:1 baudrate:115200
+  ```
+  Connect using [WinDbg](https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/) using the `Attach to kernel` option with a connection string: `com:ipport=7001,port=<host IP>`
+* To undo the changes and remove the serial setting, use `xe vm-param-remove uuid=<uuid> param-name=platform param-key=hvm_serial`
+
 ## Windows bug check 0x3B (SYSTEM_SERVICE_EXCEPTION) on systems with newer Intel CPUs
 
 ### Cause

docs/vms/vms.md

@@ -518,19 +518,6 @@ Help is welcome to help us reconcile both procedures into one.
 
 **Note**: Also have a look at our [Troubleshooting Guide - Windows PV-Tools](../troubleshooting/windows-pv-tools.md).
 
-##### VMs with INACCESSIBLE_BOOT_DEVICE error
-
-You can try to manually inject the missing drivers in recovery mode.
-
-* Get the "Drivers" folder from the XCP Tools installation path (C:\PROGRAM FILES...) - from another VM or install the tools somewhere else to get it.
-* Create an ISO-Image containing the "Drivers" folder (see [http://imgburn.com](http://imgburn.com)) and mount that ISO-Image to your VM
-* Boot to recovery mode and use the command line and the tool "dism" (see [Microsoft Docs](https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/add-and-remove-drivers-to-an-offline-windows-image)) to inject the drivers (specifically the xenbus and xenvbd drivers) - watch out for the drive letter of the Windows installation and the CD-Drive ('D' and 'E' in the following example):
-
-````
-dism /image:d:\ /add-driver /driver:e:\Drivers\xenbus\x64\xenbus.inf
-dism /image:d:\ /add-driver /driver:e:\Drivers\xenvbd\x64\xenvbd.inf
-````
-
 #### Contributing
 ##### Linux / xBSD
 If you would like to contribute improvements to the `install.sh` script so that it supports your distro, create a pull request against: https://github.com/xcp-ng/xe-guest-utilities/tree/master/mk. Relevant files are usually `xe-linux-distribution` and `install.sh`.