Skip to content

Commit

Permalink
Add XenClean and XenBootFix instructions
Browse files Browse the repository at this point in the history 
These tools are already usable regardless of the driver's signing state.

Signed-off-by: Tu Dinh <ngoc-tu.dinh@vates.tech>
  • Loading branch information
@dinhngtu
dinhngtu committed Mar 3, 2025
1 parent 8d2aaa3 commit 18c93ab
Show file tree
Hide file tree
Showing 2 changed files with 84 additions and 84 deletions.
96 changes: 82 additions & 14 deletions docs/troubleshooting/windows-pv-tools.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Windows Agent / PV-Tools
# Windows PV Tools

Common issues with Windows PV tools.

Expand All @@ -21,19 +21,8 @@ If despite running the Windows tools installer, there's no devices visible in th
### Solutions

#### Leftovers from old Citrix XenServer Client Tools.
1. remove any xen*.* files from `C:\Windows\system32` like
* xenbus_coinst_7_2_0_51.dll
* xenvbd_coinst_7_2_0_40.dll
* xenbus_monitor_8_2_1_5.exe
* and similiar `xen*_coinst` and `xen*_monitor` files
2. remove any leftover `XenServer` devices from device manager, also display hidden `XenServer` devices and remove them!
* To show hidden devices in Device Manager: `View -> Show Hidden Devices`

#### There was an issue with the installing of the drivers certificate, so the drivers did not load silently

Resolved with version 8.2.2.200-RC1 and newer.

***
See the [XenClean guide](#completely-removing-existing-xen-pv-drivers-with-xenclean) below for instructions.

## Network PV drivers aren't working.

Expand All @@ -48,7 +37,7 @@ If the tools are installed, while XCP-ng Center says that I/O is optimized, but
* Clean your system from `Citrix Client Tools` _AND_ `XCP-ng Client Tools` to create a clean state.
* Then install the Client Tools from scratch.

[This Guide](../../vms#%EF%B8%8F-guest-tools) may help you through the process.
See the [XenClean guide](#completely-removing-existing-xen-pv-drivers-with-xenclean) below for instructions.

## Not all PV drivers are correctly installed

Expand Down Expand Up @@ -80,6 +69,85 @@ Consult your motherboard manual for details; for example, on Dell systems with i
![](../../static/img/performance-setting.png)
</div>

## Fully removing Xen PV drivers with XenClean

XenClean is an utility for cleanly removing Xen PV drivers and management agents of the following products:

* XCP-ng Windows PV Tools, versions 8.2 and 9.1
* XenServer VM Tools for Windows, versions 9.3 and 9.4
* Other Xen drivers

It is included in the installation package of XCP-ng Windows PV Tools 9.1 and above.
[See the newest releases here.](https://github.com/xcp-ng/win-pv-drivers/releases)

:::note
Before running XenClean:

* Take a snapshot of your VM.
* You should always use the latest version of XenClean regardless of your Windows PV driver version. The same applies to XenBootFix.
* You should disable the "Manage Citrix PV drivers via Windows Update" option on your VM. Otherwise, Windows may reinstall PV drivers from Windows Update after rebooting.
* If you downloaded XenClean from the internet, you may need to unblock the script file before running it. This can be done by right-clicking the file, then choosing **Properties** - **Unblock** - **OK**.
:::

To use XenClean, simply run the `Invoke-XenClean.ps1` script **as Administrator**. Your system will automatically reboot.

:::tip
XenClean leaves its log files at the current directory and at `%TEMP%\xenclean-<time>`. Please provide these logs in case of uninstallation failure.
:::

## Windows fails to boot (hanging at boot or BSOD with Stop code `INACCESSIBLE_BOOT_DEVICE`)

In some situations (failed uninstallation, major Windows version upgrades), Xen PV drivers (whether Citrix or XCP-ng) may cause Windows to fail to start (hanging at boot, BSOD with Stop code `INACCESSIBLE_BOOT_DEVICE`).
The XenBootFix utility included with XCP-ng Windows PV Tools 9.1 and above helps you disable any active Xen PV drivers and get your system to a bootable state before running XenClean.

:::note
The utility only runs in Windows Preinstallation Environment (PE) or Windows Recovery Environment (RE). It will not run from Safe Mode.
:::

Below is a procedure for using XenBootFix to recover a non-booting VM:

1. Take a snapshot of your VM in case uninstallation fails.
2. Disable the "Manage Citrix PV drivers via Windows Update" option on your VM if it's enabled.
3. Boot into Windows PE or Windows RE in command line-only mode. There are a few ways to accomplish this:
* If your Windows installation BSODs on boot multiple times, it will automatically boot into Windows RE. Choose **Troubleshoot** - **Command Prompt**.
* When running Windows Server, press **F8** before Windows starts, then choose **Repair Your Computer**. Choose **Troubleshoot** - **Command Prompt**.
* Boot your VM using a Windows Setup or Windows PE CD image. If you don't see a command line, press **Shift+F10**.
4. Identify your Windows installation drive letter.
* Use the `dir` command to list files in a given drive letter. For example: `dir C:\` (the backslash is required)
* In some cases, your Windows partition should already be mounted. Try the first few letters (`C:`, `D:`, `E:`).
* If you cannot find your Windows drive letter, you may need to assign a new drive letter with Diskpart.
* Type `diskpart` at the command line. The command prompt should change to `DISKPART>`
* Type `list vol` then make a note of your Windows partition and its drive letter (if any).
* If your Windows partition does not have a drive letter, type `sel vol N` where `N` is the volume number shown in Diskpart, then type `assign letter=W`. Your Windows partition will be assigned the drive letter `W:`.
* Finally, at the `DISKPART>` prompt, type `exit` to exit Diskpart.
5. Obtain XenBootFix.
* If you're using XCP-ng Windows PV Tools 9.1 or later, it is located at `W:\Program Files\XCP-ng\Windows PV Drivers\XenBootFix\XenBootFix.exe` where `W:` is your Windows drive letter.
* If you have PowerShell, run the following command: `Invoke-WebRequest https://<download path of XenBootFix.exe> -OutFile XenBootFix.exe`
* Failing all that, you can create a new ISO image containing XenBootFix using WinCDEmu, ImgBurn or a similar tool, then attach it to your VM.
* **Note**: If using Windows PE, do not remove its CD image when it's running. You may encounter unexpected errors otherwise.
6. Run the command `<path to XenBootFix.exe> W:\Windows` where `W:` is your Windows drive letter.
* **Note**: Make sure the drive letter belongs to your actual Windows installation and not Windows PE/RE. By default, Windows PE/RE use the drive letter **X:**.
7. Type `exit` to close Command Prompt. If using Windows RE, choose **Continue** to boot into Windows. Windows should now start normally.
8. **You must immediately run XenClean from within Windows to remove the remaining Xen drivers**. See instructions above.

## Connecting to guests using serial console

In some situations, you might want to expose a VM's serial console to network for troubleshooting purposes (e.g. Windows kernel debugging).
You can use the following procedure:

* Shut down the VM.
* On your host, run the following command: `xe vm-param-set uuid=<uuid> platform:hvm_serial=tcp::<port>,server,nodelay,nowait` where `<uuid>` is the VM's UUID and `<port>` is the TCP port on the host that exposes the VM's serial port, e.g. 7001.
* You need to open the selected port on the host's firewall: `/etc/xapi.d/plugins/firewall-port open 7001`
* Note that any host running the VM will need this command.
* Connecting to port 7001 on the host will connect to your VM's serial port. You can use Telnet, PuTTY or any similar tools.
* To configure kernel debugging on Windows, disable Secure Boot and BitLocker then run the following commands within the VM:
```
bcdedit /debug on
bcdedit /dbgsettings serial debugport:1 baudrate:115200
```
Connect using [WinDbg](https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/) using the `Attach to kernel` option with a connection string: `com:ipport=7001,port=<host IP>`
* To undo the changes and remove the serial setting, use `xe vm-param-remove uuid=<uuid> param-name=platform param-key=hvm_serial`

## Windows bug check 0x3B (SYSTEM_SERVICE_EXCEPTION) on systems with newer Intel CPUs

### Cause
Expand Down
72 changes: 2 additions & 70 deletions docs/vms/vms.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -460,76 +460,8 @@ xe vm-param-get param-name=has-vendor-device uuid={VM-UUID}

Our installer is not able currently to cleanly uninstall Citrix tools. Citrix tools' uninstaller itself isn't either: it leaves various things behind.

So we need to perform a complete manual clean-up of the tools:
* either entirely manually
* or using the experimental PowerShell script contributed by one of our users at [https://github.com/siodor/win-tools-cleanup](https://github.com/siodor/win-tools-cleanup)

:warning: In any case, first disable "Windows Update tools" for the VM (Xen Orchestra, advanced tab) and reboot it.

Following is the manual process.

###### The confident option

You can try a simple process first with some chances of success.

0. Make a snapshot so you can rollback. Windows can get unstable/unbootable if things go wrong.
1. Uninstall Citrix :registered: XenServer :registered: Client Tools
2. Reboot
3. Uninstall `XenServer PV`-Drivers in Device Manager in following order (reboots may be needed):
* `XenServer PV Network Device` (one ore more Devices)
* `XenServer PV Storage Host Adapter`
* `XenServer PV Network Class`
* `XenServer Interface`
* `XenServer PV Bus (c000)` (if present)
* `XenServer PV Bus (0002)` or `XenServer PV Bus (0001)`
4. Reboot
5. Check that you see this unknown device in Device Manager:
* `SCSI-Controller` - PCI-Device ID `5853:0002`
6. Unpack ZIP file
7. Start setup.exe
8. Follow the install wizard

**Note**: Restart can take a while if your windows is currently updating. Restart only occurs after windows has the updates finished.

###### The nuclear option

If the *confident option* above didn't yield the expected results, then we switch to a more aggressive attitude towards the old tools.

:::tip
What follows works in many cases, but some users occasionally still meet the following issues: XCP-ng tools not installing (but Citrix tools install well, so that is a solution to have working tools), and occasional BSODs in some cases or versions of Windows.

Through many tests, a user came up with a similar yet slightly different procedure that allowed them to avoid Blue Screens Of Death in their situation: https://xcp-ng.org/forum/post/27602.

Help is welcome to help us reconcile both procedures into one.
:::

* Follow the steps 0 to 4 of the "confident option" above if not done yet.
* Follow this (ignore steps 6 and 7, do not try to install the tools yet) [https://support.citrix.com/article/CTX215427](https://support.citrix.com/article/CTX215427)
* Now open regedit and go to HKLM\SYSTEM\CurrentControlSet\Services and delete entries for all xen* services.
* In regedit, also go to HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\DIFx\DriverStore and remove ONLY xennet* xenvif*
* Go to C:\Windows\System32 and remove: (you may not have all these)
* xenbus_coinst*.dll
* xenvbd_coinst*.dll
* liteagent.exe
* Now go to C:\Windows\System32\drivers and remove any files named xen*
* Go to C:\Windows\system32\DriverStore\FileRepository and remove xennet* and Xenvif* directories.
* Open the Device Manager and Click View --> Show Hidden Devices. Select Other Devices and Right click on XENBUS VIF and select uninstall. If it asks to delete the driver, check yes. Do this for any xen related thing you see in device manager. Also do the same for any unknown devices.
* Lastly, reboot the VM. You should now hopefully be able to install xen tools regularly.

**Note**: Also have a look at our [Troubleshooting Guide - Windows PV-Tools](../troubleshooting/windows-pv-tools.md).

##### VMs with INACCESSIBLE_BOOT_DEVICE error

You can try to manually inject the missing drivers in recovery mode.

* Get the "Drivers" folder from the XCP Tools installation path (C:\PROGRAM FILES...) - from another VM or install the tools somewhere else to get it.
* Create an ISO-Image containing the "Drivers" folder (see [http://imgburn.com](http://imgburn.com)) and mount that ISO-Image to your VM
* Boot to recovery mode and use the command line and the tool "dism" (see [Microsoft Docs](https://docs.microsoft.com/en-us/windows-hardware/manufacture/desktop/add-and-remove-drivers-to-an-offline-windows-image)) to inject the drivers (specifically the xenbus and xenvbd drivers) - watch out for the drive letter of the Windows installation and the CD-Drive ('D' and 'E' in the following example):

````
dism /image:d:\ /add-driver /driver:e:\Drivers\xenbus\x64\xenbus.inf
dism /image:d:\ /add-driver /driver:e:\Drivers\xenvbd\x64\xenvbd.inf
````
So we need to perform a complete clean-up of it using the XenClean utility.
You will find the detailed instructions in our [Troubleshooting Guide - Windows PV Tools](../troubleshooting/windows-pv-tools.md).

#### Contributing
##### Linux / xBSD
Expand Down

0 comments on commit 18c93ab

Please sign in to comment.