A Citrix XenServer host that is managed by Apache CloudStack will reboot itself along with all running virtual machines running on the host if there is a problem with the primary storage after a period of time. In general this is a good protection method, if the primary storage is no longer available it can cause problems for the virtual machines, Windows VMs may blue screen of death (BSOD) while Linux VMs may enter a read only file system state to protect against data loss. However you may have multiple NFS mount points to act as primary storage and may not want every VM on the XenServer host to power off with the host should this happen. It is possible to work around this by modifying the heartbeat script created by CloudStack on the XenServer host.
Category Archives: XenServer
XenServer Tools Removed by Windows Update
After performing a regular Windows update and restart recently I found that XenServer tools appeared to have been removed. After trying to reinstall the tools I found that the process would hang indefinitely preventing the problem from being resolved.
After contacting Citrix support they advised that this was a known issue caused by Amazon accidentally uploading a set of their WHQL PV tools to Microsoft Windows update which overwrote the existing PV drivers in place, here’s how to fix this.
Read more »
How to remove pending updates in Citrix XenServer
A few weeks ago I accidentally attempted to apply a XenServer hotfix intended for XenServer 6.2 to a host running XenServer 6.5. Ever since this accidental mistake, XenCenter has been reporting that there is a new update to apply, which should not be the case. At first I thought it would go away after the next reboot or after the next hotfix had been applied and fix itself up, however this did not happen, here is how to fix it.
Free up disk space in Citrix XenServer
A Citrix XenServer dom0 currently runs with a 4GB root partition which is pretty small by today’s standards. A small amount of usable storage space can be quite easy to quickly fill. It is therefore important that dom0 has free space in order for it to operate correctly. Here we will cover some different methods that can be used to free up disk space within XenServer.
Virtual machine fails to power on in Citrix XenServer
After rebooting a virtual machine running CentOS through Apache CloudStack it appeared to be running, however it failed to boot up. The status through CloudStack showed the virtual machine as running, however the console did not load any content and connections to it failed. After checking the virtual machine directly through XenCenter it was clear that it was not actually running.
In XenCenter the virtual machine had the red stop icon on it and was definitely stopped. Performing a reboot through CloudStack did nothing, and stopping the instance through CloudStack resulted in the virtual machine being removed from XenCenter as expected. When starting it back up again and watching XenCenter it did appear to power on for a couple of seconds as shown with the green play icon however it quickly went back to the stopped state.
Can’t boot ISO as HVM in Apache CloudStack
Recently while trying to boot from an ISO in Apache CloudStack I received the error “Unable to start instance ‘hostname’ (UUID), see management server log for details”. The ISO media did not support installation as a PV guest in XenServer 6.2, it needed to be installed in PVHVM mode which required the following work around in order to boot correctly.
Fixing Blue Screen in Windows after XenServer Tools Update
I had a Windows Server 2008 R2 virtual machine running on XenServer 6.2 SP1 with all updates and patches applied, however it was still running an older version of XenServer tools. The tools were version 6.0.2, and this was causing problems with backups from Arcserve freezing and not completing. To fix the problem I needed to upgrade XenServer tools to the latest version of 6.2, easy right? Think again.
XenServer Virtual Machines Stop or Pause During Migration
Recently I had a whole lot of problems migrating virtual machines running on XenServer 6.2 and 6.0.2. Sometimes the migration would fail and the virtual machine would stop or pause resulting in down time, here is how the problem was investigated and fixed.
Fix a CloudStack instance stuck in an incorrect state
In Apache CloudStack it is possible for the database to become out of sync with what is actually happening. For instance if there is a network issue preventing CloudStack from correctly connecting to the hypervisor or virtual machine and you try to shutdown or reboot a server, the action may not actually take place despite the state in the CloudStack database being modified.
This can cause the instance to show as being in the ‘starting’ or ‘stopping’ states for instance, however the virtual machine may already be fully booted or completely powered off. To fix this we can manually update the management database to the correct state.
Virtual machine not powering on in Citrix XenServer 6.2
Today I connected to a CentOS 6 server via SSH and quickly noticed that the file system was in read only mode, after checking a few other Linux servers on the same XenServer host it quickly became apparent that there had been a network issue between the storage and compute layers which caused the Linux file systems to go read only in order to protect themselves.
After not being able to do anything useful within the operating system such as remounting the file system as read/write, I decided that it was time to reboot and force a file system check to pick up and fix any problems, however once the server had shut down it did not power back on as part of the restart task, it also did not power back on when attempting to start it up. This only happened to one VM, all of the others powered back on fine and worked as expected.