Change of Plans

As I've mentioned in previous posts, my original idea was to have a "master" bus driver which would manage initializing VMX on each CPU as well as creating the VMs, and then a "child" function driver for controlling each VM. However, I recently realized it would make more sense (and would be much easier in the long run) to just have everything in one driver. Instead of representing VMs as device handles, user-space applications need only a simple ID integer; to operate on a VM, this ID would be passed as a parameter, along with whatever other parameters, into the master driver itself. Furthermore, this avoids potential future issues, such as "half-creation", where a VM has resources allocated, but is not in a state where applications can free them.

I've also done a bit of optimizations with DPCs; instead of creating a new one every time I want to execute a task on a VM, I now have a single DPC per processor that is given an argument which is the task to execute which is called by a main DPC entry. This also means that the task functions no longer have to signal completion - this is done by the DPC entry itself.

Making these changes, I'm now at a point where I can create VM objects and interact with them from user-space code. The next step is to learn how the VMCS region works, how to set up guest registers/memory, and how to actually launch the VM.

In other news, I (finally) head back to school in a couple of hours - not exactly looking forward to the cold!

DPCs to the Rescue!

I finally fixed the CPU affinity issue, and have reason to believe I can now accurately initialize VMX on all active CPUs.

The trick was, instead of creating a thread for each CPU and trying to set the affinity, I can create a DPC object for each CPU. There's a nice little routine called KeSetTargetProcessorDpcEx which allows me to issue a DPC on a specific CPU. So, all I have to do is create a DPC for each CPU, target that CPU, and when the DPC runs, it just initializes VMX - no need to adjust the affinity mask. Perhaps one drawback is that, at least from my tests so far, they all run sequentially on the same thread; however, I don't think this'll be an issue because the initialization code is pretty fast and straightforward. Also, DPCs run at DISPATCH_LEVEL, meaning everything the DPC accesses (including the function itself!) must be non-paged.

Another difference is that I have to create a new DPC anytime I want to run code on a particular processor (in the future, this will include VM operations that are tied to a CPU, such as starting the VM), but again, this is no big deal, as DPCs are relatively straightforward.

I think I'm gonna call it quits for tonight - perhaps tomorrow, I can start working on actual VM creation.

Progress!

I've successfully implemented child device creation! Whenever the bus driver receives a IOCTL_SWIVL_CREATE_VM ioctl, it creates a new VM object and registers a new VM device with the PnP (plug-and-play) manager, which is then able to load the function driver. This is a pretty big accomplishment because it means a good portion of the bus driver is already complete - all that's missing is the IOCTL_SWIVL_DELETE_VM ioctl, plus maybe a few other. Most of the actual code will be in the VM instance driver - starting the VM, setting up guest registers/memory, etc.

Now, there's still one bug that's been quite troublesome since the beginning - every once in a while, when uninstalling the hypervisor driver, it crashes with STATUS_ILLEGAL_INSTRUCTION when trying to execute the vmxoff instruction. From what I could discern from the documentation, the main cause for this is if VMX is not enabled when the instruction is executed. I've made several little patches/updates to my code (as well as the 'VMX Basics' post), hoping to fix it; alas, I was only treating the symptoms, not the problem.

Basically, as part of the driver startup, it needs to initialize VMX on every active CPU. As far as I'm aware, the only way to do this is to create a system thread for every CPU, and set the affinity for each thread to only run on its corresponding CPU. Based on KeSetSystemGroupAffinityThread on MSDN, the thread should be moved to the correct CPU by the time that function returns (because the thread's IRQL is at the lowest possible level, which is below APC level). Sometimes, however, the thread remains on the CPU it started on. One time, I even saw it move to the correct CPU and immediately jump back to the original CPU before initializing VMX. Therefore, I believe the root cause of the issue is perhaps the thread's IRQL isn't high enough, some sort of kernel interrupt is preempting it and restoring the affinity mask (and thus possibly moving it back to the original CPU). As a result, it'll try to clean up VMX on a CPU that isn't running VMX anymore, thus resulting in the crash.

I want to test some more on the whole thread affinity issue, and I definitely need to do more testing with the child device creation; however, it's currently about 4:50 AM here, and I should probably get some sleep.

Maybe tomorrow I can figure out how to attach a few pictures showing the registered devices and logs.