If the system owner wanted to, they could use their own firmware/ comprimise firmware, then fake the firmware version to something new, the vm could not even be interacting with the cpu at all. Also, if the keys are in the cpu, then the keys can be extracted.