I wonder if it is remotely related to this bug?
https://bugzilla.redhat.com/show_bug.cgi?id=1350390
If I happen to cross over a virt-viewer window on my way
to some other window, the virt-viewer keeps the keyboard
focus. (Probably not the same, other than obviously
screwed up grabbing in some app).