Implementing a custom hypercall in kvm

Question

I am very new to Virtualization and of late I have been trying to familiarize myself with the way VMMs operate and how hypercalls are made.

Talking about which I plan to implement a new hypercall in KVM which is installed on my Ubuntu desktop, and in turn can be callable from the guest environment.With this hypercall I plan to just return a string saying "Hello World". At this point,I am clueless about how to make it happen.It would be really helpful if you could please guide me as to how do I go about implementing such a hypercall.Thank you!

Answer 1

You can use vmcall instruction in the user program to make a hypercall in KVM. You need to write a handler for this VMCALL in the kvm. If you run a code in guest;

#define VMCALL_ID 100
do_vmcall ()
{
   asm volatile ("vmcall" : "eax"(VMCALL_ID));
}

it will result a trap in the KVM. The kvm will call handle_vmcall function. In the handle_vmcall function you need to write a handler corresponding to this.

int handle_vmcall(struct kvm_vcpu *vcpu)
{
    unsigned eax = kvm_read_register(vcpu, VCPU_REGS_RAX);

    switch (eax) {
        case VMCALL_ID:
            BLAH; break;
        default: BLAH; BLAH;
    }
    return 0;
}

Answer 2

The patches below implement a hypercall that will trace_printk "Hello World" to the host's ftrace. (The base code is Linux 4.10.)

1. (In guest kernel source code) Add the system call that calls the hypercall:

    diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl
    index e93ef0b..2ff3b3f 100644
    --- a/arch/x86/entry/syscalls/syscall_64.tbl
    +++ b/arch/x86/entry/syscalls/syscall_64.tbl
    @@ -338,6 +338,7 @@
     329    common  pkey_mprotect       sys_pkey_mprotect
     330    common  pkey_alloc      sys_pkey_alloc
     331    common  pkey_free       sys_pkey_free
    +332    common  hello_hypercall     sys_hello_hypercall

    diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h
    index 91a740f..19208d5 100644
    --- a/include/linux/syscalls.h
    +++ b/include/linux/syscalls.h
    @@ -902,5 +902,6 @@ asmlinkage long sys_pkey_mprotect(unsigned long start, size_t len,
                      unsigned long prot, int pkey);
     asmlinkage long sys_pkey_alloc(unsigned long flags, unsigned long init_val);
     asmlinkage long sys_pkey_free(int pkey);
    +asmlinkage long sys_hello_hypercall(void);

     #endif

    diff --git a/hello_hypercall/hello_hypercall.h b/hello_hypercall/hello_hypercall.h
    new file mode 100644
    index 0000000..cc727ee8
    --- /dev/null
    +++ b/hello_hypercall/hello_hypercall.h
    @@ -0,0 +1 @@
    +asmlinkage long sys_hello_hypercall(void);

    diff --git a/hello_hypercall/Makefile b/hello_hypercall/Makefile
    new file mode 100644
    index 0000000..6247351
    --- /dev/null
    +++ b/hello_hypercall/Makefile
    @@ -0,0 +1 @@
    +obj-y:=hello_hypercall.o

    diff --git a/Makefile b/Makefile
    index f1e6a02..6a84315 100644
    --- a/Makefile
    +++ b/Makefile
    @@ -910,7 +910,7 @@ export mod_sign_cmd

     ifeq ($(KBUILD_EXTMOD),)
    -core-y     += kernel/ certs/ mm/ fs/ ipc/ security/ crypto/ block/
    +core-y     += kernel/ certs/ mm/ fs/ ipc/ security/ crypto/ block/ hello_hypercall/

     vmlinux-dirs   := $(patsubst %/,%,$(filter %/, $(init-y) $(init-m) \
                 $(core-y) $(core-m) $(drivers-y) $(drivers-m) \

2. (In guest kernel source code) Implement the syscall to call the hypercall

    diff --git a/hello_hypercall/hello_hypercall.c b/hello_hypercall/hello_hypercall.c
    new file mode 100644
    index 0000000..aa333f8
    --- /dev/null
    +++ b/hello_hypercall/hello_hypercall.c
    @@ -0,0 +1,17 @@
    +#include<linux/kernel.h>
    +#include<linux/syscalls.h>
    +#include<linux/init.h>
    +#include<linux/linkage.h>
    +#include "hello_hypercall.h"
    +#include<uapi/linux/kvm_para.h>
    +#include<linux/cpumask.h>
    +
    +asmlinkage long sys_hello_hypercall(void)
    +{
    +   kvm_hypercall0(KVM_HC_HELLO_HYPERCALL);
    +   return 0;
    +}

kvm_hypercall0 calls X86_FEATURE_VMMCALL with zero arguments (the other kvm_hypercall functions handle up to four arguments). The number KVM_HC_HELLO_HYPERCALL (defined in include/uapi/linux/kvm_para.h) is passed in as the hypercall number. After calling X86_FEATURE_VMMCALL with the hypercall number and possibly arguments, execution will jump to the host kernel's function kvm_emulate_hypercall in arch/x86/kvm/x86.c.

See https://elixir.bootlin.com/linux/latest/source/arch/x86/include/asm/kvm_para.h#L21 for more information.

3. (In host kernel source code) Define the hypercall number (include/uapi/linux/kvm_para.h) and print "Hello World" as the hypercall code (arch/x86/kvm/x86.c)

    diff --git a/include/uapi/linux/kvm_para.h b/include/uapi/linux/kvm_para.h
    index bf6cd7d..67304a17 100644
    --- a/include/uapi/linux/kvm_para.h
    +++ b/include/uapi/linux/kvm_para.h
    @@ -23,6 +23,7 @@
     #define KVM_HC_MIPS_GET_CLOCK_FREQ 6
     #define KVM_HC_MIPS_EXIT_VM        7
     #define KVM_HC_MIPS_CONSOLE_OUTPUT 8
    +#define KVM_HC_HELLO_HYPERCALL     9

    diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
    index e52c908..b755ccf 100644
    --- a/arch/x86/kvm/x86.c
    +++ b/arch/x86/kvm/x86.c
    @@ -6151,6 +6209,9 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
            kvm_pv_kick_cpu_op(vcpu->kvm, a0, a1);
            ret = 0;
            break;
    +   case KVM_HC_HELLO_HYPERCALL:
    +       trace_printk("Hello World");
    +       break;
        default:
            ret = -KVM_ENOSYS;
            break;

KVM_HC_HELLO_HYPERCALL stores the hypercall's number, 9 (see here for existing hypercall numbers). In arch/x86/kvm/x86.c, in the kvm_emulate_hypercall function, add the case where the the hypercall number matches KVM_HC_HELLO_HYPERCALL.

Invoke the hypercall in the guest kernel to see its output on the host's ftrace.