virtualization c qemu libvirt

TLDR ¶

GitHub - phsm/libvirt-stealtime-disabler · GitHub Contribute to phsm/libvirt-stealtime-disabler development by creating an account on GitHub.

What is CPU steal time ¶

In nutshell, “stolen” CPU time is when a virtual machine “wanted” to have access to CPU, but was denied because the host system CPU was used by something else.

Of course the virtual machine can’t know it by its own because it is not aware about the situation on the host system. Instead, the host hypervisor (KVM in my case) keeps the steal time counter, and exposes it to the virtual machine via special virtual CPU register.

In practice, high steal time observed on a virtual machine usually means that the host system is heavily overbooked, or the virtual machine CPU is being ratelimited.

Can it be disabled ¶

It is very easy and not easy at all at the same time.

Qemu allows to disable it by setting a special virtual CPU flag -kvm-steal-time (Source).

When you use Libvirt, however, you can not pass this CPU flag to Qemu without patching/recompiling Libvirt source, or resorting to some dubious hacks.

The most elegant way (on my opinion) to disable it ¶

So we seemingly have two options:

• Patch the libvirt source to allow adding this CPU flag.
• Patch the Linux kernel to break this functionality there somehow.

Both of these solutions leave you with a fork that you have to maintain.

But what if the qemu spawn call can somehow be intercepted and modified? Yes, there is a technique allowing you to do it: LD_PRELOAD function override.

So the plan is to find which function from the standard C library is used by Libvirt to execute Qemu, and override it with our own function injecting the desired CPU flag.

This is the final code that finally did the trick for me:

 1#include <stdlib.h>
 2#include <string.h>
 3#include <stdio.h>
 4#include <dlfcn.h>
 5
 6#define FLAG_TO_MODIFY "-cpu"
 7#define FLAG_VALUE_TO_ADD ",-kvm-steal-time"
 8#define CMD_TO_INJECT "qemu-system-"
 9
10typedef int (*execve_func_t)(const char *path, char *const argv[], char *const envp[]);
11static execve_func_t orig_execve = NULL;
12
13// Save a little bit of resources:
14// only exec this when the library is loaded
15void __attribute__((constructor)) on_load() {
16    orig_execve = dlsym(RTLD_NEXT, "execve");
17}
18
19int execve(const char *path, char *const argv[], char *const envp[]) {
20    int flag_val_position = -1;
21    char *flag_val = NULL;
22    char **newargv;
23    int i; // this also counts args for malloc
24
25    // Only inject argument to specific process
26    if (argv[0] && !strstr(argv[0], CMD_TO_INJECT)) return orig_execve(path, argv, envp);
27    fprintf(stderr, "[injectflag] attempt to run %s detected that matches the pattern\n", argv[0]);
28
29    for (i = 0; argv[i]; i++) { // when the argv[i] is NULL then the end of the array is reached
30        if (flag_val_position == -1 && strcmp(FLAG_TO_MODIFY, argv[i]) == 0) {
31            // Found the -cpu flag, thus the next argv member will be the one to modify
32            flag_val_position = i+1;
33            fprintf(stderr, "[injectflag] the -cpu flag value has found on position %d\n", flag_val_position);
34        }
35    }
36
37    // if the cpu flag wasn't found then run original func
38    if (flag_val_position < 0) return orig_execve(path, argv, envp);
39
40
41    // at this point, the size of argv is known, thus we can allocate a new argv
42    newargv = malloc(sizeof(char*) * (i+1)); // +1 because NULL needs to be put as the last arg
43    newargv[i] = NULL; // already terminate the array with NULL
44
45    for (int j = 0; j < i; j++) {
46        if (j == flag_val_position) {
47            // allocate memory for the new value
48            flag_val = (char*)malloc(
49                (
50                    strlen(argv[flag_val_position])
51                    + strlen(FLAG_VALUE_TO_ADD)
52                    + 1
53                ) * sizeof(char)
54            );
55            strcpy(flag_val, argv[flag_val_position]);
56            strcat(flag_val, FLAG_VALUE_TO_ADD);
57            newargv[j] = flag_val;
58            fprintf(stderr, "[injectflag] new cpu flags injected: \"%s\"\n", flag_val);
59            continue;
60        }
61        newargv[j] = argv[j];
62    }
63
64    // reuse i for the return code
65    i = orig_execve(path, newargv, envp);
66    free(newargv);
67    if (flag_val) free(flag_val);
68    return i;
69}

So, I compiled it with:

1gcc -Wall -Werror -shared -fPIC -o injectflag.so injectflag.c

Then, I created a systemd drop-in that passed the desired LD_PRELOAD environment variable:

1mkdir -p /etc/systemd/system/libvirtd.service.d
2cat <<EOF > /etc/systemd/system/libvirtd.service.d/injectflag.conf
3[Service]
4Environment=LD_PRELOAD=/opt/injectflag.so
5EOF
6
7systemctl daemon-reload
8systemctl restart libvirtd.service

Tested it by launch a virtual machine… and it worked!