Monthly Archives: Září 2023

This blogpost is the next installment of my series of hands-on no-boilerplate vulnerability research blogposts, intended for time-travelers in the future who want to do Linux kernel vulnerability research for bugs. In this blogpost I'm discussing adding psuedo-syscalls and struct definitions for ksmbd to Syzkaller, setting up an working ksmbd instance, and patching ksmbd in order to collect KCOV. Let's dive in!

1. What are Syzkaller and KCOV?

Syzkaller is an unsupervised coverage-guided kernel fuzzer, or put differently: it throws generated input at a kernel (through system calls, network packets, et cetera) based on educated-guesses in order to find bugs in that kernel. Syzkaller has builtin support for several kernels, such as Windows, Linux and OpenBSD. A full list can be found in Syzkaller's Github repository.

Syzkaller partially bases its prioritization on KCOV (kernel coverage) since it assumes that more coverage = more bugs. For a full explanation of Syzkaller and KCOV please refer to the "Collecting network coverage — KCOV" and "Integrating into syzkaller" sections of the "Looking for Remote Code Execution bugs in the Linux kernel" blogpost by Xairy.io in order to deduplicate content and keep this blogpost as technical as possible.

2. Adding Syzkaller definitions for ksmbd

In order to make Syzkaller generate input in an efficient matter, we need to give it the structure of the input. For ksmbd, those inputs are SMB requests and hence we need to provide Syzkaller with the structures for SMB requests. The file with my unauthenticated ksmbd definitions can be found here: ksmbd.txt.

The syntax for declaring Syzkaller structures and psuedo-syscalls is quite straightforward and well documented in their Github repository. For ksmbd, you can just copy ksmbd.txt to sys/linux/ksmbd.txt in the Syzkaller repository and Syzkaller will automatically index it. When defining Syzkaller structures you should be as tightly emulating the real data as possible. Hence, please note that the structures are marked as [packed], which means that Syzkaller should not add any padding to the structure, as that will ruin the validity of the SMB request.

However, defining Syzkaller psuedo-syscalls is not as well documented. In order to define your own psuedo-syscalls, you need to define the functions in executor/common_linux.h in the Syzkaller repository. The code I added for sending the requests is (with explanation below):

#if SYZ_EXECUTOR || __NR_syz_ksmbd_send_req
#include <netinet/in.h>
#include <stdio.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/types.h> #define KSMBD_BUF_SIZE 16000 static long syz_ksmbd_send_req(volatile long a0, volatile long a1, volatile long a2, volatile long a3)
{ int sockfd; int packet_reqlen; int errno; struct sockaddr_in serv_addr; char packet_req[KSMBD_BUF_SIZE]; // max frame size debug("[*]{syz_ksmbd_send_req} entered ksmbd send...\n"); if (a0 == 0 || a1 == 0) { debug("[!]{syz_ksmbd_send_req} param empty\n"); return -7; } sockfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); if (sockfd < 0) { debug("[!]{syz_ksmbd_send_req} failed to create socket\n"); return -1; } memset(&serv_addr, '\0', sizeof(serv_addr)); serv_addr.sin_family = AF_INET; serv_addr.sin_addr.s_addr = inet_addr("127.0.0.1"); serv_addr.sin_port = htons(445); errno = connect(sockfd, (struct sockaddr*)&serv_addr, sizeof(serv_addr)); if (errno < 0) { debug("[!]{syz_ksmbd_send_req} failed to connect (err: %d)\n", errno); return errno ^ 0xff80000; } // prepend kcov handle to packet packet_reqlen = a1 + 8 > KSMBD_BUF_SIZE ? KSMBD_BUF_SIZE - 8 : a1; *(unsigned long*)packet_req = procid + 1; memcpy(packet_req + 8, (char*)a0, packet_reqlen); if (write(sockfd, (char*)packet_req, packet_reqlen + 8) < 0) return -4; if (read(sockfd, (char*)a2, a3) < 0) return -5; if (close(sockfd) < 0) return -6; debug("[+]{syz_ksmbd_send_req} successfully returned\n"); return 0;
}
#endif

The function prepends the process ID of the currently running syz-executor to a TCP packet, sends the TCP packet (like an SMB request) to 127.0.0.1:445 and receives a TCP packet (like an SMB response). Prepending the process ID to the packet has todo with kcov and will be explained in section 4.

The psuedo-syscall can be exclusively whitelisted using the following value in the Syzkaller config (the full config can be found in section 5):

} // ... "enable_syscalls": [ "syz_ksmbd_send_req" ], // ...
}

3. Setting up ksmbd

In order to interact with ksmbd, we need to send SMB packets to at 127.0.0.1:445. However when we the executor tries it, it will not able to reach it due to two reasons:

1. Ksmbd needs to be started from userland
2. Networking namespaces prevent access

In order to start the ksmbd TCP server in the kernel we need to kickstart it using the userland toolset. Specifically, we need to setup an SMB user in order to start the SMB server and run the command to start the ksmbd server. In order to be able to interact with ksmbd in Syzkaller VMs, we need to run those commands in every VM at boot-time. I solved this by creating an systemctl service, which runs a script that kickstarts ksmbd.

[Unit]
Description=Ksmbd
After=network.target
StartLimitIntervalSec=0
Type=simple
Restart=always
RestartSec=1 [Service]
User=root
Group=root
ExecStart=/root/start_ksmbd.sh [Install]
WantedBy=multi-user.target

#!/usr/bin/env bash ksmbd.addshare --add-share=files --options="path = /tmp read only = no"
ksmbd.adduser -a user -p password
ksmbd.mountd -n

After enabling the service, it should be possible to connect to ksmbd using nc 127.0.0.1 445 -v.

Lastly, Syzkaller enables networking namespaces by default. The ksmbd service only works on 127.0.0.1:445 in the root-namespace, so the syz-executor instances will not be able to connect to ksmbd since they run in their own namespaces. To bypass this feature, I patched the Syzkaller source-code to avoid creating a network namespace:

static int do_sandbox_none(void)
{ // [snip]
#if SYZ_EXECUTOR || SYZ_NET_DEVICES initialize_netdevices_init();
#endif
+ /* if (unshare(CLONE_NEWNET)) { debug("unshare(CLONE_NEWNET): %d\n", errno); }
+ */ // Enable access to IPPROTO_ICMP sockets, must be done after CLONE_NEWNET. write_file("/proc/sys/net/ipv4/ping_group_range", "0 65535");
+ /*
#if SYZ_EXECUTOR || SYZ_DEVLINK_PCI initialize_devlink_pci();
#endif // [snip]
#if SYZ_EXECUTOR || SYZ_NET_DEVICES initialize_netdevices();
#endif
+ */
#if SYZ_EXECUTOR || SYZ_WIFI initialize_wifi_devices();
#endif setup_binderfs(); + /* int netns = open("/proc/1/ns/net", O_RDONLY); // [snip] close(netns);
+ */ loop(); doexit(1);
}
#endif

Now, the syz-executor instances run in the (same) root-namespace which means that the ksmbd service is available at 127.0.0.1:445 for syz-executor instances.

4. Adding KCOV support to ksmbd

When we try to fuzz ksmbd using the Syzkaller setup demonstrated above in section 2 and section 3 we will notice that the Syzkaller coverage page will not show any coverage in ksmbd, even though it gets valid SMB response. The result of that is that Syzkaller will not generate input based on the amount of ksmbd code reached and the compared request values. We do not get any KCOV because the execution flow of sending our SMB request starts at sys_send() and passes through the networking stack, where soft interrupt requests (softirq's) handles sending and receiving the packet. Because interrupt contexts do not have a process context assigned, they do not support KCOV and stop coverage tracking. Therefore, we need to either:

1. Patch the networking subsystem to not use softirq's
2. Patch ksmbd to use remote KCOV

I started out trying to patch the networking subsystem because I did not know about remote KCOV, but I ended up giving up due to the complexity of the networking subsystem. Hence, that left me exploring the web for alternatives: enter remote KCOV.

Remote KCOV allows you to (re)start KCOV using an identifier which the KCOV collector (like syz-fuzzer) uses to assign certain KCOV to a certain process (like syz-executor). Specifically, syz-fuzzer user the syz-executor instance PID as KCOV identifier, so we need to pass our process id to ksmbd per connection. Luckily, we can prepend the PID to our SMB packet in syz-executor without ugly side effects due to the way ksmbd works with sockets: we can execute an simple read(sock_fd, &pid, 8) function call at the start of the TCP connection handler to acquire the process id in ksmbd. The codeblocks below contain patches for starting remote KCOV (explanation below):

struct ksmbd_conn { // [snip] + unsigned long kcov_handle;
};

int ksmbd_conn_handler_loop(void *p)
{ struct ksmbd_conn *conn = (struct ksmbd_conn *)p; struct ksmbd_transport *t = conn->transport; unsigned int pdu_size, max_allowed_pdu_size; char hdr_buf[4] = {0,}; int size; // [snip] if (t->ops->prepare && t->ops->prepare(t)) goto out; + size = t->ops->read(t, (char*)&conn->kcov_handle, sizeof(conn->kcov_handle), -1);
+ if (size != sizeof(conn->kcov_handle))
+ goto out;
+
+ kcov_remote_start(conn->kcov_handle); conn->last_active = jiffies; while (ksmbd_conn_alive(conn)) { // [snip] } + kcov_remote_stop(); out: // [snip] return 0;
}

static void handle_ksmbd_work(struct work_struct *wk)
{ struct ksmbd_work *work = container_of(wk, struct ksmbd_work, work); struct ksmbd_conn *conn = work->conn; + kcov_remote_start(conn->kcov_handle); atomic64_inc(&conn->stats.request_served); __handle_ksmbd_work(work, conn); // [snip] + kcov_remote_stop();
}

As you can see, we add the kcov_handle property to struct ksmbd_conn to keep track of the KCOV identifier throughout the process of handling an SMB request. Secondly, when an TCP connection gets accepted (and ksmbd_conn_handler_loop gets called) we instantly read the first 8 bytes to kcov_handle, after which the rest of the code will not notice anything with regards to the prepended data. However, ksmbd_conn_handler_loop itself will defer SMB requests within the TCP session so we lose KCOV yet again. Hence, we need to start KCOV again in handle_ksmbd_work which is called for each SMB-request related TCP packet. Luckily, we saved the kcov_handle in struct ksmbd_conn so we can easily access it to restart KCOV.

5. Testing it

We can recompile syzkaller using:

make generate -j`nproc` # generate headers containing our psuedo-syscall
make -j`nproc` # build binaries

The config I used for fuzzing (it should be ironed out per system):

{ "target": "linux/amd64", "http": "0.0.0.0:56741", "workdir": "workdir", "kernel_obj": "./linux-v6.4-patched", "image": "./image/bullseye.img", "sshkey": "./image/bullseye.id_rsa", "syzkaller": "/opt/syzkaller", "procs": 16, "type": "qemu", "sandbox": "none", "enable_syscalls": [ "syz_ksmbd_send_req" ], "vm": { "count": 24, "kernel": "./linux-v6.4-patched/arch/x86/boot/bzImage", "cpu": 1, "cmdline": "net.ifnames=0 oops=panic panic_on_warn=1 panic_on_oops=1", "mem": 2048 }
}

When testing the Syzkaller and ksmbd setups demonstrated above, everything should run smoothly without any nasty errors or bugs. While using Syzkaller with the config above I found 4 unique bugs in an Linux 6.4 release:

6. Conclusion

I'm satisfied with how the fuzzer turned out as it allows me to fuzz unauthenticated SMB requests, but there's still a long way to go to improve the QoL of (ksmbd) fuzzing:

1. Extending the SMB request definitions to support more commands
2. Implementing authentication using a psuedo-syscall that returns the session id

Thank you for reading my blogpost, I hope you learned as much as I did researching and writing about these topics. For questions, job inquiries, and other things, please send an email to [email protected] (PGP key).