Introduction
This article dissects the chrt applet from Busybox 1.32.0. The command reads or modifies the scheduling attributes of a process.
chrt -m
SCHED_OTHER min/max priority : 0/0
SCHED_FIFO min/max priority : 1/99
SCHED_RR min/max priority : 1/99
SCHED_BATCH min/max priority : 0/0
SCHED_IDLE min/max priority : 0/0
SCHED_DEADLINE min/max priority : 0/0
pidof firefox
6987 6851 6825 6816 6800 6771 6767 6761 6720 6611
chrt -p 6987
pid 6987's current scheduling policy: SCHED_OTHER
pid 6987's current scheduling priority: 0
sudo chrt -f -p 1 6987
chrt -p 6987
pid 6987's current scheduling policy: SCHED_FIFO
pid 6987's current scheduling priority: 1
The Busybox applet is ten times smaller once compiled than the util-linux binary, with some limitations.
The dissection
The applet is implemented in util-linux/chrt.c, whose main function calls several helpers and has three parts:
- the first parses the command options
- the second prints the scheduler's information
- the last one, to apply scheduler changes in case of a set
At start of main, the character string containing the options are parsed to obtain a bitfield easier to use:
opt = getopt32(argv, "^"
"+" "mprfobi"
"\0"
/* only one policy accepted: */
"r--fobi:f--robi:o--rfbi:b--rfoi:i--rfob"
);
If the (-m) is set then the min and max valid priorities for each scheduling policies are shown and the command exits:
if (opt & OPT_m) { /* print min/max and exit */
show_min_max(SCHED_OTHER);
show_min_max(SCHED_FIFO);
show_min_max(SCHED_RR);
show_min_max(SCHED_BATCH);
show_min_max(SCHED_IDLE);
fflush_stdout_and_exit(EXIT_SUCCESS);
}
The function show_min_max uses the Posix functions sched_get_priority_max and sched_get_priority_min from the
standard C library to send a syscall to the kernel in order to obtain the min and max values accepted by each policy:
max = sched_get_priority_max(pol);
min = sched_get_priority_min(pol);
if ((max|min) < 0)
fmt = "SCHED_%s not supported\n";
Otherwise the required options and arguments to show or to apply real-time attributes of a process:
//if (opt & OPT_r)
// policy = SCHED_RR; - default, already set
if (opt & OPT_f)
policy = SCHED_FIFO;
if (opt & OPT_o)
policy = SCHED_OTHER;
if (opt & OPT_b)
policy = SCHED_BATCH;
if (opt & OPT_i)
policy = SCHED_IDLE;
argv += optind;
if (!argv[0])
bb_show_usage();
if (opt & OPT_p) {
pid_str = *argv++;
if (*argv) { /* "-p PRIO PID [...]" */
priority = pid_str;
pid_str = *argv;
}
/* else "-p PID", and *argv == NULL */
pid = xatoul_range(pid_str, 1, ((unsigned)(pid_t)ULONG_MAX) >> 1);
} else {
priority = *argv++;
if (!*argv)
bb_show_usage();
}
Then the applet uses the Posix function sched_getscheduler provides by the standard C library to obtain the scheduling attributes of the process specified by the pid.
print_rt_info:
pol = sched_getscheduler(pid);
if (pol < 0)
bb_perror_msg_and_die("can't %cet pid %u's policy", 'g', (int)pid);
Finally, when chrt modifies scheduling attributes, the Posix function sched_setscheduler applies them and the new attributes are shown:
if (sched_setscheduler(pid, policy, &sp) < 0)
bb_perror_msg_and_die("can't %cet pid %u's policy", 's', (int)pid);
if (!argv[0]) /* "-p PRIO PID [...]" */
goto print_rt_info;
The function sched_setscheduler and sched_getscheduler will send a syscall to the scheduler subsystem of the kernel Linux.
This subsystem also exposes this information from /proc:
cat /proc/6987/sched
WebExtensions (6987, #threads: 23)
-------------------------------------------------------------------
se.exec_start : 4421312.640001
se.vruntime : 344438.942254
se.sum_exec_runtime : 38238.466094
se.nr_migrations : 6811
nr_switches : 49452
nr_voluntary_switches : 21749
nr_involuntary_switches : 27703
se.load.weight : 1048576
se.runnable_weight : 1048576
se.avg.load_sum : 3415
se.avg.runnable_load_sum : 3415
se.avg.util_sum : 3497621
se.avg.load_avg : 74
se.avg.runnable_load_avg : 74
se.avg.util_avg : 74
se.avg.last_update_time : 4421312640000
se.avg.util_est.ewma : 75
se.avg.util_est.enqueued : 75
policy : 0
prio : 120
clock-delta : 89
mm->numa_scan_seq : 0
numa_pages_migrated : 0
numa_preferred_nid : -1
total_numa_faults : 0
current_node=0, numa_group_id=0
numa_faults node=0 task_private=0 task_shared=0 group_private=0 group_shared=0
Limitations
Limitations I observed in this applet:
Resetting scheduling policy
The chrt applet doesn't offer an option (-R) to specify if the scheduling policy should be applied or reset when a
process forks to create children. This feature, introduced since Linux 2.6.32, can be only enabled or disabled at the
build of busybox and it is applied on all scheduling attributes modifications done with this applet.
Deadline support
The chrt applet doesn't provide the required scheduling options (-d, -T, -P and -D) to set the deadline scheduling attributes of a process.
Introduction
I had not built an embedded Linux without a framework like Open Embedded from the Yocto project for years, so this guide builds a minimal one from scratch to boot a target. The examples boot a virtual Qemu target and can be adapted to a real one. The build environment is bootstrapped with a prebuilt glibc cross-toolchain from Bootlin.
Setup the environment
First, install the packages needed to use the cross-toolchain, compile the host tools and run Qemu:
- The Ncurses libraries are only required to execute the command make menuconfig.
- The certificates and wget will be used to download the prebuilt toolchain.
- In the same way, git will be used to checkout the source of Busybox and Linux.
- The Qemu packages emulate the target system and run static aarch64 binaries on the x86-64 host.
apt update
apt install -y --no-install-recommends \
bc \
build-essential \
ca-certificates \
cpio \
file \
flex \
git \
ipxe-qemu \
libncurses5-dev \
libncursesw5-dev \
libssl-dev \
qemu \
qemu-system-aarch64 \
qemu-user-static \
wget
Download and extract the prebuilt toolchain:
mkdir ~/src
cd ~/src
wget https://toolchains.bootlin.com/downloads/releases/toolchains/aarch64/tarballs/aarch64--glibc--stable-2020.08-1.tar.bz2
tar xvjf aarch64--glibc--stable-2020.08-1.tar.bz2
Then set the environment variables needed to cross-compile:
- PATH: extended so the cross-tools are available
- CROSS_COMPILE: the prefix used by the cross-tools
- ARCH: the architecture of the target platform
ls ~/src/aarch64--glibc--stable-2020.08-1/bin/*gcc
~/src/aarch64--glibc--stable-2020.08-1/bin/aarch64-linux-gcc
export PATH=~/src/aarch64--glibc--stable-2020.08-1/bin:$PATH
export CROSS_COMPILE=aarch64-linux-
The cross-tools can now be called from the shell:
aarch64-linux-gcc -v
Using built-in specs.
COLLECT_GCC=~/src/aarch64--glibc--stable-2020.08-1/bin/aarch64-linux-gcc.br_real
COLLECT_LTO_WRAPPER=~/src/aarch64--glibc--stable-2020.08-1/bin/../libexec/gcc/aarch64-buildroot-linux-gnu/9.3.0/lto-wrapper
Target: aarch64-buildroot-linux-gnu
<...>
Thread model: posix
gcc version 9.3.0 (Buildroot 2020.08-14-ge5a2a90)
Build the Linux kernel
Fetch and build the latest stable branch of the Linux kernel:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
cd linux
git checkout -b local/linux-5.4.y origin/linux-5.4.y
# git show HEAD
export ARCH=arm64
make defconfig
HOSTCC scripts/basic/fixdep
HOSTCC scripts/kconfig/conf.o
HOSTCC scripts/kconfig/confdata.o
HOSTCC scripts/kconfig/expr.o
LEX scripts/kconfig/lexer.lex.c
YACC scripts/kconfig/parser.tab.[ch]
HOSTCC scripts/kconfig/lexer.lex.o
HOSTCC scripts/kconfig/parser.tab.o
HOSTCC scripts/kconfig/preprocess.o
HOSTCC scripts/kconfig/symbol.o
HOSTLD scripts/kconfig/conf
*** Default configuration is based on 'defconfig'
#
# configuration written to .config
#
# make menuconfig
make -j$(nproc)
<...>
AR drivers/net/ethernet/built-in.a
AR drivers/net/built-in.a
AR drivers/built-in.a
GEN .version
CHK include/generated/compile.h
LD vmlinux.o
MODPOST vmlinux.o
MODINFO modules.builtin.modinfo
LD .tmp_vmlinux.kallsyms1
KSYM .tmp_vmlinux.kallsyms1.o
LD .tmp_vmlinux.kallsyms2
KSYM .tmp_vmlinux.kallsyms2.o
LD vmlinux
SORTEX vmlinux
SYSMAP System.map
Building modules, stage 2.
MODPOST 531 modules
OBJCOPY arch/arm64/boot/Image
GZIP arch/arm64/boot/Image.gz
make defconfig applies the default configuration for the target platform (ARCH=arm64) and make -j$(nproc) compiles.
git show HEAD and make menuconfig are optional:
- git show HEAD checks that the head is the latest tag of the linux-5.4.y branch.
- make menuconfig customizes the kernel configuration.
NB. The Linux kernel, Busybox and other projects use Kbuild to manage build options.
Populate the sysroot
The easiest way to bootstrap a sysroot is Busybox, a size-optimized single executable providing common UNIX tools. A few configuration files complete the sysroot. Pulling and building Busybox works like the kernel.
git clone git://git.busybox.net/busybox
cd busybox
git checkout -b local/1_32_stable origin/1_32_stable
# git show HEAD
export ARCH=aarch64
export LDFLAGS="--static"
make defconfig
# make menuconfig
make -j$(nproc)
make install
LDFLAGS forces static linking of Busybox; make menuconfig with CONFIG_STATIC=y does the same. A static executable can be tested with Qemu:
qemu-aarch64-static busybox echo "Hello!"
Hello!
qemu-aarch64-static busybox date
Sat Jun 27 15:06:41 UTC 2020
qemu-aarch64-static runs a binary built for another architecture on the host, here the aarch64 Busybox on an x86-64 host.
make install created a tree in the _install directory, used to populate the sysroot:
ls -l _install
total 4
drwxr-xr-x. 1 tperrot tperrot 974 Nov 30 15:22 bin
lrwxrwxrwx. 1 tperrot tperrot 11 Nov 30 15:22 linuxrc -> bin/busybox
drwxr-xr-x. 1 tperrot tperrot 986 Nov 30 15:22 sbin
drwxr-xr-x. 1 tperrot tperrot 14 Nov 30 15:22 usr
ls -l _install/bin
<...>
lrwxrwxrwx. 1 tperrot tperrot 7 Nov 30 15:22 umount -> busybox
lrwxrwxrwx. 1 tperrot tperrot 7 Nov 30 15:22 uname -> busybox
lrwxrwxrwx. 1 tperrot tperrot 7 Nov 30 15:22 usleep -> busybox
lrwxrwxrwx. 1 tperrot tperrot 7 Nov 30 15:22 vi -> busybox
lrwxrwxrwx. 1 tperrot tperrot 7 Nov 30 15:22 watch -> busybox
lrwxrwxrwx. 1 tperrot tperrot 7 Nov 30 15:22 zcat -> busybox
To finish the sysroot, create an rcS init script:
mkdir _install/proc _install/sys _install/dev _install/etc _install/etc/init.d
cat > _install/etc/init.d/rcS << EOF
#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
/sbin/mdev -s
[ ! -h /etc/mtab ] && ln -s /proc/mounts /etc/mtab
[ ! -f /etc/resolv.conf ] && cat /proc/net/pnp > /etc/resolv.conf
EOF
chmod +x _install/etc/init.d/rcS
Build the filesystem
This step packages the sysroot tree into a filesystem the kernel can mount, either a ramfs or a rootfs:
- the ramfs is a simple filesystem the kernel creates in RAM from an archive.
- the rootfs is a filesystem the kernel mounts from a non-volatile device.
See the kernel documentation for the difference between them.
Build a ramfs
Build the ramfs with cpio and gzip after fixing the ownership:
mkdir _rootfs
rsync -a _install/ _rootfs
chown -R root:root _rootfs
cd _rootfs
find . | cpio -o --format=newc > ../rootfs.cpio
cd ..
gzip -c rootfs.cpio > rootfs.cpio.gz
Build a rootfs
For the rootfs, create an empty image, format it as ext3 through a loop device, then copy the tree and fix the ownership.
dd if=/dev/zero of=rootfs.img bs=1M count=10
mke2fs -j rootfs.img
mkdir _rootfs
mount -o loop rootfs.img _rootfs
rsync -a _install/ _rootfs
chown -R root:root _rootfs
sync
umount _rootfs
Boot the target
The qemu commands to boot the system:
# With the ramfs
qemu-system-aarch64 -nographic -no-reboot -machine virt -cpu cortex-a57 -smp 2 -m 256 \
-kernel ~/src/linux/arch/arm64/boot/Image \
-initrd ~/src/busybox/rootfs.cpio.gz \
-append "panic=5 ro ip=dhcp root=/dev/ram rdinit=/sbin/init"
# With the rootfs
qemu-system-aarch64 -nographic -no-reboot -machine virt -cpu cortex-a57 -smp 2 -m 256 \
-kernel ~/src/linux/arch/arm64/boot/Image \
-append "panic=5 ro ip=dhcp root=/dev/vda" \
-drive file=~/src/busybox/rootfs.img,format=raw,if=none,id=hd0 -device virtio-blk-device,drive=hd0
The target boots to a shell:
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]
[ 0.000000] Linux version 5.10.0-rc5 (tperrot@27ea4a863f61) (aarch64-linux-gcc.br_real (Buildroot 2020.08-14-ge5a2a90) 9.3.0, GNU ld (GNU Binutils) 2.33.1) #1 SMP PREEMPT Mon Nov 30 14:40:05 UTC 2020
[ 0.000000] Machine model: linux,dummy-virt
<...>
[ 0.858346] Sending DHCP requests ., OK
[ 0.870558] IP-Config: Got DHCP answer from 10.0.2.2, my address is 10.0.2.15
[ 0.870909] IP-Config: Complete:
[ 0.871199] device=eth0, hwaddr=52:54:00:12:34:56, ipaddr=10.0.2.15, mask=255.255.255.0, gw=10.0.2.2
[ 0.871566] host=10.0.2.15, domain=, nis-domain=(none)
[ 0.871825] bootserver=10.0.2.2, rootserver=10.0.2.2, rootpath=
[ 0.871866] nameserver0=10.0.2.3
[ 0.872389]
[ 0.875863] ALSA device list:
[ 0.876151] No soundcards found.
[ 0.879353] uart-pl011 9000000.pl011: no DMA platform data
[ 0.920237] Freeing unused kernel memory: 5952K
[ 0.921223] Run /sbin/init as init process
Please press Enter to activate this console.