Developer Notes

Building KMPs with Retpoline Support

The following note provides details around building KMPs with retpoline support. It applies to SUSE products that were released before the retpoline based mitigations were created. For future products simply building KMPs as was done in the past (i.e. Method 1 described below) is all that is required.

Background

One of the mitigations against the Spectre Variant 2 vulnerability is to compile code without use of indirect jumps. This method is known as Retpoline. Many of the latest SUSE kernel updates have been built using the retpoline methods. For this mitigation to be fully effective, all running kernel object code, including loadable kernel modules, needs to be compiled using the retpoline methods. That requires all third party, externally delivered kernel modules to be built in a retpoline manner.

Building KMPs with Retpoline Support

SUSE recommends third parties to deliver kernel modules via Kernel Module Packages (KMPs). This document describes methods for building kernel modules using the KMP standard.

Default requirement: Updated compiler

The fundamental requirement for building retpoline based objects is a compiler that supports compiling using retpoline methods. With GCC these methods are provided by the -mindirect-branch and -mindirect-branch-register switches. The GCC releases for SUSE Linux Enterprise products that support these new switches can be found at the following locations:

SUSE Linux Enterprise 12 (same RPM release for all service packs)
https://download.suse.com/Download?buildid=T3xL9TbCM4k~
SUSE Linux Enterprise 11 (same RPM release for all service packs)
https://download.suse.com/Download?buildid=yVF9yHZmMoc~

Method 1: Building against a retpoline kernel

The first and most straightforward method to build retpoline based kernel modules is to use a compiler that supports the -mindirect-branch switches and build against a retpoline based kernel release. The build environment of these updated kernels will set the proper compiler flags automatically.

Advantages

  • No adjustments need to existing KMP sources - just re-build in the new environment

  • Kernel build environment takes care of setting the proper compiler flags

  • Can check for the external thunk symbols to verify that the module was built with retpoline support.

  • The retpoline: Y flag will be set in the module on kernels that provide the flag and check for it at module load time (currently only SUSE Linux Enterprise 12 Service Pack 2 and Service Pack 3)

Disadvantages

  • This method uses the kernel’s exported thunk routines (-mindirect-branch=thunk-extern compiler option) and the resulting kernel modules are therefore dependent on kernels supplying those new symbols. The KMPs will not work with pre-retpoline based kernels including the kernel used on SUSE Linux Enterprise installers.

Method 2: Using the inline thunk routines

The previous method of compiling the modules against the retpoline kernels using the default -mindirect-branch=thunk-extern compiler switch has the disadvantage of requiring kernels that supply the exported thunk routines. To overcome that requirement, the -mindirect-branch=thunk-inline compiler option can be used. That will instruct the compiler to inline the retpoline based methods and relieve the need to use kernels supplying the thunk symbols. Such modules should function on pre and post retpoline based kernel builds.

To use the inline thunk method requires explicitly specifying the following compiler switches in the build environment:

-mindirect-branch=thunk-inline
-mindirect-branch-register

As noted earlier, a compiler that supports these new switches is required, but unlike the Method 1 described above, an updated kernel is not required. The modules can be built using the initial kernel release for the given SUSE Linux Enterprise product and service pack.

Advantages

  • Resulting kernel modules can be used with pre and post retpoline based kernels from SUSE.

Disadvantage

  • Can’t check for the external thunk symbols to verify that the module was built with retpoline support.

  • Requires changes to the module source or KMP spec file

  • Building with older kernels will not set the retpoline: Y flag.

Setting the compiler switches

The KCFLAGS environment variable can be used to set the compiler flags for the kernel module build session. We recommend simply adding it to the %build section of the KMP spec file. For example the following %build section would be updated from:

%build
export EXTRA_CFLAGS='-DVERSION=\"%version\"'
for flavor in %flavors_to_build; do
    rm -rf obj/$flavor
    cp -r source obj/$flavor
    make -C %{kernel_source $flavor} modules M=$PWD/obj/$flavor
done

to:

%build
export KCFLAGS='-mindirect-branch=thunk-inline -mindirect-branch-register'
export EXTRA_CFLAGS='-DVERSION=\"%version\"'
for flavor in %flavors_to_build; do
    rm -rf obj/$flavor
    cp -r source obj/$flavor
    make -C %{kernel_source $flavor} modules M=$PWD/obj/$flavor
done

Using the -mindirect-branch compiler flags does require a compiler supporting those flags, so running a KMP build using the above spec file construct in environments without such compiler support will fail based on unknown compiler flags. To create a spec file that will work in both retpoline and non-retpoline build environments create a conditional build around the KCFLAGS line as follows:

%bcond_with retpoline
%if %{with retpoline}
export KCFLAGS='-mindirect-branch=thunk-inline -mindirect-branch-register'
%endif

The full %build section will then look like:

%build
export EXTRA_CFLAGS='-DVERSION=\"%version\"'
%bcond_with retpoline
%if %{with retpoline}
export KCFLAGS='-mindirect-branch=thunk-inline -mindirect-branch-register'
%endif
for flavor in %flavors_to_build; do
    rm -rf obj/$flavor
    cp -r source obj/$flavor
    make -C %{kernel_source $flavor} modules M=$PWD/obj/$flavor
done

By using that build condition, the special KCFLAGS will only take effect when passing the --with retpoline to the rpmbuild command. Refer to the RPM documentation on conditional builds for more information on using the --with flag.

NOTE: For the KCFLAGS to take effect, the ‘make’ coll must invoke a kbuild style build.

Method 2a: Setting the retpoline flag

This method is actually an extension to method 2 in that it also dynamically patches the retpoline flag into the module source.

As noted with Method 1, on kernels that support the retpoline module flag, the flag will be set, and checked for during module load. If a retpoline kernel loads a kernel module that does not have the retpoline flag set, the following warning will be output to the console:

Spectre V2 : System may be vulnerable to spectre v2

Currently only SUSE Linux Enterprise 12 SP2 and SP3 implement the flag and check.

When using Method 2 to build against older kernels, the retpoline flag will not be set even though the modules are indeed built with retpoline support. That will cause the above warning to be shown. To manually set the flag and avoid the warning, the following “patch” can be used to add the flag to most standard kernel module code:

find source -name *.c -print0 | xargs -0 sed -i '/MODULE_LICENSE(/a MODULE_INFO(retpoline, "Y");'

That line is added to the %build section of the spec file in the same location as the KCFLAGS. Using our example, the resulting %build section looks like:

%build
export EXTRA_CFLAGS='-DVERSION=\"%version\"'
%bcond_with retpoline
%if %{with retpoline}
export KCFLAGS='-mindirect-branch=thunk-inline -mindirect-branch-register'
find source -name *.c -print0 | xargs -0 sed -i '/MODULE_LICENSE(/a MODULE_INFO(retpoline, "Y");'
%endif
for flavor in %flavors_to_build; do
    rm -rf obj/$flavor
    cp -r source obj/$flavor
    make -C %{kernel_source $flavor} modules M=$PWD/obj/$flavor
done

The line simply searches all code (assuming it is under source) for the MODULE_LICENSE macro and appends the MODULE_INFO macro that defines the retpoline flag as new line after the LICENSE macro. The MODULE_LICENSE macro should exist and exist once for every module being built.

Of course the MODULE_INFO(retpoline, "Y"); line can be added directly to the module source, but it should be done in a way that that ensures the flag is only set if the module is compiled using the retpoline methods!

In the above example, the retpoline flag is set under same conditional build as the retpoline compiler flags. That should ensure that the flag is only set when also setting the proper -mindirect-branch flags.

Unfortunately, there are instances where the above construct will set the retpoline flag, even though the retpoline compiler flags did not take effect. This is the case if the make command is not invoking a standard kbuild style build. Please see the section below verifying retpoline builds for validating the integrity of your builds.

Verifying retpoline based modules

Verifying that a kernel module is indeed built with the proper compiler and compiler flags is not a straight forward task. The SolidDriver team has developed the following script to help identify proper builds.

#!/bin/bash

if [[ "$#" -ne 1 ]
then
    echo
    echo "$(basename $0): Check that kernel module does not use indirect calls or jumps (i.e. is retpoline based build)
    echo
    echo "  Usage: $(basename $0) PATH_TO_KO_FILE
fi

if objdump -d -Matt $1 | egrep -q "(jmp|call).*%"
then
    echo "unsafe"
    exit 1
else
    echo "safe"
    exit 0
fi

The script simply disassembles the kernel object code and searches for indirect calls or jumps. It will print safe if it does not find any and unsafe if it finds some.

While the check relies on the accuracy of the disassembler, the results of the SUSE SolidDriver team usage has demonstrated it to be a reliable indicator. If nothing else, a comparison between non-retpoline builds and retpoline based builds of the same module source should show a difference.

Note: We have come across some small kernel modules that made no indirect calls or jumps regardless of the compiler flags.

Note: if the kernel module code contains any inline assembly that makes indirect calls or jumps, that code must be manually updated for this test to pass.