SUSE Kernel ABI Stability
It would be great if OS kernels had support for all features now and in the future, had no bugs, and no known security holes. Then we could deploy our systems and not be concerned with updates and potential regressions The fact is, kernels do have bugs and security holes that must be addressed. In addition, new features and functionality are constantly being developed, enhanced and optimized. This is true for the SUSE Linux Enterprise kernels, and customers need to deal with the updates that come their way.
These updates, which contain kernel code changes, can lead to incompatibilities with other parts of the system if precautions are not taken. In this document we are concerned with the compatibility at the kernel ABI (kABI) level. This compatibility is sometimes termed kABI Stability and only applies to kernel level code (i.e. kernel modules). For more information on the kernel programing interfaces see the The Kernel Module Programming Interface
section earlier in this document.
When updating kernels, SUSE does take precautions with regard to the stability of the kernel ABI and has the following policy for SUSE Linux Enterprise products:
Keep the kernel ABI stable for the lifetime of each service pack. Kernel Module Packages built for a specific service pack will remain compatible with all update kernels to that service pack. The same policy holds for kernels updates between the initial release of the product and the first service pack.
With service pack or major version updates, there is no guarantee of kernel ABI compatibility with the previous releases1. Kernel Module Packages maintained for several service pack releases will need to be rebuilt for each service pack.
Given the life of general support of an individual service pack being around two years and the option to extend that by an addition three years using Long Term Service Pack Support2, the policy provides for a fairly long stable period with compatibility changes happening at predefined and well planned intervals. That allows for the introduction of substantial improvements over the life of the product, resulting in an overall better offering to our customers without sacrificing longer term stability and compatibility.
What does it mean for vendors?
If a vendor provides modules that are maintained in the upstream kernel and supported in the SUSE Linux Enterprise product, most likely kernel ABI compatibility will not be an issue. It’s important to work with SUSE to ensure the module is considered supported in the SUSE products, and gets updated to the latest upstream version at each service pack release.
If the modules are not part of the upstream kernel, they will need to be rebuilt for each service pack and possibly require source code changes to match the updated kernel interfaces. SUSE offers a beta program allowing for partners to test and prepare for any kernel module adjustments well in advance of the service pack release.
For further information on the SUSE beta program or ensuring that your upstream drivers are up to date and supported in SUSE Linux Enterprise products, contact your PartnerNet representative3.
What does it mean for users?
The most prominent use of third party kernel modules is updating the standard drivers delivered with SUSE Linux Enterprise products to enable new products or features. Such updates are typically maintained upstream, and as a result are included in the next service pack release removing the need for the updates going forward. In those cases, the kernel ABI changes coming with service pack updates will not affect end users.
If the modules are not maintained in the upstream kernel, then the user will need to take care when deploying service pack updates to ensure that new, compatible versions of the kernel modules are also applied. Users are advised to contact the vendor of the kernel modules for assistance with updating to new service packs.
Why not maintain compatibility across service packs?
SUSE Linux Enterprise is based on the community developed Linux kernel, and by that, leverages the advancements and code quality gained from the community development process4. To help customers best take advantage of the ongoing community work, we strive to keep the SUSE Linux Enterprise kernels as close as possible to the upstream community code base. Since the ABI of the community kernel is continually changing, maintaining compatibility over time would require diverging more and more from the community. As this divergence widens, the benefits that we and our customers realize by being close to and part of the general kernel community diminish.
So, while keeping the kernel ABI compatible for the life of a SUSE product is preferred by vendors that deliver their own drivers, the resulting divergence from upstream is not the best option for maintaining code quality over the long term. On the other hand, breaking the kernel ABI compatibility several times a year is neither desirable nor imperative.
SUSE’s policy for kernel ABI stability strikes a balance between compatibility and advancement that allows us and our customers to take advantage of new upstream development during the life of SUSE Linux Enterprise while minimizing divergence from general kernel community.
Does SUSE provide a kernel ABI symbol whitelist?
One tactic to maintaining a stable kernel ABI over a long period of time is to define a subset of the complete kernel ABI guaranteed to remain compatible. This subset is often termed a “whitelist” referring to only the symbols that are “safe” to use in external kernel modules. SUSE does not follow this route and instead keep virtually all5 symbols unchanged, but for a shorter period of time. We have found this policy to be better suited to most vendors maintaining kernel modules as it provides freedom and flexibility in leveraging the kernel interfaces they require to support their products.
Compatibility With User Space Applications
We have focused on the interfaces at the kernel level only (i.e. between the kernel and kernel modules); the topic of user space interfaces is beyond the scope of this document. It will simply be stated that the formal programming interfaces between kernel and user space are kept compatible for the life of a major release of SUSE Linux Enterprise Server and Desktop products. Indeed, most of the interfaces are compatible across major releases as well.
Later service packs (e.g. SUSE Linux Enterprise 10 SP4) tend to have smaller changes to the kernel, and in these cases we strive to keep the kernel ABI compatible with the previous service pack release where possible.↩
Ibrahim Haddad (Ph.D.) and Brian Warner, Upstreaming: Strengthening Open Source Development, The Linux Foundation, Section 3: Benefits of Upstreaming, http://www.linuxfoundation.org/publications/linux-foundation/upstreaming-strengthening-open-source-development↩
We do allow rare exceptions for a few symbols that clearly make no sense for an external kernel module to use.↩