SUSE SolidDriver Program
Third Party Kernel Modules
As has been pointed out in the previous section, there is the option for third party vendors to deliver their own kernel modules that are built for and can be loaded with SUSE Kernels.
The ability to deliver custom or updated kernel drivers allows hardware and software vendors to bring compatibility of their products to SUSE Linux Enterprise OS’s, but this capability doesn’t come without some challenges. Those challenges are covered in this following sections.
Challenges with Deploying Third Party Kernel Modules
As mentioned in section Power and Privilege of Kernel Modules , kernel modules are first class kernel code that have the same privileges and access to system resources as the rest of the kernel. Kernel modules deployed without care can introduce great risk to the integrity and stability of the complete system. Loading modules of unknown origin or support commitment immediately leads to a running system that is, strictly speaking, unsupported by SUSE.
In addition to the risk of installing kernel modules of unknown origin or support status, kernel modules are tricky to install. Before the SUSE SolidDriver Program, vendors were delivering drivers in many differing and incompatible manners which increased the complexities and likelihood of user error.
Traditional 3rd Party Kernel Module Installation
When open source drivers are involved, a typical method of delivering kernel modules to end users is to simply provide the source code along with build and installation instructions. This requires users to have both a level of software building expertise as well as the proper kernel development environment installed. The common way of building and installing kernel modules as updates to those delivered with the base OS is to overwrite the original modules with the updates. This works fine until the next kernel update is installed which will happily overwrite the previously driver update with the original base version. Many times this leads to an un-bootable system.
A second typical delivery method is to provide kernel drivers in a pre-compiled bundle (some form of archive like tar or even rpm) along with an installation script to help make the deployment process easy for end users. Some of these scripts are quite extensive and try to manage all the subtleties of proper kernel module installation (and hopefully removal as well if the user ever decides to un-install). Unfortunately vendors tend to work in isolation and have developed varying methods (many of which make unsafe assumptions) to install drivers. The end user is then encountered with multitudes of different, vendor-specific installation processes. Most of the methods work fine with manual installation on single, locally accessed systems, but come short when customers need to perform mass installations, or automated installations of various drivers on mixes of hardware.
Supportability
An additional challenge with deploying third party kernel modules comes with requiring support from SUSE. If the kernel is mis-behaving while a third party kernel module is installed, it’s important that SUSE engineers can work closely with the vendor of the driver in order to remedy the situation. This is not only true of third party delivered drivers but also true of many of the in-box drivers delivered with SUSE Linux Enterprise Kernels where kernel drivers are supported in partnership between SUSE and the hardware vendors.
Just as we have established partnerships and joint support processes setup to help support our in-box kernel drivers, it’s important that any third party delivered drivers are accompanied with a proper technical relationship and support process between SUSE and the vendor providing the kernel driver. The SUSE SolidDriver Program provides processes and resources specifically for those tasks. See the Joint Support Agreements section later in this document for more details.
Mitigating Complications
The SUSE SolidDriver Program was established and developed to address the challenges identified with delivering, deploying, and supporting third party kernel modules. Before moving on to describe the program in detail let’s take a moment to address a common question…
Why Not Deliver All Required Kernel Modules with the SUSE Kernels?
With all the risks and challenges associated with using third party delivered kernel modules, why not avoid the situation all together by having SUSE deliver all needed drivers with the SUSE Linux Enterprise kernels? That would indeed take care of many of the challenges. The drivers would simply be installed with our official kernels and kernel updates thereby removing the installation complications, support issues, and issues with trust. It sounds like an ideal solution, but it’s not reasonable given the following reasons:
- Additional quality assurance expense
- Additional risk to users that receive no benefit
- Additional delays in delivering the needed support
- Licensing restrictions
Before going into detail on each of these points lets briefly look at the figure below.
The top portion represents standard kernel updates that go to all SUSE customers while the bottom portion represents an individual kernel module update going to those customers requiring the new functionality. In the former case, the exposure of the update is much greater than the latter. The customers with the gray color are customers that would receive no benefit from the updated, new functionality. Only the green customers get the benefit, and further more, once the green customers are up and running with their new feature support they become gray customers when the next new features or functional support are introduced with kernel module updates. Keep this image in mind while reading the following sections.
Additional Expense of Quality Assurance
SUSE Linux Enterprise kernels go through a great amount of testing and quality assurance before being delivered to customers. The development of a service pack update entails around 6 months of testing and hardening by SUSE and SUSE’s partners. The combined testing efforts span many companies and customers and having the widest and most thorough testing coverage possible in the available time frame. Anyone familiar with the efforts and costs of any kind of product testing can appreciate the amount of expense this calls for. It wouldn’t make much sense to invest so much into testing the initial release of a product or a product service pack and not put the same level of attention to subsequent kernel updates. Indeed, SUSE kernel updates are done with care and testing and the amount of code changed for any update is kept to the bare minimum in order to achieve the specific objectives of the update while keeping risk of recessions and required testing to a minimum. These minimal, specific updates require weeks of testing before release.
Of course, kernel driver updates would not require the months of testing that a full service pack update encompasses, but could require weeks of testing by SUSE and SUSE partners in order to ensure no regressions are introduced for the existing install base (the gray customers). The frequency of new technology introductions and required driver updates would quickly create a very expensive process for all involved only to ensure that those customers, who otherwise receive no additional benefit, don’t encounter issues when deploying our standard updates. It should be clear that the cost vs risk would be a questionable investment.
Additional Risk
Delivering any kind of update introduces risk of regression. As outlined above, extensive quality assurance is required to reduce the risk of regression to a comfortable level. There is never 100% coverage and some level of risk is always present. Many regressions will unfortunately be initially identified by customers when they deploy the updates. While SUSE commits to fixing such regressions with the highest priority, we would rather avoid them all together. Therefore it’s better to restrict kernel module updates to those specific customers that require them and not expose the rest of the customers to unnecessary risk.
Additional Delay
We could avoid the additional QA and risk associated with delivering kernel module updates by integrating the updates into the service pack releases. That is exactly what happens with the standard upstream drivers that are included with the SUSE kernels. Typically all commonly used drivers are updated to the latest available version at time of code freeze for a new service pack. The challenge with this approach is that the time between code freeze, and first customer ship spans many months. Given the speed of development in the IT industry, it’s not uncommon that at time of first customer ship of a new SUSE Linux Enterprise major release or service pack, many newly released hardware components are not supported by the newly released SUSE product.
It’s of value to have, without delay, support in SUSE Linux Enterprise for the latest technologies as they are released to the market. For this reason, use of third party kernel modules is equally valuable and important for our customers.
Licensing Restrictions
Last, but not least, licensing restrictions make it difficult if not impossible for SUSE to deliver some pieces of software including some kernel modules.
Embracing Third Party Kernel Modules
As we have learned so far, the flexibility that loadable kernel modules bring to a Linux based system is a valuable asset while the same time safely applying third party kernel modules to production systems requires care and attention to details. SUSE embraces the delivery and use of third parties with SUSE Linux Enterprise products when done responsibly. The SUSE SolidDriver Program has been designed to address the complications and risks associated with deploying such module updates and provide customers a well integrated, supported, and dependable experience in the process.