Writing linux device drivers (kernel modules)

These notes are being brought up to date in May of 2009, at which time I am running a 2.6.27 kernel. I say this in case you are reading this years later and these notes have become wildly out of date, as they almost certainly will.

Part of what makes linux what it is, is active kernel development. This means that any description of how to write extensions to the kernel will need to be frequently updated (which is not likely for this document). These notes are not intended to be a tutorial. They are my personal condensed cheat-sheet (which I hope may be useful to others) for an activity that I get involved in only sporatically.

A device driver is part of the linux kernel. You could install a device driver by writing it, then compiling it into the kernel (i.e. compiling a custom kernel that included the device driver). Nobody in their right mind does this, they use modules. It used to be done this way in the days of yore with unix kernels. You still could do it this way if you just wanted to make things hard on yourself. Mentioning just one huge advantage of using modules ought to convince you. You can test multiple versions of a module without having to reboot your system each time.

Whole books can, should, and have been written about modules and device drivers. The classic is LINUX Device Drivers by Alessandro Rubini. At the time of this writing (2009), this is in its third edition which is frighteningly out of date for specific details. Any book on this topic is doomed to instant obsolescence. For example in 2009, I am running a 2.6.27 kernel, and the third edition is up to date as of 2.6.10 kernels. Nonetheless, this book is very much worth having and you just have to go online for the big details. The book itself is available online at:

The directory: /lib/modules

This directory contains a subdirectory for every kernel version that might be booted on a given machine, for example on my machine (currently running FC9), I see: 
2.6.23.15-137.fc8
2.6.24.3-34.fc8
2.6.25.14-69.fc8
2.6.26.6-79.fc9.x86_64
2.6.27.5-37.fc9.x86_64
2.6.27.5-41.fc9.x86_64
The kernel I am currently running is (from uname -a) 2.6.27.5-41.fc9.x86_64 so the modules for that kernel will be found in the subdirectory /lib/modules/2.6.27.5-41.fc9.x86_64 Each directory holds a quite complex arrangement of files and directories.

The file modules.dep is created by the depmod utility to indicate module interdependencies. I find I do not need to deal with this or to run this command by hand, but it is good to know about if you start fiddling with a module or subsystem that has dependency issues. If the dependencies in modules.dep are not set up right, you will have no end of grief.

lsmod and the file: /proc/modules

This entry in the /proc filesystem will spew out a lot of information if you type: 
cat /proc/modules
The lsmod utility reads this file and gives you a nicer listing.

The file /etc/conf.modules

In ancient times this might have been called /etc/conf.modules or perhaps /etc/modules.conf. These days /etc/modprobe.conf seems to be the thing. Who knows what the future holds given these sort of file renaming fetishes. This is a short little ascii file holding information to help guide the module loader. There is also a directory /etc/modprobe.d that holds files that are part of this game (and which may even interact with the udev system for yet more excitement).

The file extension .ko

Once a module is built, it is a .ko file, not a .o file as you might expect (and as things once were).

Utilities for loading and unloading modules

In a perfect world, insmod would be all you would need. You would boot your system and use insmod to load your device driver like so: 
insmod onion.ko
and your system would run forever.

Since the world is not so perfect, you also end up needing rmmod. After you fix your driver bug and want to test the new version, you will need to use this to "unload" the previously loaded driver module via rmmod onion.ko or just rmmod onion. modprobe is a higher level interface to insmod, rmmod, and more. For modules that have dependencies it is probably the tool of choice to install and remove modules, but for my simple tastes I do not use it.

Getting the include files right

Maybe you can be running one kernel and build modules for a different kernel. I don't know how to do that (although I am pretty sure it can be done). I only ever try to build a module for the kernel that I am currently running.

A module should only ever include files in /usr/include/linux or /usr/include/asm as things will be arranged so that these are files that correspond to the current running kernel. This is done via symbolic link trickery (or something) to files in /usr/src/kernels/2.6.27.5-41.fc9.x86_64/include/linux, which is exactly what you want. Actually there are other directories that are set up in this way, but none that I have had to be concerned with for simple drivers.

If you do not see files like this in /usr/src you probably need to load the kernel-devel rpm to get the kernel headers. On a fedora system you would do this: 

	yum install kernel-devel

Use the source

You are well advised to find and load the kernel source for your system and use it as a resource. This will provide you with a diverse set of working drivers (modules) for study and reference.

On a fedora system, you need to locate the source rpm for the kernel you are running (just the same as you would need to locate the source rpm for any other package you wanted to study the source from. For my current kernel this would be: 

kernel-2.6.27.5-41.fc9.src.rpm
What I actually found however (on our fedora mirror in fedora/updates/9/SRPMS was: kernel-2.6.27.24-78.2.53.fc9.src.rpm - a more recent version than I am actually running. But I don't intend to actually build a kernel from this, just look at the files, so this will be fine for study purposes: 
rpm -Uvh kernel-2.6.27.24-78.2.53.fc9.src.rpm
This seems to dump the whole shooting match into /usr/src/redhat/SOURCES as a bunch of patch files and the all important pristine file: linux-2.6.27.tar.bz2. You might just as well go to www.kernel.org (or one of its mirrors) and download this or any other kernel source snapshot. However you get it, you then pick a convenient spot (with 400M or so of free space) and do: 
tar xjvf /usr/src/redhat/SOURCES/linux-2.6.27.tar.bz2

Voila - the entire linux kernel source tree! You now have the source to hundreds, if not thousands of device drivers at your very fingertips. I am loathe to recommend any particular starting point since there is so much variation among drivers. Just cd to linux-2.6.27/drivers and start poking around.

boilerplate includes

Each driver will need to pull in: 
#include <linux/module.h>
#include <linux/version.h>
These provide the macros for MOD_INC_USE_COUNT, EXPORT_SYMBOL, and more.

If you are reading a tutorial that tells you that you need something called linux/modversions.h, it is way out of date and you should find other sources of information.

printk

The function printk is more or less the equivalent of printf for the kernel programmer. There is a 1024 byte limit (due to a fixed internal buffer) per call. The output ends up in /var/log/messages (actually this depends on how you have syslog configured), as well as on the system console (if you have one that you can see - try Ctrl-Alt-F2). Also note that printf calls include an embedded "priority" (log level) via one of two idioms, either you do: 
	printk ( "<1>hello world\n" );
or:
	printk ( KERN_INFO "hello world\n" );
Note that there is no comma in the second form. These things are defined in the include file linux/kernel.h - low numbers are more urgent. Values range from 0 (KERN_EMERG) to 7 (KERN_DEBUG). Level 7 may not even make it to /var/log/messages. KERN_INFO is level 6. KERN_NOTICE is level 5.

udev and special files

The good news is that udev (and related subsystems) provide a much more sophisticated approach to the special files in the /dev directory that unix programmers have grown to know and love. The bad news is that there is a substantial learning curve. You will have to figure out sysfs and the udev system and write rules to "do this right" in the 2.6 kernel way of things.

In any linux system, drivers are accessed via a special file in the /dev directory. Once upon a time, you would just use the mknod command to create such entries and life was good. In the current scheme of things (i.e with 2.6 kernels and later) this directory gets created fresh on every reboot! This means you will have to arrange for mknod to be properly invoked on every reboot, or learn how the new scheme (called udev) works.

Under the udev scheme, the files in the /dev directory are created fresh and new on every reboot. You can still use mknod to create entries in /dev, but they will vanish on every reboot, so this is not the way to do things. One of the points of the udev scheme is to allow /dev entries come and go as hot pluggable devices come and go. Another point is that the /dev directory holds only files representing devices (or drivers anyway) actually present in a given system.

Kernel events are sent to the udevd daemon, which gets notified of netlink uevents (by the kernel), and runs in user space in order to make or destroy /dev nodes as needed.

Files in /etc/udev and /lib/udev make this whole business go. If you read udev documents on the web, you will find many references to /sbin/hotplug which no longer exists. The things that used to be done by hotplug are now all done by udev.

It would seem that what you want to do is to create your own udev rules in a file like /etc/udev/rules.d/10-local.rules. What just might work is to have a single entry created for a given driver via: 

DRIVER=="dingus" NAME="frisbee"
This would create /dev/frisbee when the dingus driver was seen.

Driver "maintenance" and general tips

There are two major aspects to writing a device driver. The first is the issue of writing code to control whatever device (i.e. hardware) it is that you need to control. The second is making your code fit the module-driver API for the specific linux kernel version your driver needs to work with. The basic linux kernel interface maintains the same flavor, but changes are made from time to time, and suddenly a working driver will refuse to compile. (People will then say "the new kernel broke my driver".) The trick then is to figure out what is now expected of you. A google search is often the fastest and easiest. If you are really on the bleeding edge and among the first to be hitting the problem, you may need to read the linux kernel mailing lists to sort things out. If you are an experienced C programmer (as you should be), looking at the compiler error messages and the kernel header files should guide you in the right direction.

Have any comments? Questions? Drop me a line!

Adventures in Computing / tom@mmto.org