September 29, 2016

The Kyu real-time operating system -- borrowing code from Linux

Borrowing, stealing, this is what open source is all about. The main thing is to maintain attribution (give credit where credit is due) and maintain licensing.

Some time ago, I found myself hankering for a better printf than the one I had coded myself. It occurred to me that this might be a wheel that I wasn't really interested in reinventing. While I was at it, I thought I might look into some other "string" routines like memset() and strcpy(). This are trivial to write in a few lines of code, but there are carefully optimized versions available.

So, to make a long story short, I decided to take code from linux. What I did, in an attempt to reduce my own pain and suffering was to copy whatever directory contained a file and all of its contents. This pulled in a LOT of code I didn't need, but I did not compile all of it. I did the same thing for header files (pulled in entire directories when needed). When all was said and done I had about 40 megabytes of linux source in order to get 9 library routines.

Although ridiculous, things were working and I mostly ignored this until one day a C compiler upgrade made everything stop working because of some compiler dependencies wired into the linux header. You can read about that here:

The only library routines I currently require are: 
memset
memcpy
memcmp
strcpy
strcmp
strlen
strncmp
sprintf
vsnprintf
I took the new approach of making an empty "lib" directory with a Makefile and then copying only those files needed to satisfy my requirements. From there it is a rinse-lather-repeat exercise of trying to compile and link and finding what other files and bits from header files are required. I also comment out all the include statements in each imported file and chase down the things the compiler complains about.

Rather than pulling in myriads of header files, I am compiling one file named "linux_headers" with only the necessary definitions for the few files I am working with. So I begin with these 3 files: 

linux_headers.h
string.c
vsprintf.c
This grows to 12 files, mostly because printf wants to support 64 bit integers. Then a new issue rears its head. Some things are found in architecture dependent directories. The parts of the linux kernel source I am involved with look like this: 
lib
include/linux
arch/arm/lib
arch/arm/include/asm
arch/x86/lib
arch/x86/include/asm
I should mention in passing that if a person looks in arch/arm/lib they will find mostly assembler files, and in particular take note of these: 
memchr.S  memcpy.S  memmove.S  memset.S  memzero.S
These routines are also defined (as machine independent versions) in string.c. There are macros wrapped around each function like __HAVE_ARCH_MEMSET so that it won't be compiled if an architecture dependent version is available. These things are defined in arch/arm/include/asm/string.h.

We spend a lot of time (due to our choice NOT to import the header directories lock stock and barrel) chasing down include files. Especially since they are scattered here and there for various reasons. We have most of what we want in "include/linux" and get most of the rest from "arch/arm/include/asm", but occasionally we have to dig into "arch/arm/include/uapi/asm" to find things.

Some things are conditional on ARMEB, which is "big endian arm" and I do not care about.

When all the smoke finally cleared, here is what was added: 

#       new file:   lib/Makefile
#       new file:   lib/arm/Readme
#       new file:   lib/arm/asmlinkage.h
#       new file:   lib/arm/assembler.h
#       new file:   lib/arm/autoconf.h
#       new file:   lib/arm/bitops.h
#       new file:   lib/arm/byteorder.h
#       new file:   lib/arm/compiler.h
#       new file:   lib/arm/div64.h
#       new file:   lib/arm/div64asm.S
#       new file:   lib/arm/findbit.S
#       new file:   lib/arm/getit
#       new file:   lib/arm/kconfig.h
#       new file:   lib/arm/linkage.h
#       new file:   lib/arm/little_endian.h
#       new file:   lib/arm/unified.h
#       new file:   lib/arm/unwind.h
#       new file:   lib/bitops.h
#       new file:   lib/bitops/AA_getfiles
#       new file:   lib/bitops/AA_readme
#       new file:   lib/bitops/__ffs.h
#       new file:   lib/bitops/__fls.h
#       new file:   lib/bitops/arch_hweight.h
#       new file:   lib/bitops/atomic.h
#       new file:   lib/bitops/builtin-__ffs.h
#       new file:   lib/bitops/builtin-__fls.h
#       new file:   lib/bitops/builtin-ffs.h
#       new file:   lib/bitops/builtin-fls.h
#       new file:   lib/bitops/const_hweight.h
#       new file:   lib/bitops/count_zeros.h
#       new file:   lib/bitops/ext2-atomic-setbit.h
#       new file:   lib/bitops/ext2-atomic.h
#       new file:   lib/bitops/ffs.h
#       new file:   lib/bitops/ffz.h
#       new file:   lib/bitops/find.h
#       new file:   lib/bitops/fls.h
#       new file:   lib/bitops/fls64.h
#       new file:   lib/bitops/hweight.h
#       new file:   lib/bitops/le.h
#       new file:   lib/bitops/lock.h
#       new file:   lib/bitops/non-atomic.h
#       new file:   lib/bitops/sched.h
#       new file:   lib/ctype.c
#       new file:   lib/ctype.h
#       new file:   lib/div64.c
#       new file:   lib/hexdump.c
#       new file:   lib/kstrtox.c
#       new file:   lib/linux_headers.h
#       new file:   lib/math64.h
#       new file:   lib/string.c
#       new file:   lib/string_helpers.c
#       new file:   lib/string_helpers.h
#       new file:   lib/vsprintf.c
#       new file:   lib/zstring
Notice all the linux source bloat. Just to get printf(), strcpy() and a handful of other simple things. A lot ot it is due to printf() providing the facility to print 64 bit longs. Get rid of that and we could cut this down dramatically. This also drags in all the bitops stuff, which could certainly be useful, but we have done without thus far.

Have any comments? Questions? Drop me a line!

Kyu / tom@mmto.org