December 10, 2023

Let's learn USB! -- races and interrupts

You may have read about race conditions, but you probably haven't run head on into them unless you have done operating systems programming. I ran into one doing the USB work on the F103. It isn't directly related to USB at all, but it is an interesting topic to write about.

The situation is this. I added a queue for serial output and let the actual serial output be interrupt driven. This is good because it decouples the timing of writes and the actual output. The race arises because there are (in this case) 3 different independent entities that can be accessing the queue. We can have mainline (non-interrupt) code writing into the queue. We can have interrupt code pulling data off the queue and sending it over the serial port. We can also have USB interrupt code writing messages and putting data into the queue.

The race arises because (as an example) the code sending uart characters can read the queue count, then pull a character. If it gets interrupted between these two events we get an inconsistent situation. Similarly just incrementing (or decrementing) the queue count can give trouble. We will read the variable, increment a register, and write it back. We need locks around all of this.

If we only had mainline code and the interrupt code sending characters, we could just disable interrupts to provide locking. Once we start doing writes from the USB interrupt code, we get into trouble. And we get trouble in at least two ways.

One way is that my code in "puts()" disables then enables interrupts, expecting that it is only called from mainline code. If we call it from interrupt code, it will harmlessly disable interrupts which are already disabled, but then enables them when it is done, which is wrong! If I am calling this from interrupt code I should not reenable interrupts!

Another way we get into trouble is by not taking care about interrupt priorities. Can a USB interrupt be handled while a UART interrupt is being handled? Or vice versa?

The ARM Cortex-M3

The processor in the F103 is the ARM Cortex-M3. It has a single bit register called the PRIMASK. It is normally zero, which allows interrupts. If you set it to one, interrupts are disabled (blocked). I use these routines to handle this:
static inline void enable_irq() { __asm volatile("cpsie i"); }
static inline void disable_irq() { __asm volatile("cpsid i"); }
To inspect (or set) the current state of the PRIMASK register, we can use this:
#define get_PRIMASK(val)   asm volatile ( "mrs %0, primask" : "=r" ( val ) )
#define set_PRIMASK(val)   asm volatile ( "msr primask, %0" : : "r" ( val ) )
The Cortex-M3 also has an interrupt controller, the NVIC. It has a gang of bits, one per IRQ that can be used to enable and disable a given interrupt source. We could use the bit for the UART irq to lock out UART interrupts while a USB interrupt is being handled.

Another approach is to use an array of interrupt priority registers provided by the NVIC. Each register is 8 bits (we have 256 different priorities). Low numbers are "more important", i.e. "higher priority". Not all processors implement all levels, some only implement 4 levels, namely 0, 0x40, 0x80, 0xc0. The STM32F103 implements 16 levels (the high 4 bits).

Just for the record, the F103 has 43 maskable "external" interrupts and 16 internal interrupts (exception) from the Cortex-M3.

If two interrupts have the same NVIC priority set, the lowest numbered interrupt has the higher priority. The numbers for USB and UART are:

#define USB_LP_IRQ	20
#define	UART1_IRQ	37
So just letting the vector numbers sort things out should work fine.

BSD and interrupt priorities

It used to be that everyone knew the BSD unix code. It isn't so much anymore. Here is how things were done then with critical sections.
    int s = spltty();
     ...
    splx(s);
The spltty() call would "raise the interrupt level" and return the current level. Then the splx(s) call would restore the original level.

What I need is something like this to lock uart (tty) interrupts from the usb code. Of course I only have one level with the PRIMASK scheme on the ARM, but what I could do would be something like this:

    int s;
    get_PRIMASK ( s );
    disable_irq ();
     ...
    set_PRIMASK ( s );
This would avoid inadvertently reenabling interrupts if called in interrupt code, and I could let the NVIC priority scheme sort out the rest.
Feedback? Questions? Drop me a line!

Tom's Computer Info / tom@mmto.org