January 23, 2022

Orange Pi 4 (Rockchip 3399) multiple cores

The RK3399 has 6 ARM cores. Two are "big" (A72) and four are little (A53).

These two sorts of cores are separated into "clusters". We have cluster 0, which has the four A53 cores, and cluster 1, which has the two A72 cores.

Cores and the bootrom

The cores start up in 64 bit (aarch64) mode. Now that I have disassembled part of the bootrom, I can see that the bootrom is entirely 64 bit code, and all the cores start running at address ffff0000 which is the start of the bootrom.

The bootrom inspects the mpidr_el1 register which yields both the cluster number and the core in the cluster. If the cpu running is cluster 0, core 0, it continues to run the boot code. All other cores enter a WFE loop.

I have verified by reading the midr_el1 register that core 0 in cluster 0 is an A53 core.

The WFE loop monitors a pair of addresses in SRAM which are dubbed a "mailbox". I call these SRAM[1] and SRAM[2]. SRAM[0] is used for something else which is so far mysterious. These are 32 bit locations. To bring the cores out of this WFE loop, you place a run address into SRAM[2] and the write 0xdeadbeaf into SRAM[1].

Cores and psci

While looking at the rk3399.dtsi file, I noticed an entry for each of the 6 cores that indicated that the enable method for all 6 cores is "psci".

It turns out that PSCI is the "Power State Coordination Interface", which is an ARM standard for enabling and disabling processors and such. Being a standard, we can learn how to play the game and away we go. They say that the PSCI business relies on "firmware", and it turns out the firmware in question is ATF (i.e. bl31).

Linux has plenty of code to deal with PSCI: 

drivers/firmware/psci/psci.c
drivers/cpuidle/cpuidle-psci.c
include/linux/psci.h
include/kvm/arm_psci.h
virt/kvm/arm/psci.c
Part of the trick (like ATF) is to get past the hype and buzzwords and figure out what is going on.

Linux, "deadbeaf", and the RK3066

I'll note that this section pertains to the RK3066 (which is a dual core ARM Cortex-A9 chip, 32 bit ARMv7) and thus is not relevant to the RK3399 with 64 bit armv8. I was excited at first that I had found something manipulating a mailbox with 0xdeadbeaf for a Rockchip, but then I was perplexed by my inability to access SRAM. Now that I see that this is 32 bit ARM and a whole different arrangement, I am satisfied.

I did a search for "beaf" in the Xulong/linux-orangepi directory and found code that manipulated these locations in 

/u1/Projects/OrangePi/Xulong/linux-orangepi/arch/arm/mach-rockchip/platsmp.c
No doubt study of this entire directory would be rewarding. The code that brings the cores out of the parked WFE loop in SRAM is this: 
    writel(__pa_symbol(secondary_startup), sram_base_addr + 8);
    writel(0xDEADBEAF, sram_base_addr + 4);
I will just note that when I have attempted (using U-Boot) to access sram, I have had my hands slapped with a synchronous exception, so the kernel must do something interesting to gain access.

The symbol "secondary_startup" is in arch/arm64/kernel/head.S

This calls the first C code of secondary_start_kernel() in linux-orangepi/arch/arm64/kernel/smp.c
This eventually calls this: cpu_startup_entry(CPUHP_AP_ONLINE_IDLE) in linux-orangepi/kernel/sched/idle.c
To idle the CPU, this will call: arch_cpu_idle_enter() in linux-orangepi/arch/arm/kernel/process.c
and cpuidle_idle_call() in linux-orangepi/kernel/sched/idle.c

There is lots to study here, but most of this is not RK3399 specific.

The linux kernel gets the sram base address by inspecting a resource named "rockchip,rk3066-smp-sram". This gets involved with something called "mmio-sram". Interestingly, this resource is specified in linux-orangepi/arch/arm/boot/dts/rk3288.dtsi and the declaration is like so: 

bus_intmem@ff700000 {
                compatible = "mmio-sram";
                reg = <0x0 0xff700000 0x0 0x18000>;
                #address-cells = <1>;
                #size-cells = <1>;
                ranges = <0 0x0 0xff700000 0x18000>;
                smp-sram@0 {
                        compatible = "rockchip,rk3066-smp-sram";
                        reg = <0x00 0x10>;
                };
        };
This is shown from ff6f0000 to ff71ffff as a 192k reserved region. Could this be an address alias for the sram? Why the heck are the proper sram addresses made illegal?

In passing, I'll note that there is also "pmu-sram" whatever that is.

Have any comments? Questions? Drop me a line!

Tom's electronics pages / tom@mmto.org