1. 技术文章
  2. ARM: Start/Wakeup/Bringup the other CPU cores/APs and pass execution start address?

ARM: Start/Wakeup/Bringup the other CPU cores/APs and pass execution start address?

由 你离开我真会死。 提交于 2019-11-28 05:05:24

Ok, I'm back baby. Here are the conclusions:

  • The software component that puts the CPUs to sleep is the bootloader (in my case U-Boot)
  • Linux somehow knows how the bootloader does this (hardcoded in the Linux kernel for each board) and knows how to wake them up again

For my ODROID-XU board the sources describing this process are UBOOT ODROID-v2012.07 and the linux kernel found here: LINUX ODROIDXU-3.4.y (it would have been better if I looked into kernel version from the branch odroid-3.12.y since the former doesn't start all of the 8 processors, just 4 of them but the latter does).

Anyway, here's the source code I've come up with, I'll post the relevant source files from the above source code trees that helped me writing this code afterwards:

typedef unsigned int DWORD;
typedef unsigned char BOOLEAN;
#define FAILURE (0)
#define SUCCESS (1)
#define NR_EXTRA_CPUS (3) // actually 7, but this kernel version can't wake them up all -> check kernel version 3.12 if you need this

// Hardcoded in the kernel and in U-Boot; here I've put the physical addresses for ease
// In my code (and in the linux kernel) these addresses are actually virtual
// (thus the 'VA' part in S5P_VA_...); note: mapped with memory type DEVICE
#define S5P_VA_CHIPID (0x10000000)
#define S5P_VA_SYSRAM_NS (0x02073000)
#define S5P_VA_PMU (0x10040000)
#define EXYNOS_SWRESET ((DWORD) S5P_VA_PMU + 0x0400)
// Other hardcoded values
#define EXYNOS5410_REV_1_0 (0x10)
#define EXYNOS_CORE_LOCAL_PWR_EN (0x3)

BOOLEAN BootAllSecondaryCPUs(void* CPUExecutionAddress){

// 1. Get bootBase (the address where we need to write the address where the woken CPUs will jump to)
//    and powerBase (we also need to power up the cpus before waking them up (?))
DWORD bootBase, powerBase, powerOffset, clusterID;

asm volatile ("mrc p15, 0, %0, c0, c0, 5" : "=r" (clusterID));
clusterID = (clusterID >> 8);
powerOffset = 0;
if( (*(DWORD*)S5P_VA_CHIPID & 0xFF) < EXYNOS5410_REV_1_0 )
{
    if( (clusterID & 0x1) == 0 ) powerOffset = 4;
}
else if( (clusterID & 0x1) != 0 ) powerOffset = 4;

bootBase = S5P_VA_SYSRAM_NS + 0x1C;
powerBase = (S5P_VA_PMU + 0x2000) + (powerOffset * 0x80);

// 2. Power up each CPU, write bootBase and send a SEV (they are in WFE [wait-for-event] standby state)
for (i = 1; i <= NR_EXTRA_CPUS; i++)
{
    // 2.1 Power up this CPU
    powerBase += 0x80;
    DWORD powerStatus = *(DWORD*)( (DWORD) powerBase + 0x4);

    if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) == 0)
    {
        *(DWORD*) powerBase = EXYNOS_CORE_LOCAL_PWR_EN;
        for (i = 0; i < 10; i++) // 10 millis timeout
        {
            powerStatus = *(DWORD*)((DWORD) powerBase + 0x4);
            if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) == EXYNOS_CORE_LOCAL_PWR_EN)
                break;
            DelayMilliseconds(1); // not implemented here, if you need this, post a comment request 
        }
        if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) != EXYNOS_CORE_LOCAL_PWR_EN)
            return FAILURE;
    }
    if ( (clusterID & 0x0F) != 0 )
    {
        if ( *(DWORD*)(S5P_VA_PMU + 0x0908) == 0 )
        do { DelayMicroseconds(10); } // not implemented here, if you need this, post a comment request
        while (*(DWORD*)(S5P_VA_PMU + 0x0908) == 0);
        *(DWORD*) EXYNOS_SWRESET = (DWORD)(((1 << 20) | (1 << 8)) << i);
    }

    // 2.2 Write bootBase and execute a SEV to finally wake up the CPUs
    asm volatile ("dmb" : : : "memory");
    *(DWORD*) bootBase = (DWORD) CPUExecutionAddress;
    asm volatile ("isb");
    asm volatile ("\n   dsb\n   sev\n   nop\n");
}
return SUCCESS;
}

This successfully wakes 3 of 7 of the secondary CPUs.

And now for that short list of relevant source files in u-boot and the linux kernel:

  • UBOOT: lowlevel_init.S - notice lines 363-369, how the secondary CPUs wait in a WFE for the value at _hotplug_addr to be non-zeroed and to jump to it; _hotplug_addr is actually bootBase in the above code; also lines 282-285 tell us that _hotplug_addr is to be relocated at CONFIG_PHY_IRAM_NS_BASE + _hotplug_addr - nscode_base (_hotplug_addr - nscode_base is 0x1C and CONFIG_PHY_IRAM_NS_BASE is 0x02073000, thus the above hardcodings in the linux kernel)

  • LINUX KERNEL: generic - smp.c (look at function __cpu_up), platform specific (odroid-xu): platsmp.c (function boot_secondary, called by generic __cpu_up; also look at platform_smp_prepare_cpus [at the bottom] => that's the function that actually sets the boot base and power base values)

For clarity and future reference, there's a subtle piece of information missing here thanks to the lack of proper documentation of the Exynos boot protocol (n.b. this question should really be marked "Exynos 5" rather than "Cortex-A15" - it's a SoC-specific thing and what ARM says is only a general recommendation). From cold boot, the secondary cores aren't in WFI, they're still powered off.

The simpler minimal solution (based on what Linux's hotplug does), which I worked out in the process of writing a boot shim to get a hypervisor running on the XU, takes two steps:

  1. First write the entry point address to the Exynos holding pen (0x02073000 + 0x1c)
  2. Then poke the power controller to switch on the relevant core(s): That way, they drop out of the secure boot path into the holding pen to find the entry point waiting for them, skipping the WFI loop and obviating the need to even touch the GIC at all.

Unless you're planning a full-on CPU hotplug implementation you can skip checking the cluster ID - if we're booting, we're on cluster 0 and nowhere else (the check for pre-production chips with backwards cluster registers should be unnecessary on the Odroid too - certainly was for me).

From my investigation, firing up the A7s is a little more involved. Judging from the Exynos big.LITTLE switcher driver, it seems you need to poke a separate set of power controller registers to enable cluster 1 first (and you may need to mess around with the CCI too, especially to have the MMUs and caches on) - I didn't get further since by that point it was more "having fun" than "doing real work"...

As an aside, Samsung's mainline patch for CPU hotplug on the 5410 makes the core power control stuff rather clearer than the mess in their downstream code, IMO.

John Connor

Go to www.arm.com and download there evaluation copy of DS-5 developement suite. Once installed, under the examples there will be a startup_Cortex-A15MPCore directory. Look at startup.s.

QEMU uses PSCI

The ARM Power State Coordination Interface (PSCI) is documented at: https://developer.arm.com/docs/den0022/latest/arm-power-state-coordination-interface-platform-design-document and controls things such as powering on and off of cores.

TL;DR this is the aarch64 snippet to wake up CPU 1 on QEMU v3.0.0 ARMv8 aarch64:

/* PSCI function identifier: CPU_ON. */
ldr w0, =0xc4000003
/* Argument 1: target_cpu */
mov x1, 1
/* Argument 2: entry_point_address */
ldr x2, =cpu1_entry_address
/* Argument 3: context_id */
mov x3, 0
/* Unused hvc args: the Linux kernel zeroes them,
 * but I don't think it is required.
 */
hvc 0

and for ARMv7:

ldr r0, =0x84000003
mov r1, #1
ldr r2, =cpu1_entry_address
mov r3, #0
hvc 0

A full runnable example with a spinlock is available on the ARM section of this answer: What does multicore assembly language look like?

The hvc instruction then gets handled by an EL2 handler, see also: the ARM section of: What are Ring 0 and Ring 3 in the context of operating systems?

Linux kernel

In Linux v4.19, that address is informed to the Linux kernel through the device tree, QEMU for example auto-generates an entry of form:

    psci {
            method = "hvc";
            compatible = "arm,psci-0.2", "arm,psci";
            cpu_on = <0xc4000003>;
            migrate = <0xc4000005>;
            cpu_suspend = <0xc4000001>;
            cpu_off = <0x84000002>;
    };

The hvc instruction is called from: https://github.com/torvalds/linux/blob/v4.19/drivers/firmware/psci.c#L178

static int psci_cpu_on(unsigned long cpuid, unsigned long entry_point)

which ends up going to: https://github.com/torvalds/linux/blob/v4.19/arch/arm64/kernel/smccc-call.S#L51

标签
arm
multiprocessing
wakeup
cpu-cores
cortex-a
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!