Yeah, nah, aye

Main categories:

Porting Poky to Altera Cyclone V SoCFPGA

(A work in progress)

Preface

I purchased a Terasic DE10-Nano board, which is a minimal development platform for the Altera Cyclone V SoCFPGA - a dual core 800 MHz ARM Cortex-A9 SoC stuck on the same die as a low-end (but huge LUTs) FPGA. Terasic provide an SD card with Ångström Linux pre-loaded onto it, and this distribution boots fine, letting you play with this cool little platform. But, being an engineer, I'd rather "take it apart" than turn it on first. After making the first boot to check that the board didn't arrive faulty, I set to building my own distribution for the board, trying to stick to standard Yocto packages, rather than using custom packages like linux-altera and u-boot-socfpga. It's always more fun for me to steer away from potentially-stale vendor forks of open source components, and use the real upstreams themselves. Besides, These SoCs have been out long enough that decent support should already exist in the stock packages.

Layers

I found the original meta-de10-nano layer online, hosted by the usual suspect. I set the layer and its prerequisites up, changed the kernel and bootloader over to stock rather than vendor, and set to work stripping a bunch of (for now) unneeded packages such as LXDE and so on. Let's just get basic Linux with networking, FPGA interaction working first.

Within a couple of hours, I ironed many of the kinks, and had a rootfs produced by Yocto suitable for NFS booting. I next switched the init system over to systemd, and introduced some more problems that will need solving later. For now, my goal was getting the legacy SD card image upgraded to wic/wks images, so that I can boot the U-Boot produced by Yocto rather than the vendor one still on the SD card. I'd rather boot the current 2019.07 rather than the vendor version of the bootloader, even though it already works. It's one small part in the machine that is still "locked" into being vendor-supplied, and hopefully shouldn't be too much work to upgrade.

U-Boot saga

As established, the vendor U-Boot works fine. 2019.07 doesn't. The vendor U-Boot is actually some old thing from late 2017 with 6 patches applied on top of it, most of which seem to be hot fixes for getting the environment set up correctly, and for getting HDMI display working. Checking the patches against the commits in 2019.07, most of them have been included in mainline now. The only things missing are the HDMI setup patches, but I'm not interested in getting HDMI out of this board in U-Boot yet anyway.

So what doesn't work?

A kernel that boots fine under the vendor U-Boot doesn't boot under a stock 2019.07 U-Boot. Symptoms? Immediate freeze:

Speed: 1000, full duplex
Using ethernet@ff702000 device
TFTP from server 192.168.2.1; our IP address is 192.168.2.2
Filename 'socfpga_cyclone5_de10_nano.dtb'.
Load address: 0x2000000
Loading: ######
         4.8 MiB/s
done
Bytes transferred = 29940 (74f4 hex)
## Flattened Device Tree blob at 02000000
   Booting using the fdt blob at 0x2000000
   Loading Device Tree to 09ff5000, end 09fff4f3 ... OK

Starting kernel ...

Not good. Нехорошо. No output, no nothing. Not a sausage.

This implies something very, very early is failing. The way to open the window into the kernel's head in times like these is to recompile with CONFIG_LL_DEBUG, and enable at least earlyprintk (maybe even debug and then debug_initcall eventually) on the kernel command line. When we enable those, we get a little more information:

...
[    0.047899] CPU: Testing write buffer coherency: ok
[    0.052796] CPU0: Spectre v2: using BPIALL workaround
[    0.058041] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.064273] Setting up static identity map for 0x100000 - 0x100060
[    0.070554] rcu: Hierarchical SRCU implementation.
[    0.075829] smp: Bringing up secondary CPUs ...
[    0.081034] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.081041] CPU1: Spectre v2: using BPIALL workaround
[    0.091821] smp: Brought up 1 node, 2 CPUs
[    0.095901] SMP: Total of 2 processors activated (400.00 BogoMIPS).
[    0.102161] CPU: All CPU(s) started in SVC mode.
[    0.107747] devtmpfs: initialized
[    0.115208] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[    0.123071] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.132892] futex hash table entries: 512 (order: 3, 32768 bytes)
[    0.140115] NET: Registered protocol family 16
[    0.145480] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.153770] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[    0.161777] hw-breakpoint: maximum watchpoint size is 4 bytes.

By this point, it was time to get OpenOCD to drive the on-board JTAG blaster in order to have a look-see. Unfortunately, I wasn't able to get it to reliably halt the board, so it was more or less useless. Getting it working started to look like it would take longer than adding some debug prints to the kernel and recompiling a few times.

Eventually, with enough debug prints put into the suspect places, I narrowed the place of the lockup down to a series of readl calls that the kernel makes in amba_device_try_add. This function is called as part of the of_platform_default_populate_init initcall. initcalls are all executed very early in the kernel, so no wonder we needed CONFIG_LL_DEBUG.

With the cause of the lock-up being a readl, my sights turn away from the Linux kernel, and back to U-Boot. This is probably a hardware setup problem -the vendor U-Boot must be setting something up that the stock one isn't.

U-Boot Bisection

Since the problem is probably in U-Boot, it was time to do some digging in order to find either a common ancestor of the socfpga U-Boot and upstream which works, and then git bisect it, taking the current HEAD as the bad revision. I filtered the list of suspect commits down quite a lot (15k to about 40) by caring only about merge commits which affected socfpga. This more than halved the time required for the bisection (6 boots versus 14). For a relatively slow manual process of rewriting U-Boot to an SD card, changing the SD card into the board, and booting it up, this halving in time was important.

As it turns out, I pinned the regression down to the merge b51d103cab in U-Boot. More specifically, commit 430b42f76a. Here's the hunk of interest (aside from you, dear reader 😉):

    sysmgr_pinmux_init();
    sysmgr_config_warmrstcfgio(0);

-   /* De-assert reset for peripherals and bridges based on handoff */
-   reset_deassert_peripherals_handoff();
+   /* Set bridges handoff value */
    socfpga_bridges_set_handoff_regs(true, true, true);

    debug("Unfreezing/Thaw all I/O banks\n");

A few things to notice here:

Chapter 4 of "Cyclone V Hard Processor System Technical Reference Manual" confirms this is the base address of the SoC's Reset Manager. Additionally, it confirms the comments in the U-Boot source around some of the peripheral resets being held, requiring software deassertion since they "are likely to require software configuration before they can safely be taken out of reset."

This, plus the software architecture of U-Boot, really make final the removal of the "ad hoc" reset routine. In order to solve this problem, I'm not going to just patch it back in locally. Linux is the one trying to access the peripheral, so Linux ought to be the one taking it out of reset when it's ready.

Linux Reset Driver

Luckily, Linux has a concept of reset drivers. Reset drivers are close cousins (double cousins?) of GPIO drivers. At their hearts, they form an abstraction upon some real-world signal. In a way, the abstract reset line can be held as a specialisation of an abstract GPIO. In general, according to the docs, resets express a chip-internal reset line, while chip-external reset lines are "likely better represented as GPIOs."

Had I not taken a 3 month break between the events I'm writing about and the time of writing, I would have had to look at adding reset deassertion in the drivers myself, but as it would happen, in that 3 month gap, the work happened upstream. All that needed backporting were:

…and then the board would boot Linux correctly. Hooray!

Let's quickly nose into these patches, for what it's worth.

ARM: dts: socfpga: add missing reset-names for dma

This patch adds a declaration of a stringy reset name associated with the DMA reset already specified in the SoC's PrimeCell DMA controller's device tree node. If we look at the PrimeCell driver, we'll see that it expects a named reset line, otherwise it won't pull any reset lines, even if they are declared in the device tree node but not named.

ARM: 8906/1: drivers/amba: add reset control to amba bus probe

I haven't dug into this patch deeply yet. But it would seem on the face of it that the PrimeCell driver should already be twiddling the reset line, and this patch itself shouldn't be affected be d8c1ccac44.

More on this later once I learn more about this area of the kernel. It probably has something to do with how platform/bus drivers interact with the drivers of things that hang off them.

Up next: Polishing userspace

The boot still isn't without errors, but this is largely due to hangover from old userspace utilities that were loaded into the image to demo the board. For example, it tries to start a web server with some fancy CGI backend that expects a certain FPGA image to be loaded, and expects certain features from the kernel which I have turned off etc.

I have some minor polishing to do on this front (mostly disabling of this extra cruft) and then I'll be happy with the state of Linux on this board, such that I can start playing with the FPGA side of the SoC rather than the HPS.

More to come.