u@home:~$

La32r_linux_stb_article

Teaching Linux About a Store Buffer: Kernel Changes for the SoC2 CPU7B STB

CPU7B, the LoongArch32r core in SoC2, has gained a store buffer (STB). A store no longer stalls the core until memory has answered. It is put into a 4-entry queue and finishes three cycles after it is issued, while the queue writes it to the AXI bus in the background. That cuts a store from about 35 cycles to 3.

Software can’t see the STB when it works, but it can see what the STB changes:

  1. Ordering. A buffered store and a later load to a different address can reach the bus in the opposite order.
  2. Errors. A store whose write fails has already finished, so there is no instruction left to take the exception on.

The Linux kernel for SoC2 (la32r-linux-mmu) had a bug for each. Neither one could show up on the current hardware, where every store is synchronous. Both would break the system once buffering was switched on. This article describes the two bugs, what they would do to a running system, and the kernel changes that fix them.


1. Background: what software must know about the STB

Each access has a memory access type (MAT), taken from the direct map window (DMW) or the TLB entry it goes through:

MAT Meaning STB behaviour
CC (coherent cached) ordinary memory stores are buffered; a load may forward from the STB or overtake a buffered store to another address
SUC (strongly ordered uncached) devices a store waits for its own write response; a load waits until the STB is empty
WUC (weakly ordered uncached) frame buffers buffered, like CC

The SoC2 kernel sets up two direct map windows in head.S:

Window Virtual base CSR value MAT Kernel macro
DMW0 0xa000_0000 0xa0000011 CC CAC_BASE, PAGE_OFFSET
DMW1 0x8000_0000 0x80000001 SUC UNCAC_BASE, IO_BASE, TO_UNCAC()

Both windows map the same physical memory. ioremap() already returns addresses in the SUC window (IO_BASE + pa). On hardware without an STB, or with STB_BUFFER = 0, the two windows behave the same, which is why the first bug stayed hidden.


2. Bug 1: the UART was accessed through the cached window

What the code did

The SoC2 UART driver hard-coded its registers in the CC window:

#define SOC2_UART_BASE		0xa0020000	/* DR at +0, STATUS at +4 */

So did the fallback interrupt clear in loongson32/irq.c (readl((void __iomem *)0xa0020000)), and the hand-written serial debug macros in kernel/entry.S and mm/tlbex-32.S.

What it would do with buffering on

The polled transmit loop is:

static void soc2_putc(struct uart_port *port, int ch)
{
	while (readl(port->membase + SOC2_UART_STATUS) & SOC2_UART_TX_BUSY)
		;
	writel(ch & 0xff, port->membase + SOC2_UART_DR);
}

Printing "ab" with the registers mapped CC goes like this:

writel('a') -> DR       buffered in the STB, the core goes on
readl(STATUS)           CC load, address 0x...04
                        STB check: same 8-byte word as the DR store, but no
                        byte overlap -> no hit -> the load goes to the bus
                        now, AHEAD of the buffered write to DR
                        -> STATUS says "not busy" (the UART hasn't seen 'a')
writel('b') -> DR       buffered
...                     the UART gets 'a', then 'b' while still sending 'a'

The busy check reads the state of the UART before the previous character reached it. Characters are overwritten and lost, so console output is garbled, more so the faster the CPU prints. With CC mapping, forwarding is a second hazard. A load from a register that a buffered store has just written returns the stored value from the STB and never reaches the device. For a register whose read and write mean different things (UART DR: read = received byte, write = byte to send), that read returns the wrong value.

These are the standard reasons device registers must be mapped strongly ordered on any CPU with a write buffer.

The fix

All UART accesses now go through the SUC window:

#include <asm/addrspace.h>

#define SOC2_UART_PHYS		0x20000
#define SOC2_UART_BASE		TO_UNCAC(SOC2_UART_PHYS)	/* 0x80020000, SUC */
...
	.mapbase	= SOC2_UART_PHYS,

The same change was made to the interrupt clear in loongson32/irq.c (readl((void __iomem *)TO_UNCAC(0x20000))) and to the eight copies of the serial debug macro in entry.S and tlbex-32.S (0xa0020000/4 → 0x80020000/4). Those macros are currently not called, but they would print garbage if re-enabled.

Because both windows reach the same physical address, this change is safe on the current hardware and can be tested before the STB is enabled.


3. Bug 2: what happens when a buffered store fails

The situation

A store to a physical address that nothing decodes (on SoC2, the AXI interconnect answers with DECERR) used to get a precise bus error (ADEM): ERA pointed at the store, BADV held its address, and the kernel’s do_ade() sent SIGBUS or oopsed. With buffering, the store has finished by the time the error comes back, tens to hundreds of cycles later.

The first design, and why it would have broken the kernel

The first version of the STB hardware kept a sticky error and made the next load or store raise the bus error instead of executing. It looks reasonable, but the next memory access is often not in the program that issued the store.

Suppose a timer interrupt arrives just after a user program’s bad store. The first memory accesses after that are in the kernel’s interrupt entry, SAVE_SOME (include/asm/stackframe.h):

csrrd   t1, PRMD
...
get_saved_sp            LONG_L sp, t1, 0     <- load: takes the pending error
LONG_S  zero, sp, PT_R0 ...                  <- or one of these stores
...
csrrd   ra, LOONGARCH_CSR_EPC
LONG_S  ra, sp, PT_EPC                       <- ERA is only saved here

The nested ADEM overwrites ERA and PRMD before they have been saved. The interrupted program’s return address and privilege level are gone, and do_ade() sees a kernel-mode fault and calls die_if_kernel(). The TLB refill handler has the same problem: it reads page tables with ld.w before saving anything. So a bad store in a user program would have crashed the kernel, and the oops would point at an unrelated kernel instruction. This was caught in design review, before the STB went to hardware.

The hardware fix: an asynchronous abort

The STB now reports a failed buffered write the way Arm reports one, as an asynchronous abort that is masked like an interrupt (SError on AArch64, imprecise external abort on ARMv7, imprecise BusFault on Cortex-M):

  • The error is a level on ESTAT.IS bit 5 (HWI1), enabled by ECFG.LIE bit 5. It is only taken when CRMD.IE is set, so never inside exception entry or the refill handler.
  • STBERR (CSR 0x102): bit 0 = error pending (write 1 to clear), bit 1 = more than one error since the last clear.
  • STBEVA (CSR 0x103): virtual address of the first failed store.
  • Loads and stores are never blocked by a pending error.
  • Strongly-ordered (SUC) stores still wait for their own response and get a precise ADEM, so device writes keep exact errors.

Kernel part 1: handle the interrupt

arch_init_irq() now enables the new source:

set_csr_ecfg(SOC2_IRQ_TIMER | SOC2_IRQ_STBERR);

mach_irq_dispatch() handles bit 5 first, and soc2_stb_report() reads the two CSRs, clears the error and decides by the address:

	if (va >= TASK_SIZE)
		panic("SoC2: asynchronous write error at kernel VA 0x%08x", va);

	if (tsk->mm) {
		tsk->thread.soc2_stb_err_va = va;
		tsk->thread.soc2_stb_err = 1;
		set_tsk_thread_flag(tsk, TIF_NOTIFY_RESUME);
	}

A user address means a user program wrote to a mapping with no device behind it. It gets SIGBUS, the same signal the precise path gave. A kernel address means kernel memory or a kernel mapping is broken, and the kernel cannot know what it lost, so it panics. arm64 does the same for an uncontainable SError.

Kernel part 2: blame the right task

The interrupt doesn’t say which task issued the store, and blaming current isn’t always right:

  1. A user task does a buffered store to a bad address.
  2. A timer interrupt enters the kernel; exception entry clears CRMD.IE.
  3. The write error comes back while IE = 0: ESTAT.IS bit 5 is set, but the interrupt can’t be taken.
  4. The kernel calls schedule() and switches to another task. Interrupts are re-enabled after the switch.
  5. The STB interrupt is taken with current = the new task.

The window is short, at most the time the STB needs to drain, about STB_DEPTH write latencies or a few hundred cycles. It is still a real race. It is worst when the next task is a kernel thread: an earlier draft of the handler decided “panic” from current->mm == NULL, so a user program’s bad store could have panicked the kernel.

An asynchronous error can never say which instruction failed, but it can be tied to the task. Arm added the ESB instruction (and IESB, an implicit ESB on exception entry) in ARMv8.2 to collect pending errors at a chosen point. CPU7B’s equivalent is DBAR: it waits until every buffered store has its write response, and the sticky error is set on the same clock edge as the entry leaves the STB. So a CSR read right after DBAR is guaranteed to see it.

switch_to() now calls soc2_stb_sync(prev) before resume():

#define switch_to(prev, next, last)					\
do {									\
	lose_fpu_inatomic(1, prev);					\
	soc2_stb_sync(prev);						\
	(last) = resume(prev, next, task_thread_info(next));		\
} while (0)
void soc2_stb_sync(struct task_struct *prev)
{
	/* DBAR waits until every buffered store has its write response */
	__asm__ __volatile__("dbar 0" : : : "memory");

	if (csr_readl(SOC2_CSR_STBERR) & SOC2_STBERR_E)
		soc2_stb_report(prev, prev->mm ? task_pt_regs(prev)->csr_epc : 0UL,
				"at context switch, last user");
}

After this, an error can only be pending for the task that is running. So the interrupt handler can blame current, and the context switch blames prev.

Why the signal is deferred

switch_to() runs with the runqueue lock held. Sending a signal from there would take the signal lock and wake up the target task, which takes scheduler locks again: the classic deadlock. soc2_stb_report() therefore does only lock-free things. It logs with printk_deferred(), stores the address in two new thread_struct fields, and sets TIF_NOTIFY_RESUME. The signal itself is sent when the task next returns to user mode, in do_notify_resume():

#ifdef CONFIG_SOC2_SOC
	/* store buffer write error recorded for this task, see loongson32/irq.c */
	if (current->thread.soc2_stb_err) {
		current->thread.soc2_stb_err = 0;
		force_sig_fault(SIGBUS, BUS_ADRERR,
				(void __user *)current->thread.soc2_stb_err_va);
		thread_info_flags |= _TIF_SIGPENDING;
	}
#endif

force_sig_fault() fills si_code = BUS_ADRERR and si_addr = the failed address, so a debugger or a signal handler sees where the bad write went. The interrupt path uses the same deferred delivery, so there is one code path for both.


4. Prior art: how arm64 KVM blames the right guest

soc2_stb_sync() solves a problem that Linux already had to solve on Arm. There, a failed buffered write becomes an asynchronous SError. arm64 Linux panics on any SError it can’t attribute (do_serror() in arch/arm64/kernel/traps.c). That is acceptable for the host kernel, because an unprivileged process can’t create a mapping to a hole in the physical address map. A virtual machine is different: a guest kernel is untrusted, and it can point its “physical” addresses anywhere. If a guest’s failed store raised its SError after the CPU had left the guest, the host would panic because of the guest.

So KVM collects pending SErrors on every guest exit, in __guest_exit (arch/arm64/kvm/hyp/entry.S), before any host code runs. The guest plays the role of prev, the host that of next. There are two paths, chosen at boot.

With the RAS extension (ARMv8.2+). The hypervisor’s exception vectors execute esb (Error Synchronization Barrier). It consumes any pending SError and records its syndrome in DISR_EL1 instead of taking it. The exit path only reads the register:

	mrs_s	x2, SYS_DISR_EL1
	str	x2, [x1, #(VCPU_FAULT_DISR - VCPU_CONTEXT)]
	cbz	x2, 1f
	msr_s	SYS_DISR_EL1, xzr
	orr	x0, x0, #(1<<ARM_EXIT_WITH_SERROR_BIT)
1:	ret

Without RAS: the “abort window”. The only way to consume a pending SError is to take it, so KVM takes it on purpose, at one chosen instruction:

	dsb	sy		// Synchronize against in-flight ld/st
	isb			// Prevent an early read of side-effect free ISR
	mrs	x2, isr_el1
	tbnz	x2, #8, 2f	// ISR_EL1.A: an SError is pending
	ret
2:
	mrs	x2, elr_el2	// taking the SError overwrites these, save them
	mrs	x3, esr_el2
	mrs	x4, spsr_el2
	mov	x5, x0
	msr	daifclr, #4	// unmask SError
abort_guest_exit_start:
	isb			// the pending SError is taken here
abort_guest_exit_end:
	msr	daifset, #4	// mask again
	ret

	_kvm_extable	abort_guest_exit_start, 9997f
	_kvm_extable	abort_guest_exit_end, 9997f
9997:
	msr	daifset, #4
	mov	x0, #(1 << ARM_EXIT_WITH_SERROR_BIT)
	msr	elr_el2, x2	// restore the exit's exception context
	msr	esr_el2, x3
	msr	spsr_el2, x4
	orr	x0, x0, x5
1:	ret

The _kvm_extable lines use Linux’s exception table mechanism, the same one get_user() relies on. Each entry is a pair: an instruction that is allowed to fault, and where to continue if it does. When the SError is taken at EL2, __kvm_unexpected_el2_exception() looks up the faulting PC (ELR_EL2) in __kvm_ex_table:

  • If the PC is one of the window labels, the SError was expected. It sets ELR_EL2 to 9997: and returns there.
  • Anywhere else, it is a real hypervisor fault, and it panics.

There are two entries because the SError may be taken just before or just after the isb.

In both paths the exit code then carries ARM_EXIT_WITH_SERROR_BIT. handle_exit_early() (arch/arm64/kvm/handle_exit.c) runs before the vCPU thread can be preempted, and reflects the error back into the guest as a virtual SError (kvm_inject_vabt()). The host is never blamed. With RAS, an error whose syndrome says it was corrected or contained can even be ignored.

One gap. Before RAS, the architecture did not guarantee that an error from an earlier write is already pending once dsb sy completes. That was implementation-defined. An error that arrives later is taken in the host and panics it. Closing that gap is one of the reasons ESB was added.

Comparison

Both arm64 and SoC2 handle an asynchronous write error in two ways. During normal execution it is delivered as an exception or interrupt as soon as it is unmasked. At a synchronization point (KVM’s guest exit, SoC2’s context switch), where blaming the right context matters, it is collected on purpose.

Step arm64 KVM, no RAS arm64 KVM, RAS SoC2
Normal delivery SError exception SError exception STB interrupt (ESTAT.IS bit 5), soc2_stb_error()
Synchronization point guest exit, __guest_exit guest exit (esb in the vectors) context switch, soc2_stb_sync() in switch_to() (interrupts off)
Wait for outstanding writes dsb sy (error pending afterwards is not guaranteed) esb on exception entry (guaranteed) dbar 0 (guaranteed: the STB is empty and the error is set on the same clock edge)
Check for an error ISR_EL1.A DISR_EL1 STBERR.E (CSR 0x102)
Collect it at the synchronization point Take the SError in a one-instruction window, fix up via the exception table Read and clear DISR_EL1, no exception Read STBEVA, clear STBERR, no exception
Saved/restored registers ELR/ESR/SPSR_EL2 none none
Blamed on the guest (virtual SError) the guest, by syndrome prev, by address (SIGBUS or panic)

The SoC2 design follows the RAS model:

  • Normal delivery. Errors are still delivered asynchronously, by the STB interrupt, so an error is reported as soon as interrupts are enabled, not only at the next context switch.
  • At the synchronization point. switch_to() runs with interrupts disabled, like esb with SError masked. There the pending error is read from a status register instead of being taken, which needs no exception window, no saved registers and no exception-table fixup.
  • The guarantee. DBAR also guarantees that the error is visible, which dsb did not guarantee before RAS.

The two paths depend on each other. The interrupt reports the error promptly. The check at the context switch ensures that an error is only ever pending for the running task, so the interrupt handler can blame current.

All code above is from arch/arm64 in the same Linux 5.14 tree as la32r-linux-mmu.


5. Effects on the system, before and after

Situation Before (with STB_BUFFER = 1) After
Console output Characters lost or garbled: the STATUS poll overtakes the DR write Correct: UART is SUC, every access is ordered
User program writes through a mapping with no device behind it Nested ADEM in exception entry or TLB refill: corrupted ERA/PRMD, kernel oops at an unrelated PC SIGBUS (BUS_ADRERR, si_addr = address) to that program, a log line
Same, with a context switch before the error arrives SIGBUS to the wrong task, or a panic if the next task is a kernel thread SIGBUS to the program that stored
Kernel writes to a bad kernel address Oops at an unrelated PC Panic naming the address
Device write fails (SUC) Precise ADEM Unchanged: precise ADEM
Hardware without the STB, or STB_BUFFER = 0 — Same behaviour as before: CSR 0x102 reads 0, ESTAT.IS bit 5 never rises, DBAR finishes at once

The cost is one dbar 0 per context switch. It finishes in the same cycle when the STB is empty, and takes at most the STB drain time otherwise.


6. Files changed

File Change
drivers/tty/serial/soc2_uart.c UART base TO_UNCAC(0x20000) (0x80020000), .mapbase from SOC2_UART_PHYS, comment on why the cached window must not be used
arch/loongarch/loongson32/irq.c UART clear through TO_UNCAC. New SOC2_IRQ_STBERR, STBERR/STBEVA definitions, soc2_stb_report(), soc2_stb_error(), soc2_stb_sync(). ECFG.LIE bit 5 enabled at boot.
arch/loongarch/include/asm/switch_to.h switch_to() calls soc2_stb_sync(prev) (no-op without CONFIG_SOC2_SOC)
arch/loongarch/include/asm/processor.h thread_struct.soc2_stb_err / soc2_stb_err_va (under CONFIG_SOC2_SOC, before fpu, which must stay last)
arch/loongarch/kernel/signal.c do_notify_resume() delivers the recorded error as force_sig_fault(SIGBUS, BUS_ADRERR, va)
arch/loongarch/kernel/entry.S, arch/loongarch/mm/tlbex-32.S Serial debug macros use 0x80020000 / 0x80020004

7. Status and limitations

Tested so far: the hardware side of the error path is covered by simulation. test4_stb_err tests the LSU and STB. test109_stb_write_error runs a program on the whole core with STB_BUFFER = 1: a buffered store to an unmapped address raises the interrupt, and the handler sees the right ESTAT, STBERR and STBEVA and clears it.

Not tested yet: the kernel changes have not been compiled or run; this was written without a LoongArch32r toolchain. To test on hardware:

  1. Boot with STB_BUFFER = 0. Everything should behave as before, including the console.
  2. Boot with STB_BUFFER = 1. Run a long console stress test, such as a large cat or dmesg, and look for lost characters.
  3. From user space, map a physical address with no device behind it (/dev/mem) and write to it. Expect SIGBUS with that si_addr and the log line.
  4. Repeat 3 with a sched_yield() right after the store, to exercise the switch_to() path.

Limitations:

  • The instruction is never known. The PC in the log is where the error was taken (or the task’s last user PC at a context switch), not the store. Only no buffering gives that: STB_BUFFER = 0, like Cortex-M’s DISDEFWBUF.
  • Threads share an mm. The SIGBUS goes to the thread that was running when the error was synchronized. With soc2_stb_sync() in switch_to(), that is the thread that issued the store.
  • Kernel threads that use a user mm (kthread_use_mm) are recorded but never return to user mode, so they don’t get a signal. The error is still logged.
  • The early console is still broken. loongson32/early_printk.c writes to physical 0x20000 with 8250 register offsets. That is wrong with paging on and on the SoC2 UART, independent of the STB, and was left unchanged.