La32r_linux_stb_article
Teaching Linux About a Store Buffer: Kernel Changes for the SoC2 CPU7B STB
CPU7B, the LoongArch32r core in SoC2, has gained a store buffer (STB). A store no longer stalls the core until memory has answered. It is put into a 4-entry queue and finishes three cycles after it is issued, while the queue writes it to the AXI bus in the background. That cuts a store from about 35 cycles to 3.
Software can’t see the STB when it works, but it can see what the STB changes:
- Ordering. A buffered store and a later load to a different address can reach the bus in the opposite order.
- Errors. A store whose write fails has already finished, so there is no instruction left to take the exception on.
The Linux kernel for SoC2 (la32r-linux-mmu) had a bug for each. Neither
one could show up on the current hardware, where every store is
synchronous. Both would break the system once buffering was switched on.
This article describes the two bugs, what they would do to a running
system, and the kernel changes that fix them.
1. Background: what software must know about the STB
Each access has a memory access type (MAT), taken from the direct map window (DMW) or the TLB entry it goes through:
| MAT | Meaning | STB behaviour |
|---|---|---|
| CC (coherent cached) | ordinary memory | stores are buffered; a load may forward from the STB or overtake a buffered store to another address |
| SUC (strongly ordered uncached) | devices | a store waits for its own write response; a load waits until the STB is empty |
| WUC (weakly ordered uncached) | frame buffers | buffered, like CC |
The SoC2 kernel sets up two direct map windows in head.S:
| Window | Virtual base | CSR value | MAT | Kernel macro |
|---|---|---|---|---|
| DMW0 | 0xa000_0000 |
0xa0000011 |
CC | CAC_BASE, PAGE_OFFSET |
| DMW1 | 0x8000_0000 |
0x80000001 |
SUC | UNCAC_BASE, IO_BASE, TO_UNCAC() |
Both windows map the same physical memory. ioremap() already returns
addresses in the SUC window (IO_BASE + pa). On hardware without an STB,
or with STB_BUFFER = 0, the two windows behave the same, which is why
the first bug stayed hidden.
2. Bug 1: the UART was accessed through the cached window
What the code did
The SoC2 UART driver hard-coded its registers in the CC window:
#define SOC2_UART_BASE 0xa0020000 /* DR at +0, STATUS at +4 */
So did the fallback interrupt clear in loongson32/irq.c
(readl((void __iomem *)0xa0020000)), and the hand-written serial debug
macros in kernel/entry.S and mm/tlbex-32.S.
What it would do with buffering on
The polled transmit loop is:
static void soc2_putc(struct uart_port *port, int ch)
{
while (readl(port->membase + SOC2_UART_STATUS) & SOC2_UART_TX_BUSY)
;
writel(ch & 0xff, port->membase + SOC2_UART_DR);
}
Printing "ab" with the registers mapped CC goes like this:
writel('a') -> DR buffered in the STB, the core goes on
readl(STATUS) CC load, address 0x...04
STB check: same 8-byte word as the DR store, but no
byte overlap -> no hit -> the load goes to the bus
now, AHEAD of the buffered write to DR
-> STATUS says "not busy" (the UART hasn't seen 'a')
writel('b') -> DR buffered
... the UART gets 'a', then 'b' while still sending 'a'
The busy check reads the state of the UART before the previous character reached it. Characters are overwritten and lost, so console output is garbled, more so the faster the CPU prints. With CC mapping, forwarding is a second hazard. A load from a register that a buffered store has just written returns the stored value from the STB and never reaches the device. For a register whose read and write mean different things (UART DR: read = received byte, write = byte to send), that read returns the wrong value.
These are the standard reasons device registers must be mapped strongly ordered on any CPU with a write buffer.
The fix
All UART accesses now go through the SUC window:
#include <asm/addrspace.h>
#define SOC2_UART_PHYS 0x20000
#define SOC2_UART_BASE TO_UNCAC(SOC2_UART_PHYS) /* 0x80020000, SUC */
...
.mapbase = SOC2_UART_PHYS,
The same change was made to the interrupt clear in loongson32/irq.c
(readl((void __iomem *)TO_UNCAC(0x20000))) and to the eight copies of
the serial debug macro in entry.S and tlbex-32.S
(0xa0020000/4 → 0x80020000/4). Those macros are currently not called,
but they would print garbage if re-enabled.
Because both windows reach the same physical address, this change is safe on the current hardware and can be tested before the STB is enabled.
3. Bug 2: what happens when a buffered store fails
The situation
A store to a physical address that nothing decodes (on SoC2, the AXI
interconnect answers with DECERR) used to get a precise bus error (ADEM):
ERA pointed at the store, BADV held its address, and the kernel’s
do_ade() sent SIGBUS or oopsed. With buffering, the store has finished
by the time the error comes back, tens to hundreds of cycles later.
The first design, and why it would have broken the kernel
The first version of the STB hardware kept a sticky error and made the next load or store raise the bus error instead of executing. It looks reasonable, but the next memory access is often not in the program that issued the store.
Suppose a timer interrupt arrives just after a user program’s bad store.
The first memory accesses after that are in the kernel’s interrupt entry,
SAVE_SOME (include/asm/stackframe.h):
csrrd t1, PRMD
...
get_saved_sp LONG_L sp, t1, 0 <- load: takes the pending error
LONG_S zero, sp, PT_R0 ... <- or one of these stores
...
csrrd ra, LOONGARCH_CSR_EPC
LONG_S ra, sp, PT_EPC <- ERA is only saved here
The nested ADEM overwrites ERA and PRMD before they have been saved. The
interrupted program’s return address and privilege level are gone, and
do_ade() sees a kernel-mode fault and calls die_if_kernel(). The TLB
refill handler has the same problem: it reads page tables with ld.w
before saving anything. So a bad store in a user program would have
crashed the kernel, and the oops would point at an unrelated kernel
instruction. This was caught in design review, before the STB went to
hardware.
The hardware fix: an asynchronous abort
The STB now reports a failed buffered write the way Arm reports one, as an asynchronous abort that is masked like an interrupt (SError on AArch64, imprecise external abort on ARMv7, imprecise BusFault on Cortex-M):
- The error is a level on ESTAT.IS bit 5 (HWI1), enabled by ECFG.LIE bit 5. It is only taken when CRMD.IE is set, so never inside exception entry or the refill handler.
- STBERR (CSR 0x102): bit 0 = error pending (write 1 to clear), bit 1 = more than one error since the last clear.
- STBEVA (CSR 0x103): virtual address of the first failed store.
- Loads and stores are never blocked by a pending error.
- Strongly-ordered (SUC) stores still wait for their own response and get a precise ADEM, so device writes keep exact errors.
Kernel part 1: handle the interrupt
arch_init_irq() now enables the new source:
set_csr_ecfg(SOC2_IRQ_TIMER | SOC2_IRQ_STBERR);
mach_irq_dispatch() handles bit 5 first, and soc2_stb_report() reads
the two CSRs, clears the error and decides by the address:
if (va >= TASK_SIZE)
panic("SoC2: asynchronous write error at kernel VA 0x%08x", va);
if (tsk->mm) {
tsk->thread.soc2_stb_err_va = va;
tsk->thread.soc2_stb_err = 1;
set_tsk_thread_flag(tsk, TIF_NOTIFY_RESUME);
}
A user address means a user program wrote to a mapping with no device behind it. It gets SIGBUS, the same signal the precise path gave. A kernel address means kernel memory or a kernel mapping is broken, and the kernel cannot know what it lost, so it panics. arm64 does the same for an uncontainable SError.
Kernel part 2: blame the right task
The interrupt doesn’t say which task issued the store, and blaming
current isn’t always right:
- A user task does a buffered store to a bad address.
- A timer interrupt enters the kernel; exception entry clears CRMD.IE.
- The write error comes back while IE = 0: ESTAT.IS bit 5 is set, but the interrupt can’t be taken.
- The kernel calls
schedule()and switches to another task. Interrupts are re-enabled after the switch. - The STB interrupt is taken with
current= the new task.
The window is short, at most the time the STB needs to drain, about
STB_DEPTH write latencies or a few hundred cycles. It is still a real
race. It is worst when the next task is a kernel thread: an earlier draft
of the handler decided “panic” from current->mm == NULL, so a user
program’s bad store could have panicked the kernel.
An asynchronous error can never say which instruction failed, but it
can be tied to the task. Arm added the ESB instruction (and IESB, an
implicit ESB on exception entry) in ARMv8.2 to collect pending errors at a
chosen point. CPU7B’s equivalent is DBAR: it waits until every buffered
store has its write response, and the sticky error is set on the same
clock edge as the entry leaves the STB. So a CSR read right after DBAR is
guaranteed to see it.
switch_to() now calls soc2_stb_sync(prev) before resume():
#define switch_to(prev, next, last) \
do { \
lose_fpu_inatomic(1, prev); \
soc2_stb_sync(prev); \
(last) = resume(prev, next, task_thread_info(next)); \
} while (0)
void soc2_stb_sync(struct task_struct *prev)
{
/* DBAR waits until every buffered store has its write response */
__asm__ __volatile__("dbar 0" : : : "memory");
if (csr_readl(SOC2_CSR_STBERR) & SOC2_STBERR_E)
soc2_stb_report(prev, prev->mm ? task_pt_regs(prev)->csr_epc : 0UL,
"at context switch, last user");
}
After this, an error can only be pending for the task that is running. So
the interrupt handler can blame current, and the context switch blames
prev.
Why the signal is deferred
switch_to() runs with the runqueue lock held. Sending a signal from
there would take the signal lock and wake up the target task, which takes
scheduler locks again: the classic deadlock. soc2_stb_report() therefore
does only lock-free things. It logs with printk_deferred(), stores the
address in two new thread_struct fields, and sets TIF_NOTIFY_RESUME.
The signal itself is sent when the task next returns to user mode, in
do_notify_resume():
#ifdef CONFIG_SOC2_SOC
/* store buffer write error recorded for this task, see loongson32/irq.c */
if (current->thread.soc2_stb_err) {
current->thread.soc2_stb_err = 0;
force_sig_fault(SIGBUS, BUS_ADRERR,
(void __user *)current->thread.soc2_stb_err_va);
thread_info_flags |= _TIF_SIGPENDING;
}
#endif
force_sig_fault() fills si_code = BUS_ADRERR and si_addr = the failed
address, so a debugger or a signal handler sees where the bad write went.
The interrupt path uses the same deferred delivery, so there is one code
path for both.
4. Prior art: how arm64 KVM blames the right guest
soc2_stb_sync() solves a problem that Linux already had to solve on Arm.
There, a failed buffered write becomes an asynchronous SError. arm64 Linux
panics on any SError it can’t attribute (do_serror() in
arch/arm64/kernel/traps.c). That is acceptable for the host kernel,
because an unprivileged process can’t create a mapping to a hole in the
physical address map. A virtual machine is different: a guest kernel is
untrusted, and it can point its “physical” addresses anywhere. If a
guest’s failed store raised its SError after the CPU had left the guest,
the host would panic because of the guest.
So KVM collects pending SErrors on every guest exit, in __guest_exit
(arch/arm64/kvm/hyp/entry.S), before any host code runs. The guest
plays the role of prev, the host that of next. There are two paths,
chosen at boot.
With the RAS extension (ARMv8.2+). The hypervisor’s exception vectors
execute esb (Error Synchronization Barrier). It consumes any pending
SError and records its syndrome in DISR_EL1 instead of taking it. The
exit path only reads the register:
mrs_s x2, SYS_DISR_EL1
str x2, [x1, #(VCPU_FAULT_DISR - VCPU_CONTEXT)]
cbz x2, 1f
msr_s SYS_DISR_EL1, xzr
orr x0, x0, #(1<<ARM_EXIT_WITH_SERROR_BIT)
1: ret
Without RAS: the “abort window”. The only way to consume a pending SError is to take it, so KVM takes it on purpose, at one chosen instruction:
dsb sy // Synchronize against in-flight ld/st
isb // Prevent an early read of side-effect free ISR
mrs x2, isr_el1
tbnz x2, #8, 2f // ISR_EL1.A: an SError is pending
ret
2:
mrs x2, elr_el2 // taking the SError overwrites these, save them
mrs x3, esr_el2
mrs x4, spsr_el2
mov x5, x0
msr daifclr, #4 // unmask SError
abort_guest_exit_start:
isb // the pending SError is taken here
abort_guest_exit_end:
msr daifset, #4 // mask again
ret
_kvm_extable abort_guest_exit_start, 9997f
_kvm_extable abort_guest_exit_end, 9997f
9997:
msr daifset, #4
mov x0, #(1 << ARM_EXIT_WITH_SERROR_BIT)
msr elr_el2, x2 // restore the exit's exception context
msr esr_el2, x3
msr spsr_el2, x4
orr x0, x0, x5
1: ret
The _kvm_extable lines use Linux’s exception table mechanism, the
same one get_user() relies on. Each entry is a pair: an instruction that
is allowed to fault, and where to continue if it does. When the SError is
taken at EL2, __kvm_unexpected_el2_exception() looks up the faulting PC
(ELR_EL2) in __kvm_ex_table:
- If the PC is one of the window labels, the SError was expected. It sets
ELR_EL2to9997:and returns there. - Anywhere else, it is a real hypervisor fault, and it panics.
There are two entries because the SError may be taken just before or just
after the isb.
In both paths the exit code then carries ARM_EXIT_WITH_SERROR_BIT.
handle_exit_early() (arch/arm64/kvm/handle_exit.c) runs before the vCPU
thread can be preempted, and reflects the error back into the guest as a
virtual SError (kvm_inject_vabt()). The host is never blamed. With RAS,
an error whose syndrome says it was corrected or contained can even be
ignored.
One gap. Before RAS, the architecture did not guarantee that an error
from an earlier write is already pending once dsb sy completes. That was
implementation-defined. An error that arrives later is taken in the host
and panics it. Closing that gap is one of the reasons ESB was added.
Comparison
Both arm64 and SoC2 handle an asynchronous write error in two ways. During normal execution it is delivered as an exception or interrupt as soon as it is unmasked. At a synchronization point (KVM’s guest exit, SoC2’s context switch), where blaming the right context matters, it is collected on purpose.
| Step | arm64 KVM, no RAS | arm64 KVM, RAS | SoC2 |
|---|---|---|---|
| Normal delivery | SError exception | SError exception | STB interrupt (ESTAT.IS bit 5), soc2_stb_error() |
| Synchronization point | guest exit, __guest_exit |
guest exit (esb in the vectors) |
context switch, soc2_stb_sync() in switch_to() (interrupts off) |
| Wait for outstanding writes | dsb sy (error pending afterwards is not guaranteed) |
esb on exception entry (guaranteed) |
dbar 0 (guaranteed: the STB is empty and the error is set on the same clock edge) |
| Check for an error | ISR_EL1.A |
DISR_EL1 |
STBERR.E (CSR 0x102) |
| Collect it at the synchronization point | Take the SError in a one-instruction window, fix up via the exception table | Read and clear DISR_EL1, no exception |
Read STBEVA, clear STBERR, no exception |
| Saved/restored registers | ELR/ESR/SPSR_EL2 |
none | none |
| Blamed on | the guest (virtual SError) | the guest, by syndrome | prev, by address (SIGBUS or panic) |
The SoC2 design follows the RAS model:
- Normal delivery. Errors are still delivered asynchronously, by the STB interrupt, so an error is reported as soon as interrupts are enabled, not only at the next context switch.
- At the synchronization point.
switch_to()runs with interrupts disabled, likeesbwith SError masked. There the pending error is read from a status register instead of being taken, which needs no exception window, no saved registers and no exception-table fixup. - The guarantee.
DBARalso guarantees that the error is visible, whichdsbdid not guarantee before RAS.
The two paths depend on each other. The interrupt reports the error
promptly. The check at the context switch ensures that an error is only
ever pending for the running task, so the interrupt handler can blame
current.
All code above is from arch/arm64 in the same Linux 5.14 tree as
la32r-linux-mmu.
5. Effects on the system, before and after
| Situation | Before (with STB_BUFFER = 1) |
After |
|---|---|---|
| Console output | Characters lost or garbled: the STATUS poll overtakes the DR write | Correct: UART is SUC, every access is ordered |
| User program writes through a mapping with no device behind it | Nested ADEM in exception entry or TLB refill: corrupted ERA/PRMD, kernel oops at an unrelated PC | SIGBUS (BUS_ADRERR, si_addr = address) to that program, a log line |
| Same, with a context switch before the error arrives | SIGBUS to the wrong task, or a panic if the next task is a kernel thread | SIGBUS to the program that stored |
| Kernel writes to a bad kernel address | Oops at an unrelated PC | Panic naming the address |
| Device write fails (SUC) | Precise ADEM | Unchanged: precise ADEM |
Hardware without the STB, or STB_BUFFER = 0 |
— | Same behaviour as before: CSR 0x102 reads 0, ESTAT.IS bit 5 never rises, DBAR finishes at once |
The cost is one dbar 0 per context switch. It finishes in the same cycle
when the STB is empty, and takes at most the STB drain time otherwise.
6. Files changed
| File | Change |
|---|---|
drivers/tty/serial/soc2_uart.c |
UART base TO_UNCAC(0x20000) (0x80020000), .mapbase from SOC2_UART_PHYS, comment on why the cached window must not be used |
arch/loongarch/loongson32/irq.c |
UART clear through TO_UNCAC. New SOC2_IRQ_STBERR, STBERR/STBEVA definitions, soc2_stb_report(), soc2_stb_error(), soc2_stb_sync(). ECFG.LIE bit 5 enabled at boot. |
arch/loongarch/include/asm/switch_to.h |
switch_to() calls soc2_stb_sync(prev) (no-op without CONFIG_SOC2_SOC) |
arch/loongarch/include/asm/processor.h |
thread_struct.soc2_stb_err / soc2_stb_err_va (under CONFIG_SOC2_SOC, before fpu, which must stay last) |
arch/loongarch/kernel/signal.c |
do_notify_resume() delivers the recorded error as force_sig_fault(SIGBUS, BUS_ADRERR, va) |
arch/loongarch/kernel/entry.S, arch/loongarch/mm/tlbex-32.S |
Serial debug macros use 0x80020000 / 0x80020004 |
7. Status and limitations
Tested so far: the hardware side of the error path is covered by
simulation. test4_stb_err tests the LSU and STB. test109_stb_write_error
runs a program on the whole core with STB_BUFFER = 1: a buffered store to
an unmapped address raises the interrupt, and the handler sees the right
ESTAT, STBERR and STBEVA and clears it.
Not tested yet: the kernel changes have not been compiled or run; this was written without a LoongArch32r toolchain. To test on hardware:
- Boot with
STB_BUFFER = 0. Everything should behave as before, including the console. - Boot with
STB_BUFFER = 1. Run a long console stress test, such as a largecatordmesg, and look for lost characters. - From user space, map a physical address with no device behind it
(
/dev/mem) and write to it. Expect SIGBUS with thatsi_addrand the log line. - Repeat 3 with a
sched_yield()right after the store, to exercise theswitch_to()path.
Limitations:
- The instruction is never known. The PC in the log is where the error
was taken (or the task’s last user PC at a context switch), not the
store. Only no buffering gives that:
STB_BUFFER = 0, like Cortex-M’sDISDEFWBUF. - Threads share an mm. The SIGBUS goes to the thread that was running
when the error was synchronized. With
soc2_stb_sync()inswitch_to(), that is the thread that issued the store. - Kernel threads that use a user mm (
kthread_use_mm) are recorded but never return to user mode, so they don’t get a signal. The error is still logged. - The early console is still broken.
loongson32/early_printk.cwrites to physical0x20000with 8250 register offsets. That is wrong with paging on and on the SoC2 UART, independent of the STB, and was left unchanged.