The function that never ran: chasing a 4-byte bug in CPU7B's fetch unit

04 Nov 2026

SoC2 can finally boot Linux 5.14 with the MMU on and run a small interactive shell as /init. Getting there took a while, and one bug took longer than all the others put together. This post is about that one.

The symptom

Boot, wait for the prompt, type ps:

soc2:/# ps
  PID  PPID S CMD
do_page_fault(): sending SIGSEGV to init for invalid read access from 00000008
epc = 77e155c8 in libc.so.6[77dcb000+162000]
ra  = 77e154c4 in libc.so.6[77dcb000+162000]
Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

But type cd or ls first, and then ps works fine. It crashed only on the first run, at the same place every time, and sometimes boot itself died earlier in ld.so.

The crash address was always libc offset 0x4a5c8, inside vfprintf:

4a5c0: ld.w  r4, r26, 72      ; fp->_lock
4a5c8: ld.w  r13, r4, 8       ; fp->_lock->owner  -> NULL + 8

vfprintf decided it had to lock the stream. The stream was the temporary FILE that snprintf builds on the stack, which has no lock because it is marked _IO_USER_LOCK. So _flags did not have that bit set when it was read back.

A value written a moment earlier, read back wrong. That looks like memory corruption, and that’s where I went first.

Suspect 1: the TLB

CPU7B has software-managed, separate ITLB and DTLB, 32 entries each. The obvious story was a stale or duplicate TLB entry sending a load to the wrong physical page. Along the way I found real TLB bugs:

The 4-entry priority encoder returned the OR of the matching indices when two entries matched, so a lookup could return an unrelated page’s translation.
INVTLB only cleared the TLB selected by TLBIDX.I_D. The kernel flushes pages without setting I_D, so after copy-on-write a stale entry was left in the other TLB.
csrrd TLBIDX right after tlbsrch read the value from before the search, because the IFU was only stalled for one cycle.

Fixing those made things better: ps crashed less often. But it still crashed.

So I wrote a kernel helper that reads all 64 TLB entries at the crash and compares each one with the page tables. Result:

TLB check: 0 duplicate, 0 stale

Then I made every kernel TLB flush invalidate all user entries, and stopped __update_tlb() from writing any entry at all, so the refill handler was the only thing writing user TLB entries. Same crash, same address. The TLB was not it.

Suspect 2: memory

Turning off the instruction cache changed nothing. A memory test (fresh pages, fork and copy-on-write) passed. Then I dumped the crashing FILE twice: once through the user mapping, and once by reading the physical frame directly from the kernel. Both agreed. The loads were reading exactly what was in memory. Memory really held the wrong values.

The clue: a function that left no trace

vsnprintf builds the string FILE in a few steps:

sf.f._sbf._f._lock = NULL;
_IO_no_init (&sf.f._sbf._f, _IO_USER_LOCK, -1, NULL, NULL);
_IO_JUMPS (&sf.f._sbf) = &_IO_strn_jumps;
_IO_str_init_static_internal (&sf.f, string, maxlen - 1, string);

So I dumped the whole FILE and checked every field:

Who writes it	Field	In memory
`vsnprintf`	`_lock = NULL`	✓
`_IO_no_init`	`_flags`, `_mode`, `_wide_data`, save/backup pointers	✗ old stack data
`vsnprintf`	vtable	✓
`_IO_str_init_static_internal`	read/write/buffer pointers	✓

Everything before and after the call was there. Everything _IO_no_init writes was missing. I also scanned the stack for any word that looked like a FILE flags value (0xfbadxxxx), in case the stores had gone to the wrong address. There were none.

The stores weren’t lost or misdirected. _IO_no_init never ran.

Narrowing it down

That shifted the question from data to instruction fetch. Why would a call be skipped, and only the first time?

“First time” in a fresh process means pages that have never run. The kernel maps code pages lazily, and the first fetch from such a page takes a TLB refill and/or a page-invalid fetch fault (PIF). So I wrote small user-space tests and ran each one in a fresh process (fork + exec):

Test	What it does	Result
far1, far2	call a function on an unmapped page	pass
far3	run straight through into an unmapped 64 KB block	pass
far4	call into the not-yet-valid half of an ITLB pair	pass
far5	call that needs a refill then a PIF on the target	pass

All passed. Then I looked at libc’s call site again:

6ffd4: bne    r5, r0, 16        -> 0x6ffe4
...
6fffc: st.w   r12, r3, 12       <- last word of a page
70000: bl     _IO_no_init       <- first word of the next page

The bl is the first instruction of a page, reached by falling through from the previous page. In far3 the instruction at the page boundary was a nop, so a lost instruction wouldn’t show. So three more tests put a real instruction there:

far6: b 1f   ...   0xffc: st.w ...   0x1000: bl far6_callee
far7: b 1f   ...   0xffc: st.w b     0x1000: st.w c
far8: b 1f   ...  0xfffc: st.w b    0x10000: st.w c

far6: ret WRONG, stores WRONG   got a=61616161 b=00000000 c=00000000 d=64646464
far7: ret ok,    stores WRONG   got a=71717171 b=72727272 c=00000000 d=74747474
far8: ret ok,    stores WRONG   got a=81818181 b=82828282 c=00000000 d=84848484

Every time, in every process: the first instruction of the new page was skipped.

The root cause: one bit

The IFU fetches 8 bytes (two instructions) at a time. It keeps the fetch address in pf_vaddr_q, and only bits [31:3] go to the cache or bus.

After a branch, pf_vaddr_q is the exact branch target. In far6..far8 that’s 0xffc, and in libc it’s 0x6ffe4: both are the second word of an 8-byte block, so bit 2 is set. That’s fine for the first block.

The trouble was the next address:

assign pf_vaddr_inc = pf_vaddr_q + 4'h8;

Adding 8 keeps bit 2. So after a branch to ...4, every following fetch address was 4 bytes into its block:

`pf_vaddr_q`	block actually fetched
`0x6ffe4` (branch target)	`0x6ffe0`
`0x6ffec`	`0x6ffe8`
`0x6fff4`	`0x6fff0`
`0x6fffc`	`0x6fff8`
`0x70004`	`0x70000`

The instructions were fetched correctly, because bit 2 isn’t used for fetching. The bug stayed invisible until a fetch faulted. The fetch exception logic reports the fault at pf_vaddr_q, and that address becomes ERA:

dffe_ns #(32) fexc_addr_reg (
   .din (pf_vaddr_q),      // 0x70004, should be 0x70000
   ...

Here’s what happens in the ps case:

bne jumps to 0x6ffe4. From here on fetch addresses are block + 4.
The fetch of block 0x70000 hits a page that isn’t valid yet. PIF is raised at 0x70004.
The kernel’s PIF handler marks the page valid (BADV 0x70004 is in the right page, so that part works) and returns to ERA = 0x70004.
bl _IO_no_init at 0x70000 never runs.
vsnprintf carries on with an uninitialized FILE, and vfprintf dereferences NULL + 8.

And why only the first ps? Once any code on that page has run, the page is valid and no fault happens. Running cd or ls first happened to run code on that page.

There’s a slightly embarrassing detail. The wrong low bits had always been there. They only started causing crashes after an earlier fix in the same hunt: fetch exceptions used to take their PC from the instruction in decode, which was wrong in a different way. When I changed fetch exceptions to carry their own fetch address, the stale bit 2 became ERA.

The fix

One line:

assign pf_vaddr_inc = {pf_vaddr_q[31:3] + 29'd1, 3'b000};

The first block after a branch still uses the exact target, so the instruction queue still drops the first half of a ...4 target block. Every block after that starts on an 8-byte boundary, so a fault on it is reported at the right instruction.

After rebuilding:

soc2:/# ps
  PID  PPID S CMD
    1     0 R /init
    2     0 S [kthreadd]
  ...

ps, snp (a single snprintf), snptest 20 (the first snprintf in 20 fresh processes) and the fartest cases all pass.

What I took away

“Memory corruption” isn’t always memory. Every memory-side check came back clean, and that was the result to trust. Data that is missing in a consistent pattern (everything one function writes, and nothing else) points to code that didn’t run.
Dump the whole structure and check every field against the code that should have written it. That was the turning point.
Small, targeted user-space tests beat staring at waveforms when the trigger depends on what the OS has mapped. The fartest cases went from “can’t reproduce” to “fails every time” just by putting a real instruction at the page boundary, and entering it through a branch to 0xffc.
Fixing one bug can expose another. The fetch exception change was correct, and it turned a harmless wrong bit into a crash.

Next steps: a core-level simulation test for this case, checking that the ld.so boot crash is gone too, and removing the debug code from the kernel.

u@home:~$

Archive

About

RSS