Appendix A: Debugging the User/Kernel Boundary

Everything in the debugging appendices of Project 2 and Project 3 still applies, and so do the CHECK() and DEBUG() macros from Project 2’s Appendix D. This project adds one new category of wrong: the CPU is at a different exception level, or on a different stack, than you think. Most of the bugs in this project show up as an exception in an unexpected place, and almost all of them can be diagnosed from three numbers: the exception class in ESR_EL1, the address in ELR_EL1, and the mode bits of the saved SPSR_EL1.

Exception classes

ESR_EL1 bits 31:26 hold the exception class (EC): what kind of synchronous exception happened. These are the ones you are likely to meet:

EC	Meaning	Typical cause in this project
`0x00`	Unknown reason; in practice, an undefined instruction	A process executed an instruction that does not exist, or one that is not allowed at EL0 at all (`eret`, or `msr`/`mrs` on an EL1 register such as `VBAR_EL1`).
`0x01`	Trapped `wfi` or `wfe`	Your idle process executed `wfi` at EL0 while there was nothing to wake it.
`0x0E`	Illegal execution state	An `eret` whose saved `SPSR` names a mode it cannot return to (EL2 or EL3, a reserved mode, or garbage). The exception is taken at EL1 (your `0x200` entry), at the address you returned to. Note that an all-zero `SPSR` is not illegal: it means EL0t with every interrupt unmasked, so a context whose `SPSR` was never filled in usually “returns” to EL0 instead—often at a kernel address (see the box below).
`0x15`	`svc` from AArch64: a system call	Expected. The low 16 bits of `ESR_EL1` hold the system call number.
`0x18`	Trapped system register access	A process at EL0 touched `DAIF` (masking interrupts) or the timer registers. Also kernel code running at EL0 by mistake—see below.
`0x20`	Instruction abort from a lower EL	A process jumped to an address it cannot execute (rare with the MMU off).
`0x21`	Instruction abort at the current EL	The kernel jumped to a bad address—a corrupted function pointer or return address.
`0x22`	PC alignment fault	A jump to an address that is not a multiple of 4.
`0x24`	Data abort from a lower EL	A process made a bad memory access. With the MMU off, this is almost always an unaligned access: the low six bits of the ISS (the DFSC) are `0x21`, “alignment fault”, and bit 6 (WnR) tells you whether it was a write.
`0x25`	Data abort at the current EL	The same, in the kernel.
`0x26`	SP alignment fault	Cannot happen with our boot code, which leaves the CPU’s stack alignment check off. A misaligned stack shows up as an alignment data abort instead (see Failure modes).
`0x3C`	`brk` instruction	A software breakpoint.

The full list is in the ARM Architecture Reference Manual, in the description of the ESR_EL1 register, which also describes what the remaining 25 bits (the instruction-specific syndrome, ISS) mean for each class.

The most confusing exception in this project is an EC 0x18 (or 0x00) whose ELR_EL1 points into your kernel’s code. It means that kernel code ran at EL0. The usual culprits are: a process called a kernel function through a leftover trampoline pointer; or restore_context was given a context whose saved SPSR is EL0t but whose ELR is a kernel address—for example, a kernel continuation saved with the user SPSR.

Reading QEMU’s exception log

make qemu-verbose logs every exception the CPU takes, and every eret. Here is a system call from EL0 (call number 0x2a = 42), and the return to user mode:

Taking exception 2 [SVC] on CPU 0
...from EL0 to EL1
...with ESR 0x15/0x5600002a
...with SPSR 0x600001c0
...with ELR 0x2000150
...to EL1 PC 0x81c00 PSTATE 0x3c5
Exception return from AArch64 EL1 to AArch64 EL0 PC 0x2000150

Read it line by line:

from EL0 to EL1: the exception came from user mode. If you expected a system call and see from EL1 to EL1, your process was not running at EL0.
ESR 0x15/0x5600002a: the exception class (0x15, svc), then the whole ESR_EL1. Its low 16 bits are the svc immediate.
SPSR 0x600001c0: the PSTATE the process had. The low four bits (0) say EL0t; 0x1c0 are the interrupt mask bits (here, FIQ, IRQ, and SError masked—a real process of yours should have IRQs unmasked, i.e., 0x140).
ELR 0x2000150: where the process will resume: the instruction after its svc.
to EL1 PC 0x81c00: where the CPU jumped. This should be your vector table plus 0x400. If it is your table plus 0x200, the exception came from EL1.
Exception return ... to AArch64 EL0 PC ...: your restore_context’s eret back into the process.

A few things QEMU prints that are easy to misread:

A trapped system register access (EC 0x18) is logged as Taking exception 1 [Undefined Instruction], with ESR 0x18/.... QEMU’s label in square brackets is its own name for how it delivers the exception; trust the EC in the ESR line.
A timer interrupt is Taking exception 5 [IRQ]. With your processes at EL0, it should always say from EL0 to EL1; an IRQ from EL1 to EL1 means your kernel was running with interrupts unmasked. Ignore the ...with ESR line of an IRQ: QEMU prints whatever syndrome was left over from an earlier exception, and it says nothing about the interrupt.
A data abort prints ...with FAR 0x... as well: the address the process tried to access.
The log is long, and it goes to standard error, not to the serial console. Save it with make qemu-verbose 2> int.log (your serial console stays in the terminal), and search it from a second terminal, e.g., grep -A6 'from EL0' int.log.

Failure modes

Symptom	Usual cause
The first process starts, then the machine hangs silently	An exception from EL0 hit a vector entry that is still `b .`. Check that `0x400` and `0x480` are filled in, and at the right offsets (`aarch64-elf-objdump -d kernel/kernel8.elf`).
A data abort (EC `0x25`) `from EL1 to EL1` while a process’s first saved context is built or restored, with `FAR` inside your kernel stacks	A kernel stack top that is not 8-byte aligned (it must be 16-byte aligned). Check how you compute each top.
Your process “runs at EL0”, but `make qemu-verbose` shows its exceptions `from EL1 to EL1`	The initial saved context’s `SPSR` is still your Project 3 `SPSR_PROC` (EL1h), not `SPSR_USER` (EL0t).
Every process dies with EC `0x18` at an address in the kernel	They are still calling kernel code through the trampoline. Expected in Quest 1, and in Quest 2 until you silence `printf`. Later, it means a program still calls a trampoline function you have not replaced yet: `info symbol` on the `ELR` tells you which.
Every diabolical program dies right away with EC `0x18`, all at the same small offset in their slots	Their start-up delay, `diab_delay()`, reads the timer’s counter, and your kernel does not let EL0 do that. Write `CNTKCTL_VALUE` to `CNTKCTL_EL1` in `kernel_main` (Quest 1, Part B).
A process’s local variables are garbage after its first system call or interrupt	`SP_EL0` is not saved and restored, or is restored after the `eret`—or two processes’ saved contexts overlap because their kernel stacks do.
A system call returns garbage	The handler’s return value never reached the saved context’s `x0`, or (for `yield`) was stored after the call that never returns.
`svc` arrives with the wrong number	Your kernel extracts the number from the wrong bits of `ESR_EL1`, your `SYS_` number in `kernel/syscall.h` differs from the Syscall ABI (check the stubs with `aarch64-elf-objdump -d user/syscalls.o`), or `user/syscalls.S` is stale (`make clean`).
After a `yield()`, the kernel takes an IRQ `from EL1 to EL1`, or `handle_syscall` seems to return twice	`sys_yield` called `yield_entry`, which saved a second, kernel-mode context with interrupts enabled. Call the function that marks the process `RUNNABLE` and runs the scheduler instead.
A process blocked in `read()` crashes when it wakes up	Its kernel continuation was saved with the user `SPSR`, so `restore_context` returned to EL0 at a kernel address. Use `SPSR_KERNEL`.
A process blocked in `read()` wakes up, and later the kernel takes an IRQ `from EL1 to EL1`	Its continuation was saved with IRQs enabled (your Project 3 `SPSR_PROC`).
Under QEMU, keyboard input stops for good after a while (no panic, and programs keep running), more often when you type fast	Your `uart_interrupt()` acknowledges in `ICR` after draining the FIFO. A byte that arrives in between loses its interrupt, and QEMU’s UART never interrupts for it again. Write `ICR` first, then drain.
`write()` of a valid buffer returns `-1`	Your `user_range_ok()` is off by one at the end of the slot, or compares against the wrong process’s slot.
A spawned child runs the wrong program, or another process’s data changes	The child’s memory slot was chosen by load order instead of by its process table index, or `allocproc` hands out an entry that is still in use.
`wait()` never returns	The child never became a `ZOMBIE` (killed processes must go through your `exit` code), or `exit` woke up a different wait channel from the one `wait` blocks on.
A pipeline never finishes (Quest 5)	Some process still has the pipe’s write end open: usually the shell.
A `gterm` started from a shell never gets the keyboard focus, or its keys show up in the other terminal (Quest 5)	It reads its keys from descriptor 0, a pipe from the other terminal, instead of from the descriptor that `open_console()` returned. Or your focus rules still look only at a process’s parent, which is now a shell.
After `exit`, `vtty` does not start a new shell until some other program exits, and what you type meanwhile disappears (Quest 5)	That program still holds both of the old shell’s pipes: typically a `gterm` started from that shell that did not close its descriptor 0.

`gdb` recipes for this project

Goal	How
Which exception level am I at?	`p/x $cpsr`. The low four bits are `0` for EL0t and `5` for EL1h. (A process at EL0 cannot read `CurrentEL` itself; it is not accessible at EL0.)
Which stack am I on?	`p/x $sp` and compare against your kernel stacks and the process’s slot. In the kernel, it should always be inside the current process’s kernel stack.
Catch every system call	`break handle_syscall`, then `p number` and `p/x *ctx`. `x/2i ctx->elr_el1 - 4` shows the `svc` that got you here.
Watch one system call’s arguments	`p/x ctx->regs[0]`, `ctx->regs[1]`, … (or whatever you called them), and `x/s ctx->regs[1]` for a string—but remember that a process can pass any address at all.
See a process’s registers from inside the kernel	At a breakpoint in the kernel, `$x0`, `$x1`, … are the kernel’s registers, not the process’s. The process’s are in its saved context: `p/x ctx` in your handlers, or `p/x processes[N].context` anywhere. For the process that just trapped, that context sits right at the top of its kernel stack.
Follow a process back to user mode	`break restore_context`, `continue`, then `stepi` until you pass the `eret`. `p/x $pc` is now a user address, and `p/x $cpsr` says EL0.
See user code with symbols	See Using GDB with user programs.
See what a blocked process is waiting in	`p/x processes[N].context`, then `p/x processes[N].context->elr_el1` and `info symbol` on it. For a process blocked in a system call, it is an address inside `block()`.
Check a kernel stack for overflow	Fill each kernel stack with a pattern (e.g., `0xDEADBEEF`) when you allocate it, and later `x/8wx` its bottom: if the pattern is gone, something used it all.

Using GDB with user programs

In Quest 4, you will implement the spawn syscall, which will allow your kernel to start up new programs! This is great, but it also makes debugging with GDB a bit tricky: if new processes are created at runtime, GDB can’t easily know where your kernel loaded it into memory, which means it won’t know where to look for its functions or variables!

To fix this, we’ve added a command to your project’s GDB configuration to allow you to tell GDB which programs you’re loading, and which PIDs they use. Since the starting address for each program is based on the PID number (i.e., PROC_START + pid*PROC_SIZE), we can use this to compute where to look for the program in memory. (You don’t need it for the programs your kernel starts at boot: the build reads your boot_programs array from init.c, and GDB loads their symbols by itself.) Here’s how to use it (and, generally, how to run gdb to debug a user program):

Start QEMU with GDB as you normally would (i.e., make qemu-gdb, make qemu-fb-gdb or similar)
In another terminal, start GDB (gdb or gdb-multiarch)
Think about which programs you want to launch, and which PIDs they’ll have. Each program you run will get the next PID in the sequence. For example, at the end of Quest 4, new programs launched will start from PID 2 (because your kernel starts up with idle as PID 0, and a terminal as PID 1). At the end of Quest 5 (CS 1690/2670), vtty and its shell take PIDs 1 and 2, so new programs start from PID 3 (a gterm you start takes one PID, and its shell the next). If in doubt, run ps.
For each program you want to debug, run the following command in the GDB prompt:
```
load_user_symbols <program name> <PID number>
```
… where <program name> and <PID number> are the name of the program you want to run, and the PID number you expect it to have, respectively. For example:
```
load_user_symbols basic 2
```
… would tell GDB expect the code for basic will live in the slot for PID 2. (Note that the process does not need to be running in your kernel when you run this command–you’re just telling GDB the range of addresses where the program will live. From there, GDB will configure its debug information for that program to match.)
Set a breakpoint in the program you want to run (e.g., main in basic.c):
```
b basic.c:main
```
Tell GDB to run your kernel with c:
```
c
```
Switch back to the terminal where you started QEMU. You should now see your serial terminal’s prompt! Next, start your program in your kernel’s terminal, i.e.,
```
run basic
```
Your kernel’s console should now hang–this is because you hit a breakpoint! Switch back to GDB, and you should now see it paused at your breakpoint!

Techniques

Here are some general debugging techniques to consider:

Test the mechanism before the system calls. Before you implement any real system call, make every system call return its own number, and check it in gdb. If yield returns 0 and ticks returns 1, your stubs, your dispatch, and your saved context are right, and every later bug is in a handler.
Print every system call, once. A DEBUG() line in handle_syscall that prints the PID, the system call number, and its first three arguments, and another one that prints the return value, turns most bugs in this project into something you can read. Build with DEBUG_LVL=1 when you need it (e.g., make qemu DEBUG_LVL=1; switching rebuilds everything once); it costs nothing otherwise. (Turn it off before you play Snake: it makes a lot of system calls.)
Kill loudly. Your kernel’s message when it kills a process is your best debugging tool for user programs. Make it include the PID, the program name, the exception class in words, ELR, and, for aborts, FAR.
Check the boundary in both directions. Many bugs in this project are a value that crossed the boundary in the wrong form: a kernel pointer handed to a process, a user pointer used without copy_from_user, a struct pinfo whose two definitions have drifted apart. When something odd crosses the boundary, CHECK() both sides.
Reduce to one process. As always, most bugs reproduce with a single program (plus idle). Add the others back once it works.