Appendix C: Debugging concurrency

Everything in Project 1’s testing and debugging appendix still applies. What follows is the additional toolkit for this project, where bugs are no longer purely about what your code does but also about when.

Five common failure modes

Almost every non-trivial bug in this project falls into one of the following categories. Learning to tell them apart from the symptom saves enormous amounts of time.

Debugging a context switch in gdb

The single most useful thing you can do is look at a context frame directly. It is 34 8-byte values (“words”), so you can read it on one screen.

(gdb) break restore_context
(gdb) continue
(gdb) p/x $x0                # the context pointer
(gdb) x/34gx $x0             # the whole frame: x29, x30, x0..x28, sp_el0, elr, spsr

Reading that dump against the layout table in Quest 2 will tell you almost immediately whether a context is plausible: the last three words should be a stack-ish address (or 0), a code address in a process’s region, and a small value ending in 5 (EL1h). Zeroes or wild values in the middle mean a save/restore offset bug.

Other recipes:

Techniques that work better than a debugger here

• Print, but print carefully. A printf in the middle of your switch path changes the timing of the thing you are debugging, and (until Quest 4 Part E) can itself be preempted. Prefer printing around the interesting region, or record events into a global array and dump it later.
• A ring buffer of events is a classic kernel technique: create a global array of {pid, event, timestamp}, write into it in your context switch path with no formatting or I/O at all, and dump its contents when something goes wrong. This perturbs timing far less than printing as saving data into the array is fast and the formatting only happens when you dump debug output later, and you get a history rather than a snapshot.
• Fail aggressively when impossible things happen. Add panic() calls for the things that must never happen: pick_next finding a RUNNING process, yield returning, an unrecognized interrupt source, a resume on a process with a null context. A panic at the moment an invariant breaks is much better than trying to figure out what went wrong from garbage execution further down the line. The CHECK() macro in kernel/debug.h can help you with this; see Appendix D for how to use it.
• Reduce the problem. Use two processes instead of three, such as hello and counter instead of primecheck. This combination requires cooperative scheduling only, and thus works with the timer disabled. This simpler setting tells you whether general context switch and yield infrastructure or your timer code is at fault.
• Predict the output before you run it. With two cooperatively-scheduled processes, your OS is entirely deterministic. Writing down what you expect and comparing catches misunderstandings that reading code does not.