Okay, the next stage of experiments was trying to boot the int64-patched version with a real ISO. I started with the oldest one I could find on the Internet: the Solaris Nevada Image, snv134, from here. IMO it’s a Holy Grail for QEMU experiments at this stage: this specific version is close to the latest available OpenSolaris code: it’s roughly some months before the first Solaris Express build in 10/2010.
dpim@vmbox:~/build/qemu-sparc64-19a589cc31/build$ ./qemu-system-sparc64 -M niagara -L /home/dpim/images/S10image -nographic -m 256 \
-drive if=pflash,readonly=on,file=/zroot/iso/osol-dev-134-ai-sparc.iso \
-d int,guest_errors -D /home/dpim/build/qemu-sparc64-19a589cc31/mmu-miss-trace.log
cpu Probing I/O buses
Sun Fire T2000, No Keyboard
Copyright 2005 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.20.0, 256 MB memory available, Serial #1122867.
[mo23723 obp4.20.0 #0]
Ethernet address 0:80:3:de:ad:3, Host ID: 80112233.
ok boot -kdv
Boot device: /virtual-devices/disk@0 File and args: -kdv
hsfs-file-system
Loading: /platform/sun4v/boot_archive
ramdisk-root ufs-file-system
Loading: /platform/sun4v/kernel/sparcv9/unix
module /platform/sun4v/kernel/sparcv9/unix: text at [0x1000000, 0x10d9cef] data at 0x1800000
module /platform/sun4v/kernel/sparcv9/genunix: text at [0x10d9cf0, 0x12d4d17] data at 0x18d93c0
module /platform/sun4v/kernel/misc/sparcv9/platmod: text at [0x12d4d18, 0x12d4d1f] data at 0x192f670
module /platform/sun4v/kernel/cpu/sparcv9/SUNW,UltraSPARC-T1: text at [0x12d4d20, 0x12d8e77] data at 0x192fd40
Loading kmdb...
module /platform/sun4v/kernel/misc/sparcv9/kmdbmod: text at [0x12d8e80, 0x13553bf] data at 0x19487c0
module /kernel/misc/sparcv9/ctf: text at [0x13553c0, 0x135cda7] data at 0x1965f30
-
Welcome to kmdb
kmdb: unable to determine terminal type: assuming `vt100'
Loaded modules: [ unix krtld genunix ]
[0]> :c
ERROR: Last Trap: Fast Data Access MMU Miss
[Exception handlers interrupted, please file a bug]
[type 'resume' to attempt a normal recovery]
ok
Okay, this is a early boot trap, but where?
[0]> ::bp panic
[0]> :c
kmdb: stop at panic
kmdb: target stopped at:
panic: save %sp, -0xb0, %sp
[0]> $r
%g0 = 0x0000000000000000 %l0 = 0xffffffffffffffff
%g1 = 0x000000000000002c %l1 = 0x0000000000000045
%g2 = 0x00000000fff34260 %l2 = 0x0000000000000089
%g3 = 0x0000000000000122 %l3 = 0x000000000183cf48 n_mblocks
%g4 = 0x0000000000000001 %l4 = 0x8000000083000740
%g5 = 0x4d61636844657363 %l5 = 0x8000000083110750
%g6 = 0x0000000000000000 %l6 = 0x0000000000000000
%g7 = 0x000000000180e000 t0 %l7 = 0x00000000010d1400
%o0 = 0x00000000010d16e0 %i0 = 0x00000000fff32000
%o1 = 0x000000000180b8a8 t0stack+0xb8a8 %i1 = 0x0000000000000001
%o2 = 0x00000000010d1588 %i2 = 0x00000000fff42000
%o3 = 0x00000000010d1438 %i3 = 0x000000000183cc00
%o4 = 0x000000000180b8a8 t0stack+0xb8a8 %i4 = 0x0000000000000000
%o5 = 0x0000000000000000 %i5 = 0x00000000010d1708
%o6 = 0x000000000180afc1 t0stack+0xafc1 %i6 = 0x000000000180b0b1 t0stack+0xb0b1
%o7 = 0x0000000001048b30 lgrp_traverse+0x520 %i7 = 0x0000000001048b94
plat_lgrp_init+0x14
%ccr = 0x00 xcc=nzvc icc=nzvc
%fprs = 0x00 fef=0 du=0 dl=0
%asi = 0x00
%y = 0x0000000000000000
%pc = 0x0000000001078224 panic
%npc = 0x0000000001078228 panic+4
%sp = 0x000000000180afc1 unbiased=0x000000000180b7c0
%fp = 0x000000000180b0b1
%tick = 0x00000000a376242d
%tba = 0x00000000f0200000
%tt = 0x17e
%tl = 0x0
%pil = 0xd
%pstate = 0x016 cle=0 tle=0 mm=TSO red=0 pef=1 am=0 priv=1 ie=1 ag=0
%cwp = 0x04 %cansave = 0x06
%canrestore = 0x00 %otherwin = 0x00
%wstate = 0x00 %cleanwin = 0x07
[0]> 0x00000000010d16e0/s
0x10d16e0: lgrp_traverse: No memory blocks found
[0]> <o0/s
0x10d16e0: lgrp_traverse: No memory blocks found
It’s points us to OpenSolaris/Illumos mpo.c source code with panic on line 743, and the system are not able to determine the memory blocks.
fail:
if (n_cpunodes > 0)
md_free_scan_dag(md, &cpunodes);
if (n_mblocks > 0)
md_free_scan_dag(md, &mblocknodes);
else
panic("lgrp_traverse: No memory blocks found");
Then, try to look the conditions of the panic:
[0]> :c
kmdb: stop at lgrp_traverse
kmdb: target stopped at:
lgrp_traverse: save %sp, -0xf0, %sp
[0]> lgrp_traverse::dis
lgrp_traverse: save %sp, -0xf0, %sp
lgrp_traverse+4: call -0x360 <md_get_root>
lgrp_traverse+8: mov %i0, %o0
lgrp_traverse+0xc: sethi %hi(0x10d1400), %g1
lgrp_traverse+0x10: sethi %hi(0x183cc00), %i3
lgrp_traverse+0x14: add %g1, 0x308, %i5
lgrp_traverse+0x18: cmp %o0, -0x1
lgrp_traverse+0x1c: bne,pn %xcc, +0x14 <lgrp_traverse+0x30>
lgrp_traverse+0x20: add %i3, 0x348, %l3
lgrp_traverse+0x24: mov -0x1, %l0
lgrp_traverse+0x28: ba +0x4a4 <lgrp_traverse+0x4cc>
lgrp_traverse+0x2c: ld [%l3], %l6
lgrp_traverse+0x30: mov %o0, %i4
lgrp_traverse+0x34: add %fp, 0x7f7, %o4
lgrp_traverse+0x38: add %i5, -0x1c0, %o2
lgrp_traverse+0x3c: add %i5, -0x2d0, %o3
lgrp_traverse+0x40: mov %i0, %o0
lgrp_traverse+0x44: call -0x9d70 <md_alloc_scan_dag>
lgrp_traverse+0x48: mov %i4, %o1
lgrp_traverse+0x4c: cmp %o0, 0x0
lgrp_traverse+0x50: bg,pn %icc, +0x28 <lgrp_traverse+0x78>
lgrp_traverse+0x54: st %o0, [%l3]
lgrp_traverse+0x58: sethi %hi(0x183cc00), %l6
[0]> ::bp md_alloc_scan_dag
[0]> :c
The first call is md_alloc_scan_dag, we set the bp on it,and continue
kmdb: stop at md_alloc_scan_dag
kmdb: target stopped at:
md_alloc_scan_dag: save %sp, -0xb0, %sp
[0]> :z
[0]> $r
%g0 = 0x0000000000000000 %l0 = 0x00000000fff340e0
%g1 = 0x00000000010d1400 %l1 = 0x0000000000000045
%g2 = 0x000000000000000b %l2 = 0x0000000000000089
%g3 = 0x00000000fff34010 %l3 = 0x000000000183cf48 n_mblocks
%g4 = 0x000000000000000b %l4 = 0x8000000083000740
%g5 = 0x4d61636844657363 %l5 = 0x8000000083110750
%g6 = 0x0000000000000000 %l6 = 0x00000000fff34010
%g7 = 0x000000000180e000 t0 %l7 = 0x000000000183cb18 startup_memops
%o0 = 0x00000000fff32000 %i0 = 0x00000000fff32000
%o1 = 0x0000000000000000 %i1 = 0x0000000000000001
%o2 = 0x00000000010d1548 %i2 = 0x00000000fff42000
%o3 = 0x00000000010d1438 %i3 = 0x000000000183cc00
%o4 = 0x000000000180b8a8 t0stack+0xb8a8 %i4 = 0x0000000000000000
%o5 = 0x00000000000001b0 %i5 = 0x00000000010d1708
%o6 = 0x000000000180afc1 t0stack+0xafc1 %i6 = 0x000000000180b0b1 t0stack+0xb0b1
%o7 = 0x0000000001048654 lgrp_traverse+0x44 %i7 = 0x0000000001048b94
plat_lgrp_init+0x14
%ccr = 0x00 xcc=nzvc icc=nzvc
%fprs = 0x00 fef=0 du=0 dl=0
%asi = 0x00
%y = 0x0000000000000000
then, we looking %o2 and %o3 registers, they contain the values after md_alloc_scan_dag call execution:
[0]> 0x00000000fff32000/s
[0]> 0x00000000010d1548/s
0x10d1548: mblock
[0]> 0x00000000010d1438/s
0x10d1438: fwd
[0]> n_mblocks/X
n_mblocks:
n_mblocks: 0
So, we have a proof: “n_mblocks” is zeroed here, The call returns “mblock” and “fwd” before the crash through md_alloc_scan_dag call.
But what is all this for? What is ‘md’, what are ‘mblock’ and ‘fwd’, and how do all of these interact at the earliest level of boot?
Hopefully, Sun published some documents at this time that can help determine the sequence and behavior — all of this is described in FWARC 2005/115.
The idea differs from the x86 world, where things mostly operate through parsing the ACPI tables
In case of sun4v world all hosts are de-facto are the guests controled by tiny firmware Type-1 hypervisor : and each machine (any term - had a pre-described firmware part named a Machine Description:
A machine description (“MD”) contains both explicit information about resources within a machine - detailed by specific nodes within the MD, and implicit information about the relationship of those resources - detailed by how nodes are interconnected into a relationship graph
and below in 3.1 Graph Contents:
The intrinsic Machine Description (MD) is a directed acyclic graph (DAG) of nodes describing resources or information available within a machine. This information is provided upon request to a guest operating system via the machine description request API
okay, we know what is the DAG, but how elements are connect to each other?
The default DAG described within the MD is defined by arcs (element type PROP_ARC) with a name of “fwd”. For convenience in walking this DAG. Arcs named “back” are also provided that define the inverse DAG. Thus for every node A that has a “fwd” arc pointing to another node B, there is a corresponding “back” arc for node B pointing back to A.
It’s described the relationship between two nodes through “fwd” directive:
So we should have DAG very similar to:
And it seems we have a “memory” -> “fwd” -> “mblock” request with an unconditional answer (‘0’), and a panic — and it’s proven by the QEMU MD/’firmware’ part, using the mdlint tool (described in Part 2):
~/images/S10image$ ./mdlint -t 1up-md.bin | egrep "fwd|node_0x1c|node_0x25"
fwd -> node_0x1c;
fwd -> node_0x25;
node platform node_0x1c { /* next @ 0x25 */
node virtual-devices node_0x25 { /* next @ 0x28 */
back -> node_0x25;
back -> node_0x25;
back -> node_0x25;
back -> node_0x25;
and we have “mblock0” definition but no “memory” node and no any “fwd”/”back” directive for both of it
dpim@vmbox:~/images/S10image$ ./mdlint -t 1up-md.bin | egrep memory
dpim@vmbox:~/images/S10image$ ./mdlint -t 1up-md.bin | egrep mblock
node mblock node_0x53 { /* next @ 0x57 */