After enabling the QEMU 256MB pagesize for sparc64, we have some progress on MD configurations with more than 1024MB of memory. Solaris 11 boots, but in contrast to Solaris 10, it is extremely slow: init on 11.1 has to bring up ~260 modules (::modinfo), which takes hours of boot time. The whole saga is covered in the previous two posts.
a loose pile of links, notes, and things worth keeping.
feed
Continuing from Part 1.
Let’s look at what this looks like in practice, and where QEMU’s handling of it actually falls over.
The latest good breakpoint was alloc_page_freelists+0x54 (I spent some time catching it):
[0]> alloc_page_freelists+0x54::bp
[0]> :c
kmdb: stop at alloc_page_freelists+0x54
kmdb: target stopped at:
alloc_page_freelists+0x54: stx %i0, [%l6 + %l7]
[0]> mmu_exported_pagesize_mask/X
mmu_exported_pagesize_mask:
mmu_exported_pagesize_mask: 2b
[0]> kmem64_szc/X
kmem64_szc:
kmem64_szc: 5 - <this is 256M page for this specific allocation >
[0]> kmem64_base/J
kmem64_base:
kmem64_base: 70000000000
[0]> kmem64_aligned_end/J
kmem64_aligned_end:
kmem64_aligned_end: 70010000000
[0]> 70000000000/J
0x70000000000: 0
[0]> 7000000F000/J
0x7000000f000: 0
[0]> 7000000FF00/J
0x7000000ff00: 0
[0]> 7000000FFF0/J
0x7000000fff0: 0
[0]> 7000000FFFE/J -<hang where we try to look the code out of 64K>
Here, kmem64_szc is 5, which corresponds directly to TTE256M from the table in Part 1 - confirming that the kernel is indeed trying to map the kmem64 segment using 256M pages, driven by the broken_md_flag logic, but it can’t do that.
The hang happens exactly at the boundary where we step outside the 64K range - right where QEMU would need to service a translation for the 256M-sized mapping. This lines up with the suspicion from Part 1: QEMU’s MMU emulation may not correctly handle the 256M page size that the guest kernel enables as a side effect of the missing domaining-enabled property. The guest isn’t doing anything unusual - it’s following the exact logic in niagara.c - but QEMU’s handling of that particular [sz] encoding looks like the likely point of failure, producing the endless stream of Hyperprivileged Trap Instruction / Data Access MMU Miss entries seen in the log at the start of this investigation. This remains a suspicion rather than a confirmed root cause.
When we tried to suppress the pagesize to 0x9 (8K+4M), like so:
[0]> ::bp alloc_kmem64
[0]> :c
[0]> mmu_exported_pagesize_mask/W b
the kernel booted successfully, but without 256MB pages.
I then started looking at the QEMU code. The TTE page-size field is defined in QEMU’s /target/sparc/cpu.h as:
#define TTE_PGSIZE(tte) (((tte) >> 61) & 3ULL)
and the sun4v_tte_to_sun4u() conversion inside QEMU’s target/sparc/ldst_helper.c only preserves 2 bits for the page size:
sun4u_tte |= (sun4v_tte & 3ULL) << 61; /* TTE_PGSIZE */
It seems the logic only preserves two bits (61/62), giving a value of 0-3, when the highest bit - bit 48 (szh) - is also supposed to be part of it, which makes the operation a bit more involved: we need to reconstruct it as
| szh[48] | szl[62] | szl[61] | |—|—|—|
to build the table described below in the UltraSPARC T1 Supplement draft:
to hold all the properties in our case.
Interestingly, when I tried to figure out when this extension was introduced into the CPU spec, I found the Panther/USIV CPU listed as the first to support it, then T1 supported the same TTE pagesize bits, but T2 changed the format:
| Document | Where it lands |
|---|---|
| JPS1 Commonality, Working Draft 1.0.5, TABLE F-1, p.441 | The state before the extension: 2-bit size, Data<49:47> “Reserved, read as 0” (bit 48 included). Joint Sun+Fujitsu, so it covers US-III and SPARC64 V. |
| Panther Impl. Supplement (UltraSPARC IV+), Preliminary Draft 17 Mar 2006, TABLE F-1-4, p.337 | The citation to use. States plainly: “Bit 48 is the most significant bit of the page size and is concatenated with bits <62:61>”, with the full 000-101 encoding. |
| UltraSPARC T1 Supplement to the UltraSPARC Architecture 2005, Draft D2.1, sec. 13.1.2, TABLE 13-1, p.182 | Same bit in T1’s sun4u-format TTE, named szl/szh. |
| UltraSPARC T2 Supplement, TABLE 12-4, p.107 | T2 dropped the sun4u format entirely: one contiguous 4-bit field, and only 4 sizes (512K and 32M Reserved). |
When I started prototyping the fix with Claude, it took five iterations to land on this patch and send it to the qemu-devel community. Each of the first four had something wrong: they used the wrong bitfields for storing the [szh] bit - the first time we built it on unused bit 42, which is out of UltraSparc spec :), and it even worked, sort of; or used suboptimal extra operations to store this bit in the TTE. All of these worked, but…
With it applied, the guest started to boot:
[0]> alloc_page_freelists+0x54::bp
[0]> :c
kmdb: stop at alloc_page_freelists+0x54
kmdb: target stopped at:
alloc_page_freelists+0x54: stx %i0, [%l6 + %l7]
[0]> kmem64_szc/X
kmem64_szc:
kmem64_szc: 5
[0]> kmem64_base/J
kmem64_base:
kmem64_base: 70000000000
[0]> kmem64_aligned_end/J
kmem64_aligned_end:
kmem64_aligned_end: 70010000000
[0]> mmu_exported_pagesize_mask/X
mmu_exported_pagesize_mask:
mmu_exported_pagesize_mask: 2b
[0]> 7000000FFFE/J <- it's survived!
0x7000000fffe: 0
[0]> 70000010000/J
0x70000010000: 0
[0]> 70000100000/J <- also survived
0x70000100000: 0
[0]> :c
kmdb: stop at alloc_page_freelists+0x54
kmdb: target stopped at:
alloc_page_freelists+0x54: stx %i0, [%l6 + %l7] - <then it stopped the second time
[0]> 70000100000/J
0x70000100000: 0
[0]> mmu_exported_pagesize_mask/X
mmu_exported_pagesize_mask:
mmu_exported_pagesize_mask: 2b
[0]> :c
SunOS Release 5.11 Version snv_151a 64-bit
Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
And here’s how it looks under the QEMU monitor: you can see 256MB chunk/page!
(qemu) info tlb
(qemu) info tlb
MMU contexts: Primary: 0, Secondary: 0
DMMU Tag Access: 180a000, TSB Tag Target: 0
DMMU dump
[00] VA: 70000000000, PA: d0000000, 256M, priv, RW, unlocked, ie no, ctx 0 local <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
[01] VA: ffffffffffffc000, PA: 801ee000, 8k, priv, RW, unlocked, ie no, ctx 0 local
[02] VA: ffffffffff300000, PA: f1f0c000, 8k, priv, RW, unlocked, ie no, ctx 0 local
[03] VA: fffffffcc000a000, PA: f1394000, 8k, priv, RW, unlocked, ie no, ctx 0 local
[04] VA: 1930000, PA: ff530000, 8k, priv, RW, unlocked, ie no, ctx 0 local
One of the more interesting and puzzling things I ran into was a suspicious hang on any configuration with 2048MB or more of memory under QEMU and Machine Description (MD) configuration. It was a glass wall (matrix here) I couldn’t break through for any Solaris 10 release higher than S10u1.
The symptoms were odd: it starts to boot and then completely hangs, without any further response to kmdb.
QEMU itself generates tons of logs - traps between hypervisor and MMU, like this:
58552984: Hyperprivileged Trap Instruction (v=0183)
58552985: Data Access MMU Miss (v=0068)
58552986: Hyperprivileged Trap Instruction (v=0183)
58552987: Data Access MMU Miss (v=0068)
58552988: Hyperprivileged Trap Instruction (v=0183)
58552989: Data Access MMU Miss (v=0068)
This made me think something was wrong with the software MMU code execution inside QEMU - but I had no idea what.
Moreover, it looked like a memory pagesize problem: the guest OS trying to reach a pagesize bigger than the host supports. This was a suspicion, not proven at the time: for decades, every new paging feature introduced in silicon was a bit of “fanfare” inside the Solaris community, and I suspect we have some limitations in our software emulation/QEMU also.
Let’s start from the guest/Solaris side with a bit of theory (i’m sorry):
A parameter named “mmu-pagesize-list” is read from the MD at a predetermined point before boot, and it uses the standard bitmask supported by the 3-bit [sz] parameter on the TSB Translation Table Entry (TTE):
This is described cleanly in UltraSPARC Architecture 2005, page 362 - though there it’s actually a 4-bit field:
Data – 3:0 sz The page size of this entry, encoded as shown below.
sz Page Size
0000 8 Kbyte
0001 64 Kbyte
0010 Reserved
0011 4 Mbyte
0100 Reserved
0101 256 Mbyte
0110 Reserved
0111 Reserved
1000-1111 Reserved
This converts to the following code table for T1:
| code | size | bit value |
|---|---|---|
| 0 | 8K | 0x01 |
| 1 | 64K | 0x02 |
| 2 | 512K | 0x04 |
| 3 | 4M | 0x08 |
| 4 | 32M | 0x10 |
| 5 | 256M | 0x20 |
So, to enable the standard support for 8K+64K+4M, we should have 0xb at this stage; the MD should be 0x9 without 64K support (and even that works!), and 0xb also seems valid with 64K pages enabled.
But in practice, the picture was the opposite:
The kernel mystically overrides the default values from 0x9 (or even 0xb) and silently changes them to 0x2b (+256M support) at an early stage of booting:
[0]> mmu_exported_pagesize_mask/X
mmu_exported_pagesize_mask:
mmu_exported_pagesize_mask: 2b
Why does the kernel insist on “0x2b” instead of “0xb”, which is recorded cleanly in our “firmware” / MD property?
I started tracing through Solaris and found a number of interesting things. Seems in OpenSolaris/Solaris there are two branches of control selected by “broken_md_flag”, set when the domaining-enabled parameter (p.62) is absent from the MD. “Domaining” refers to the host acting as the primary domain (of course, under the hypervisor as well - any sun4v system runs under the control of the hypervisor code from the sun4v firmware; this is the “q.bin” file underneath the Sun Legion/QEMU part).
The absence of the “domaining-enabled” property raises the broken_md_flag here:
[0]> broken_md_flag/D
broken_md_flag:
broken_md_flag: 1
And it comes from os/fillsysinfo.c in the OpenSolaris/Illumos code:
init_md_broken(md_t *mdp, mde_cookie_t *cpulist)
{
int nrnode;
mde_cookie_t *platlist, rootnode;
uint64_t val = 0;
char *namebuf;
int namelen;
rootnode = md_root_node(mdp);
ASSERT(rootnode != MDE_INVAL_ELEM_COOKIE);
ASSERT(cpulist);
nrnode = md_alloc_scan_dag(mdp, rootnode, "platform", "fwd",
&platlist);
if (nrnode < 1)
cmn_err(CE_PANIC, "init_md_broken: platform node missing");
if (md_get_prop_data(mdp, cpulist[0],
"compatible", (uint8_t **)&namebuf, &namelen)) {
cmn_err(CE_PANIC, "init_md_broken: "
"Cannot read 'compatible' property of 'cpu' node");
}
if (md_get_prop_val(mdp, platlist[0],
"domaining-enabled", &val) == -1 &&
strcmp(namebuf, "SUNW,UltraSPARC-T1") == 0)
broken_md_flag = 1;
md_free_scan_dag(mdp, &platlist);
}
[0]> :c
kmdb: stop at init_md_broken+0x88
[0]> <o0/s
0xfff34ac0: SUNW,UltraSPARC-T1
[0]>
(This is beside the point, but if we try to enable the “domaining-enabled” property, the kernel starts but panics in the CPU platform module when it tries to configure some cache/TLB/MMU properties extracted from the MD - i.e., it behaves as if it were a real control LDOM. It’s a good starting point for the next stage of MD structure research, to open up a new level of possibilities, IMO.)
So, since we don’t have the domaining-enabled property in the MD’s cpu node, as expected, broken_md_flag gets set:
cpu0: UltraSPARC-T1 (chipid 0, clock 5 MHz)
Loaded modules: [ scsi_vhci mac neti ufs s1394 hook ip usba specfs sctp arp
sockfs ]
kmdb: target stopped at:
kmdb_enter: ta %icc, 0x7d
[0]> broken_md_flag/D
broken_md_flag:
broken_md_flag: 1
[0]>
This triggers the setup of a specific memory pagesize mask in /usr/src/uts/sun4v/cpu/niagara.c:
if (broken_md_flag) {
/*
* Turn on the missing bits supported by Niagara CPU in
* MMU pagesize mask returned by MD.
*/
mmu_exported_pagesize_mask |= NI_MMU_PAGESIZE_MASK;
} else {
if ((mmu_exported_pagesize_mask &
DEFAULT_SUN4V_MMU_PAGESIZE_MASK) !=
DEFAULT_SUN4V_MMU_PAGESIZE_MASK)
cmn_err(CE_PANIC, "machine description"
" does not have required sun4v page sizes"
" 8K, 64K and 4M: MD mask is 0x%x",
mmu_exported_pagesize_mask);
So, when broken_md_flag=1, the MMU pagesizes come from NI_MMU_PAGESIZE_MASK:
#define NI_MMU_PAGESIZE_MASK ((1 << TTE8K) | (1 << TTE64K) | (1 << TTE4M) \
| (1 << TTE256M))
otherwise, the predefined parameters above are used - the default CPU pagesize mask with 8K/64K/4M, from /usr/src/uts/sun4v/sys/cpu_module.h (not our case):
#define DEFAULT_SUN4V_MMU_PAGESIZE_MASK ((1 << TTE8K) | (1 << TTE64K) \
| (1 << TTE4M))
(it seems the control domain didn’t use 256M at this point in Solaris Express 10/2011)
and the bitmask pagesize flags come from /usr/src/uts/sun4v/sys/pte.h:
/* Defines for sz field in tte */
#define TTE8K 0x0
#define TTE64K 0x1
#define TTE512K 0x2
#define TTE4M 0x3
#define TTE32M 0x4
#define TTE256M 0x5
#define TTE2G 0x6
#define TTE16G 0x7
NI_MMU_PAGESIZE_MASK = (1<<0)|(1<<1)|(1<<3)|(1<<5) = { 8K, 64K, 4M, 256M } = binary 0010 1011 = 0x2b
DEFAULT_SUN4V_MMU_PAGESIZE_MASK = (1<<0)|(1<<1)|(1<<3) = {8K, 64K, 4M } = 0000 1011 = 0xb
i.e., 00101011 -> 0x2b (256M enabled) vs. 00001011 -> 0xb for the basic value (8K+64K+4M).
It seems our host is trying to enable 256M pages. In other words, here’s the interesting logic: on a domaining-capable host (i.e., one that controls the hypervisor), we support 8K/64K/4M paging; but when “broken_md_flag” is set, we also get 256M support on T1. Note that 32M pages aren’t implemented in T1 silicon at all, as described in the niagara.c logic above.
Part 2 continues with what this looks like in practice, and what the QEMU side reveals.
Came out with a booting text installer across all three Solaris 10 releases and a clean Solaris 11.0 11/11 kernel init.
Proof: Solaris 10u10 Solaris 10u11 Solaris Express 11/10 Solaris 11.0
Whole progress/matrix list
Came out with a booting text installer across all three Solaris 10 releases and a clean Solaris 11.0 11/11 kernel init.
That led me to the next candidate for review after snv_134: snv_151, aka Solaris Express 11/2010, built a year before the official 11.0 release:
[0]> ::msgbuf
MESSAGE
SunOS Release 5.11 Version snv_151a 64-bit
Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
panic[cpu0]/thread=180e000:
cpu0: failed to get MMU CTX domain index
Warning - stack not written to the dumpbuf
After walking through sfmmu_cpu_init -> plat_cpuid_to_mmu_ctx_info -> load_tlb_cpu_mappings and md_alloc_scan_dag system wants the fwd and tlb entries it ended by:
[0]> 0x10d1230/s
0x10d1230: tlb
[0]> 0x10d0ae0/s
0x10d0ae0: fwd
it’s similar to the previous:
--- a/common.pdesc
+++ b/common.pdesc
@@ -38,6 +38,7 @@
mmu-#va-bits = 0x40; \
back -> cpus; \
fwd -> cache; \
+ fwd -> tlb; \
}
node options XXoptions {
--- a/1up.pdesc
+++ b/1up.pdesc
@@ -49,3 +49,7 @@
node cache cache {
back -> cpu0;
}
+
+node tlb tlb {
+ back -> cpu0;
+}
After bumping the RAM from 256 MB to 1024 MB in the MD, OpenSolaris build 134 boots successfully on QEMU SPARC64 with sun4v emulation. And Solaris 10u8 reaches the install screen!
Proof:
snv_134 s10u8
--- a/1up.pdesc
+++ b/1up.pdesc
@@ -33,7 +33,7 @@
node mblock p0_memlist0 {
base = 0x0000000080000000;
- size = 256M;
+ size = 1024M;
}
node memory memory {
--- a/1up.hdesc
+++ b/1up.hdesc
@@ -32,7 +32,7 @@
#include "common.hdesc"
CPU(0,guest0,0)
-GUEST(0, 0x1, 0x80000000, 256M, 0x80000000, 0x1f12000000, 0x1f40000000)
+GUEST(0, 0x1, 0x80000000, 1024M, 0x80000000, 0x1f12000000, 0x1f40000000)
node cpus cpus {
cpu -> cpu0
snv_134 successfully booted:
Result: PASS
Output:
cpu Probing I/O buses
Sun Fire T2000, No Keyboard
Copyright 2005 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.20.0, 1024 MB memory available, Serial #1122867.
[mo23723 obp4.20.0 #0]
Ethernet address 0:80:3:de:ad:3, Host ID: 80112233.
ok boot -kdv
Boot device: /virtual-devices/disk@0 File and args: -kdv
hsfs-file-system
Loading: /platform/sun4v/boot_archive
ramdisk-root ufs-file-system
Loading: /platform/sun4v/kernel/sparcv9/unix
module /platform/sun4v/kernel/sparcv9/unix: text at [0x1000000, 0x10d9cef] data at 0x1800000
module /platform/sun4v/kernel/sparcv9/genunix: text at [0x10d9cf0, 0x12d4d17] data at 0x18d93c0
module /platform/sun4v/kernel/misc/sparcv9/platmod: text at [0x12d4d18, 0x12d4d1f] data at 0x192f670
module /platform/sun4v/kernel/cpu/sparcv9/SUNW,UltraSPARC-T1: text at [0x12d4d20, 0x12d8e77] data at 0x192fd40
Loading kmdb...
module /platform/sun4v/kernel/misc/sparcv9/kmdbmod: text at [0x12d8e80, 0x13553bf] data at 0x19487c0
module /kernel/misc/sparcv9/ctf: text at [0x13553c0, 0x135cda7] data at 0x1965f30
-
Welcome to kmdb
kmdb: unable to determine terminal type: assuming `vt100'
Loaded modules: [ unix krtld genunix ]
[0]> :c
SunOS Release 5.11 Version snv_134 64-bit
Copyright 1983-2010 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
os-io Ethernet address = 0:80:3:de:ad:3
Using default device instance data
mem = 1048576K (0x40000000)
avail mem = 696803328
root nexus = Sun Fire T2000
pseudo0 at root
pseudo0 is /pseudo
scsi_vhci0 at root
scsi_vhci0 is /scsi_vhci
NOTICE: Disabling watchdog as watchdog services are not available
ramdisk0 at root
ramdisk0 is /ramdisk-root
root on /ramdisk-root:a fstype ufs
pseudo-device: dld0
dld0 is /pseudo/dld@0
cpu0: UltraSPARC-T1 (chipid 0, clock 5 MHz)
pseudo-device: audio0
audio0 is /pseudo/audio@0
pseudo-device: devinfo0
devinfo0 is /pseudo/devinfo@0
iscsi0 at root
iscsi0 is /iscsi
pseudo-device: pseudo1
pseudo1 is /pseudo/zconsnex@1
pseudo-device: drctl0
drctl0 is /pseudo/drctl@0
pseudo-device: zfs0
zfs0 is /pseudo/zfs@0
pseudo-device: fcode0
fcode0 is /pseudo/fcode@0
pseudo-device: llc10
llc10 is /pseudo/llc1@0
pseudo-device: lofi0
lofi0 is /pseudo/lofi@0
pseudo-device: ramdisk1024
ramdisk1024 is /pseudo/ramdisk@1024
pseudo-device: mdesc0
mdesc0 is /pseudo/mdesc@0
WARNING: kernel debugger was booted; application watchdog is not available.
pseudo-device: trapstat0
trapstat0 is /pseudo/trapstat@0
pseudo-device: dtrace0
dtrace0 is /pseudo/dtrace@0
pseudo-device: fbt0
fbt0 is /pseudo/fbt@0
pseudo-device: dcpc0
dcpc0 is /pseudo/dcpc@0
pseudo-device: lockstat0
lockstat0 is /pseudo/lockstat@0
pseudo-device: profile0
profile0 is /pseudo/profile@0
pseudo-device: fasttrap0
fasttrap0 is /pseudo/fasttrap@0
pseudo-device: systrace0
systrace0 is /pseudo/systrace@0
pseudo-device: sdt0
sdt0 is /pseudo/sdt@0
pseudo-device: fcp0
fcp0 is /pseudo/fcp@0
pseudo-device: fcsm0
fcsm0 is /pseudo/fcsm@0
Hostname: opensolaris
Remounting root read/write
Probing for device nodes ...
Preparing automated install image for use
Requesting System Maintenance Mode
(See /lib/svc/share/README for more information.)
Console login service(s) cannot run
Enter user name for system maintenance (control-d to bypass): root
Enter root password (control-d to bypass):
single-user privilege assigned to root on /dev/console.
Entering System Maintenance Mode
Jul 16 07:30:51 su: 'su root' succeeded for root on /dev/console
-bash: /usr/sbin/quota: No such file or directory
Sun Microsystems Inc. SunOS 5.11 snv_134 February 2010
-bash: /bin/mail: No such file or directory
-bash: /usr/bin/hostname: No such file or directory
root@:~#
So, in Part 1 we decided that the problem comes from a wrongly formed Machine Description (MD) file: the mblock doesn’t have an ‘fwd’ directive to the root node.
And we need to patch the blob, or find a way to change all of this the regular way.
The original OpenSPARC T1 archive from 2006 contains the source code/toolchain for building it, named mdgen and mdlint; it’s located under the hypervisor/src/md directory inside the tar.bz2 file.
Also, legion/src/config/niagara contains the machine description blobs (and what’s important here: one is specially used by the QEMU uniprocessor 1up.*), and the proper conf files used for these blob generations. When we build it using the specific configuration, at the end we can get a 1up-md/1up-hv bin pair which is identical to the one shipped with the original T1 OpenSPARC archive from 2006 (and used by QEMU). The toolchain with the generated files located here it contain some byteswap modifications for building on Linux
Each pair is built from the lexical directives based on four files: two of them describe hypervisor properties (hdesc), and the other two describe the machine description (pdesc). Two of these are common — literally common — for the whole architecture, while the other two describe the specific system (i.e., 1up (uniprocessor) or 32-thread 1g32p).
so, we use pair
- 1up.hdesc/pdesc
- common.hdesc/pdesc
These two files are loaded via hw/sparc64/niagara.c in QEMU code:
#define NIAGARA_MD_ROM_BASE 0x1f12000000ULL
#define NIAGARA_MD_ROM_SIZE 0x2000
#define NIAGARA_HV_ROM_BASE 0x1f12080000ULL
#define NIAGARA_HV_ROM_SIZE 0x2000i
Now, let’s focus on the ‘lost’ memory in the MD, and let’s look at common.pdesc:
node root root {
<...>
max-#tsb-entries = 0x2;
reset-reason = 0; // power-on
fwd -> platform_data;
fwd -> vdev;
}
node platform platform_data {
We have a fwd directive for platform_data and vdev, but not for memory here But per 5.5.4 Memory node config on (page 7)[https://sun4v.github.io/ARChive/FWARC/2005/115/final.materials/md-bindings.pdf] At the same time we have pre-defined a mblock0 in the configuration:
node mblock p0_memlist0 {
base = 0x0000000080000000;
size = 256M;
}
The mblock directive is literally ‘hanged in the air,’ without any real references to any other pdesc/hdesc file(s). Let’s try to describe the memory node as the parent for the p0_memlist0 mblock node.
node mblock p0_memlist0 {
base = 0x0000000080000000;
size = 256M;
}
+node memory memory {
+ fwd -> p0_memlist0;
+ back -> root;
+}
And push the fwd directive for the memory node from root, as it is required by root per md binding
node root root {
<...>
fwd -> platform_data;
fwd -> vdev;
+ fwd -> memory;
}
We built the blob binaries, and we passed the mblock allocation in them!
Loaded modules: [ unix krtld genunix ]
[0]> ::bp md_alloc_scan_dag
[0]> ::bp panic
[0]> :c
kmdb: stop at md_alloc_scan_dag
kmdb: target stopped at:
md_alloc_scan_dag: save %sp, -0xb0, %sp
[0]> <o2/s
0x10d1548: mblock
[0]> n_mblocks/X
n_mblocks:
n_mblocks: 0
[0]> :c
kmdb: stop at md_alloc_scan_dag
kmdb: target stopped at:
md_alloc_scan_dag: save %sp, -0xb0, %sp
[0]> n_mblocks/X
n_mblocks:
n_mblocks: 1
… and then immidiately failing on cpu node
[0]> <o2/s
0x10ce3a8: cpu
[0]> :c
ERROR: Last Trap: Fast Data Access MMU Miss
So, we need to add the CPU node also, i.e:
add to common.pdesc
node cpus cpus {
fwd -> cpu0;
back -> root;
}
and adding the fwd directive after memory node too:
{
fwd -> vdev;
fwd -> memory;
+ fwd -> cpus;
}
Next, it passed the CPU node properties, and it fails on the ‘platform’ node.
md_alloc_scan_dag: save %sp, -0xb0, %sp
[0]> <o2/s
0x10ce848: platform
[0]> :c
ERROR: Last Trap: Fast Data Access MMU Miss
Next thing, we walk to the platform and set up an md_get_prop_val breakpoint, because we need to know the specific variable that leads to a panic in md_alloc_scan_dag:
[0]> <o2/s
0x10ce848: platform
[0]> ::bp md_alloc_scan_dag
[0]> ::bp md_get_prop_val
[0]> :c
kmdb: stop at md_get_prop_val
kmdb: target stopped at:
md_get_prop_val:save %sp, -0xb0, %sp
[0]> <o2/s
0x10ce9b0: domaining-enabled
[0]> :c
kmdb: stop at md_get_prop_val
kmdb: target stopped at:
md_get_prop_val:save %sp, -0xb0, %sp
[0]> <o2/s
0x10ce580: mmu-page-size-list
[0]>
it’s points us to usr/src/uts/sun4v/os/fillsysinfo.c
static uint64_t
get_cpu_pagesizes(md_t *mdp, mde_cookie_t cpu_node_cookie)
{
uint64_t mmu_page_size_list;
if (md_get_prop_val(mdp, cpu_node_cookie, "mmu-page-size-list",
&mmu_page_size_list))
mmu_page_size_list = 0;
if (mmu_page_size_list == 0 || mmu_page_size_list > MAX_PAGESIZE_MASK)
cmn_err(CE_PANIC, "Incorrect 0x%lx pagesize mask returned"
"by MD", mmu_page_size_list);
return (mmu_page_size_list);
}
In a similar way, mmu-#va-bits was added to the node (after a bit of walking through breakpoints) — the routine cyclically walks across memory/cpu/platform/cache/exec_unit, and then walks around different values across md_get_prop_val:
on common.pdesc - both parameters is cpu node properties
+ mmu-page-size-list = 0x9; \
+ mmu-#va-bits = 0x40; \
In between these two mmu- properties, due to the platform node’s unavailability, we had a cache node failure the same way as with mnode and cpu, and we need to declare it properly
at the end we have very simple directives for md nodes in 1up.conf:
--- a/1up.pdesc
+++ b/1up.pdesc
@@ -35,3 +35,17 @@
base = 0x0000000080000000;
size = 256M;
}
+
+node memory memory {
+ fwd -> p0_memlist0;
+ back -> root;
+}
+
+node cpus cpus {
+ fwd -> cpu0;
+ back -> root;
+}
+
+node cache cache {
+ back -> cpu0;
+}
and folliwing fwd/back directives +mmu-# bits on common.pdesc
--- a/common.pdesc
+++ b/common.pdesc
@@ -34,9 +34,12 @@
isalist = "sparcv9","sparcv8plus","sparcv8","sparcv8-fsmuld","sparcv7","sparc"; \
compatible = { "SUNW,UltraSPARC-T1", "SUNW,sun4v-cpu", "sun4v" }; \
max-#tsb-entries = 0x2; \
+ mmu-page-size-list = 0x9; \
+ mmu-#va-bits = 0x40; \
+ back -> cpus; \
+ fwd -> cache; \
}
-
node options XXoptions {
input-device = "ttya";
output-device = "ttya";
@@ -66,6 +69,8 @@
reset-reason = 0; // power-on
fwd -> platform_data;
fwd -> vdev;
+ fwd -> memory;
+ fwd -> cpus;
}
Then, snv_134 kernel initialized properly….
And “normally” paniced due the lack of RAM:
Loaded modules: [ unix krtld genunix ]
[0]> :c
panic - kernel: bop_alloc_chunk failed
Program terminated
at the end we have very simple directives for md nodes in 1up.conf:
Okay, the next stage of experiments was trying to boot the int64-patched version with a real ISO. I started with the oldest one I could find on the Internet: the Solaris Nevada Image, snv134, from here. IMO it’s a Holy Grail for QEMU experiments at this stage: this specific version is close to the latest available OpenSolaris code: it’s roughly some months before the first Solaris Express build in 10/2010.
dpim@vmbox:~/build/qemu-sparc64-19a589cc31/build$ ./qemu-system-sparc64 -M niagara -L /home/dpim/images/S10image -nographic -m 256 \
-drive if=pflash,readonly=on,file=/zroot/iso/osol-dev-134-ai-sparc.iso \
-d int,guest_errors -D /home/dpim/build/qemu-sparc64-19a589cc31/mmu-miss-trace.log
cpu Probing I/O buses
Sun Fire T2000, No Keyboard
Copyright 2005 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.20.0, 256 MB memory available, Serial #1122867.
[mo23723 obp4.20.0 #0]
Ethernet address 0:80:3:de:ad:3, Host ID: 80112233.
ok boot -kdv
Boot device: /virtual-devices/disk@0 File and args: -kdv
hsfs-file-system
Loading: /platform/sun4v/boot_archive
ramdisk-root ufs-file-system
Loading: /platform/sun4v/kernel/sparcv9/unix
module /platform/sun4v/kernel/sparcv9/unix: text at [0x1000000, 0x10d9cef] data at 0x1800000
module /platform/sun4v/kernel/sparcv9/genunix: text at [0x10d9cf0, 0x12d4d17] data at 0x18d93c0
module /platform/sun4v/kernel/misc/sparcv9/platmod: text at [0x12d4d18, 0x12d4d1f] data at 0x192f670
module /platform/sun4v/kernel/cpu/sparcv9/SUNW,UltraSPARC-T1: text at [0x12d4d20, 0x12d8e77] data at 0x192fd40
Loading kmdb...
module /platform/sun4v/kernel/misc/sparcv9/kmdbmod: text at [0x12d8e80, 0x13553bf] data at 0x19487c0
module /kernel/misc/sparcv9/ctf: text at [0x13553c0, 0x135cda7] data at 0x1965f30
-
Welcome to kmdb
kmdb: unable to determine terminal type: assuming `vt100'
Loaded modules: [ unix krtld genunix ]
[0]> :c
ERROR: Last Trap: Fast Data Access MMU Miss
[Exception handlers interrupted, please file a bug]
[type 'resume' to attempt a normal recovery]
ok
Okay, this is a early boot trap, but where?
[0]> ::bp panic
[0]> :c
kmdb: stop at panic
kmdb: target stopped at:
panic: save %sp, -0xb0, %sp
[0]> $r
%g0 = 0x0000000000000000 %l0 = 0xffffffffffffffff
%g1 = 0x000000000000002c %l1 = 0x0000000000000045
%g2 = 0x00000000fff34260 %l2 = 0x0000000000000089
%g3 = 0x0000000000000122 %l3 = 0x000000000183cf48 n_mblocks
%g4 = 0x0000000000000001 %l4 = 0x8000000083000740
%g5 = 0x4d61636844657363 %l5 = 0x8000000083110750
%g6 = 0x0000000000000000 %l6 = 0x0000000000000000
%g7 = 0x000000000180e000 t0 %l7 = 0x00000000010d1400
%o0 = 0x00000000010d16e0 %i0 = 0x00000000fff32000
%o1 = 0x000000000180b8a8 t0stack+0xb8a8 %i1 = 0x0000000000000001
%o2 = 0x00000000010d1588 %i2 = 0x00000000fff42000
%o3 = 0x00000000010d1438 %i3 = 0x000000000183cc00
%o4 = 0x000000000180b8a8 t0stack+0xb8a8 %i4 = 0x0000000000000000
%o5 = 0x0000000000000000 %i5 = 0x00000000010d1708
%o6 = 0x000000000180afc1 t0stack+0xafc1 %i6 = 0x000000000180b0b1 t0stack+0xb0b1
%o7 = 0x0000000001048b30 lgrp_traverse+0x520 %i7 = 0x0000000001048b94
plat_lgrp_init+0x14
%ccr = 0x00 xcc=nzvc icc=nzvc
%fprs = 0x00 fef=0 du=0 dl=0
%asi = 0x00
%y = 0x0000000000000000
%pc = 0x0000000001078224 panic
%npc = 0x0000000001078228 panic+4
%sp = 0x000000000180afc1 unbiased=0x000000000180b7c0
%fp = 0x000000000180b0b1
%tick = 0x00000000a376242d
%tba = 0x00000000f0200000
%tt = 0x17e
%tl = 0x0
%pil = 0xd
%pstate = 0x016 cle=0 tle=0 mm=TSO red=0 pef=1 am=0 priv=1 ie=1 ag=0
%cwp = 0x04 %cansave = 0x06
%canrestore = 0x00 %otherwin = 0x00
%wstate = 0x00 %cleanwin = 0x07
[0]> 0x00000000010d16e0/s
0x10d16e0: lgrp_traverse: No memory blocks found
[0]> <o0/s
0x10d16e0: lgrp_traverse: No memory blocks found
It’s points us to OpenSolaris/Illumos mpo.c source code with panic on line 743, and the system are not able to determine the memory blocks.
fail:
if (n_cpunodes > 0)
md_free_scan_dag(md, &cpunodes);
if (n_mblocks > 0)
md_free_scan_dag(md, &mblocknodes);
else
panic("lgrp_traverse: No memory blocks found");
Then, try to look the conditions of the panic:
[0]> :c
kmdb: stop at lgrp_traverse
kmdb: target stopped at:
lgrp_traverse: save %sp, -0xf0, %sp
[0]> lgrp_traverse::dis
lgrp_traverse: save %sp, -0xf0, %sp
lgrp_traverse+4: call -0x360 <md_get_root>
lgrp_traverse+8: mov %i0, %o0
lgrp_traverse+0xc: sethi %hi(0x10d1400), %g1
lgrp_traverse+0x10: sethi %hi(0x183cc00), %i3
lgrp_traverse+0x14: add %g1, 0x308, %i5
lgrp_traverse+0x18: cmp %o0, -0x1
lgrp_traverse+0x1c: bne,pn %xcc, +0x14 <lgrp_traverse+0x30>
lgrp_traverse+0x20: add %i3, 0x348, %l3
lgrp_traverse+0x24: mov -0x1, %l0
lgrp_traverse+0x28: ba +0x4a4 <lgrp_traverse+0x4cc>
lgrp_traverse+0x2c: ld [%l3], %l6
lgrp_traverse+0x30: mov %o0, %i4
lgrp_traverse+0x34: add %fp, 0x7f7, %o4
lgrp_traverse+0x38: add %i5, -0x1c0, %o2
lgrp_traverse+0x3c: add %i5, -0x2d0, %o3
lgrp_traverse+0x40: mov %i0, %o0
lgrp_traverse+0x44: call -0x9d70 <md_alloc_scan_dag>
lgrp_traverse+0x48: mov %i4, %o1
lgrp_traverse+0x4c: cmp %o0, 0x0
lgrp_traverse+0x50: bg,pn %icc, +0x28 <lgrp_traverse+0x78>
lgrp_traverse+0x54: st %o0, [%l3]
lgrp_traverse+0x58: sethi %hi(0x183cc00), %l6
[0]> ::bp md_alloc_scan_dag
[0]> :c
The first call is md_alloc_scan_dag, we set the bp on it,and continue
kmdb: stop at md_alloc_scan_dag
kmdb: target stopped at:
md_alloc_scan_dag: save %sp, -0xb0, %sp
[0]> :z
[0]> $r
%g0 = 0x0000000000000000 %l0 = 0x00000000fff340e0
%g1 = 0x00000000010d1400 %l1 = 0x0000000000000045
%g2 = 0x000000000000000b %l2 = 0x0000000000000089
%g3 = 0x00000000fff34010 %l3 = 0x000000000183cf48 n_mblocks
%g4 = 0x000000000000000b %l4 = 0x8000000083000740
%g5 = 0x4d61636844657363 %l5 = 0x8000000083110750
%g6 = 0x0000000000000000 %l6 = 0x00000000fff34010
%g7 = 0x000000000180e000 t0 %l7 = 0x000000000183cb18 startup_memops
%o0 = 0x00000000fff32000 %i0 = 0x00000000fff32000
%o1 = 0x0000000000000000 %i1 = 0x0000000000000001
%o2 = 0x00000000010d1548 %i2 = 0x00000000fff42000
%o3 = 0x00000000010d1438 %i3 = 0x000000000183cc00
%o4 = 0x000000000180b8a8 t0stack+0xb8a8 %i4 = 0x0000000000000000
%o5 = 0x00000000000001b0 %i5 = 0x00000000010d1708
%o6 = 0x000000000180afc1 t0stack+0xafc1 %i6 = 0x000000000180b0b1 t0stack+0xb0b1
%o7 = 0x0000000001048654 lgrp_traverse+0x44 %i7 = 0x0000000001048b94
plat_lgrp_init+0x14
%ccr = 0x00 xcc=nzvc icc=nzvc
%fprs = 0x00 fef=0 du=0 dl=0
%asi = 0x00
%y = 0x0000000000000000
then, we looking %o2 and %o3 registers, they contain the values after md_alloc_scan_dag call execution:
[0]> 0x00000000fff32000/s
[0]> 0x00000000010d1548/s
0x10d1548: mblock
[0]> 0x00000000010d1438/s
0x10d1438: fwd
[0]> n_mblocks/X
n_mblocks:
n_mblocks: 0
So, we have a proof: “n_mblocks” is zeroed here, The call returns “mblock” and “fwd” before the crash through md_alloc_scan_dag call.
But what is all this for? What is ‘md’, what are ‘mblock’ and ‘fwd’, and how do all of these interact at the earliest level of boot?
Hopefully, Sun published some documents at this time that can help determine the sequence and behavior — all of this is described in FWARC 2005/115.
The idea differs from the x86 world, where things mostly operate through parsing the ACPI tables
In case of sun4v world all hosts are de-facto are the guests controled by tiny firmware Type-1 hypervisor : and each machine (any term - had a pre-described firmware part named a Machine Description:
A machine description (“MD”) contains both explicit information about resources within a machine - detailed by specific nodes within the MD, and implicit information about the relationship of those resources - detailed by how nodes are interconnected into a relationship graph
and below in 3.1 Graph Contents:
The intrinsic Machine Description (MD) is a directed acyclic graph (DAG) of nodes describing resources or information available within a machine. This information is provided upon request to a guest operating system via the machine description request API
okay, we know what is the DAG, but how elements are connect to each other?
The default DAG described within the MD is defined by arcs (element type PROP_ARC) with a name of “fwd”. For convenience in walking this DAG. Arcs named “back” are also provided that define the inverse DAG. Thus for every node A that has a “fwd” arc pointing to another node B, there is a corresponding “back” arc for node B pointing back to A.
It’s described the relationship between two nodes through “fwd” directive:
So we should have DAG very similar to:
And it seems we have a “memory” -> “fwd” -> “mblock” request with an unconditional answer (‘0’), and a panic — and it’s proven by the QEMU MD/’firmware’ part, using the mdlint tool (described in Part 2):
~/images/S10image$ ./mdlint -t 1up-md.bin | egrep "fwd|node_0x1c|node_0x25"
fwd -> node_0x1c;
fwd -> node_0x25;
node platform node_0x1c { /* next @ 0x25 */
node virtual-devices node_0x25 { /* next @ 0x28 */
back -> node_0x25;
back -> node_0x25;
back -> node_0x25;
back -> node_0x25;
and we have “mblock0” definition but no “memory” node and no any “fwd”/”back” directive for both of it
dpim@vmbox:~/images/S10image$ ./mdlint -t 1up-md.bin | egrep memory
dpim@vmbox:~/images/S10image$ ./mdlint -t 1up-md.bin | egrep mblock
node mblock node_0x53 { /* next @ 0x57 */
UPD 13/07,14/07: Patch are reviewed and on/queue for qemu-devel
For more than a decade I’ve followed QEMU SPARC emulation with hope, especially Artyom Tarasenko’s experiments. Claude has helped unlock a lot of possibilities for progressing the SPARC64 emulation project, for a relatively small fee.
When I started reviewing the possibilities, it seemed like a good idea to test bootability across [Open]Solaris releases, from the earliest to the latest, and systematically track the status of each one. It’s a relatively small cost of effort — you just need a list of [Open]Solaris builds and check each one for progress.
Most of Artyom’s blog intensive work is based on Solaris Express images (or “snv” in Sun-era terminology), built for the Sun OpenSPARC project. All of these OpenSPARC images use ufs ramdrive with relatively tiny sizes, hundreds of megabytes. At this time, two well-known bootable drives are disk.s10hw2 from the OpenSPARC T1 emulator archive dated circa 2006, and another one, ramdisk.snv-b77-nd.no-boot-time-network.gz. With b77 it should be not too late than mid 2007.
When trying to load a full-capable ISO image, i.e. Solaris 10, it looks like this:
/home/dpim/build/qemu-sparc64/qemu-system-sparc64 -M niagara -L /home/dpim/images/S10image -nographic -m 256 -drive if=pflash,readonly=on,file=/zroot/iso/s10u11/V36434-01.iso
qemu-system-sparc64: could not load ram disk '/zroot/iso/s10u11/V36434-01.iso'
Our first touch of the code directly unlocked the ability to use >2GB vdisks during the copy into memory. It just prevents large (>2 GiB) vdisk images from silently getting corrupted or truncated when copied into the ramdisk region.
And now we’re able to boot, i.e.:
/home/dpim/build/qemu-sparc64/qemu-system-sparc64 -M niagara -L /home/dpim/images/S10image -nographic -m 256 -drive if=pflash,readonly=on,file=/zroot/iso/s10u11/V36434-01.iso
<..>
cpu Probing I/O buses
Sun Fire T2000, No Keyboard
Copyright 2005 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.20.0, 256 MB memory available, Serial #1122867.
[mo23723 obp4.20.0 #0]
Ethernet address 0:80:3:de:ad:3, Host ID: 80112233.
ok boot /virtual-devices/disk@0:f
Boot device: /virtual-devices/disk@0:f File and args:
hsfs-file-system
Loading: /platform/sun4v/boot_archive
<...>
The qemu now starts successfully and recognizes the iso image for booting!
Yesterday, after 18 years, I left Oracle — spanning two countries and several roles along the way. Looking back, it was a genuinely exciting stretch of my life; I met a lot of interesting people over these years.
I’m starting this small note here because I want to share some things about the technology and products I worked with. And also — I’m open and looking forward to whatever comes next.
Not everything I wanted to touch, I got to touch — not for lack of interest, but because the day job left no room for it. There are things I’ve watched from the sidelines for years, with real hope, that I simply couldn’t get to while working on my past job. Some of that I’m quietly starting now, on my own time and terms.
Stay tuned!
