One of the more interesting and puzzling things I ran into was a suspicious hang on any configuration with 2048MB or more of memory under QEMU and Machine Description (MD) configuration. It was a glass wall (matrix here) I couldn’t break through for any Solaris 10 release higher than S10u1.

The symptoms were odd: it starts to boot and then completely hangs, without any further response to kmdb.

QEMU itself generates tons of logs - traps between hypervisor and MMU, like this:

58552984: Hyperprivileged Trap Instruction (v=0183)
58552985: Data Access MMU Miss (v=0068)
58552986: Hyperprivileged Trap Instruction (v=0183)
58552987: Data Access MMU Miss (v=0068)
58552988: Hyperprivileged Trap Instruction (v=0183)
58552989: Data Access MMU Miss (v=0068)

This made me think something was wrong with the software MMU code execution inside QEMU - but I had no idea what.

Moreover, it looked like a memory pagesize problem: the guest OS trying to reach a pagesize bigger than the host supports. This was a suspicion, not proven at the time: for decades, every new paging feature introduced in silicon was a bit of “fanfare” inside the Solaris community, and I suspect we have some limitations in our software emulation/QEMU also.

Let’s start from the guest/Solaris side with a bit of theory (i’m sorry):

A parameter named “mmu-pagesize-list” is read from the MD at a predetermined point before boot, and it uses the standard bitmask supported by the 3-bit [sz] parameter on the TSB Translation Table Entry (TTE):

This is described cleanly in UltraSPARC Architecture 2005, page 362 - though there it’s actually a 4-bit field:

Data – 3:0 sz The page size of this entry, encoded as shown below.
sz Page Size
0000 8 Kbyte
0001 64 Kbyte
0010 Reserved
0011 4 Mbyte
0100 Reserved
0101 256 Mbyte
0110 Reserved
0111 Reserved
1000-1111 Reserved

This converts to the following code table for T1:

So, to enable the standard support for 8K+64K+4M, we should have 0xb at this stage; the MD should be 0x9 without 64K support (and even that works!), and 0xb also seems valid with 64K pages enabled.

But in practice, the picture was the opposite:

The kernel mystically overrides the default values from 0x9 (or even 0xb) and silently changes them to 0x2b (+256M support) at an early stage of booting:

[0]> mmu_exported_pagesize_mask/X
mmu_exported_pagesize_mask:
mmu_exported_pagesize_mask:     2b

Why does the kernel insist on “0x2b” instead of “0xb”, which is recorded cleanly in our “firmware” / MD property?

I started tracing through Solaris and found a number of interesting things. Seems in OpenSolaris/Solaris there are two branches of control selected by “broken_md_flag”, set when the domaining-enabled parameter (p.62) is absent from the MD. “Domaining” refers to the host acting as the primary domain (of course, under the hypervisor as well - any sun4v system runs under the control of the hypervisor code from the sun4v firmware; this is the “q.bin” file underneath the Sun Legion/QEMU part).

The absence of the “domaining-enabled” property raises the broken_md_flag here:

[0]> broken_md_flag/D
broken_md_flag:
broken_md_flag: 1

And it comes from os/fillsysinfo.c in the OpenSolaris/Illumos code:

init_md_broken(md_t *mdp, mde_cookie_t *cpulist)
{
        int nrnode;
        mde_cookie_t *platlist, rootnode;
        uint64_t val = 0;
        char *namebuf;
        int namelen;

        rootnode = md_root_node(mdp);
        ASSERT(rootnode != MDE_INVAL_ELEM_COOKIE);
        ASSERT(cpulist);

        nrnode = md_alloc_scan_dag(mdp, rootnode, "platform", "fwd",
            &platlist);

        if (nrnode < 1)
                cmn_err(CE_PANIC, "init_md_broken: platform node missing");

        if (md_get_prop_data(mdp, cpulist[0],
            "compatible", (uint8_t **)&namebuf, &namelen)) {
                cmn_err(CE_PANIC, "init_md_broken: "
                    "Cannot read 'compatible' property of 'cpu' node");
        }

        if (md_get_prop_val(mdp, platlist[0],
            "domaining-enabled", &val) == -1 &&
            strcmp(namebuf, "SUNW,UltraSPARC-T1") == 0)
                broken_md_flag = 1;

        md_free_scan_dag(mdp, &platlist);
}

[0]> :c
kmdb: stop at init_md_broken+0x88
[0]> <o0/s
0xfff34ac0:     SUNW,UltraSPARC-T1
[0]>

(This is beside the point, but if we try to enable the “domaining-enabled” property, the kernel starts but panics in the CPU platform module when it tries to configure some cache/TLB/MMU properties extracted from the MD - i.e., it behaves as if it were a real control LDOM. It’s a good starting point for the next stage of MD structure research, to open up a new level of possibilities, IMO.)

So, since we don’t have the domaining-enabled property in the MD’s cpu node, as expected, broken_md_flag gets set:

cpu0: UltraSPARC-T1 (chipid 0, clock 5 MHz)
Loaded modules: [ scsi_vhci mac neti ufs s1394 hook ip usba specfs sctp arp
sockfs ]
kmdb: target stopped at:
kmdb_enter:     ta        %icc, 0x7d
[0]> broken_md_flag/D
broken_md_flag:
broken_md_flag: 1
[0]>

This triggers the setup of a specific memory pagesize mask in /usr/src/uts/sun4v/cpu/niagara.c:

  if (broken_md_flag) {
                /*
                 * Turn on the missing bits supported by Niagara CPU in
                 * MMU pagesize mask returned by MD.
                 */
                mmu_exported_pagesize_mask |= NI_MMU_PAGESIZE_MASK;
        } else {
                if ((mmu_exported_pagesize_mask &
                    DEFAULT_SUN4V_MMU_PAGESIZE_MASK) !=
                    DEFAULT_SUN4V_MMU_PAGESIZE_MASK)
                        cmn_err(CE_PANIC, "machine description"
                            " does not have required sun4v page sizes"
                            " 8K, 64K and 4M: MD mask is 0x%x",
                            mmu_exported_pagesize_mask);

So, when broken_md_flag=1, the MMU pagesizes come from NI_MMU_PAGESIZE_MASK:

#define	NI_MMU_PAGESIZE_MASK	((1 << TTE8K) | (1 << TTE64K) | (1 << TTE4M) \
				    | (1 << TTE256M))

otherwise, the predefined parameters above are used - the default CPU pagesize mask with 8K/64K/4M, from /usr/src/uts/sun4v/sys/cpu_module.h (not our case):

#define DEFAULT_SUN4V_MMU_PAGESIZE_MASK ((1 << TTE8K) | (1 << TTE64K) \
                                            | (1 << TTE4M))

(it seems the control domain didn’t use 256M at this point in Solaris Express 10/2011)

and the bitmask pagesize flags come from /usr/src/uts/sun4v/sys/pte.h:

/* Defines for sz field in tte */
#define TTE8K                   0x0
#define TTE64K                  0x1
#define TTE512K                 0x2
#define TTE4M                   0x3
#define TTE32M                  0x4
#define TTE256M                 0x5
#define TTE2G                   0x6
#define TTE16G                  0x7

NI_MMU_PAGESIZE_MASK = (1<<0)|(1<<1)|(1<<3)|(1<<5) = { 8K, 64K, 4M, 256M } = binary 0010 1011 = 0x2b
DEFAULT_SUN4V_MMU_PAGESIZE_MASK = (1<<0)|(1<<1)|(1<<3) = {8K, 64K, 4M } = 0000 1011 = 0xb

i.e., 00101011 -> 0x2b (256M enabled) vs. 00001011 -> 0xb for the basic value (8K+64K+4M).

It seems our host is trying to enable 256M pages. In other words, here’s the interesting logic: on a domaining-capable host (i.e., one that controls the hypervisor), we support 8K/64K/4M paging; but when “broken_md_flag” is set, we also get 256M support on T1. Note that 32M pages aren’t implemented in T1 silicon at all, as described in the niagara.c logic above.

Part 2 continues with what this looks like in practice, and what the QEMU side reveals.