Thursday, April 05, 2007

Time to Build a Guest Domain!

So now it's time to build a guest domain! A guest domain is made up of the following components:

  • CPU
  • MAU (Cryptographic Thread)
  • Memory
  • Networking
  • Storage
The control domain will partition CPU threads as VCPU's for the guest domain. Each CPU core has an MAU for cryptographic processing. Only one logical domain using the CPU threads in a core can have control over this thread. So it's important to decide if your guest domain will require one. Memory is partitioned in 8K segments. Networking is handled by connecting a virtual network interface to a virtual switch in one of the service domains. In our example, I configured each physical interface as a separate virtual switch in the control/service (a.k.a. primary) domain. Storage can come from a wide variety of sources:

  • Local Disk
  • SAN LUN
  • Virtual Disk Image File
  • Disk Slice
  • ZFS Volume
The T2000 for example has four drive bay that could be used, but obviously that doesn't leave us with a lot of flexibility or space. SAN storage can be used with greater flexibility since it's remote and can easily be migrated or replicated. It's possible to create a sparse file and use it as a virtual disk. This has the advantage of being stored on local, SAN, or even NAS. The fact that files can be used opens the door for very flexible options. Using a disk slice is also possible, but it can not be used for jumpstart installation. One could create ZFS volumes and use them as storage for logical domains as well. However, it can not be used for jumpstart installation. However, it makes for easy allocation of storage for applications. You can even take SAN LUN's and create a ZFS pool and export it into a logical domain. For our example, I'll use two virtual disk image files created on a ZFS file system and use SVM mirroring:) The following will be configured:

  • 4 x VCPU's
  • 1 x MAU
  • 4GB's RAM
  • 2 x 10GB Virtual Disk Image Files
  • 2 x Network Ports

# ldm add-domain ldom1
# ldm add-vcpu 4 ldom1
# ldm add-mau 1 ldom1
# ldm add-memory 4G ldom1
# mkfile 10g /ldoms/vdisk1_10gb.img
# mkfile 10g /ldoms/vdisk2_10gb.img
# ldm add-vdiskserverdevice /ldoms/vdisk1_10gb.img vdisk1@primary-vds0
# ldm add-vdiskserverdevice /ldoms/vdisk2_10gb.img vdisk2@primary-vds0
# ldm add-vdisk vdisk1 vdisk1@primary-vds0 ldom1
# ldm add-vdisk vdisk2 vdisk2@primary-vds0 ldom1
# ldm add-vnet vnet0 primary-vsw0 ldom1
# ldm add-vnet vnet1 primary-vsw2 ldom1
# ldm set-variable auto-boot\?=false ldom1
# ldm set-variable local-mac-address\?=true ldom1
# ldm set-variable boot-device=/virtual-devices@100/channel-devices@200/disk@0 ldom1
# ldm bind-domain ldom1
# ldm start ldom1


So with the above commands we allocated the vcpu's, mau, and the memory. Then created the virtual disk images files, added them as virtual disk devices to the primary domain's VDS service, and finally added them as virtual disks to the guest domain. Then attached virtual networks, set auto-boot to false in the OBP(yes that's right each logical domain gets its own OBP), set local-mac-address to true, and set the default boot device. Finally we've bound the configuration and started the guest domain. So what do we get?

# ldm list-bindings ldom1
Name: ldom1
State: active
Flags: transition
OS:
Util: 0.2%
Uptime: 1d 6h 43m
Vcpu: 4
vid pid util strand
0 4 0.7% 100%
1 5 0.1% 100%
2 6 0.1% 100%
3 7 0.0% 100%
Mau: 1
mau cpuset (4, 5, 6, 7)
Memory: 4G
real-addr phys-addr size
0x4800000 0x104800000 4G
Vars: auto-boot?=false
boot-device=/virtual-devices@100/channel-devices@200/disk@0
local-mac-address?=true
Vldcc: vldcc0 [Domain Services]
service: primary-vldc0 @ primary
[LDC: 0x0]
Vnet: vnet0 [LDC: 0x2]
mac-addr=0:14:4f:fb:c4:ef
service: primary-vsw0 @ primary
[LDC: 0x1]
Vnet: vnet1 [LDC: 0xd]
mac-addr=0:14:4f:fb:24:b6
service: primary-vsw2 @ primary
[LDC: 0xc]
Vdisk: vdisk1 vdisk1@primary-vds0
service: primary-vds0 @ primary
[LDC: 0x17]
Vdisk: vdisk2 vdisk2@primary-vds0
service: primary-vds0 @ primary
[LDC: 0x18]
Vcons: [via LDC:25]
ldom1@primary-vcc0 [port:5000]


As you can see, everything that's been previously configured is listed. Some important things to note are the MAC addresses for the network interfaces (which are assigned automatically) and the Vcons port for the console. So now we can jumpstart our domain:

# telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Connecting to console "ldom1" in group "ldom1" ....
Press ~? for control options ..

Sun Fire T200, No Keyboard
Copyright 2007 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.26.0.build_07, 4096 MB memory available, Serial #66831599.
Ethernet address 0:14:4f:fb:c4:ef, Host ID: 83fbc4ef.



{0} ok show-nets
a) /virtual-devices@100/channel-devices@200/network@1
b) /virtual-devices@100/channel-devices@200/network@0
q) NO SELECTION
Enter Selection, q to quit: a
/virtual-devices@100/channel-devices@200/network@1 has been selected.
Type ^Y ( Control-Y ) to insert it in the command line.
e.g. ok nvalias mydev ^Y
for creating devalias mydev for /virtual-devices@100/channel-devices@200/network@1
{0} ok boot /virtual-devices@100/channel-devices@200/network@1 - install
Boot device: /virtual-devices@100/channel-devices@200/network@1 File and args:
- install
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
SunOS Release 5.10 Version Generic_118833-33 64-bit
Copyright 1983-2006 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
WARNING: machine_descrip_update: new MD has the same generation (1) as the old MD
whoami: no domain name
Configuring devices.
Using RPC Bootparams for network configuration information.
Attempting to configure interface vnet1...
Configured interface vnet1
Attempting to configure interface vnet0...
Skipped interface vnet0
Setting up Java. Please wait...
Extracting windowing system. Please wait...
Beginning system identification...
Searching for configuration file(s)...
...
So after the guest domain is finished jumpstarting, we can take a look around.

# psrinfo -vp
The physical processor has 4 virtual processors (0-3)
UltraSPARC-T1 (cpuid 0 clock 1000 MHz)
# psrinfo -v
Status of virtual processor 0 as of: 04/05/2007 22:17:04
on-line since 04/05/2007 22:16:15.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 1 as of: 04/05/2007 22:17:04
on-line since 04/05/2007 22:16:16.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 2 as of: 04/05/2007 22:17:04
on-line since 04/05/2007 22:16:16.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 3 as of: 04/05/2007 22:17:04
on-line since 04/05/2007 22:16:16.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
# prtdiag -v
System Configuration: Sun Microsystems sun4v Sun Fire T200
Memory size: 4096 Megabytes

========================= CPUs ===============================================

CPU CPU
Location CPU Freq Implementation Mask
------------ ----- -------- ------------------- -----
MB/CMP0/P0 0 1000 MHz SUNW,UltraSPARC-T1
MB/CMP0/P1 1 1000 MHz SUNW,UltraSPARC-T1
MB/CMP0/P2 2 1000 MHz SUNW,UltraSPARC-T1
MB/CMP0/P3 3 1000 MHz SUNW,UltraSPARC-T1


========================= IO Configuration =========================

IO
Location Type Slot Path Name Model
----------- ----- ---- --------------------------------------------- ------------------------- ---------

========================= HW Revisions =======================================

System PROM revisions:
----------------------
OBP 4.26.0.build_07 2007/02/14 19:20

IO ASIC revisions:
------------------
Location Path Device Revision
-------------------- ---------------------------------------- ------------------------------ ---------
# df -h
Filesystem size used avail capacity Mounted on
/dev/md/dsk/d0 7.8G 2.2G 5.5G 30% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 5.1G 1.1M 5.1G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
/platform/SUNW,Sun-Fire-T200/lib/libc_psr/libc_psr_hwcap1.so.1
7.8G 2.2G 5.5G 30% /platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,Sun-Fire-T200/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
7.8G 2.2G 5.5G 30% /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd 0K 0K 0K 0% /dev/fd
swap 1.6G 0K 1.6G 0% /tmp
swap 5.1G 32K 5.1G 1% /var/run
# metastat
d1: Mirror
Submirror 0: d11
State: Okay
Submirror 1: d21
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 4194600 blocks (2.0 GB)

d11: Submirror of d1
State: Okay
Size: 4194600 blocks (2.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0d0s1 0 No Okay No


d21: Submirror of d1
State: Okay
Size: 4194600 blocks (2.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0d1s1 0 No Okay No


d0: Mirror
Submirror 0: d10
State: Okay
Submirror 1: d20
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 16644000 blocks (7.9 GB)

d10: Submirror of d0
State: Okay
Size: 16644000 blocks (7.9 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0d0s0 0 No Okay No


d20: Submirror of d0
State: Okay
Size: 16644000 blocks (7.9 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0d1s0 0 No Okay No


Device Relocation Information:
Device Reloc Device ID
c0d1 No -
c0d0 No -
# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
vnet0: flags=9040843 mtu 1500 index 2
inet 192.168.1.2 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp1
ether 0:14:4f:fb:c4:ef
vnet0:1: flags=1000843 mtu 1500 index 2
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
vnet1: flags=9040843 mtu 1500 index 3
inet 192.168.1.3 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp1
ether 0:14:4f:fb:24:b6
# uptime
10:20pm up 5 min(s), 1 user, load average: 0.02, 0.11, 0.06


So now we have a guest domain running Solaris 10 Update 3, with SVM mirrored boot drives that are really sparse files, IPMP on virtual NICs, four CPU's, 4GB's RAM, etc. Pretty kewl, eh?

In the next post I'll talk about virtual devices in more detail and go from there!

Disclaimer: This post only covers the Early Access RC3 of the LDOM technology, which is in pre-release.

16 comments:

Chhandomay said...

This is an excellent series of articles on Solaris LDOM. Great work.

-- Chhandomay

Octave Orgeron said...

Thanks!

Real said...

First of all I want to thank you for the great coverage of LDOMs. Great job!
But there are some minors mistakes in commands you have published here.

1. ldm add-vnet vnet1 primary-vsw2 ldom1
should be
ldm add-vnet vnet1 primary-vsw1 ldom1

2. ldm set local-mac-address?=true ldom1
should be
ldm set-variable local-mac-address\?=true ldom1

Octave Orgeron said...

Thanks for catching the missing "\" in the local-mac-address. I've corrected this. I've had issues with cutting and pasting some commands, let alone the formatting is not WYSIWYG:( As a result, I end up spending a day or two trying to fix this manually for each post that has CLI output, and as such it's error prone:(

As for the primary-vsw2 switch being used instead of the primary-vsw1 switch, that is intentional. I have the public network connected on e1000g0 and e1000g2. This is because ports 0 and 1 are on one broadcom controller on PCI-E bus A. Ports 2 and 3 are on a different broadcom controller on PCI-E bus B. I did this for performance and redundancy.

That brings up the topic of Split PCI-E configurations where you have two service domains. I'll have some postings on that later on.

Real said...

OK. I've got it about primary-vsw2.

But there is still typo in "ldm set local-mac-address\?=true ldom1"
There should be "set-variable" instead of "set".

Also, I cannot use disk slices to make them as virtual disk devices.
When I do "ldm add-vdiskserverdevice /dev/rdsk/c1t1d0s2 vdisk2@primary-vds0" and "ldm add-vdisk vdisk2 vdisk2@primary-vds0 ldom1" I've got
error messages:
Apr 8 11:47:35 t2000 vds: [ID 416625 kern.info] vd_setup_vd(): ldi_get_size() failed for /dev/rdsk/c1t1d0s2
Apr 8 11:47:35 t2000 vds: [ID 556514 kern.info] vds_add_vd(): Failed to add vdisk ID 1

I tried to use /dev/dsk/c1t1d0s2 with the same result. But your example with /ldom/files10g works OK for me. Looks like vds thinks about c1t1d0s2 as a regular file, not a device.

Octave Orgeron said...

Thanks for catching the missing "set-variable" error. I've fixed that.

As for using a disk slice, I'll have to test this out again with RC3. Are you using Solaris 10 U3 or Nevada?

Chris said...

Where do you setup the JumpStart server? Is it in the Control Domain or another physical server on the same network as the Control Domain?

In my case, the guest domain has problem finding the server.

{0} ok boot /virtual-devices@100/channel-devices@200/network@0 - install
Boot device: /virtual-devices@100/channel-devices@200/network@0 File and args: - install
Requesting Internet Address for 0:14:4f:f8:d0:c
Requesting Internet Address for 0:14:4f:f8:d0:c

Octave Orgeron said...

In my examples, the jumpstart server is on a separate server. However, you can jumpstart a guest domain from another guest domain. The key is to make sure that the jumpstart server is on the same virtual switch or network that the guest domain is connected to.

Dan said...

After running into BUG 6514091, I created a few files instead...

-rw------T 1 root root 3956202496 Jun 7 14:11 root
-rw------T 1 root root 4294967296 Jun 7 14:11 swap
-rw------T 1 root root 4294967296 Jun 7 14:11 user-local

added them to the vds:

vdsdev: root device=/smtp4-vdisks/root
vdsdev: swap device=/smtp4-vdisks/swap
vdsdev: user-local device=/smtp4-vdisks/user-local

added them to the guest domain:

Vdisk: root root@primary-vds0
service: primary-vds0 @ primary
Vdisk: swap swap@primary-vds0
service: primary-vds0 @ primary
Vdisk: user-local user-local@primary-vds0
service: primary-vds0 @ primary

and it looks like the JumpStart went OK, but it will not boot. Any ideas???

Boot device: /virtual-devices@100/channel-devices@200/disk@0:a File and args:
SunOS Release 5.10 Version Generic_118833-33 64-bit
Copyright 1983-2006 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
disk: no major number
channel-devices: no major number
Cannot load drivers for /virtual-devices@100/channel-devices@200/disk@0:a
Can't load the root filesystem
Debugging requested; hardware watchdog suspended.
Type 'go' to resume

אחיק said...

I'm probably missing the obvious - but can I install a guest ldom from the T2000 cdrom drive, or am I limited to installing from a Jumpstart server ?
Thanks,
Marvin.

Octave Orgeron said...

Dan,

I would need some more information. How did you jumpstart the ldom and what's the disk configuration. What order did you use to add your virtual disk images? Can you see your virtual disk images in the guest domain OBP or in single-user mode over netboot?

Octave Orgeron said...

Marvin,

You'll definitely want to use Jumpstart to get Solaris installed on your LDOM's. The CD-ROM will only be visible to the control/service domain.

thulhu said...

Thanks a lot for your examples,
There is something i am missing :
From what i read in the beginner LDOM guide i can't use virtual disk files to do regular install because the format command can't partition such disk (hence whole disks have to be allocated to each virtual servers). So your Jumpstart profiles can't include partionning and filesys keywords ?
Is there any way to make virtual disks files (or anything else minus real disks) acting like "real disks" ?

clilmoney said...

I am unable to jumpstart my guest domain. Are there any requirements for the jumpstart server? The jumpstarted is located on a seperate server...

Rahul said...

After setting up jumpstart, I am getting following error:

{0} ok boot ldom1-vnet0
Boot device: /virtual-devices@100/channel-devices@200/network@0le and args:
Requesting Internet Address for 0:14:4f:fb:3b:cc
SunOS Release 5.10 Version Generic_118833-17 64-bit
Copyright 1983-2005 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
network: no major number
channel-devices: no major number
Cannot load drivers for /virtual-devices@100/channel-devices@200/network@0
Can't load the root filesystem
Type 'go' to resume
{0} ok

Any clue of what's going wrong?

HP said...

Hi, Reall a good one.

Can you suggest the way we can configure the vlan tagging in ldm 1.1, is that still required to name the vnetxxxxx in the gdom based on the vlan id and the interface details.

Thanks