Excellent Flash Video of LDOM's

I came across this great flash video of 31 logical domains booting on Ashley Saulsbury's blog. It demonstrates what is possible with logical domains. However, it's unlikely that one would create that many, as it would under power the primary (control/service) domain. But it does demonstrate how each logical domain is separate and independent. But as Niagara 2 and later ROCK servers come into play, it should be possible to see well over a 100 logical domains running on a single server.

Project Blackbox

Yesterday at the Sam Houston Race Park, I had a the opportunity to check out Sun's Project Blackbox! I was very impressed with the product. It has a lot of potential in today's market where space, energy, cooling, etc are major concerns. Within one shipping container, you can pack a lot of servers and storage. There were many presentations from Sun and lots of guests! I just want to say that the Sun Folks did a great job and I had a blast! So now to the pictures!

Here are the power and cooling hookups:


This is one of the breaker boxes:


Here is the Bulkhead you must open after the container doors are opened:


And now for the door behind the Bulkhead:


The BIG RED button:


Nice Cable channels above. The lighting is great and is on a timer:

I want one of these under my desk to keep my cables organized:


Ah.. some servers:


A nice cut-away so you can see the servers in the cabinet:


This is the device that allows you to pull the cabinet out:


This is a cut away of the one of the heat exchanger units:


The shock absorbers:


The networking end of things:


The dehumidifier:


The front of the truck:


Big Sun Logo on the side of BlackBox:


Pretty kewl eh? Well I recommend checking out Blackbox if you have the chance. Check out the website here.

Logical Domains 1.0 Production Release!

Awesome news for everyone! The production release of Logical Domains is now available for download! This is really great news! As such, I'll be upgrading today and I'll post some blog entries this weekend. I also have an entry on virtual devices and services that I'm working on finishing:)

In any case, here are the downloads:

• Logical Domains 1.0
• Logical Domains 1.0 Release Notes
• Logical Domains 1.0 Administration Guide
  

SPARC Enterprise Line Released!

Well the SPARC Enterprise server line is now released! Here are the max specs:

• T1000 (1 x UltraSPARC-T1 8 core CPU, 32GB's, 2 x SAS Drives, 1 x PCI-E Slot)
• T2000 (1 x UltraSPARC-T1 8 core CPU, 64GB's, 2 x SAS Drives, 3 x PCI-E, 2 x PCI-X Slots)
• M4000 (4 x SPARC64 VI Dual core CPU's, 128GB's RAM, 2 x SAS Drives, 1 x IOU's)
• M5000 (8 x SPARC64 VI Dual core CPU's, 256GB's RAM, 4 x SAS Drives, 2 x IOU's)
• M8000 (16 x SPARC64 VI Dual core CPU's, 512GB RAM, 16 x SAS Drives, 4 x IOU's)
• M9000 (64 x SPARC64 VI Dual core CPU's, 2 TB RAM, 64 x SAS Drives, 16 x IOU's)

The T1000 and the T2000 are just newer revisions of the older systems and a label change. One of the kewl things about the OPL systems is the IOU's (I/O Units), which have cartridges for the PCI-E/PCI-X slots. So you don't have to open the box to install/replace cards! Very similar to the E15/25K in this regard. Speaking of I/O, the OPL systems also have the option of attaching an I/O Expansion Unit. This is basically an external box that had additional PCI-E/PCI-X slots (with the cards in cartridges of course). I hope this will be supported with the T1000/T2000 or the Niagara 2 systems. With Logical Domains, you'll want more I/O expansion.

All very kewl stuff and I've been waiting for this announcement. Now we just have to wait for the Niagara 2 systems.

Check out the link to the event!

ROCK, UltraSparc IV+, APL, and other news!

Well a lot of buzz is going around about next week when Sun will announce some new products. Sun has given the UltraSparc IV+ a speed increase, gotten ROCK silicon, and is about to release the APL line next week. So lots of interesting stuff going on. Here are some articles, I'd recommend:

• UltraSparc IV+ Speed Bump
• Plenty of Sun Users to Skip Fujitsu March
• Sun CEO shows off ROCK ahead of Fujitsu launch
• Pictures of ROCK
• APL Info

Definitely lots of info! The speed up in the UltraSparc IV+ will definitely help customers who are looking to wait for ROCK or protect their current investment in SunFire servers. But I'm sure the APL/OPL servers will offer some performance opportunities in the meantime. Indeed, the APL/OPL servers may be more geared towards traditional applications that are more sensitive to single-threaded performance and are not optimized for CMT. That would seem to make the most sense, because Niagara and ROCK are CMT designs that work best with multi-threaded applications. Time will tell. Hopefully some benchmarks will come out to help customers figure out which platform to choose for which application.

It's great to see the ROCK silicon! This is obviously ahead of schedule and means that ROCK servers are not far off from being tested. In one of the articles above, you'll see a slide presented that shows Sun will release a 2, 4, and 8 way ROCK servers. Even Schwartz has not confirmed the number of threads per core, but has atleast confirmed that each ROCK chip will have 16 cores. So if we were to assume that each core were to have 4 threads that we mean the Platinum server would have 512 threads!

Also it would appear that people have looked at the Niagara 2 additions to OpenSolaris and the trademarks Sun has registered to figure out the model names for upcoming Niagara servers. From what I can tell, the models should be as follows:

• T5120 - 1U, 1 way Niagara 2
  
• T5220 - 2U, 1 way Niagara 2
  
• T5140 - 1U, 2 or 4 way Niagara 3
  
• T5240 - 2U, 2 or 4 way Niagara 3
  

This is my best guess since the first UltraSparc-T2(Niagara 2) servers will be 1 socket servers and the follow on servers will be 2 socket servers. The model names also follow what's been added to OpenSolaris over the past few months and what John Fowler has presented in the past about Niagara 2 and 3.

For those following OpenSolaris builds, you'll notice that the T5120 and the T5220 support is already present under /usr/platform. Also, you'll notice the "SUNW,SPARC-Enterprise" platform. Considering the above articles, it would appear that the new APL and Niagara servers will be known as "SPARC Enterprise" servers. Another interesting piece of info in OpenSolaris is that there is an addition to the sun4u platform called "sun4u-opl". This would suggest that the APL/OPL (Fujitsu Olympus SPARC64) servers will be "sun4u" and not "sun4us" which has traditionally been used for Fujitsu SPARC64. This is how UltraSparc 3 support was added to the "sun4u" line, you'll see "sun4u-us3" under /usr/platform. Hopefully, this will be the case and end the differences in the architectures:)

Logical Domain Links

I've added some links for Logical Domains to the far right for everyone. Direct links to the software download, documents, and on-line forum.

Enjoy!

Time to Build a Guest Domain!

So now it's time to build a guest domain! A guest domain is made up of the following components:

• CPU
  
• MAU (Cryptographic Thread)
• Memory
• Networking
• Storage

The control domain will partition CPU threads as VCPU's for the guest domain. Each CPU core has an MAU for cryptographic processing. Only one logical domain using the CPU threads in a core can have control over this thread. So it's important to decide if your guest domain will require one. Memory is partitioned in 8K segments. Networking is handled by connecting a virtual network interface to a virtual switch in one of the service domains. In our example, I configured each physical interface as a separate virtual switch in the control/service (a.k.a. primary) domain. Storage can come from a wide variety of sources:

• Local Disk
  
• SAN LUN
• Virtual Disk Image File
• Disk Slice
• ZFS Volume

The T2000 for example has four drive bay that could be used, but obviously that doesn't leave us with a lot of flexibility or space. SAN storage can be used with greater flexibility since it's remote and can easily be migrated or replicated. It's possible to create a sparse file and use it as a virtual disk. This has the advantage of being stored on local, SAN, or even NAS. The fact that files can be used opens the door for very flexible options. Using a disk slice is also possible, but it can not be used for jumpstart installation. One could create ZFS volumes and use them as storage for logical domains as well. However, it can not be used for jumpstart installation. However, it makes for easy allocation of storage for applications. You can even take SAN LUN's and create a ZFS pool and export it into a logical domain. For our example, I'll use two virtual disk image files created on a ZFS file system and use SVM mirroring:) The following will be configured:

• 4 x VCPU's
• 1 x MAU
• 4GB's RAM
  
• 2 x 10GB Virtual Disk Image Files
• 2 x Network Ports

# ldm add-domain ldom1
# ldm add-vcpu 4 ldom1
# ldm add-mau 1 ldom1
# ldm add-memory 4G ldom1
# mkfile 10g /ldoms/vdisk1_10gb.img
# mkfile 10g /ldoms/vdisk2_10gb.img
# ldm add-vdiskserverdevice /ldoms/vdisk1_10gb.img vdisk1@primary-vds0
# ldm add-vdiskserverdevice /ldoms/vdisk2_10gb.img vdisk2@primary-vds0
# ldm add-vdisk vdisk1 vdisk1@primary-vds0 ldom1
# ldm add-vdisk vdisk2 vdisk2@primary-vds0 ldom1
# ldm add-vnet vnet0 primary-vsw0 ldom1
# ldm add-vnet vnet1 primary-vsw2 ldom1
# ldm set-variable auto-boot\?=false ldom1
# ldm set-variable local-mac-address\?=true ldom1
# ldm set-variable boot-device=/virtual-devices@100/channel-devices@200/disk@0 ldom1
# ldm bind-domain ldom1
# ldm start ldom1

So with the above commands we allocated the vcpu's, mau, and the memory. Then created the virtual disk images files, added them as virtual disk devices to the primary domain's VDS service, and finally added them as virtual disks to the guest domain. Then attached virtual networks, set auto-boot to false in the OBP(yes that's right each logical domain gets its own OBP), set local-mac-address to true, and set the default boot device. Finally we've bound the configuration and started the guest domain. So what do we get?

# ldm list-bindings ldom1
Name: ldom1
State: active
Flags: transition
OS:
Util: 0.2%
Uptime: 1d 6h 43m
Vcpu: 4
vid pid util strand
0 4 0.7% 100%
1 5 0.1% 100%
2 6 0.1% 100%
3 7 0.0% 100%
Mau: 1
mau cpuset (4, 5, 6, 7)
Memory: 4G
real-addr phys-addr size
0x4800000 0x104800000 4G
Vars: auto-boot?=false
boot-device=/virtual-devices@100/channel-devices@200/disk@0
local-mac-address?=true
Vldcc: vldcc0 [Domain Services]
service: primary-vldc0 @ primary
[LDC: 0x0]
Vnet: vnet0 [LDC: 0x2]
mac-addr=0:14:4f:fb:c4:ef
service: primary-vsw0 @ primary
[LDC: 0x1]
Vnet: vnet1 [LDC: 0xd]
mac-addr=0:14:4f:fb:24:b6
service: primary-vsw2 @ primary
[LDC: 0xc]
Vdisk: vdisk1 vdisk1@primary-vds0
service: primary-vds0 @ primary
[LDC: 0x17]
Vdisk: vdisk2 vdisk2@primary-vds0
service: primary-vds0 @ primary
[LDC: 0x18]
Vcons: [via LDC:25]
ldom1@primary-vcc0 [port:5000]

As you can see, everything that's been previously configured is listed. Some important things to note are the MAC addresses for the network interfaces (which are assigned automatically) and the Vcons port for the console. So now we can jumpstart our domain:

# telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Connecting to console "ldom1" in group "ldom1" ....
Press ~? for control options ..

Sun Fire T200, No Keyboard
Copyright 2007 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.26.0.build_07, 4096 MB memory available, Serial #66831599.
Ethernet address 0:14:4f:fb:c4:ef, Host ID: 83fbc4ef.



{0} ok show-nets
a) /virtual-devices@100/channel-devices@200/network@1
b) /virtual-devices@100/channel-devices@200/network@0
q) NO SELECTION
Enter Selection, q to quit: a
/virtual-devices@100/channel-devices@200/network@1 has been selected.
Type ^Y ( Control-Y ) to insert it in the command line.
e.g. ok nvalias mydev ^Y
for creating devalias mydev for /virtual-devices@100/channel-devices@200/network@1
{0} ok boot /virtual-devices@100/channel-devices@200/network@1 - install
Boot device: /virtual-devices@100/channel-devices@200/network@1 File and args:
- install
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
Requesting Internet Address for 0:14:4f:fb:24:b6
SunOS Release 5.10 Version Generic_118833-33 64-bit
Copyright 1983-2006 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
WARNING: machine_descrip_update: new MD has the same generation (1) as the old MD
whoami: no domain name
Configuring devices.
Using RPC Bootparams for network configuration information.
Attempting to configure interface vnet1...
Configured interface vnet1
Attempting to configure interface vnet0...
Skipped interface vnet0
Setting up Java. Please wait...
Extracting windowing system. Please wait...
Beginning system identification...
Searching for configuration file(s)...
...

So after the guest domain is finished jumpstarting, we can take a look around.

# psrinfo -vp
The physical processor has 4 virtual processors (0-3)
UltraSPARC-T1 (cpuid 0 clock 1000 MHz)
# psrinfo -v
Status of virtual processor 0 as of: 04/05/2007 22:17:04
on-line since 04/05/2007 22:16:15.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 1 as of: 04/05/2007 22:17:04
on-line since 04/05/2007 22:16:16.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 2 as of: 04/05/2007 22:17:04
on-line since 04/05/2007 22:16:16.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
Status of virtual processor 3 as of: 04/05/2007 22:17:04
on-line since 04/05/2007 22:16:16.
The sparcv9 processor operates at 1000 MHz,
and has a sparcv9 floating point processor.
# prtdiag -v
System Configuration: Sun Microsystems sun4v Sun Fire T200
Memory size: 4096 Megabytes

========================= CPUs ===============================================

CPU CPU
Location CPU Freq Implementation Mask
------------ ----- -------- ------------------- -----
MB/CMP0/P0 0 1000 MHz SUNW,UltraSPARC-T1
MB/CMP0/P1 1 1000 MHz SUNW,UltraSPARC-T1
MB/CMP0/P2 2 1000 MHz SUNW,UltraSPARC-T1
MB/CMP0/P3 3 1000 MHz SUNW,UltraSPARC-T1


========================= IO Configuration =========================

IO
Location Type Slot Path Name Model
----------- ----- ---- --------------------------------------------- ------------------------- ---------

========================= HW Revisions =======================================

System PROM revisions:
----------------------
OBP 4.26.0.build_07 2007/02/14 19:20

IO ASIC revisions:
------------------
Location Path Device Revision
-------------------- ---------------------------------------- ------------------------------ ---------
# df -h
Filesystem size used avail capacity Mounted on
/dev/md/dsk/d0 7.8G 2.2G 5.5G 30% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 5.1G 1.1M 5.1G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
/platform/SUNW,Sun-Fire-T200/lib/libc_psr/libc_psr_hwcap1.so.1
7.8G 2.2G 5.5G 30% /platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,Sun-Fire-T200/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
7.8G 2.2G 5.5G 30% /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd 0K 0K 0K 0% /dev/fd
swap 1.6G 0K 1.6G 0% /tmp
swap 5.1G 32K 5.1G 1% /var/run
# metastat
d1: Mirror
Submirror 0: d11
State: Okay
Submirror 1: d21
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 4194600 blocks (2.0 GB)

d11: Submirror of d1
State: Okay
Size: 4194600 blocks (2.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0d0s1 0 No Okay No


d21: Submirror of d1
State: Okay
Size: 4194600 blocks (2.0 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0d1s1 0 No Okay No


d0: Mirror
Submirror 0: d10
State: Okay
Submirror 1: d20
State: Okay
Pass: 1
Read option: roundrobin (default)
Write option: parallel (default)
Size: 16644000 blocks (7.9 GB)

d10: Submirror of d0
State: Okay
Size: 16644000 blocks (7.9 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0d0s0 0 No Okay No


d20: Submirror of d0
State: Okay
Size: 16644000 blocks (7.9 GB)
Stripe 0:
Device Start Block Dbase State Reloc Hot Spare
c0d1s0 0 No Okay No


Device Relocation Information:
Device Reloc Device ID
c0d1 No -
c0d0 No -
# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
vnet0: flags=9040843 mtu 1500 index 2
inet 192.168.1.2 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp1
ether 0:14:4f:fb:c4:ef
vnet0:1: flags=1000843 mtu 1500 index 2
inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
vnet1: flags=9040843 mtu 1500 index 3
inet 192.168.1.3 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp1
ether 0:14:4f:fb:24:b6
# uptime
10:20pm up 5 min(s), 1 user, load average: 0.02, 0.11, 0.06

So now we have a guest domain running Solaris 10 Update 3, with SVM mirrored boot drives that are really sparse files, IPMP on virtual NICs, four CPU's, 4GB's RAM, etc. Pretty kewl, eh?

In the next post I'll talk about virtual devices in more detail and go from there!

Disclaimer: This post only covers the Early Access RC3 of the LDOM technology, which is in pre-release.