We recently helped IBM with a long-time customer who was in process of transitioning to the latest generation of IBM System x servers. The customer had just received a new IBM System x3850 X5 and was surprised to find that it included a blue I/O adapter. They hadn’t ordered any I/O options, so they were a bit confused as to how the card ended up in their server, and why it was blue. Don’t get me wrong: in the grand scheme of things, this was a good problem to have. I would much rather have a customer receive our product and ask what it is than never know about it and not buy the right server as a result. The thing that keeps me up at night is the customers who could be getting a lot of benefit from our products but don’t know about the great technology we have available. (It is 5 a.m. as I type this. I am not an early riser.)

The blue adapter in question was the Emulex 10GbE Integrated Virtual Fabric Adapter (VFA). The Integrated VFA is included with most models of the IBM System x3850 X5 and is available for integration into several other System x and BladeCenter servers as well. The Emulex Integrated VFA is a custom PCIe card with a dedicated slot and delivers all of the benefits of Emulex 10GbE technology to IBM’s new servers. We are very excited about it and we’ve been putting a lot of effort into making customers aware that it is included and the great features that it has. As I illustrated above, I’ve recently discovered we need to put more effort into customer awareness.

Clearly, this design is unique. Server vendors generally integrate their standard I/O technology directly on the motherboard rather than in a custom integrated card. IBM chose to do things a little differently for a simple reason. IBM knows that 10GbE is soon to become the de facto standard in I/O networking, but they also know it is still ramping and that most customers today still use 1GbE. The integrated card gives IBM a lot of design flexibility with this generation of servers. They can choose where to incorporate 10GbE and they can choose when, without resorting to redesigning the motherboard. If you look at IBM’s server platforms today, you will see that the more powerful the server, the more likely it will have integrated 10GbE. At some point in the future, as 10GbE adoption takes off, IBM will have an easy way to transition more and more of their existing models to include low-cost 10GbE integrated into the system.

This is a smart move on IBM’s part, but it does beg the question of why the interest in 10gbE and what makes IBM and others in the industry believe that widespread adoption of it is imminent. To answer that question, Steven Hill and Tim Dales took an in-depth look at the 10GbE market and the new IBM systems in their industry brief, “Paving the New Data Center Autobahn.” They do a great job of analyzing the industry transition to 10GbE, and how IBM and Emulex fit into it. I hope that, among other things, this paper will serve as a great way to build awareness on our solution.

It is this concept that turns 10GbE infrastructures from nice to have to essential for today’s data centers. This is a single adapter that can support all networking and storage traffic and do it in a way that is flexible to the needs of the administrators of each system. The ability to converge onto a single I/O infrastructure provides the capability to reduce costs, reduce complexity and improve performance. Is there anyone out there who does not want to get better performance in a simpler configuration at lower cost?

What is it about Emulex Virtual Fabric Adapters that reduces costs, reduces complexity and improves performance? Emulex VFAs are part of IBM’s Virtual Fabric Solution, and it is this solution that delivers all of those benefits. Let’s take a look at each of these.

Reduces costs. Very simply, by implementing the IBM Virtual Fabric solution, organizations will purchase a lot less equipment than they would for a traditional 1GbE Infrastructure. Think about the I/O recommendations for a typical VMware environment, as an example. VMware recommends six Ethernet ports and two storage ports. In order to achieve that in an IBM BladeCenter environment, each blade server would have to be fully populated with two I/O adapters and the chassis would need up to eight switch modules to support the VMware infrastructure. With the Virtual Fabric solution, each blade server only needs the Emulex VFA and the chassis only needs two BNT Virtual Fabric switches. The Virtual Fabric implementation is up to 49% less expensive, even though it is 10GbE-based rather than 1GbE-based.

Reduces complexity. Using the same example as the last paragraph, an IBM BladeCenter chassis would have half the number of adapters and 75% less switches. There are a lot less parts to manage, which makes the administrator’s job a lot easier.

Improves performance. Remember that Virtual Fabric allows you to provision bandwidth into up to four vNICs per channel, or eight per adapter. As a 10GbE infrastructure, this averages out to 2.5Gb/s per channel when using all eight vNICs. As I discussed in part one, bandwidth can be allocated, but is needed in 100Mb/s increments from 100Mb/s up to the maximum 10Gb/s bandwidth of the adapter. Even better, this bandwidth can be allocated dynamically and scheduled without the need for a reboot. Consider the possibility of allocated 5Gb/s to the vNIC handling e-mail traffic during the work day, with other vNIC splitting the remaining bandwidth as needs dictate. After hours, when the need for e-mail bandwidth goes down, most of that bandwidth can be allocated to ports running another essential process such as backup. Virtual Fabric opens up a whole new world of possibilities for administrators.

Emulex 10GbE Virtual Fabric Adapters provide a single source for all I/O needs with better performance for lower cost and less complexity. With all of these advantages, why would anyone want to continue using the more expensive, feature-limited 1GbE infrastructures that they are eventually going to need to replace some day anyway? Myth busted.

We busted this myth last week in San Francisco at a fantastic eX5 launch event with special guests Jamie Hyneman and Adam Savage, stars of Discovery Channel’s Mythbusters. Jamie and Adam have proven their ability to solve hundreds of urban legends and other commonly held beliefs, but in this case we were lucky to have them just for entertainment purposes; and they did not disappoint. But the real myth busting that day was being done by Emulex and IBM.

IBM got us started by busting a myth about memory capacity by introducing their groundbreaking Max5 technology. This would prove to be a key point in Emulex’s own myth busting exercise a bit later. Max5 allows IBM System x and BladeCenter servers to scale memory to up to twice what other x86 systems can offer. We learned that the new Intel processors have so much performance that a typical x86 will run out of memory before maximizing the processor’s capabilities. With Max5, IBM system x and BladeCenter servers can get higher utilization from the processor, giving that server the ability to run more VMs and more applications than other servers in the same class. This can result in lower costs for software licenses as well as the hardware itself.

With the memory capacity myth officially busted, Emulex took on the myth that 10GbE Infrastructure is unnecessary. We had already learned that IBM’s new eX5 servers delivered a huge (or, as my father-in-law would say, “yooj”) increase in processing capability. But where was all that newfound horsepower supposed to go? A server is pretty useless if it doesn’t have the ability to serve users on the network. As system performance and utilization increases to never before seen levels, network traffic increases proportionally. Yesterday’s I/O cannot handle today’s bandwidth requirements. Enter Emulex 10GbE Virtual Fabric Adapters for IBM System x and BladeCenter. Yes, the Emulex VFA is a line-rate 10Gb/s Ethernet adapter with TCP Offload, but there are two features that make Emulex VFAs truly unique and essential for today’s datacenter deployments. I’ll talk about the first one today, and the second one in a future entry. The first is virtual NIC, or vNIC. vNIC is the ability for a single dual-port VFA to function as 8 separate NICs to the operating system or hypervisor.

Think about this for a minute. Prior to the VFA, you basically had 2 options when you needed more I/O bandwidth for the server. You could add more 1GbE ports, or you could upgrade to a 10GbE Infrastructure. Each of these had drawbacks. More 1GbE ports meant more cables, more switches, and more parts to manage, and you’re still operating at 1GbE so you’re still bandwidth constrained. Everyone is going to have to eventually switch to 10GbE or greater. But do every VM and every server need full 10GbE bandwidth? Probably not. With straight 10GbE, you could change all the 1GbE ports to 10GbE and solve the bandwidth issue but do nothing to fix the complexity issue; or you could try to get by with fewer 10GbE ports and sacrifice Quality of Service. Emulex VFAs for IBM System x and BladeCenter offer a better solution. Every physical port delivers 4 separate vNIC ports to the system; ports that do everything a separate physical connection can do but they do it all from a single adapter and a single 10GbE cable. Bandwidth of the vNIC ports can be tailored from as little as 100Mb/s to up to 10Gb/s (up to 10Gb/s per physical port) so it is easy to provision bandwidth to exactly what the applications requires. The vNIC capability of Emulex VFAs are what truly make them essential for any deployment using the newest IBM system x servers.

In our next edition, we’ll talk about the second unique feature of Emulex Virtual Fabric Adapters and bust the 10GbE myth once and for all.

At least, that’s what I thought.  Then I started having people react to our solution and tell me how ‘nice’ it is we actually did something as simple as NIC Teaming.  So I wanted to list out the functionality we offer on our 10Gb NIC part of the solution.

If you look at the traditional NIC function that has been in any server platform now for almost ten years there are a list of basic features you expect to have to ‘just work’  they can include:

• NIC Teaming
• VLAN Support
• Wake on LAN Support
• Serial over LAN support (IBM BladeCenter Feature)
• Software iSCSI Initiator and iSCSI Software Boot Support
• Preboot Execution Envrionment (PXE) support

The OneConnect platform, whether it be mezzanine card or PCIe based, offers this functionality on its NIC ports, whether physical or virtual, out of the box. In addition to that functionality, where we go above and beyond is at the hardware layer we also provide TCP Offload Engine support.   With all of these technologies enabled we can still drive over 3.6m packets per second in an ethernet environment.

Moving forward with an entitlement you can easily turn on FCoE or iSCSI HBA support within the adapter as well.  The upgrade process is a simple flash update that can be applied remotely or locally on the systems using our OneCommand tools.

So back in June, QLogic asked a question about whether or not they thought encryption of data at the Fibre Channel adapter level was necessary. What interested me most about this was how they said you should buy something they don’t make in order to get the best deal. It seems counterintuitive to me for a company who makes storage networking products to say ‘this sucks, go buy it from someone else’.

Even stranger to me was that this technology they recommended wasn’t deemed ‘good enough’ by a number of customers I have met and talked about security options with storage area networks.Now why would QLogic tell people out of the blue that something they don’t make is a bad idea? Why would anybody buy products from a company that doesn’t understand their needs? Wouldn’t their partners be concerned that they may be at a competitive disadvantage because QLogic doesn’t provide solutions that customers want?

Most likely they felt the need to post this because they heard through the grapevine about a new technology we are working on, Emulex Secure.

At Emulex, we understand the importance of security in your data center. Introducing Emulex Secure, the first 8Gb/s Encryption HBA. We demonstrated this adapter at the RSA Conference in Europe.

The business case for customers with security issues is extremely strong for the host-based security solutions. With this type of technology, you don’t need an expensive security infrastructure, or devices that specifically are built for secure purposes. You can use a commercial off-the-shelf (COTS) solution where you only secure the nodes that need securing, and no longer have to manage destruction of failed devices to adhere to any governmental regulation. So if only 10 hosts out of 1,000 need to be secured, you only have to secure them at the adapter level to have both in flight and at rest encryption.

For more details and to learn a little about Emulex Secure, you can read more here.

It appears that the appropriate sacrifices were made to the gods of computer shows and our demonstration of the eVFA adapter on the IBM BladeCenter at Storage Networking World: Europe.

Unfortunately, due to my need to be in the States to help finish up our final testing before general availability, I was unable to attend in person. Instead, I helped walk the team over the phone through enabling the adapter & getting VMware up and running.

I’ve been following vicariously through people that have gotten updates and posted their thoughts. One author, Nigel Poulton, was lucky enough to gain some nice pictures of the HS22 with the eVFA card installed in his write-up, and he touches on something I wanted to expand on a little bit in this quote:

NOTE: Of particular interest to me was the fact that the core features, as well as the base cost, of this adapter are 10Gbps Ethernet. This is very interesting when you consider Emulex are traditionally a Fibre Channel company. Clearly Emulex are moving with the market here and recognising Ethernet as the dominant technology and building on that. Emulex also have people on IEEE 802.1 committees such as DCB. Now that’s what I call not betting against Ethernet.

I love it when a plan comes together and people ‘get’ it. This has been something I’ve been trying to communicate to the technical community within IBM since I’ve come on board.

As with most new product announcements, some details take time to permeate in the industry before they are fully understood. There will be questions, and change can take quite some time (See: It is not an Inflection Point…it is an Inflection Process).

I was around and working in datacenters when Fibre Channel hit the world back in the late 90s. While we don’t have the problem now of ‘it’s expensive’ we do have the concerns of how will I manage? Will it do everything I need? and lots of other questions that aren’t completely answered yet. I’ve traded comments on Twitter with Cisco’s Brad Hedlund about how I think Multi-Hop CEE/DCE is that last step and how multiple FCFs (Fibre Channel Forwarders) are my two ‘must have’ features. What I mean by that is if you look at this white paper from Cisco you will notice that they have two separate and redundant networks. The reason for that is right now you can only have one device in your FCoE network that forwards traffic to FCoE. Now, granted, it’ll take a large network to require more than one FCF today but if you look at some SAN infrastructures that are out there they would necessitate multiple isolated networks to completely rip & replace their FC infrastructure.

I’ve also had a bit of time to bounce some comments to Dell’s Jeff Sullivan over a blog post that came out comparing FCoE to iSCSI. After some clarification from the authors, I see where they are headed, but I do think the idea is fundamentally flawed in that FC in some way had to be forced into the OSI model and that because of that there might be some concerns. I do agree iSCSI is more mature than FCoE and probably why they are seeing success with it. But you also have a complete end to end solution maturity here. FCoE while has had some successes with announcements around products in the adapter, switch, and storage parts of the market it’s not nearly as ubiquitous as FC and iSCSI are. Slight plug here, but my OneConnect OCe10102-I card is the only adapter that is hardware offload is the only uCNA that does that, and extremely well I might add.

That post is a must read too, it has some great points though about the state of the FCoE space. The ‘all’ part isn’t there 100%. Which is why a move to 10Gb now for data traffic, and migrate to FCoE for storage traffic is actually resonating so well with a lot of our customers. I’m aware of large opportunities that are buying both an FC card and the eVFA card already in the pipeline. So while going entirely to FCoE does work for some, there are a lot of more conservative companies that will put the stuff in place for a migration, but aren’t ready for it today.

So those are my thoughts for the month of October. Now that things are wrapping up and the eVFA card is becoming readily available in a short amount of time. I’m will continue my planned discussion about the eVFA in future posts.

vNIC is a feature of the adapter to present multiple adapters to the operating system through the PCI Function ID of the adapter.   When an operating system scans the adapter and the vNIC function is enabled through the UEFI BIOS (which is on by default) the operating system will see up to eight PCI function IDs with eight unique MAC addresses.   The PCI Functions are numbered 0 – 7 and are always enumerated with functions 0,2,4,6 being assigned to Port 0 as VNIC 1 – 4, and then 1,3,5,7 being assigned to Port 1 as VNIC 1-4.

In the initial launch, you will have some basic configuration options at the card level. However, primarily you will be configuring your vNIC solution from the switch as the switch controls the virtual groups and the bandwidth assigned to each virtual NIC.   The switch in this solution is provided to IBM by Blade Network Technologies.   It’s their BNT 10-Port 10Gb Ethernet Switch Module.  They have some detailed information similar to our pages on the IBM BladeCenter Virtual Fabric solution along with a solution brief.

The BNT switch also provides 1Gb or 10Gb connections just like the eVFA adapter does.   Giving you the ability to start with 1Gb network and grow into 10Gb.   You could easily provide 10Gb of bandwidth to the blades and between your IBM BladeCenters and still connect to the corporate network at 1Gb.

We will be working on a IBM Redbook that will be available when the solution ships later in October.  If you have any scenarios you’d like to see discussed in the RedBook feel free to give me some ideas either here or you can e-mail me as always.

I wanted to start at the easy side, and work my way into the more complex features.   So the easiest side of this is that at it’s core, the eVFA (my short name for this) is the base function of the adapter.  At it’s core, this adapter is a two port 10Gb Ethernet adapter with full offload functionality.   So let’s take a look at that function first.

As we’ve previously announced back in May, Emulex chose to partner with ServerEngines to deliver converged networking solutions.   This is the first adapter to come out of that partnerships.   The ASIC used is Emulex’s ‘Tigershark’. 

The core features of this NIC is that it is a PCI Gen2 interface, and a dual port 10Gb ethernet out of the ASIC.   The adapter also has the NCSI support for WoL an SoL so you can provide full services via this adapter for those protocols as well.  Some key things to keep in mind is that this is a PCI Gen2 device and to get the full bandwidth out of it you do need a PCI Gen2 aware blade.  Currently that is IBM’s BladeCenter HS22.   At launch this will be the only server supported however since Gen2 is backwards compatible with Gen1 in theory it would work, albeit not at full performance capability due to the bandwidth limitations of the Gen1 slot in older blades.

As a standard NIC the eVFA adapter has full TOE offload.   This is mostly exploited using the Windows Chimney Stack AKA The MS Windows 2003 Scalable Networking Pack or a default feature of Windows 2008.   These allow the adapter to do most of the heavy lifting of packetizing and de-packetizing the data.   The end result is lower CPU utilization & PCI Bus traffic for network traffic.   This is especially helpful for heavy network traffic solutions.  Details of how it’s enabled on Windows 2003/2008 can be found here.  It is enabled by default on Windows 2008 Server R2.  If you’re using Linux it’s currently not a supported option.   We have no plans to bring anything custom out for linux so as long as TOE is viewed as non-viable by the kernel developers we will not offer any support for it.  Because of that, the idea of TOE enabled adapters is not that exciting for linux servers.  VMware is schedule to bring Asynchronous support for 10Gb iSCSI offloads about the time the iSCSI software key will be available so we should line up nicely for that functionality.

What is a Cloud? It’s Cloud Computing that a lot of vendors have jumped onto whole heartedly, and with good reason, as this is where a lot of growth potential can come from.

But for the nerds on the ground, what makes cloud computing any different from a hardware perspective than the computing we’ve been doing for the last ten years, or longer?

In my sarcastic ‘been there done that’ view of the world, I say not a whole lot.Now certainly there is a lot of new software layers being implemented here with all sorts of fancy new features like provisioning and a host of other features, but at the end of the day, there is still a computer with a processor, memory, and I/O doing something. What becomes more important now is how hard or how easy it is to remotely manage this computer. This is where hardware makers are going to have to improve & implement new designs that allow for as much remote access as possible, because one of the key features of Cloud Computing is that the source of the cloud is not where you are.

From the perspective of hardware management there is a few areas where Emulex based technology will be used to work ‘in the clouds’ However one of the key features of cloud computing where I think Emulex’s value add will be strongest is in our remote management capabilities using HBAnyware, or it’s forthcoming converged management tool of OneCommand.

While stand alone is great, so is remote scripting capabilities to allow you to integrate into larger class management practices. This is something HBAnyware/OneCommand has excelled at for years with such tools as PowerShell in Windows or through your standard scripting capabilities in the *nix operating systems.

So while the marketing guys are dancing around talking about clouds and what not, keep in mind that the remote capabilities of the components in the servers – just like always – are important.

IBM today announced the Emulex Virtual Fabric Adapter for IBM BladeCenter. This adapter in conjunction with the Blade Network Technologies 10-port 10G Ethernet Switch Module introduces IBM BladeCenter’s new Virtual Fabric Architecture. The eVFA Adapter is based on Emulex’s OneConnect family of Universal Converged Network Adapters (uCNA)

With the introduction of these products, IBM enters the Virtual NIC environment. Based on ethernet standards that are available the new eVFA adapter provides the ability to take a standard dual port multi-protocol adapter and carve it into up to eight independent networking adapters.

This is regardless of the operating system on top of it, and is in no way tied to any virtualization software. You can use this technology on Microsoft Windows Server, Red Hat Enterprise Linux, SUSE Enterprise Linux, and VMware. So any high I/O environment you have can benefit from this solution allowing you to bring to bear enough I/O to keep Intel’s Xeon 5500 based BladeCenter HS22.

Next week at VMworld 2009 I, along with a few others, will be demonstrating this technology in the IBM Booth. So if you are able to attend VMworld I highly recommend you stop by. We will be demonstrating one of VMware’s best practice designs for a VMotion based solution using the eVFA adapter.

This is just the first step in convergence. As I’ve rambled on about before 10Gb is the base to the converged network, and building on that moving forward is the goal. The eVFA Adapter will have licensed upgrades to enable either iSCSI or FCoE protocols as well due in the first half of 2010. It will be a software download that you apply to the adapter, so there will be no need to replace any hardware if you are migrating to FCoE or iSCSI based technologies.

In conjunction with a software license for the BNT 10-Port switch you will be able to seamlessly migrate into the world of FCoE when you are ready.

However if you don’t plan on doing that any time soon, you can always add the Emulex LPe1205V-CIOV adapter for two 8Gb ports of ‘regular old’ Fibre Channel, giving you eight ethernet, and two fibre channel ports per HS22.

My goal over the next few weeks is to highlight the various components of this solution and do a bit of a ‘deep dive’ in the technology behind them to provide some education to those of you who like to play along at home.