Many enterprise data centers rely on Ethernet for their LAN and data traffic, and on Fibre Channel (FC) networks for their storage infrastructure. With increased adoption of 10GbE in the data center, the availability of Fibre Channel over Ethernet (FCoE), and new lossless 10GbE technologies, it is now possible to consolidate FC SAN storage data flows with LAN and data traffic onto the same unified Ethernet cable. Network convergence will enable enterprises to preserve their existing investments in FC storage, reduce data center costs and complexity, and simplify network management.
Although the benefits of using FCoE are compelling, many companies are still waiting to deploy the technology. This article provides a summary of FCoE and addresses questions about the technology. It concludes with information on how enterprises can make the move to FCoE using a gradual, phased approach.
The majority of today’s data centers maintain multiple networks for separate purposes. Most use:
- Ethernet for their LANs to transfer small amounts of information across short or long distances or in clustered computing environments. Ethernet provides a cost-effective and efficient way to support a variety of data types, including corporate LANs, Voice over IP telephony, and storage with NFS, CIFS, and iSCSI.
- Fibre Channel is used for SANs to provide access to block I/O for applications such as booting over SANs, mail servers, and large data-intensive databases. FC SANs provide storage consolidation, centralized storage management, high performance, reliability, and business continuance.
Ethernet/IP data networks and FC SANs each fill an essential role in the data center, but they are very different in design and functionality. The two networks have their own security constraints and traffic patterns and utilize separate management toolsets. As a result, each network must be built and maintained on its own dedicated, isolated infrastructure, requiring separate cabling and network interfaces on each server.
Managing two separate networks adds complexity and costs to the data center. Enterprises are now looking for new ways to converge their IP and SAN networks to enable the data center to run more efficiently and cost-effectively, while preserving investments in the FC infrastructure.
Fibre Channel over Ethernet
FCoE enables organizations to transport LAN and FC SAN traffic on a single, unified Ethernet cable. The converged network can support LAN and SAN data types, reducing equipment and cabling in the data center while simultaneously lowering the power and cooling load associated with that equipment. There are also fewer support points when converging to a unified network, which helps reduce the management burden.
FCoE is enabled by an enhanced 10GbE technology commonly referred to as data center bridging (DCB) or Converged Enhanced Ethernet (CEE). Tunneling protocols, such as FCiP and iFCP, use IP to transmit FC traffic over long distances, but FCoE is a layer-2 encapsulation protocol that uses Ethernet physical transport to transmit FC data. Recent advances and planned enhancements in the Ethernet standard, such as the ability to provide lossless fabric characteristics over a 10GbE link, are what enable FCoE.
FCoE components
Some of the components needed to implement FCoE in the data center include:
- Converged network adapters (CNAs). These combine the functionality of Ethernet NICs and Fibre Channel HBAs in an FCoE environment, enabling users to reduce the number of server adapters they need to buy, reduce the port count, and decrease the number of cables required.
- FCoE cables. There are currently two options for FCoE cables: the optical cabling generally found in FC SANs and a new type of Twinax copper cabling. FCoE twin cables require less power and are less expensive, but because their distance is limited to fewer than 10 meters, users will likely use optical cabling to go from the top-of-rack switches to the LAN.
- FCoE switches. Users can’t go to FCoE directly from their servers to their storage arrays, so they must buy switches that support FCoE/DCB, which connect servers to corporate LANs, SANs, or FCoE storage systems. For early adopters, that means top-of-rack switches or end-of-row blades where possible.
- FCoE/DCB storage systems. These are storage systems that support FCoE and converged traffic natively. There are also storage systems that support Fibre Channel to an FCoE switch and FCoE from the switch to the host servers.
FCoE requires minimal changes to existing IT infrastructure in enterprise data centers. It is a natural evolution of FC network, service, and protocol technology, designed to carry data over Ethernet physical and data link layers.
In traditional data center environments, the storage group owns and operates the FC SAN and the networking group owns the Ethernet LAN. Since the two groups have been historically separate, introducing FCoE into the data center may bring with it possible changes to some IT practices.
Cultural, political, organizational, and behavioral concerns in data center and provisioning paradigms can present obstacles to FCoE adoption. Some new business processes and procedures may need to be adopted to ensure that proper control mechanisms are in place for FCoE networks. Purchasing patterns may also have to be modified and the reliability of what were traditionally Ethernet networks will have to be increased.
With the convergence of FC and Ethernet on FCoE, these two traditionally separate network realms overlap. Implementing FCoE requires little if any additional IT training. FCoE leverages the existing IT expertise and skill sets of the enterprise’s IP data and FC teams. Role-based management features in management applications allow the FC group to continue owning and operating the SAN, and the IP networking group to continue owning and operating the data network.
Where to deploy FCoE
FCoE convergence does not require a “rip and replace” upgrade in the data center and is not a disruptive process. Moving to FCoE can be conducted in a gradual, phased approach. Early FCoE deployments will take place mostly as part of new server deployments in Windows and Linux environments where virtualized tier-3 and some tier-2 applications are deployed.
Considering that FCoE is a relatively new technology, initial FCoE deployment will be best suited for access-layer server I/O consolidation. As storage traffic requires the new lossless Ethernet, the 10GbE transport requires Link Layer (L2) multi-pathing and multi-hop capabilities. Such features are currently under development, and should become available later in 2010. These capabilities will enable the deployment of larger FCoE networks, which will expand the reach of FCoE beyond access-layer server connectivity and I/O consolidation.
Best practices for determining where to deploy FCoE in the enterprise include:
Environments that already have an FC skill base and infrastructure.
- “Greenfield” deployments, where new infrastructure is being introduced to accommodate data growth.
- Many enterprises will begin the transition to FCoE in tier-3 or tier-2 applications; with the experience gained in labs or other less mission-critical environments, knowledge may then be applied to tier-2 and, in some instances, tier-1 applications.
- Enterprises should start implementing FCoE on the access-layer server I/O consolidation side, and that step may be combined with native FCoE storage deployment. Extending FCoE beyond access-layer servers should take place when multi-pathing and multi-hop standards become available.
How to begin
Migration to FCoE can be accomplished in a gradual, phased approach, typically starting at the edge or switch, then moving on to native FCoE storage, and eventually moving deeper into the corporate network.
Figure 1 (on p. 25) depicts a typical data center architecture before network convergence with FCoE. The FC SAN (illustrated by the orange line) is a parallel network requiring network ports and cabling separate from those required for the Ethernet IP LAN (illustrated by the blue line).
Phase 1: Transition to DCB/FCoE at the edge or switch
Moving to a converged or unified Ethernet infrastructure can be done gradually and will likely begin at the edge (illustrated by the green lines in Figure 2) where the greatest ROI will be realized. With FCoE convergence, port count at the servers and edge switches can be reduced by half, driving significant capital and operational cost reductions as well as improvements in management.
Phase 2: Transition to native DCB/FCoE storage systems
Move to an end-to-end DCB/FCoE solution from the host to network to native DCB/FCoE storage. FCoE and converged traffic is supported throughout the infrastructure, providing optimal savings.
Phase 3: Transition to DCB/FCoE at the core
After implementing FCoE at the edge or switch, enterprises can migrate to an all-10GbE enhanced Ethernet network at the core and then gradually move to supporting FCoE in the storage devices.
FCoE brings together two leading technologies—the Fibre Channel protocol and an enhanced Ethernet physical transport—to provide a compelling option for SAN connectivity and networking. FCoE enables IT team members to use the same tools and techniques they use today for managing both the IP and FC networks.
Companies planning data center builds or storage-network upgrades should seriously look at FCoE. By taking a phased approach to consolidating their data centers around Ethernet, they can economically build out their Ethernet infrastructure over time while protecting FC infrastructure investments.
MIKE McNAMARA is the marketing chairman of the Fibre Channel Industry Association (and a senior manager, product marketing, at NetApp) andAHMAD ZAMER is an FCIA member (and senior product marketing manager at Brocade). For more information on the FCIA, visit www.fibrechannel.org.