ARTICLE POSTED APRIL 9, 2001
InfiniBand I/O standard will enhance storage networking
By Robert Williams
The demand for greater speed across network connections has been restricted by a number of bottlenecks. Arguably, the greatest restriction to data flow is the nature of the I/O architecture. While data can buzz around the world with astonishing speed, it must still have a point of orientation and delivery. In many cases, an organization may have tremendous amounts of bandwidth, yet have systems with extremely slow and primitive I/O buses.
InfiniBand technology addresses the shortfall of current I/O buses. The performance gains could range from 500M bits/sec. to 6G bits/sec. per link. The implications for storage networking are enormous.
The InfiniBand Trade Association (IBTA) developed and published the specification for the InfiniBand architecture in the fall of 2000. The development of this specification had industry-wide support, with over 215 companies participating. The new point-to-point linking technology is expected to be transformed into a variety of interconnect and computer server commercial products by the end of this year. Companies like Intel, for example, already have InfiniBand prototype devices in development and have established dedicated interoperability labs.
InfiniBand enables increasing data flow rates on servers, storage devices and interconnect devices through the I/O fabric. The term fabric is used in order to illustrate the nature of the architecture that relies on communication strands or links. As with any fabric, these strands can be interwoven to provide different patterns of data flow. In theory, the I/O throughput with InfiniBand technology should be increased initially by two to four times. The simple design offers increased system performance, enhanced reliability, and greater availability. It also provides independent scalability of fabric elements.
Cram course on InfiniBand Architecture Technology
A fabric of InfiniBand switches and links is used to provide connections between servers, remote storage and networking devices.
The InfiniBand fabric will both coexist with, and replace, existing I/O technology standards. While "standards" such as Ethernet, Fibre Channel and Peripheral Component Interconnect (PCI) have existed side by side and continue to evolve, none of them effectively provide true interoperability or manageability. The InfiniBand architecture can theoretically provide operating system and hardware platform interoperability while improving system configurations.
The key to InfiniBand is switching technology that links high-channel adapters (HCA) to target channel adapters (TCA). The switch operates between the HCA and TCA to manage and direct data packets. Architecturally, the high-channel adapter typically resides near the CPU and memory. In contrast, the target channel adapter supports the storage and peripheral device I/O.
InfiniBand switching technology offers manageability that does not exist in traditional buses. With InfiniBand, it is possible to utilize variables on data packets that include service levels and other identifiers. And through the InfiniBand fabric, it is also possible to configure InfiniBand links between multiple tiers of servers. The objective is to more appropriately allocate the resources of the entire system and connected devices.
InfiniBand was created to dramatically increase the speeds of n-tier architectures. InfiniBand links will transcend connections between first-tier Web servers, second-tier application servers and third-tier database servers. The multiple data strands that comprise the InfiniBand fabric can be combined in 1x, 4x and 12x strands in order to achieve higher-speed interconnects. This technology will be particularly valuable when used in clustering environments for classic failover, workload adjustments,and general manageability. Each of the three levels of fabric strands will be applied to specific technology needs:
- The 1x link will most likely be used for simple connectivity. For example, front-end devices like Web hosting, file or print services can effectively utilize the throughput of a single 1x link.
- The two-tier computing environment could find the 4x link particularly beneficial. On this level, it is common to find multiple processor systems on which workload balancing is an underlying requirement. Therefore, the ability to configure the data flow accordingly could dramatically impact overall performance.
- The 12x link will be best used for third-tier and n-tier environments. Latency is always a concern in heavy transactional environments and cannot be tolerated. The InfiniBand architecture helps eliminate data flow latency by virtual of the availability of multiple link strands.
InfiniBand replaces shared bus architectures. As a result, I/O can be removed from the server chassis itself. This will enable significantly greater server chassis design density. Moreover, it will result in less physical space being needed to host more computing power. Freed from strictly confining shared I/O buses, data centers can be constructed with more scalable and flexible infrastructure configuration.
Driving forces behind InfiniBand technology adoption
InfiniBand technology connects servers with remote storage and networking devices, and other servers. In addition, it can be used inside servers for inter-processor communication (IPC) in parallel clusters. Internet Service Providers (ISPs) and other enterprise users that require dense server deployments will benefit from the small form factors of InfiniBand devices. Industry supporters of InfiniBand believe that many other potential benefits exist, including enhanced performance, reduced system latency, streamlined data sharing, built-in security and quality of service and advanced usability features.
The problem InfiniBand is resolving involves mismatched technologies combined with lagging, outmoded components. With processors eclipsing 1GHz, and server farms scaling as needs require, speed is being achieved safely in peer environments. However, when these environments have to push data outside their confines, the issue of interoperability immediately raises it ugly head. Dealing with mixed equipment, operating systems, storage methodologies and data streams all play a part in the current slowdown.
Revolutionizing Internet data centers
As a new server I/O technology specification, InfiniBand is expected to revolutionize Internet data centers. It simplifies and expedites server-to-server connections. It also makes links to other server-related systems such as remote storage and networking devices less cumbersome.
The InfiniBand specification is designed to overcome current interoperability limitations. In so doing, InfiniBand will address traditional bandwidth hogs like streaming video, graphics and multimedia bandwidth like any other data transmission.
InfiniBand positively overcomes hardware bottlenecks. The speed of server virtual memory and storage memory communications has always been an issue. The bus technology in servers simply has not been up to the task. InfiniBand link-based technology will reduce this problem.
An interesting and possibly major advance made possible by InfiniBand is the innovation of remote clustering. Scalable and highly available clusters could conceivably exist through the interconnection of servers in remote locations. The current requirement to co-locate all clustered servers could be eliminated.
Another important factor in the adoption of InfiniBand technology is cost. According to Framingham, Mass.-based, International Data Corporation (IDC), cost will play a significant role in promoting InfiniBand adoption. IDC analysts expect that InfiniBand will drive down the cost of servers, interconnects, bridges and switches. Users of these products will need to refocus on providing highly available solutions that take full advantage of this new system architecture.
SAN implications
The potential for InfiniBand to transform computing is tremendous. The first significant change is that the CPU and memory can be separated from storage. This will permit the individual management of application loading on the CPU, memory and storage. The system will no longer be a monolithic beast, but a highly configurable set of components in which data flow can move with appropriately defined speed.
The initial use of InfiniBand will probably be traditional and restricted to bus improvement within the same server. However, this should rapidly evolve into clustering environments and first-tier Web server applications. The next likely utilization will occur with storage area networks (SANs) where data retrieval is critical. The next wave will include radical new designs of computer systems that embrace more specialized functionality and are not constrained by the current "everything-in-the-same-box" architecture.
IDC view of InfiniBand
IDC is bullish on the future of InfiniBand. They have defined a number of markets in which InfiniBand should have the greatest impact. In their 2001 white paper entitled, InfiniBand Architecture: The industry's choice for I/O Architecture, IDC envisions the following computing marketplace activity:
- General Purpose Entry Server Market - IDC foresees a substantial adoption of InfiniBand for general purpose server utilization. As a segment that constitutes 90% of all server units, it is believed that migration can occur to InfiniBand without major impacts on existing infrastructures.
- Application Server Market - A rapidly growing segment of the server market (projected by IDC to top $11 billion by 2004) includes special purpose servers that are usually pre-configured and pre-installed. Since these systems have narrowly defined computing purposes, they are ideal for factory-based InfiniBand installation. Arrays of systems could be installed using InfiniBand fabric.
- RISC Server Market - Vendors of proprietary RISC systems will probably be slow to adopt InfiniBand technology. IDC believes that these vendors are at least in part locked into the ASIC-controlled subsystems that are "burned into" silicon. When adoption does take place, look for it on the lower end of the RISC based offerings first.
- Server Provider Adoption - The emerging service provider segment should be among the earliest wide-scale adopters of InfiniBand. In part, this will occur because service providers do not have significant legacy system investments. They will also benefit for the flexibility, dense server designs, performances and cost of ownership that is promised with the InfiniBand technology fabric.
The role of the InfiniBand Trade Association
The IBTA was created to ensure that industry I/O standards could address costly computing environment bottlenecks. The IBTA has three primary objectives:
- Plan and develop comprehensive specifications that meet current and future requirements.
- Draw on existing technology to advance the concept of peer-to-peer interconnection
- Utilize its governance authority to balance the development process in an open and fair manner.
The goals of the IBTA are broader than mere administrative objectives. Ultimately, the group hopes the resulting technical specification will result in a number of solid benefits for the computing and networking industry. These include:
- InfiniBand technology will be initially used to connect servers with remote storage networking devices and with other peer servers. The technology will also be used within servers for Internet protocol communication (IPC) in parallel clusters. This will benefit organizations such as ISPs.
- Greater performance, reduced latency, enhanced built-in security and improved quality of service
- Total cost of ownership. By addressing issues of speed, reliability and configuration, costs can be lowered and performance increased.
- Scalability to increase in two areas. First, the fabric is designed to eliminate latency in I/O transmission. Second, the physical modularity will result in a reduced need to purchase extra capacity upfront. Scaling can occur in what the IBTA calls a "pay-as-they-grow" approach.
Postscript
New software and hardware industry standards appear so often that it is difficult to evaluate them all. Many standards die before giving birth to commercial products. InfiniBand will likely not fall into that category. In fact, its high level acceptance seems already assured by endorsements from Compaq, Dell, Hewlett Packard, Microsoft and Sun Microsystems. The technology appears to have both technological and economic merits.
Challenges still lie ahead for InfiniBand. This is one of many I/O industry standards. The investment already made in bus architectures is tremendous, and therefore resistance to change could be significant. Regardless of the technical value-add associated with InfiniBand, change takes time and legacy environments will not quickly be altered.
InfiniBand is an important technology. However, its commercial success is still not clear.
Robert Williams is CEO of Enterprise Certified Corporation and the co-author of #1 Sys Admin best seller The Ultimate Windows 2000 System Administrator's Guide (see our recent review of the book) and Windows NT and UNIX: Administration, Co-existence, Integration and Migration.
Feedback: feedback@snwonline.com
Questions: questions@snwonline.com
Technical difficulties: webmaster@snwonline.com
| 
