Configuring NIC teaming with Intel prosetcl

If you’re configuring Intel NICs in your servers instead of Broadcom, I’ve tested Intels prosetcl-utility too… Seemed unfair that just the Broadcom-running people could create teams from command line 🙂

Intel has a different approach when you install proset, because you get an additional tab on the NIC in device manager. In Server Core there is no device manager with tabs, so you can’t get to the team configuration… Enter prosetCL.exe!

Before you start using prosetcl you’ll need to install the drivers for your NICs and Intel PROSet for Windows, otherwise it won’t work. PROSet has a bunch of unattended switches, but for this exercise we’ll just use the teaming and VLAN commands. To install this we’d run “setup /qn” from the command line.

Once you’ve got it installed it’s time to create your teams! Compared to Broadcom Intel actually supports creating a team from the command line without exporting configurations to files (just a hint…).

If you want to import and export settings you can use the SavResDX.vbs script. It’s stored in the DMIX folder, the path is normally C:Program FilesIntelDMIX. To use the script you’ll need the proset installed too. If you wanna run restore you’ll need the same version of the OS as you exported from.

Export: cscript SavResDX.vbs save c:tempnicsettings.txt
Import: cscript SavResDX.vbs restore c:tempnicsettings.txt

Another nice feature is that Intel supports the creation of teams with NICs from different vendors, as long as at least one NIC is from Intel. (For certain team types, see Intel documentation for full info)

Team creation with prosetcl.exe

List all adapters (you’ll need the index numbers to create the team…)
c:prosetcl.exe Adapter_Enumerate

Create the team
c:prosetcl.exe Team_Create 1,2 Team1 SFT

AdapterList – Comma delimited list of adapter indices (i.e., 1,2,3). To obtain adapter indices, use the Adapter_Enumerate command.
TeamName – Name to apply to the team.
TeamMode – Type of team to create. Valid team modes are: ALB, AFT, SLA, 8023AD, and SFT. Note: SFT teams can only be created with 2 or fewer adapters. (see further down for more info on teams)

List teams (will give you the index of the teams, you’ll need it for vlans and stuff…)
c:prosetcl.exe Team_Enumerate

Add vlan to team
c:prosetcl.exe Team_CreateVlan 1 5

TeamIndex (1 in the example above) – Index of the team. The team indices are obtained by running the Team_Enumerate command.
VlanID (5 in the example above) – ID number of the VLAN. Valid ranges for the VlanID are 1-4094 for a maximum of 64 VLANs. Only one VLAN can be ‘Untagged’ (0) per device.

Note 1: If a VLAN ID is already in use or an untagged VLAN already exists, an
error message is displayed.

Note2: A tagged VLAN must be present before an untagged VLAN can be created.

What’s the different team types?

Adapter Fault Tolerance (AFT)
Allows mixed models and mixed connection speeds as long as there is at least one Intel® PRO server adapter in the team. A ‘failed’ Primary adapter will pass its MAC and Layer 3 address to the failover (secondary) adapter. All adapters in the team should be connected to the same hub or switch with Spanning Tree (STP) set to Off.

Switch Fault Tolerance (SFT)
Uses two adapters connected to two switches to provide a fault tolerant network connection in the event that the first adapter, its cabling or the switch fail. Only two adapters can be assigned to an SFT team. Note Do not put clients on the SFT team link partner switches, because they will not pass traffic to the partner switch at fail.
Spanning Tree (STP) must be running on the network to ensure that loops are eliminated.
STP should be turned off on the incoming ports of the switches directly connected to the adapters in the team, or these ports should be configured for PortFast.
Only 802.3ad DYNAMIC mode allows failover between teams.

Adaptive Load Balancing (ALB)
Offers increased network bandwidth by allowing transmission over 2-8 ports to multiple destination addresses, and also incorporates Adapter Fault Tolerance. Only the primary receives incoming traffic. Only the primary transmits broadcasts/multicasts and non routed protocols. The ANS software load balances transmissions, based on Destination Address, and can be used with any switch. Simultaneous transmission only occurs at multiple addresses. This mode can be connected to any switch.

Receive Load Balancing (RLB)
Offers increased network bandwidth by allowing reception over 2-8 ports from multiple addresses.
Can only be used in conjunction with ALB.
RLB is enabled by default when an RLB team is configured.
Only the adapters connected at the fastest speed will be used to load balance incoming TCP/IP traffic. The primary, regardless of speed, will receive all other RX traffic.
Can be used with any switch. Any failover will increase network latency until ARPs are re-sent. Simultaneous reception only occurs from multiple clients.
Available for Microsoft Windows.
If using NetWare*1 Load Balancing, you can use ALB but not RLB.

Virtual Machine Load Balancing (VMLB)
Provides transmit and receive traffic load balancing across Virtual Machines bound to the team interface, as well as fault tolerance in the event of switch port, cable, or adapter failure.This teaming type works with any switch.

The driver analyzes the transmit and receive load on each member adapter and balances the traffic across member adapters. In a VMLB team, each Virtual Machine is associated with one team member for its TX and RX traffic. If only one virtual NIC is bound to the team, or if Hyper-V is removed, then the VMLB team will act like an AFT team.

Note VMLB does not load balance non-routed protocols such as NetBEUI and some IPX* traffic.
Note VMLB supports from 2 to 8 ports per team.
Note You can create an VMLB team with mixed speed adapters. The load is balanced according to the lowest common denominator of adapter capabilities and the bandwidth of the channel.

IEEE 802.3ad
This standard has been implemented in two ways:

Static Link Aggregation (SLA):
Equivalent to EtherChannel/Intel’s Link Aggregation
Must be used with an 802.3ad, FEC/GEC/ or Intel Link Aggregation capable switch.

Requires 802.3ad DYNAMIC capable switches.
Active aggregators in software determine team membership between the switch and the ANS software (or between switches).
There is a maximum of 2 aggregators per server and you must choose either maximum bandwidth or maximum adapters.
Both 802.3ad modes include adapter fault tolerance and load balancing capabilities. However in DYNAMIC mode load balancing is within only one team at a time.

To see Intels documentation on the subject:

Configuring NIC teaming with Broadcom BACScli

Configuring NIC teaming in Windows Server Core or Hyper-V Server isn’t as easy as it sounds from the command line. What’s good is that BACS runs on Core if you just install .Net framework 2.0 (dism /online /enable-feature /featurename:NetFX2-ServerCore). After installation of BACS you can then start BACS from c:program filesBroadcomBACSbacs.exe. That eliminates the need for CLI and gives you a nice graphical application…

But for scripting purposes or the hardcore CLI admin this won’t do, you want the BACScli!

What kind of teams does BACS support?

Smart Load Balance and Failover

In this type of team, a standby member handles the traffic if all of the load balance members fail (a failover event). All load balance members have to fail before the standby member takes over. When one or more of the load balance members is restored (fallback), the restored team member(s) resumes the handling of the traffic. The LiveLink feature is supported for this type of team.

Link Aggregation (802.3ad)

In this type of team, you can dynamically configure the network adapters that have been selected to participate in a given team. If the link partner is not correctly configured for IEEE 802.3ad link configuration, errors are detected and noted. All adapters in the team are configured to receive packets for the same MAC address. The outbound load balancing scheme is determined by the BASP driver. The link partner of the team determines the load balancing scheme for inbound packets. In this mode, at least one of the link partners must be in active mode.

NOTE: TOE is not applicable for Link Aggregation team type. NetXtreme II network adapters with iSCSI enabled is not supported for Link Aggregation team type.

Generic Trunking (FEC/GEC)/802.3ad-Draft Static

This type of team is very similar to the link aggregation type, in that all adapters in the team must be configured to receive packets for the same MAC address. This mode does not provide link aggregation control protocol (LACP) or marker protocol support. This mode supports a variety of environments where the link partners are statically configured to support a proprietary trunking mechanism. Trunking supports load balancing and failover for both outbound and inbound traffic.

NOTE: TOE is not applicable for Generic Trunking (FEC/GEC)/802.3ad-Draft Static team type. NetXtreme II network adapters with iSCSI enabled is not supported for Generic Trunking (FEC/GEC)/802.3ad-Draft Static team type.

Disable iSCSI

As you can see above you’ll need to disable iSCSI for this type of team. To do this with BACScli, run the following:
BACScli.exe -t vbd -f mac -i 0015c5f9d79b “cfg Resource “ISCSI”=”Disable””

Note the double quotes!

SLB (Auto-Fallback Disable)

This team is identical to Smart Load Balance and Failover, with the following exception: when the standby member is active, if a primary member comes back online, the team continues using the standby member rather than switching back to the primary member. This type of team is supported only for situations in which the network cable is disconnected and reconnected to the network adapter. It is not supported for situations in which the adapter is removed/installed through Device Manager or Hot-Plug PCI. If any primary adapter assigned to a team is disabled, the team functions as a Smart Load Balancing and Failover type of team in which auto-fallback occurs. The LiveLink feature is supported for this type of team.

Creating teams

To create teams you’ll need a configuration file (or know how to write one yourself).

The answer file for a load balancing team without standby looks like this:

name: Team 1
type: 0
pnic: 03:00.0
pnic: 05:00.0

So what’s in that file?

Starting from the top:

Name: Team 1 <- The name of your team
Type: 0 <- There are three different types:
0 = Load Balacing
1 = Generic Trunk
2 = Link Aggregation

pnic: 03:00.0 <- pnic = Primary. In the examples below there’s also snic = Secondary. The numbers are the PCI-position of the NIC.

So how do you find out which PCI-position your NICs have? The command below will list all devices in your system:

BACScli -bdf

BDF is the Bus:Device.Function of the NIC. For example, if the NIC has bus number of 5, device number of 0 and function number of 1, use “-bdf 5:0.1” to select the NIC for the command to execute on.

If you have servers that you’d like to script installation and such you could export a configuration either from the GUI or with the help of BACScli: BACScli -t team “save -f BDF”

Once it’s exported you can create your team by importing the configuration:

BACScli -t team “restore”

Please note that the configuration files contains the PCI-position of your NICs so if your servers aren’t configured identically it won’t work. You could of course script it, detecting the nics and dynamically creating the configuration file. After all, it’s just text.

There are a bunch of different types of teams, below is a list of how the configuration files look if you don’t feel like doing the research:

Generic trunk without VLAN
name: Team 1
type: 1
pnic: 03:00.0
pnic: 05:00.0

Generic trunk with VLAN
name: Team 1
type: 1
pnic: 03:00.0
pnic: 05:00.0
vname: VLAN 1
vid: 1

Link aggregation with VLAN
name: Team 1
type: 2
pnic: 03:00.0
pnic: 05:00.0
vname: VLAN 1
vid: 0

Load balancing without standby

name: Team 1
type: 0
pnic: 03:00.0
pnic: 05:00.0

Load balacing with standby
name: Team 1
type: 0
pnic: 03:00.0
pnic: 05:00.0
snic: 08:00.0

Load balancing with LiveLink

name: Team 1
type: 0
retry: 5
freq: 2000
retry_freq: 1000
livelink_vid: 0
pnic: 03:00.0
pnic: 05:00.0

Link aggregation without VLAN

name: Team 1
type: 2
pnic: 03:00.0
pnic: 05:00.0

If you’re scripting this you’ll find the exit codes for BACSCLI below:

BACSCLI_OK 0 Upgrade firmware OK
BACSCLI_QUIT 1 Quit program
BACSCLI_PARAM_ERROR 2 Not correct parameters
BACSCLI_INIT_FAILED 6 Initialization failed
BACSCLI_MALLOC_ERROR 9 memory allocation error
BACSCLI_BMAPI_ERROR 10 BMAPI call returns error
BACSCLI_OS_NOT_SUPPORTED 11 OS is not supported
BACSCLI_NO_ADVANCED_PARAMS 12 No Advanced Parameter for the NIC
BACSCLI_INVALID_ADVANCED_PARAM_DETECTED 13 Invalid Advanced Parameter detected
BACSCLI_INVALID_ADVANCED_PARAM_SPECIFIED 14 Invalid Advanced Parameter specified
BACSCLI_FEATURE_NOT_SUPPORTED_FOR_NIC 16 Feature not supported for the NIC
BACSCLI_SET_ADVANCED_PARAM_ERROR 17 Failed to set the Advanced Parameter with new value
BACSCLI_SYSTEM_REBOOT 18 System Reboot required
BACSCLI_UNSUPPORT_PLATFORM 19 System platform is not supported
BACSCLI_NOT_ENOUGH_PRIVILEGE 20 Current user does not have enough privilege
BACSCLI_READ_LICENSE_FILE_ERROR 21 Error in reading license file
BACSCLI_INVALID_LICENSE_KEY 22 Invalid license key
BACSCLI_INVALID_ISCSI_PARAM_SPECIFIED 23 Invalid iSCSI Management Parameter specified
BACSCLI_INVALID_ISCSI_VALUE_SPECIFIED 24 Invalid iSCSI Management Value specified
BACSCLI_INVALID_RSC_PARAM_SPECIFIED 25 Invalid Resource Parameter specified
BACSCLI_INVALID_RSC_VALUE_SPECIFIED 26 Invalid Resource Value specified
BACSCLI_FEATURE_NOT_SUPPORTED_IN_FCFS 27 Feature not supported in FCFS mode
BACSCLI_PARAM_IS_READ_ONLY 28 This parameter can’t be modified, Administrator authority required.
BACSCLI_NULL_IP_ADDRESS 29 The current IP address is NULL.
BACSCLI_CANNOT_UNLOCK_ADAPTER 30 Failed to unlock adapter.
BACSCLI_INVALID_VALUE_SPECIFIED 31 Invalid value specified
BACSCLI_REGISTRY_ACCESS_ERROR 33 Error in accessing Registry.
BACSCLI_NOT_AN_ISCSI_BOOT_DEVICE 34 This is not an iSCSI Boot device.
BACSCLI_TEAM_DRIVER_NOT_LOAD 37 nic( %s ) driver has to be loaded to make it a member of a team.
BACSCLI_NDIS6_DRIVER_REQUIRED 38 Ndis6 driver is required for the NIC to join the Team in Windows Vista and later.
BACSCLI_INVALID_SUBNET_MASK 40 Invalid subnet mask.
BACSCLI_INVALID_CMD 41 Invalid command.
BACSCLI_NOT_YET_IMPL 1000 This functionality is not yet implemented.
BACSCLI_UNWIND 1001 Unwind to the parent processor
BACSCLI_NOT_APPLICABLE 1002 Command not applicable
BACSCLI_NO_TARGET_SEL 1003 No active target selection.
BACSCLI_INVALID_CONTEXT 1004 Not a valid context
BACSCLI_INVALID_FORMAT 1005 Invalid format selection
BACSCLI_INVALID_TARGET_ID 1006 Invalid target identifier
BACSCLI_FILE_DOES_NOT_EXIST 1007 File does not exist
BACSCLI_INVALID_TEAM_NAME 1008 Supplied team name is invalid
BACSCLI_TEAM_COMMIT_FAILED 1009 Failed to commit the team operation.
BACSCLI_TEAM_REMOVE_FAILED 1010 Failed to remove the team.
BACSCLI_CANNOT_OPEN_FILE 1011 Failed to open the file handle.
BACSCLI_ERR_CANNOT_SET_IPADDR 1012 Failed to set ip address.
BACSCLI_ERR_CANNOT_GET_NIC_PCI_INFO 1013 Failure retrieving NIC information
BACSCLI_ERR_RETRIEVE_IP_ADDR 1014 Error retrieving IP address information
BACSCLI_FAILED_GET_INFO 1015 Failed to get info from the DataContainer
BACSCLI_WRONG_OPTION_FLAG 1016 General team config file parsing error.
BACSCLI_EXCEEDMAXVLAN 1017 Only a maximum of 64 VLANs are allowed.
BACSCLI_CANNOT_CREATE_LIVE_LINK 1018 Live link support only applied to SLB team
BACSCLI_EXCEED_MAX_TARGET_IP 1019 Live link support allows up to 4 link
BACSCLI_TOO_MANY_PHY_NIC 1020 Only up to a maximum of 8 nics are allowed in a team.
BACSCLI_CANNOT_CREATE_FECGEC_8023AD 1021 Cannot create FECGEC or 802.3ad team with standby adapter.
BACSCLI_INVALID_RANGE 1023 Value is out of range.
BACSCLI_INVALID_INTERVAL 1024 Invalid probe retry frequency
BACSCLI_DUPLICATE_OPTION 1025 The same option has been specified previously.
BACSCLI_DUPLICATE_MAC_ADDRESS 1026 Duplicate adapter physical MAC address
BACSCLI_TEAM_ALREADY_EXISTS 1028 Team with the specified name already exists
BACSCLI_CANNOT_MATCH_MAC_ADDR 1030 Cannot find device using the specified MAC address.
BACSCLI_NO_TEAM_TO_CONFIG 1031 No team to configure.
BACSCLI_CANNOT_CREATE_TEAM 1032 Failure while creating team
BACSCLI_NO_LINK_FOR_IP_CFG 1033 No link is present in team to set ip address.
BACSCLI_ONE_OR_MORE_CREATE_FAILED 1034 Creation of one or more teams failed.
BACSCLI_TEAM_NO_MEMBER 1035 A team is required to have at least one valid member.
BACSCLI_NO_BRCM_NIC_IN_TEAM 1036 Team requires at least one Broadcom nic.
BACSCLI_ONLY_BROADCOM_NIC_FOR_VLAN 1037 Only Broadcom certified adapters are supported in VLAN.
BACSCLI_CANNOT_SET_IPADDR 1038 Failed to assign ip address on the virtual adapter.
BACSCLI_INVALID_CFG 1039 Invalid configuration
BACSCLI_SET_ADVANCE_PARAM_FAILED 1040 Failed to set advanced parameter
BACSCLI_INVALID_PARAMETER 1042 Invalid Parameter. Parameter is too few
BACSCLI_NIC_NOT_SUPPORTED 1043 The current NIC is not supported for this operation.
BACSCLI_SET_ASF_FAILED 1044 Failed to set the ASF Table
BACSCLI_SET_POWER_MGMT_FAILED 1045 Failed to set Power Management configuration.
BACSCLI_INVALID_TARGET_CMD 1046 command/target identifier is invalid
BACSCLI_NO_ISCSI_SESSIONS 1047 No iSCSI sessions exists on the system
BACSCLI_SET_MGMT_OBJ_FAILED 1048 Failed to set Management Object
BACSCLI_SHOW_USAGE 1049 If we want to show usage
BACSCLI_EXCEEDMAXTAGGEDVLAN 1051 Only a maximum of 63 tagged VLANs are allowed.
BACSCLI_NO_TEAM_AVALIABLE 1052 There is no team to save.
BACSCLI_CANNOT_SUSPEND_RESUME_DRIVER 1054 Failed to suspend or resume the driver.
BACSCLI_CANNOT_RESTART_DRIVER 1055 Failed to restart the driver.