Tag Archives: broadcom

Configuring NIC teaming with Broadcom BACScli

Configuring NIC teaming in Windows Server Core or Hyper-V Server isn’t as easy as it sounds from the command line. What’s good is that BACS runs on Core if you just install .Net framework 2.0 (dism /online /enable-feature /featurename:NetFX2-ServerCore). After installation of BACS you can then start BACS from c:program filesBroadcomBACSbacs.exe. That eliminates the need for CLI and gives you a nice graphical application…

But for scripting purposes or the hardcore CLI admin this won’t do, you want the BACScli!

What kind of teams does BACS support?

Smart Load Balance and Failover

In this type of team, a standby member handles the traffic if all of the load balance members fail (a failover event). All load balance members have to fail before the standby member takes over. When one or more of the load balance members is restored (fallback), the restored team member(s) resumes the handling of the traffic. The LiveLink feature is supported for this type of team.

Link Aggregation (802.3ad)

In this type of team, you can dynamically configure the network adapters that have been selected to participate in a given team. If the link partner is not correctly configured for IEEE 802.3ad link configuration, errors are detected and noted. All adapters in the team are configured to receive packets for the same MAC address. The outbound load balancing scheme is determined by the BASP driver. The link partner of the team determines the load balancing scheme for inbound packets. In this mode, at least one of the link partners must be in active mode.

NOTE: TOE is not applicable for Link Aggregation team type. NetXtreme II network adapters with iSCSI enabled is not supported for Link Aggregation team type.

Generic Trunking (FEC/GEC)/802.3ad-Draft Static

This type of team is very similar to the link aggregation type, in that all adapters in the team must be configured to receive packets for the same MAC address. This mode does not provide link aggregation control protocol (LACP) or marker protocol support. This mode supports a variety of environments where the link partners are statically configured to support a proprietary trunking mechanism. Trunking supports load balancing and failover for both outbound and inbound traffic.

NOTE: TOE is not applicable for Generic Trunking (FEC/GEC)/802.3ad-Draft Static team type. NetXtreme II network adapters with iSCSI enabled is not supported for Generic Trunking (FEC/GEC)/802.3ad-Draft Static team type.

Disable iSCSI

As you can see above you’ll need to disable iSCSI for this type of team. To do this with BACScli, run the following:
BACScli.exe -t vbd -f mac -i 0015c5f9d79b “cfg Resource “ISCSI”=”Disable””

Note the double quotes!

SLB (Auto-Fallback Disable)

This team is identical to Smart Load Balance and Failover, with the following exception: when the standby member is active, if a primary member comes back online, the team continues using the standby member rather than switching back to the primary member. This type of team is supported only for situations in which the network cable is disconnected and reconnected to the network adapter. It is not supported for situations in which the adapter is removed/installed through Device Manager or Hot-Plug PCI. If any primary adapter assigned to a team is disabled, the team functions as a Smart Load Balancing and Failover type of team in which auto-fallback occurs. The LiveLink feature is supported for this type of team.

Creating teams

To create teams you’ll need a configuration file (or know how to write one yourself).

The answer file for a load balancing team without standby looks like this:

name: Team 1
type: 0
pnic: 03:00.0
pnic: 05:00.0

So what’s in that file?

Starting from the top:

Name: Team 1 <- The name of your team
Type: 0 <- There are three different types:
0 = Load Balacing
1 = Generic Trunk
2 = Link Aggregation

pnic: 03:00.0 <- pnic = Primary. In the examples below there’s also snic = Secondary. The numbers are the PCI-position of the NIC.

So how do you find out which PCI-position your NICs have? The command below will list all devices in your system:

BACScli -bdf

BDF is the Bus:Device.Function of the NIC. For example, if the NIC has bus number of 5, device number of 0 and function number of 1, use “-bdf 5:0.1” to select the NIC for the command to execute on.

If you have servers that you’d like to script installation and such you could export a configuration either from the GUI or with the help of BACScli: BACScli -t team “save -f BDF c:tempteaming.bcg”

Once it’s exported you can create your team by importing the configuration:

BACScli -t team “restore c:tmpteaming.bcg”

Please note that the configuration files contains the PCI-position of your NICs so if your servers aren’t configured identically it won’t work. You could of course script it, detecting the nics and dynamically creating the configuration file. After all, it’s just text.

There are a bunch of different types of teams, below is a list of how the configuration files look if you don’t feel like doing the research:

Generic trunk without VLAN
name: Team 1
type: 1
pnic: 03:00.0
pnic: 05:00.0

Generic trunk with VLAN
name: Team 1
type: 1
pnic: 03:00.0
pnic: 05:00.0
vname: VLAN 1
vid: 1

Link aggregation with VLAN
name: Team 1
type: 2
pnic: 03:00.0
pnic: 05:00.0
vname: VLAN 1
vid: 0

Load balancing without standby

name: Team 1
type: 0
pnic: 03:00.0
pnic: 05:00.0

Load balacing with standby
name: Team 1
type: 0
pnic: 03:00.0
pnic: 05:00.0
snic: 08:00.0

Load balancing with LiveLink

name: Team 1
type: 0
target_ip: 172.16.4.15
target_ip: 172.16.0.1
retry: 5
freq: 2000
retry_freq: 1000
livelink_vid: 0
pnic: 03:00.0
livelink_ip: 172.16.9.88
pnic: 05:00.0
livelink_ip: 172.16.9.89

Link aggregation without VLAN

name: Team 1
type: 2
pnic: 03:00.0
pnic: 05:00.0

If you’re scripting this you’ll find the exit codes for BACSCLI below:

BACSCLI_OK 0 Upgrade firmware OK
BACSCLI_QUIT 1 Quit program
BACSCLI_PARAM_ERROR 2 Not correct parameters
BACSCLI_ADAPTER_NOT_FOUND 3 Adapter not found
BACSCLI_CANNOT_LOCK_ADAPTER 4 Cannot lock adapter
BACSCLI_GET_CLOSE_EVENT 5 Get close event
BACSCLI_INIT_FAILED 6 Initialization failed
BACSCLI_UNSUPPORTED_BMAPI_VER 7 BMAPI is too old
BACSCLI_UNKNOWN_COMMAND 8 Unknown command
BACSCLI_MALLOC_ERROR 9 memory allocation error
BACSCLI_BMAPI_ERROR 10 BMAPI call returns error
BACSCLI_OS_NOT_SUPPORTED 11 OS is not supported
BACSCLI_NO_ADVANCED_PARAMS 12 No Advanced Parameter for the NIC
BACSCLI_INVALID_ADVANCED_PARAM_DETECTED 13 Invalid Advanced Parameter detected
BACSCLI_INVALID_ADVANCED_PARAM_SPECIFIED 14 Invalid Advanced Parameter specified
BACSCLI_INVALID_ADVANCED_VALUE_SPECIFIED 15 Invalid Advanced Value specified
BACSCLI_FEATURE_NOT_SUPPORTED_FOR_NIC 16 Feature not supported for the NIC
BACSCLI_SET_ADVANCED_PARAM_ERROR 17 Failed to set the Advanced Parameter with new value
BACSCLI_SYSTEM_REBOOT 18 System Reboot required
BACSCLI_UNSUPPORT_PLATFORM 19 System platform is not supported
BACSCLI_NOT_ENOUGH_PRIVILEGE 20 Current user does not have enough privilege
BACSCLI_READ_LICENSE_FILE_ERROR 21 Error in reading license file
BACSCLI_INVALID_LICENSE_KEY 22 Invalid license key
BACSCLI_INVALID_ISCSI_PARAM_SPECIFIED 23 Invalid iSCSI Management Parameter specified
BACSCLI_INVALID_ISCSI_VALUE_SPECIFIED 24 Invalid iSCSI Management Value specified
BACSCLI_INVALID_RSC_PARAM_SPECIFIED 25 Invalid Resource Parameter specified
BACSCLI_INVALID_RSC_VALUE_SPECIFIED 26 Invalid Resource Value specified
BACSCLI_FEATURE_NOT_SUPPORTED_IN_FCFS 27 Feature not supported in FCFS mode
BACSCLI_PARAM_IS_READ_ONLY 28 This parameter can’t be modified, Administrator authority required.
BACSCLI_NULL_IP_ADDRESS 29 The current IP address is NULL.
BACSCLI_CANNOT_UNLOCK_ADAPTER 30 Failed to unlock adapter.
BACSCLI_INVALID_VALUE_SPECIFIED 31 Invalid value specified
BACSCLI_NIC_IS_PART_OF_GEC_LACP_TEAM 32 NIC is part of a GEC/LACP Team.
BACSCLI_REGISTRY_ACCESS_ERROR 33 Error in accessing Registry.
BACSCLI_NOT_AN_ISCSI_BOOT_DEVICE 34 This is not an iSCSI Boot device.
BACSCLI_INVALID_IP_ADDRESS 35 Invalid IP Address.
BACSCLI_DUPLICATE_IP_ADDRESS 36 Duplicate IP Address.
BACSCLI_TEAM_DRIVER_NOT_LOAD 37 nic( %s ) driver has to be loaded to make it a member of a team.
BACSCLI_NDIS6_DRIVER_REQUIRED 38 Ndis6 driver is required for the NIC to join the Team in Windows Vista and later.
BACSCLI_TEAM_UNKNOW_NIC 39 Unknown NIC ( %s ).
BACSCLI_INVALID_SUBNET_MASK 40 Invalid subnet mask.
BACSCLI_INVALID_CMD 41 Invalid command.
BACSCLI_NOT_YET_IMPL 1000 This functionality is not yet implemented.
BACSCLI_UNWIND 1001 Unwind to the parent processor
BACSCLI_NOT_APPLICABLE 1002 Command not applicable
BACSCLI_NO_TARGET_SEL 1003 No active target selection.
BACSCLI_INVALID_CONTEXT 1004 Not a valid context
BACSCLI_INVALID_FORMAT 1005 Invalid format selection
BACSCLI_INVALID_TARGET_ID 1006 Invalid target identifier
BACSCLI_FILE_DOES_NOT_EXIST 1007 File does not exist
BACSCLI_INVALID_TEAM_NAME 1008 Supplied team name is invalid
BACSCLI_TEAM_COMMIT_FAILED 1009 Failed to commit the team operation.
BACSCLI_TEAM_REMOVE_FAILED 1010 Failed to remove the team.
BACSCLI_CANNOT_OPEN_FILE 1011 Failed to open the file handle.
BACSCLI_ERR_CANNOT_SET_IPADDR 1012 Failed to set ip address.
BACSCLI_ERR_CANNOT_GET_NIC_PCI_INFO 1013 Failure retrieving NIC information
BACSCLI_ERR_RETRIEVE_IP_ADDR 1014 Error retrieving IP address information
BACSCLI_FAILED_GET_INFO 1015 Failed to get info from the DataContainer
BACSCLI_WRONG_OPTION_FLAG 1016 General team config file parsing error.
BACSCLI_EXCEEDMAXVLAN 1017 Only a maximum of 64 VLANs are allowed.
BACSCLI_CANNOT_CREATE_LIVE_LINK 1018 Live link support only applied to SLB team
BACSCLI_EXCEED_MAX_TARGET_IP 1019 Live link support allows up to 4 link
BACSCLI_TOO_MANY_PHY_NIC 1020 Only up to a maximum of 8 nics are allowed in a team.
BACSCLI_CANNOT_CREATE_FECGEC_8023AD 1021 Cannot create FECGEC or 802.3ad team with standby adapter.
BACSCLI_LL_IP_TARGET_IP_TYPE_MISMATCH 1022 Invalid IPv6 Address
BACSCLI_INVALID_RANGE 1023 Value is out of range.
BACSCLI_INVALID_INTERVAL 1024 Invalid probe retry frequency
BACSCLI_DUPLICATE_OPTION 1025 The same option has been specified previously.
BACSCLI_DUPLICATE_MAC_ADDRESS 1026 Duplicate adapter physical MAC address
BACSCLI_DUPLICATE_VLANID 1027 Duplicate VLAN name
BACSCLI_TEAM_ALREADY_EXISTS 1028 Team with the specified name already exists
BACSCLI_CANNOT_MATCH_MAC_ADDR 1030 Cannot find device using the specified MAC address.
BACSCLI_NO_TEAM_TO_CONFIG 1031 No team to configure.
BACSCLI_CANNOT_CREATE_TEAM 1032 Failure while creating team
BACSCLI_NO_LINK_FOR_IP_CFG 1033 No link is present in team to set ip address.
BACSCLI_ONE_OR_MORE_CREATE_FAILED 1034 Creation of one or more teams failed.
BACSCLI_TEAM_NO_MEMBER 1035 A team is required to have at least one valid member.
BACSCLI_NO_BRCM_NIC_IN_TEAM 1036 Team requires at least one Broadcom nic.
BACSCLI_ONLY_BROADCOM_NIC_FOR_VLAN 1037 Only Broadcom certified adapters are supported in VLAN.
BACSCLI_CANNOT_SET_IPADDR 1038 Failed to assign ip address on the virtual adapter.
BACSCLI_INVALID_CFG 1039 Invalid configuration
BACSCLI_SET_ADVANCE_PARAM_FAILED 1040 Failed to set advanced parameter
BACSCLI_INTERNAL_ERROR_INVALID_DATA 1041 Invalid or NULL data found
BACSCLI_INVALID_PARAMETER 1042 Invalid Parameter. Parameter is too few
BACSCLI_NIC_NOT_SUPPORTED 1043 The current NIC is not supported for this operation.
BACSCLI_SET_ASF_FAILED 1044 Failed to set the ASF Table
BACSCLI_SET_POWER_MGMT_FAILED 1045 Failed to set Power Management configuration.
BACSCLI_INVALID_TARGET_CMD 1046 command/target identifier is invalid
BACSCLI_NO_ISCSI_SESSIONS 1047 No iSCSI sessions exists on the system
BACSCLI_SET_MGMT_OBJ_FAILED 1048 Failed to set Management Object
BACSCLI_SHOW_USAGE 1049 If we want to show usage
BACSCLI_CANNOT_CONNECT_WMI 1050 Cannot connect WMI
BACSCLI_EXCEEDMAXTAGGEDVLAN 1051 Only a maximum of 63 tagged VLANs are allowed.
BACSCLI_NO_TEAM_AVALIABLE 1052 There is no team to save.
BACSCLI_NOT_CONFIGURABLE 1053 Not Configurable.
BACSCLI_CANNOT_SUSPEND_RESUME_DRIVER 1054 Failed to suspend or resume the driver.
BACSCLI_CANNOT_RESTART_DRIVER 1055 Failed to restart the driver.

NIC Teaming in Server Core or Hyper-V Server

Update
Teaming with Intel ProsetCL
Teaming with Broadcom BACScli

If you’re running Server Core or Hyper-V Server 2008 R2 you’ve probably come across the problem of teaming nics. No matter which hardware vendor you choose, they all have they’re special way of doing things. Helping an old colleague out the other day it made me realize that it’s not as straightforward as it is in the full version, so I’ve tried it out with both Intel and Broadcom nics. Which you of course know covers the servers from both HP and Dell (where I work, shameless plug).

Installing the Broadcom software to support network teaming in Server Core / Hyper-V Server

Before you start you must install the prereq’s for the drivers, that comes down to .Net 2.0, .Net 2.0 WOW64 and SNMP.
The easiest way is to use OcSetup to install them:

Start /w ocsetup NetFx2-ServerCore
Start /w ocsetup NetFx2-ServerCore-WOW64
Start /w ocsetup SNMP-SC

The “/w” will let you wait during installation so you know when it’s finished, please note that roles/features are case sensitive for ocsetup so type it as it looks…

When that’s done you’ll need to download the 14.1.x-version of the BACS from Dell’s site and extract them to somewhere on your drive, default is c:broadcom. Navigate to the c:broadcomdriver_management_apps_installer and run setup.exe.

The wizard kicks in and when you’re done you can team your nics (and change other things too) through the c:/program files/broadcom/bacs.exe utility.

Installing the Intel software to support network teaming in Server Core / Hyper-V Server

Turns out that Intel has their own great post on the subject of command line installations which you can find at http://www.intel.com/support/network/sb/cs-016040.htm.

A short rundown otherwise is that you’ll need the setup.exe program for your nic, then you have multiple choices on how to install them. The base driver can be installed through the included pnputil.exe for Server Core or you could use the Intel setup.exe instead.

This is from Intel’s site and shows you what switches does what:

Setup.exe support the following command line parameters:

Parameter Definition
ANS Advanced Network Services
“0”, do not install ANS. If ANS is already installed, it will be uninstalled.

“1”, install ANS (default).

NOTE: If the ANS parameter is set to ANS=1, both Intel PROSet and ANS will be installed.

DMIX PROSet for Windows Device Manager
“0”, do not install Intel PROSet feature. If the Intel PROSet feature is already installed, it will be uninstalled.

“1”, install Intel PROSet feature (default).

NOTE: If DMIX=0, ANS will not be installed. If DMIX=0 and Intel PROSet and ANS are already installed, Intel PROSet and ANS will be uninstalled.

SNMP Intel SNMP Agent
“0”, do not install SNMP. If SNMP is already installed, it will be uninstalled.

“1”, install SNMP (default).

NOTE: Although the default value for the SNMP parameter is 1 (install), the SNMP agent will only be installed if:
The Intel SNMP Agent is already installed. In this case, the SNMP agent will be updated.
The Windows SNMP service is installed. In this case, the SNMP window will pop up and you may cancel the installation if you do not want it installed.

BD Base Driver and IOATDMA Driver
“0”, do not install the base driver.

“1”, install the base driver (default).

LOG [log file name]
LOG allows you to enter a file name for the installer log file. The default name is C:UmbInst.log.

-a Extract the components required for installing the base driver and I/OAT driver to C:Program FilesIntelDrivers. The directory where these files will be extracted to can be modified unless silent mode (/qn) is specified. If this parameter is specified, the installer will exit after the base driver and I/OAT driver are extracted. Any other parameters will be ignored.
-f Force a downgrade of the components being installed. NOTE: If the installed version is newer than the current version, this parameter needs to be set.

How to install the base driver on Windows Server 2008 and Windows Server 2003:

:Setup DMIX=0 ANS=0 SNMP=0

How to install the base driver on Windows Server 2008 and Windows Server 2003 using the LOG option:

:Setup LOG=C:installBD.log DMIX=0 ANS=0 SNMP=0

How to install Intel PROSet and ANS silently on Windows Server 2008 and Windows Server 2003 (32-bit version):

:Setup DMIX=1 ANS=1 /qn

How to install Intel PROSet without ANS silently on Windows Server 2008 and Windows Server 2003 x64 Edition:

:Setup DMIX=1 ANS=0 /qn

How to install components but deselect ANS for Windows Server 2008 and Windows Server 2003:

:Setup DMIX=1 ANS=0 /qn /liew C:install.log

The /liew log option provides a log file for the DMIX installation.

To install teaming and VLAN support on a system that has adapter base drivers and Intel PROSet for Windows Device Manager installed, type the command line :Setup ANS=1.