Utilizing Out-of-Band Management for Server and Network Hardware

Out-of-band management (OOBM) of network devices is nothing new, but it can be an extremely powerful tool to assist when something needs to be modified on a server or a piece of networking hardware.

Utilizing OOBM will allow you to perform functions such as server initial setup and loading of an operating system (OS), or simply installing hardware updates that require you to reboot into a bootable ISO or firmware update for the hardware. I even have used this feature to run a bare metal restore from backup on a physical server that failed to boot into its OS. You might have heard of an IP KVM (keyboard/video/mouse); this is very similar to that technology, but without any additional hardware components required.

So why am I sharing this information? Well, one reason is that I performed all the above tasks without physically being in the same building, or even state in some cases, as the devices I worked on. OOBM also allows for extensive visibility into the health of each component of the server. Some other key features are the ability to modify BIOS options, change networking configurations, and do any other work that otherwise would require you to be physically with the server (other than adding or replacing components, of course).

Generally, with Dell and Hewlett Packard Enterprise (HPE) Servers, the iDRAC or iLO server components are included in the server bundle, but not all features are licensed. For example, for HPE you need to purchase the iLO Advanced License to unlock the ability to remote boot to ISOs or to manage the server from the iLO interface after the server starts loading the OS. These licenses cost only a few hundred dollars but their return on investment is substantial, as you will experience less downtime and reduce the travel time required for someone to get in front of the hardware and start diagnosing the issue.

Over the last few years, I have come up with some best practices for utilizing this technology in a secure manner to gain a high-level overview and management of critical equipment. I will highlight some of these below with specific examples.

Best Practices for Out of Band Management, with Examples

1. Locate all OOBM interfaces in a network separate from the production network and set their IP address statically
   1. Production network is 192.168.1.0/24
   2. OOBM network should be different – e.g., 192.168.100.0/24
   3. Confirm that no access between networks is allowed except for that involving authorized devices. Use Access Control Lists (ACLs) to lock down to servers, IT staff, VPN users, etc., as needed)
2. Use a single isolated switch or additional interfaces on a router or firewall to plug in all OOBM Ethernet cables. Use a single uplink to a router if using a switch.
   1. This is very important for remote diagnosis and investigation. It allows you to reboot any network equipment remotely and see if a device has failed, is offline, or is experiencing other issues.
   2. I have utilized OOBM in an instance in which I could see a client firewall from our office, but I was unable to connect to anything internally on their production LAN. The client also was not able to get to the Internet. I connected to the client’s VPN and I was able to utilize the management IP of their core switch to confirm it was not passing traffic to the firewall. I rebooted their core switch remotely and resolved the issue. This saved two hours of driving and allowed the client to get back online quickly and cost-effectively.
3. Keep the default username and password as labeled on the server, and create a new one for yourself and vendors
   1. Leave the default username and password for the original iLO or iDRAC configuration, as they are unique to each server.
   2. Each HPE or Dell server has a sticker that will have the login information printed on it. If you change those credentials, the default login will not work, but we can always use a secondary login.
   3. On network hardware (switches, firewalls, routers, etc.), you should change the password, as the default is usually very simple (e.g., “password,” or sometimes even no password). Make sure to document your new credentials!
4. Maintenance and licensing
   1. Install the premium licenses immediately so all features and components of the OOBM are available as needed.
   2. Patch management procedures should be in place to check for new and upgraded firmware. Over the last few years, we have seen an increase in malicious exploits and vulnerabilities in OOBM features. By ensuring these technologies are up-to-date and patched, you can mitigate that risk significantly.
   3. Security should be the highest priority on these OOBM devices, as they provide full control of hardware, software, data, and networking, and someone who gains unauthorized access can do significant damage.

Now that I have given you all these best practices in network management, what are you going to tackle next?

Contact the Sikich Tech Team to learn more about how we can build a strong partnership together to strengthen and empower your business and employees by utilizing the latest technology to meet your needs in this rapidly changing culture.