Server Setup

This guide walks you through a procedure for serversSetup for on-prem deployments. Every deployment on-site should follow this requirements for the physical servers that will run HyperCX.

Minimum hardware requisites for physical server

• CPU: 2x 6 core processors
  
• Memory RAM: 128GB
  
• Storage: Direct-attached storage (DAS).
  
  • OS: Need a minimum of (2) drives 480GB or bigger (SSD or NVMe are preferred), RAID1 mode (hardware controller or Software controller).
    
  • Local: Requires at least 2 drives 980GB or bigger (SSD or NVMe are preferred). The modalities for this storage solutions are RAID0 for each disk if the storage controller does not allow HBA/passthrough mode.
    
  • SDS: Requires at least 1 drive 980GB or bigger (SSD or NVMe are preferred). It is required a minimum of 3 servers with this amount of disks to create the desired replica level 3.
    

Recommended BIOS configurations

Upgrade Firmware

To ensure optimal performance and reliability and the absence of bugs, it is advised to visit the official website to check whether the firmware version of the current computer is the latest one. If not, it is advised to upgrade it to the latest one. Server firmware includes BIOS, BMC, and CPLD.

Optimal Configuration of BIOS

For the application to all scenarios, various factors will be considered for the general BIOS default configura-tion. If the optimal performance is required, it is advised to provide the following configuration.

CPU Hardware Prefetcher

Also, server needs to be configured in HDD passthrough (HBA mode) and be bootable from HDD/SSD or AHCI mode for on-board controller config.

Hyper-Threading (HT) Technology

It is generally advised by Intel® not to disable HT. Intel® HT improves the overall system throughput by increas-ing the number of CPU cores. For multi-thread applications, it is advised to enable HT, therefore enhancing CPU usage and overall performance. However, for a single thread, HT will degrade its performance. Therefore, for some applications sensitive to delay and with not high concurrency, it is advised to disable HT. Moreover, some applications or system bugs limit the total number of CPU cores, and it is advised to disable HT to reduce the number.

Lockstep Mode

The Lockstep mode carries out a higher-level check for the memory, improving the system reliability, but low-ering the bandwidth and delay of memory access. It has high real-time requirements, and is not applicable to service scenarios with high throughput. For such scenarios, there are sound protection mechanisms, from sys-tem to software, and to solutions. Therefore, it is advised to disable the Lockstep mode.

EIST and Turbo Boost

EIST is an intelligent frequency control technology of Intel® which automatically adjusts the voltage and fre-quency of processors based on the system workload to reduce power consumption and heat productivity. The Turbo Boost Technology of Intel®, or commonly known as automatic overclocking technology, is mainly used to improve the frequency of processors and maximize the performance of processors. The latest CPU supports the Turbo Boost which is not dependent on OS and features single core.

NUMA Technology

NUMA is short for Non-uniform Memory Access architecture, that is, the Non-uniform Memory is concept in-troduced from the multiprocessor system. In the conventional single-processor system, the time during which the processor accesses external memories is the same. In the multiprocessor system, however, each processor has its own memory. When a processor accesses the memory of another processor, the access time differs from time during which it accesses its own memory. The efficiency of accessing the local memory is higher than that of accessing farther memories.

The Xeon CPUs of Intel® are designed based on the NUMA architecture. When it is enabled, the OS will identify the NUMA architecture of hardware to optimize applications and improve the processing efficiency of applica-tions. It is advised to enable NUMA for computers with more than 2 sockets or Virtualization Hosts.

Hardware Virtualization

To boost the virtualization performance, Intel® provides CPU hardware virtualization features. If a server does not use virtualization, it is advised to disable all virtualization features, preventing additional system overheads, improving the performance, and reducing the energy consumption.

Energy Consumption Management

The CPU dynamic energy management technology (DEMT) is used to reduce the server power consumption. The server power consumption can be reduced to a certain extent by selecting different power management policies if the system is idle. Lower power consumption means slower CPU wake-up and greater impact on the performance.

It is advised to disable the C-States of CPU to prevent delay increase and performance reduction due to the switch between different states. Keep that all CPU cores are operating.

BIOS configuration recommendations for servers in a Dell EMC Solutions for HyperCX HCI

Configuration recommendations for servers with intel-based BIOS for HyperCX HCI

BIOS configuration recommendations for servers in a HPE Solutions for HyperCX HCI

For HPE Proliant Gen10/Gen10plus generation

This gen has a predefined workload profile to do the whole configuration. The workload is called “Virtualization: Static High Performance”.

For HPE Proliant G9/G8 generation

For HP Proliant G6/G7 genaration

Node OS installation

This documentation describes how should be installed the Operative System (OS) Ubuntu Server 22.04 ISO. Follow the right indications to make the OS ready for the automated deployment.

OS curtin installation

Start the installation of the Ubuntu Server:

Set the default language to English:

Same for keyboard layaout:

Select Ubuntu Server for installation:

Continue without configure the network:

Do not configure a proxy:

Leave the default mirror for Ubuntu:

Configure a custom storage layout:

The configuration must use a Soft-RAID 1, with the specified partitions and FS, but with the bootloader outside of the Soft-RAID.

Select the disk that will be used as primary boot device:

Select the disk that will be used as secondary boot device:

After previous actions, the configuration must looks like follow:

For each device (those for the OS), create a new partition (DO NOT FORMAT THE PARTITION!):

• swap: 24G
  
• /var/log: 100G
  
• /: The rest of the available space.
  

After previous actions, the configuration must looks like follow:

Create a Soft-RAID for the created swap partition:

Create a Soft-RAID for the created /var/log partition:

Create a Soft-RAID for the created / partition:

The following screenshot shows how is going the configuration:

Format md0 device for swap (DO NOT PARTITIONATE THE SOFTRAID DEVICE!):

Format md1 device for ext4 and mount it on /var/log:

Format md2 device for ext4 and mount it on /:

Select md0 device for swap.

The custom storage layout should look like follow, after all previous configurations:

Confirm the configuration in order to continue with the installation:

Complete de profile setup, adapting the hostname to the deployment:

Skip Ubuntu Pro:

Enable the installation of OpenSSH server:

As the server is still not connected to the network, it wont e able to reach the snap store. Continue without updating:

Reboot when the installation finish:

Once the system is up, the lsblk command output should look like follow:

Set the right hostname for the deployment:

CLUSTER_NAME=ECASA-NC-CLOUD
hostnamectl set-hostname ${CLUSTER_NAME}-node1
sed -i "s#127.0.1.1.*#127.0.1.1 $HOSTNAME ${CLUSTER_NAME}-node1#" /etc/hosts

Node OS bootable on both devices

From previous screenshot we can appreciate that OS drivers sda and sdb are being used for the Soft-RAID and the primary ESP is being used by sdb.

At this point so far, the server has a redundant setup. The Soft-RAID 1 of each partition could be checked as follow:

For md0 (where is the swap)

mdadm --detail /dev/md0

For md1 (where is the /var/log)

mdadm --detail /dev/md1

For md2 (where is the /)

mdadm --detail /dev/md2

Check the Soft-RAID status:

cat /proc/mdstat

Once finished the resyncing, it should look like follow:

Beside of this, the custom storage layout is not reeady yet, as the bootloader is only enabled for the primary disk. While having this redundancy is fine, there is one potential lurking issue. If one drive is removed, the server might be unable to boot the system. So let’s make sure the ESP is the same on both drives, and that the system will try to boot from either of the hard drives and not just one.

The lsblk command output should look like follow:

Check which disk is currently using the bootloader:

mount | grep boot

In this case, the output shows that the boot partition is mounted on sdb1 (same conclusion lsbk output).

OS drives sda and sdb are being used for the Soft-RAID and the primary ESP is being used by partition sdb1. Lets define the following variables:

part_boot_on=sdb1
part_boot_backup=sda1

Check for the PARTUUID for those partitions:

blkid | egrep "${part_boot_on}|${part_boot_backup}"

or

ls -la /dev/disk/by-partuuid/ | egrep "${part_boot_on}|${part_boot_backup}"

From previous outputs we have gathered the PARTUUIDs for each partition device:

• sda1: 2f5db39f-9944-4eda-b15c-d6fc45313480
  
• sdb1: f0072633-9c78-4fa5-9212-594574fa6f2e
  

Clone /dev/sdb1 into /dev/sda1:

dd if=/dev/${part_boot_on} of=/dev/${part_boot_backup}

The server has now a working ESP on both drives, so the next step is to make sure both ESP exists in the boot-list:

efibootmgr -v | grep ubuntu

It should be shown two entries called ubuntu. Make sure the PARTUUIDs are the same as the two taken note of earlier.

efibootmgr --create --disk /dev/sda --part 1 --label "ubuntu" --loader "\EFI\ubuntu\shimx64.efi"

You should now be able to remove any of the two drives and still boot the system.

More information about this procedure on this article.

Network configuration

The network will be configured on each server, using the netplan. For this, become root user and follow the next steps:

rm -rm /etc/netplan/*.yaml
nano /etc/netplan/01-netcfg.yaml

In the following picture, are exposed all needed configurations for the netplan configuration. Take in mind that VLANs and IP could differ on each deployment.

Example for HCX1 deployment:

# Network configuration
network:
  version: 2
  renderer: networkd

  ## Physical NICs configuration
  ethernets:
    eno1:
      dhcp4: no
      optional: true
    eno2:
      dhcp4: no
      optional: true

  ## Bonds configuration
  bonds:
    ### MGM_NET, PUBLIC_NET, CLIENT_NET and cluster private networks
    bond0:
      mtu: 9000
      interfaces:
        - eno1
        - eno2
      parameters:
        mode: 802.3ad
        lacp-rate: fast
        mii-monitor-interval: 100

  ## VLANs configuration
  vlans:
    ### MGM_NET
    bond0.2971:
      mtu: 9000
      id: 2971
      link: bond0

    ### PUBLIC_NET
    bond0.950:
      mtu: 9000
      id: 950
      link: bond0

    ### CLIENT_NET
    bond0.2972:
      mtu: 9000
      id: 2972
      link: bond0

  ## Bridges configuration
  bridges:
    ### MGM_NET
    br0:
      interfaces: [bond0.2971]
      addresses: [172.17.71.11/24]
      routes:
        - to: default
          via: 172.17.71.1
      nameservers:
        addresses: [1.1.1.1,8.8.8.8]

    ## PUBLIC_NET
    br1:
      interfaces: [bond0.950]

    ## CLIENT_NET
    br100:
      interfaces: [bond0.2972]

Example for BENTO deployment:

# Network configuration
network:
  version: 2
  renderer: networkd

  ## Physical NICs configuration
  ethernets:
    eno1:
      dhcp4: no
      optional: true
    eno2:
      dhcp4: no
      optional: true
    eno3:
      dhcp4: true
      optional: true
    eno4:
      dhcp4: true
      optional: true
    enp4s0f0:
      dhcp4: no
      optional: true
    enp4s0f1:
      dhcp4: no
      optional: true

  ## Bonds configuration
  bonds:
    ### MGM_NET, PUBLIC_NET, CLIENT_NET, MIGRATION_NET and cluster private networks
    bond0:
      mtu: 9050
      interfaces:
        - eno1
        - eno2
      parameters:
        mode: 802.3ad
        lacp-rate: fast
        mii-monitor-interval: 100

    ### STORAGE_NET
    bond1:
      mtu: 9000
      interfaces:
        - enp4s0f0
        - enp4s0f1
      parameters:
        mode: 802.3ad
        lacp-rate: fast
        mii-monitor-interval: 100
      addresses: [172.16.66.11/24]

  ## VLANs configuration
  vlans:
    ### MGM_NET
    bond0.2966:
      mtu: 9000
      id: 2966
      link: bond0

    ### PUBLIC_NET, MIGRATION_NET
    bond0.950:
      mtu: 9000
      id: 950
      link: bond0

    ### CLIENT_NET
    bond0.2967:
      mtu: 9000
      id: 2967
      link: bond0

    ### CLIENT_VXLAN
    vxlan_vtep:
      mtu: 9050
      id: 2968
      link: bond0
      addresses: [172.18.66.11/24]

  ## Bridges configuration
  bridges:
    ### MGM_NET
    br0:
      interfaces: [bond0.2966]
      addresses: [172.17.66.11/24]
      routes:
        - to: default
          via: 172.17.66.1
      nameservers:
        addresses: [1.1.1.1,8.8.8.8]

    ### PUBLIC_NET
    br1:
      interfaces: [bond0.950]
      addresses: [172.19.100.41/24]

    ### CLIENT_NET
    br100:
      interfaces: [bond0.2967]

• The network interfaces for the server are bonded and configured in LACP mode for redundancy.
• These are the required VLANs for the deployment:
  • MGM_NET VLAN
  • PUBLIC_NET VLAN
  • CLIENT_NET VLAN The VLAN numbering is at your discretion. The choice of the private IP segments to be used for the MGM and Client VLANs are also at your discretion, but we recommend as /24 to allow the cluster expansion.

• (3) Bridges need to be created, which are: o br0 for MGM o br1 for Public o br100 for Client

In a single node installation, the Client bridge and client VLAN are not mandatory, but recommended if there are other networks from outside the clusters that need to stablish communication with the VMs in-side the cluster.

Apply network configuration:

netplan apply