Deploying MetalSoft storage appliance

This guide will cover setting up a MetalSoft storage appliance.

Required hardware

Entry-level 8 core Xeon processor with a decent amount of memory, depending on the number of iSCSI sessions and clients connecting to it. Better start with at least 64GB of RAM, at least 4 disks set to AHCI mode, 1 or 2 x10Gbit NIC for iSCSI client traffic, 1 x 1Gbit port for management access and a lights-out port.

Cabling

The cabling can be seen in the below diagram:

../_images/preparing_to_setup_a_datacenter_01.svg

OS installation

Centos 7.4 1708 Minimal will be used as a installation media. Installation will be deployed using two simmilar disks and creating mirror mdraid partition sets for boot, / and swap partitions.

Configure network

Set up out-of-band , management and SAN network configurition.

The lights-out network interface can be configured by rebooting the server and accesing the out-of-band firmware menu or by using the ipmitool package and cli suite, from the OS.

Setting lights-out IP address using ipmitool

Configuration example:

[root@test_server ~]# ipmitool lan set 1 ipsrc static
[root@test_server ~]# ipmitool lan set 1 ipaddr 192.168.1.211
Setting LAN IP Address to 192.168.1.211
[root@test_server ~]# ipmitool lan set 1 netmask 255.255.255.0
Setting LAN Subnet Mask to 255.255.255.0
[root@test_server ~]# ipmitool lan set 1 defgw ipaddr 192.168.1.254
Setting LAN Default Gateway IP to 192.168.1.254

Creating a new user and setting-up a password

Configuration example:

[root@test_server ~]# ipmitool user set name 2 admin
[root@test_server ~]# ipmitool user set password 2
Password for user 2: 
Password for user 2:

Setting up management interface IP address

Configuration example:

[root@test_server ~]# ip addr add 192.168.100.153/24 dev eth1
[root@test_server ~]# ip route add 192.168.100.0/24 via 192.168.100.1 dev eth1

Management network IP address will be used afterwards also as a API storage endpoint.

Setting up SAN IP address

Configuration example:

[root@test_server ~]# ip addr add 100.96.1.156/30 dev eth2

For the SAN iSCSI traffic, one 10Gbit or 2 x 10Gbit interfaces can be set up in a bond fashion. Depending on the network configuration, iSCSI traffic may need to be directed through a static route to the SAN clients subnet.

Enabling zfs services at boot-time

Issue the below command to enable zfs related services at boot time

[root@test_server ~]# systemctl preset zfs-import-cache zfs-import-scan zfs-mount zfs-share zfs-zed zfs.target

Disable RP filter

Rp filter should be disabled as well on all interfaces

[root@test_server ~]# for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 0 > ${i} ; done

Increase zfs max usable ram space

Below there’s an example for a system running with 256 GB of memory for which 205543648256 bytes will be reserved for the arc memory used by zfs.

Modify it on the fly

Configuration example:

[root@test_server ~]# echo "205543648256" > /sys/module/zfs/parameters/zfs_arc_max

Make it default at boot-time

Configuration example:

[root@test_server ~]# echo "options zfs zfs_arc_max=205543648256" > /etc/modprobe.d/zfs.conf

Creating zpool

Below there’s an example of a one, 3 disks, vdev raidz1 zpool creation command. Using /dev/disk/by-id disk terminology in order to avoid problems with possible disk renaming at boot-time.

Configuration example:

[root@test_server ~]# zpool create -f Pool-A raidz1 /dev/disk/by-id/scsi-3600508b1001c30f7f6eb0e71ed4fa217 /dev/disk/by-id/scsi-3600508b1001c5c4c279dd5d9b6d7c703 /dev/disk/by-id/scsi-3600508b1001cc79180aad3504bc84d2c

Manually create datasets and set all properties

Configuration example:

  zfs create Pool-A/Data
  zfs create Pool-A/Data/Drives
  zfs create Pool-A/Data/Templates
  zfs create Pool-A/Internal
  zfs create Pool-A/Internal/Logs
  zfs set reservation=8M Pool-A/Data
  zfs set reservation=8M Pool-A/Data/Drives
  zfs set reservation=8M Pool-A/Data/Templates
  zfs set reservation=8M Pool-A/Internal
  zfs set reservation=8M Pool-A/Internal/Logs
  zfs set reservation=16M Pool-A/Internal
  zfs set quota=1G Pool-A/Internal
  zfs set quota=1G Pool-A/Internal/Logs
  zfs set compression=lz4 Pool-A
  zfs set checksum=fletcher4 Pool-A
  zfs set xattr=sa Pool-A
  zfs set atime=off Pool-A
  zfs mount -a
  zfs list
  zfs set bsa:type=Filesystem Pool-A
  zfs set bsa:type=Filesystem Pool-A/Data
  zfs set bsa:type=Filesystem Pool-A/Data/Drives
  zfs set bsa:type=Filesystem Pool-A/Data/Templates
  zfs set bsa:type=Filesystem Pool-A/Internal
  zfs set bsa:type=Filesystem Pool-A/Internal/Logs

Create saveconfig json file

Create LIO saveconfig json file and populate it with the below, to create an empty config.

[root@test_server ~]# vim /Pool-A/Internal/saveconfig.json.pool

{
  "fabric_modules": [],
  "storage_objects": [],
  "targets": []
}

Sync API with the manually created zpool and targetcli config file.

BSA api can be accessed at http://[mgmt_ip_address]/api

[root@test_server ~]# echo '{"method":"target_config_reload","params":["\/etc\/target\/saveconfig.json"],"id":9,"jsonrpc":"2.0"}' | bsa_python.py

Check if BSA API has recognized the newly created zfs pool

storage.storage_pools()

Which should return information about the zpool ()

../_images/pool_layout.png

Create iSCSI target IQN and set it’s IP address

Use the IP address previously set up on the SAN interface

storage.target_create(["[san_ip_address]"])

Benchmark (if needed)

The script creates three large volumes and then performs a series of tests simulating these types of IO operations (read, write, randread, randwrite, rw, randrw) using as well two types of blocksizes to simulate small and large blocksize applications (4k and 128k).

Prerequisites:

fio test tool needs to be installed In the same path as the script a file containing the IO types as a column

#!/bin/bash
zfs create -s -V 900G Pool-A/Data/Drives/fio_test_drive_3
sleep 5
zfs create -s -V 900G Pool-A/Data/Drives/fio_test_drive_4
sleep 5
zfs create -s -V 900G Pool-A/Data/Drives/fio_test_drive_5
sleep 5
a=`ls -lrth /dev/zvol/Pool-A/Data/Drives | grep 'fio_test_drive_3' | awk -F "/" '{print $NF}'`
b=`ls -lrth /dev/zvol/Pool-A/Data/Drives | grep 'fio_test_drive_4' | awk -F "/" '{print $NF}'`
c=`ls -lrth /dev/zvol/Pool-A/Data/Drives | grep 'fio_test_drive_5' | awk -F "/" '{print $NF}'`
iod="128"
bs1="4k"
bs2="128k"
for io in `cat /root/benchmark_script/io_type`; do

fio --filename=/dev/$a --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs1 --direct=0 --size=10G --numjobs=85 --runtime=60 > /root/benchmark_script/results/"$io$bs1$a.txt"
echo "fio --filename=/dev/$a --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs1 --direct=0 --size=10G --numjobs=85 --runtime=60" >> /root/benchmark_script/results/"$io$bs1$a.txt"

fio --filename=/dev/$b --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs1 --direct=0 --size=10G --numjobs=85 --runtime=60 > /root/benchmark_script/results/"$io$bs1$b.txt"
echo "fio --filename=/dev/$b --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs1 --direct=0 --size=10G --numjobs=85 --runtime=60" >> /root/benchmark_script/results/"$io$bs1$b.txt"

fio --filename=/dev/$c --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs1 --direct=0 --size=10G --numjobs=85 --runtime=60 > /root/benchmark_script/results/"$io$bs1$c.txt"
echo "fio --filename=/dev/$c --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs1 --direct=0 --size=10G --numjobs=85 --runtime=60" >> /root/benchmark_script/results/"$io$bs1$c.txt"

fio --filename=/dev/$a --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs2 --direct=0 --size=10G --numjobs=85 --runtime=60 > /root/benchmark_script/results/"$io$bs2$a.txt"
echo "fio --filename=/dev/$a --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs2 --direct=0 --size=10G --numjobs=85 --runtime=60" >> /root/benchmark_script/results/"$io$bs2$a.txt"

fio --filename=/dev/$b --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs2 --direct=0 --size=10G --numjobs=85 --runtime=60 > /root/benchmark_script/results/"$io$bs2$b.txt"
echo "fio --filename=/dev/$b --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs2 --direct=0 --size=10G --numjobs=85 --runtime=60" >> /root/benchmark_script/results/"$io$bs2$b.txt"

fio --filename=/dev/$c --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs2 --direct=0 --size=10G --numjobs=85 --runtime=60 > /root/benchmark_script/results/"$io$bs2$c.txt"
echo "fio --filename=/dev/$c --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs2 --direct=0 --size=10G --numjobs=85 --runtime=60" >> /root/benchmark_script/results/"$io$bs2$c.txt"

done

zfs destroy Pool-A/Data/Drives/fio_test_drive_3
zfs destroy Pool-A/Data/Drives/fio_test_drive_4
zfs destroy Pool-A/Data/Drives/fio_test_drive_5

echo "Finished.Results can be found in path /root/benchmark_script/results"

Adding the storage device to application

Check out the guide on how to add the newly created storage pool to application