Deploying MetalSoft storage appliance¶
This guide will cover setting up a MetalSoft storage appliance.
Required hardware¶
Entry-level 8 core Xeon processor with a decent amount of memory, depending on the number of iSCSI sessions and clients connecting to it. Better start with at least 64GB of RAM, at least 4 disks set to AHCI mode, 1 or 2 x10Gbit NIC for iSCSI client traffic, 1 x 1Gbit port for management access and a lights-out port.
OS installation¶
Centos 7.4 1708 Minimal will be used as a installation media. Installation will be deployed using two simmilar disks and creating mirror mdraid partition sets for boot, / and swap partitions.
Configure network¶
Set up out-of-band , management and SAN network configurition.
The lights-out network interface can be configured by rebooting the server and accesing the out-of-band firmware menu or by using the ipmitool package and cli suite, from the OS.
Setting lights-out IP address using ipmitool
Configuration example:
[[email protected]_server ~]# ipmitool lan set 1 ipsrc static
[[email protected]_server ~]# ipmitool lan set 1 ipaddr 192.168.1.211
Setting LAN IP Address to 192.168.1.211
[[email protected]_server ~]# ipmitool lan set 1 netmask 255.255.255.0
Setting LAN Subnet Mask to 255.255.255.0
[[email protected]_server ~]# ipmitool lan set 1 defgw ipaddr 192.168.1.254
Setting LAN Default Gateway IP to 192.168.1.254
Creating a new user and setting-up a password
Configuration example:
[[email protected]_server ~]# ipmitool user set name 2 admin
[[email protected]_server ~]# ipmitool user set password 2
Password for user 2:
Password for user 2:
Setting up management interface IP address
Configuration example:
[[email protected]_server ~]# ip addr add 192.168.100.153/24 dev eth1
[[email protected]_server ~]# ip route add 192.168.100.0/24 via 192.168.100.1 dev eth1
Management network IP address will be used afterwards also as a API storage endpoint.
Setting up SAN IP address
Configuration example:
[[email protected]_server ~]# ip addr add 100.96.1.156/30 dev eth2
For the SAN iSCSI traffic, one 10Gbit or 2 x 10Gbit interfaces can be set up in a bond fashion. Depending on the network configuration, iSCSI traffic may need to be directed through a static route to the SAN clients subnet.
Enabling zfs services at boot-time¶
Issue the below command to enable zfs related services at boot time
[[email protected]_server ~]# systemctl preset zfs-import-cache zfs-import-scan zfs-mount zfs-share zfs-zed zfs.target
Disable RP filter¶
Rp filter should be disabled as well on all interfaces
[[email protected]_server ~]# for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 0 > ${i} ; done
Increase zfs max usable ram space¶
Below there’s an example for a system running with 256 GB of memory for which 205543648256 bytes will be reserved for the arc memory used by zfs.
Modify it on the fly
Configuration example:
[[email protected]_server ~]# echo "205543648256" > /sys/module/zfs/parameters/zfs_arc_max
Make it default at boot-time
Configuration example:
[[email protected]_server ~]# echo "options zfs zfs_arc_max=205543648256" > /etc/modprobe.d/zfs.conf
Creating zpool¶
Below there’s an example of a one, 3 disks, vdev raidz1 zpool creation command. Using /dev/disk/by-id disk terminology in order to avoid problems with possible disk renaming at boot-time.
Configuration example:
[[email protected]_server ~]# zpool create -f Pool-A raidz1 /dev/disk/by-id/scsi-3600508b1001c30f7f6eb0e71ed4fa217 /dev/disk/by-id/scsi-3600508b1001c5c4c279dd5d9b6d7c703 /dev/disk/by-id/scsi-3600508b1001cc79180aad3504bc84d2c
Manually create datasets and set all properties¶
Configuration example:
zfs create Pool-A/Data
zfs create Pool-A/Data/Drives
zfs create Pool-A/Data/Templates
zfs create Pool-A/Internal
zfs create Pool-A/Internal/Logs
zfs set reservation=8M Pool-A/Data
zfs set reservation=8M Pool-A/Data/Drives
zfs set reservation=8M Pool-A/Data/Templates
zfs set reservation=8M Pool-A/Internal
zfs set reservation=8M Pool-A/Internal/Logs
zfs set reservation=16M Pool-A/Internal
zfs set quota=1G Pool-A/Internal
zfs set quota=1G Pool-A/Internal/Logs
zfs set compression=lz4 Pool-A
zfs set checksum=fletcher4 Pool-A
zfs set xattr=sa Pool-A
zfs set atime=off Pool-A
zfs mount -a
zfs list
zfs set bsa:type=Filesystem Pool-A
zfs set bsa:type=Filesystem Pool-A/Data
zfs set bsa:type=Filesystem Pool-A/Data/Drives
zfs set bsa:type=Filesystem Pool-A/Data/Templates
zfs set bsa:type=Filesystem Pool-A/Internal
zfs set bsa:type=Filesystem Pool-A/Internal/Logs
Create saveconfig json file¶
Create LIO saveconfig json file and populate it with the below, to create an empty config.
[[email protected]_server ~]# vim /Pool-A/Internal/saveconfig.json.pool
{
"fabric_modules": [],
"storage_objects": [],
"targets": []
}
Sync API with the manually created zpool and targetcli config file.¶
BSA api can be accessed at http://[mgmt_ip_address]/api
[[email protected]_server ~]# echo '{"method":"target_config_reload","params":["\/etc\/target\/saveconfig.json"],"id":9,"jsonrpc":"2.0"}' | bsa_python.py
Check if BSA API has recognized the newly created zfs pool¶
storage.storage_pools()
Which should return information about the zpool ()
Create iSCSI target IQN and set it’s IP address¶
Use the IP address previously set up on the SAN interface
storage.target_create(["[san_ip_address]"])
Benchmark (if needed)¶
The script creates three large volumes and then performs a series of tests simulating these types of IO operations (read, write, randread, randwrite, rw, randrw) using as well two types of blocksizes to simulate small and large blocksize applications (4k and 128k).
Prerequisites:
fio test tool needs to be installed In the same path as the script a file containing the IO types as a column
#!/bin/bash
zfs create -s -V 900G Pool-A/Data/Drives/fio_test_drive_3
sleep 5
zfs create -s -V 900G Pool-A/Data/Drives/fio_test_drive_4
sleep 5
zfs create -s -V 900G Pool-A/Data/Drives/fio_test_drive_5
sleep 5
a=`ls -lrth /dev/zvol/Pool-A/Data/Drives | grep 'fio_test_drive_3' | awk -F "/" '{print $NF}'`
b=`ls -lrth /dev/zvol/Pool-A/Data/Drives | grep 'fio_test_drive_4' | awk -F "/" '{print $NF}'`
c=`ls -lrth /dev/zvol/Pool-A/Data/Drives | grep 'fio_test_drive_5' | awk -F "/" '{print $NF}'`
iod="128"
bs1="4k"
bs2="128k"
for io in `cat /root/benchmark_script/io_type`; do
fio --filename=/dev/$a --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs1 --direct=0 --size=10G --numjobs=85 --runtime=60 > /root/benchmark_script/results/"$io$bs1$a.txt"
echo "fio --filename=/dev/$a --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs1 --direct=0 --size=10G --numjobs=85 --runtime=60" >> /root/benchmark_script/results/"$io$bs1$a.txt"
fio --filename=/dev/$b --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs1 --direct=0 --size=10G --numjobs=85 --runtime=60 > /root/benchmark_script/results/"$io$bs1$b.txt"
echo "fio --filename=/dev/$b --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs1 --direct=0 --size=10G --numjobs=85 --runtime=60" >> /root/benchmark_script/results/"$io$bs1$b.txt"
fio --filename=/dev/$c --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs1 --direct=0 --size=10G --numjobs=85 --runtime=60 > /root/benchmark_script/results/"$io$bs1$c.txt"
echo "fio --filename=/dev/$c --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs1 --direct=0 --size=10G --numjobs=85 --runtime=60" >> /root/benchmark_script/results/"$io$bs1$c.txt"
fio --filename=/dev/$a --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs2 --direct=0 --size=10G --numjobs=85 --runtime=60 > /root/benchmark_script/results/"$io$bs2$a.txt"
echo "fio --filename=/dev/$a --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs2 --direct=0 --size=10G --numjobs=85 --runtime=60" >> /root/benchmark_script/results/"$io$bs2$a.txt"
fio --filename=/dev/$b --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs2 --direct=0 --size=10G --numjobs=85 --runtime=60 > /root/benchmark_script/results/"$io$bs2$b.txt"
echo "fio --filename=/dev/$b --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs2 --direct=0 --size=10G --numjobs=85 --runtime=60" >> /root/benchmark_script/results/"$io$bs2$b.txt"
fio --filename=/dev/$c --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs2 --direct=0 --size=10G --numjobs=85 --runtime=60 > /root/benchmark_script/results/"$io$bs2$c.txt"
echo "fio --filename=/dev/$c --name=$io --ioengine=libaio --iodepth=$iod --rw=$io --bs=$bs2 --direct=0 --size=10G --numjobs=85 --runtime=60" >> /root/benchmark_script/results/"$io$bs2$c.txt"
done
zfs destroy Pool-A/Data/Drives/fio_test_drive_3
zfs destroy Pool-A/Data/Drives/fio_test_drive_4
zfs destroy Pool-A/Data/Drives/fio_test_drive_5
echo "Finished.Results can be found in path /root/benchmark_script/results"
Adding the storage device to application¶
Check out the guide on how to add the newly created storage pool to application