Notes on Head-less Server Set-up

I've got this ubuntu (linux mint actually) and want to make a decent headless server out of it.  I am thinking about:

• ssh server - setup instructions
• vnc server

The latter proved to be

• hard to set-up as a service
• Image compression artifacts were ruining it for me.

So the solution proved to be xrdp!  Easy to setup and works perfectly.  Standard Windows client works just fine.

Analog DC Load Build

Inspiration

https://www.youtube.com/watch?v=2jSdfRM_Mm8

Schematics

Assembled PCB

ebay

 Enclosure

ebay

Reading Temperatures - CPU and HDs

FreeBSD

On my pfSense router and NAS box the following works:

nasa: /etc# sysctl dev.cpu | grep temperatu
dev.cpu.3.temperature: 39.0C
dev.cpu.2.temperature: 39.0C
dev.cpu.1.temperature: 38.0C
dev.cpu.0.temperature: 41.0C
nasa: /etc# sysctl -n dev.cpu.0.temperature
38.0C
nasa: /etc# sysctl -n dev.cpu.1.temperature
37.0C
nasa: /etc# sysctl -n dev.cpu.2.temperature
39.0C
nasa: /etc# sysctl -n dev.cpu.3.temperature
40.0C
nasa: /etc# sysctl -n dev.cpu.4.temperature
sysctl: unknown oid 'dev.cpu.4.temperature'

Linux

On my VM server (RHEL7) acpi proved to be useless. lm-sensors proved to be useful, especially drivers coretemp-isa-0000 for reading CPU temps and (on this mb) it8728-isa-0a30 for reading fan RPM

[root@hyper alex]# yum install lm-sensors
[root@hyper alex]# sensors-detect
[root@hyper alex]# sensors
acpitz-virtual-0
Adapter: Virtual device
temp1:        +27.8°C  (crit = +105.0°C)
temp2:        +29.8°C  (crit = +105.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +46.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:         +41.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:         +40.0°C  (high = +80.0°C, crit = +100.0°C)

it8728-isa-0a30
Adapter: ISA adapter
in0:          +0.04 V  (min =  +0.00 V, max =  +3.06 V)
in1:          +2.03 V  (min =  +0.00 V, max =  +3.06 V)
in2:          +2.03 V  (min =  +0.00 V, max =  +3.06 V)
in3:          +2.00 V  (min =  +0.00 V, max =  +3.06 V)
in4:          +0.01 V  (min =  +0.00 V, max =  +3.06 V)
in5:          +1.75 V  (min =  +0.00 V, max =  +3.06 V)
in6:          +1.50 V  (min =  +0.00 V, max =  +3.06 V)
3VSB:         +3.41 V  (min =  +0.00 V, max =  +6.12 V)
Vbat:         +3.29 V
fan1:        1108 RPM  (min =   10 RPM)
fan2:         528 RPM  (min =    0 RPM)
fan3:           0 RPM  (min =    0 RPM)
fan4:         531 RPM  (min =    0 RPM)
fan5:           0 RPM  (min =    0 RPM)
temp1:        +32.0°C  (low  =  +0.0°C, high = +70.0°C)  sensor = thermistor
temp2:        +25.0°C  (low  = +127.0°C, high = +127.0°C)  sensor = thermistor
temp3:        +37.0°C  (low  =  +0.0°C, high = +80.0°C)  sensor = Intel PECI
intrusion0:  ALARM

[root@hyper alex]# sensors -A coretemp-isa-0000|grep "id 0"|awk '{print $4}'
+42.0°C

Windows

On my workstation the following works:

nasa: ??

Testing

I used stress-ng package to stress CPU:

[root@hyper ~]# stress-ng -c 4  -v

Making Some pfSense

Not that I really need it, but an idea of getting rid the household of ads was compelling enough.  So I got myself a little "industrial PC", added a stick of RAM and mSATA SSD from an old laptop and installed pfSense in no time!

As you can see, a SATA cable is provided - this is important because the other end uses a proprietary connector.  This cable created a rattling noise which made me worry when I first got my hands on the box.  The cable goes unused.

The enclosure is all metal, and there are no moving parts inside.  The CPU heat-sink is touching external wall of the enclosure - you can see the PCB is flexing noticeably.

There are 4 Intel NICs onboard!  pfSense assigns names in a somewhat unexpected manner:

Port#	pfSense Default Assignment
1	Internet
2	Opt2
3	Opt1
4	LAN

mSATA SSD and RAM installed.  There is a slot for a WiFi card which I left unpopulated.  I rely on Ubiquity Access Point for that.

Power supply brick.  Very unimpressive.
It idles at 7W, spikes to 14W.

pfSense installation went smoothly, all components were recognized during the install - you do need to tell pfSense to use AES hardware. It does dissipate a decent amount of heat, so that after running a night in an enclosed cabinet the temperature went up to 50C. Adding a vent reduced temperature 35C. CPU cores temps reported at 36C. 

I run DNSmasq, DHCP server, Squid, ntpd and ntopng (highly recommended). I was surprised how much HTTPS and how little HTTP is there, but Squid did manage to cache iTunes updates. CPU use spikes to 32% but most of the time it is below 5%.

Replacing HDs in RAIDZ2

Here is a plan on how to replace a defective HD in RAIDZ2:

1. Take the Defective Disk offline

# zpool offline 'tank' '/dev/ada1'

  pool: tank
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
 Sufficient replicas exist for the pool to continue functioning in a
 degraded state.
action: Online the device using 'zpool online' or replace the device with
 'zpool replace'.
  scan: none requested
config:

 NAME                      STATE     READ WRITE CKSUM
 tank                      DEGRADED     0     0     0
   raidz2-0                DEGRADED     0     0     0
     ada0                  ONLINE       0     0     0
     15564787101436659472  OFFLINE      0     0     0  was /dev/ada1
     ada2                  ONLINE       0     0     0
     ada3                  ONLINE       0     0     0
     ada4                  ONLINE       0     0     0

errors: No known data errors

2. Physically Remove the Old Drive

Skipped.

3. Physically Add the New Drive

Done.

4. Partition the New Drive

# gpart create -s GPT ada5
ada5 created

5. Replace the Old Drive with the New One in the Pool

# zpool replace tank 15564787101436659472 ada5
# zpool status
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Jun  5 10:51:44 2018
        3.46G scanned out of 9.48T at 322M/s, 8h34m to go
        706M resilvered, 0.04% done
config:

        NAME                        STATE     READ WRITE CKSUM
        tank                        DEGRADED     0     0     0
          raidz2-0                  DEGRADED     0     0     0
            ada0                    ONLINE       0     0     0
            replacing-1             OFFLINE      0     0     0
              15564787101436659472  OFFLINE      0     0     0  was /dev/ada1
              ada5                  ONLINE       0     0     0  (resilvering)
            ada2                    ONLINE       0     0     0
            ada3                    ONLINE       0     0     0
            ada4                    ONLINE       0     0     0

errors: No known data errors

6. Wait for the Resilver Process to Finish

# zpool status
  pool: tank
 state: ONLINE
  scan: resilvered 1.90T in 6h8m with 0 errors on Tue Jun  5 17:00:00 2018
config:

 NAME        STATE     READ WRITE CKSUM
 tank        ONLINE       0     0     0
   raidz2-0  ONLINE       0     0     0
     ada0    ONLINE       0     0     0
     ada4    ONLINE       0     0     0
     ada1    ONLINE       0     0     0
     ada2    ONLINE       0     0     0
     ada3    ONLINE       0     0     0

errors: No known data errors

Big Reshuffle

Those attempts to run VirtualBox on nas4free were a real nice try.  But now I want to have a dedicated VM server.  So I decided... to build a new NAS box.  In its previous life NASA (as it will be called) used to be my HTPC.  In this new incarnation it will get a better Ethernet and SATA adapter.   New NASA box will be:

• smaller on the outside, yet bigger on the inside
• run nas4free
• will serve disks to a separate VM server - we shall name it Hyperion
• Hyperion will be built out of part of the old NAS box

Here is the current spec for NASA:

CPU	Intel Core i3-3225 Ivy Bridge 3.3 GHz 55W LGA1155
Motherboard	ASRock H77 Pro4-M mATX
RAM	Team 8GB (2 x 4GB) 240-Pin DDR3 1600 (PC3 12800) 9-9-9-24 1.5V
HBA	HighPoint Rocket 640L 4-Port PCI-Express 2.0 x4 SATA 6Gb/s
Ethernet	Dell Intel I350-T2 Dual Port 1GBE Network Card XP0NY
Video	Intel HD4000
PSU	Seasonic FOCUS 450 Gold SSR-450FM 450W 80+ Gold ATX12V & EPS12V 140mm
Case	Fractal Design Define Mini Black Silent mATX

The best part of the build is the case.  It is awesome.  The details, such as dampening material on the cover, HD cage, provisions for cable  management are fantastic.

Intel Ethernet NIC is a top notch and provides two channels!  I may even play with LACP.

Motherboard has 2 SATA3 ports provided by Intel H77 chipset.  2 more SATA3 ports are provided by on-board  ASMedia ASM1061 chipset - I decided against usint it.  H77 has 4 more SATA2 ports.  I thought those will not cut it.  I am reconsidering.  a $20 HighPoint Rocket 640L uses Marvell 88SE9230 chipset and the more I read about it the less confident I feel about running NAS on it.


nas4free immediately recognized all the hardware and I managed to build myself a RAIDZ2 pool using 5 WD Reds.  Too bad I went cheap and bought used drives.  Two of them have to be replaced.  Already.

I will try to move HDs to SATA2 ports and if the box will be still able to sustain 100MB/sec transfer rate (which I think it should) I will stick to motherboard SATA only.

Setting up VirtualBox on NAS4Free

Turn VirtualBox On

Make sure the selected folder does exist.


No matter how I tried, I can not shut VM gracefully. So I would just send ACPI shutdown to a VM. It would be marked as Stopping and just hang there. I usually restart VistualBox using this GUI.

Guest VM Settings

Few comments on VM settings:

• Disable nested paging
• CD is mounted to an image of the Ubuntu server installation CD.

Guest Set-up

Install hung on me in preparing headers phase at 83%.  Seems to common.  Another time it told me linux install failed.  Finally I went back to the VM settings and set the HD to be SSD.  Then install completed.

Helpful: instructions to set Ubuntu as a guest OS.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install dkms
sudo apt-get install virtualbox-guest-dkms

The latter seem to install 5.1.34 modules while my host runs VirtualBox 5.2.8.  Hmmm .  I dig have a guest additions CD for 5.2.8.  Should have used that!  Tried it.  Install script succeeds but upon reboot VirtualBox service fails to start and SATA erros are abound.  I will be sticking with older guest add-ons - at least those start.

To install default desktop GUI I did:

sudo apt-get install ubuntu-desktop

But during this installation I get:

ata3.00: exception Emask 
ata3.00: failed command: FLUSH CACHE

And if I resume and continue the installation, it does get completed, but then upon the reboot into graphical mode, the VM is aborted.
So, it is not an unmitigated success by any stretch.

What's New?

By now I run NAS4Free 11.1.0.4 (5321).

The hardware was replaced more than once with little to no disruptions.  This is a mixed bag of recycled desktop parts, but it does the job:

CPU	Intel Core i3-4130 Haswell 3.4GHz LGA 1150 54W Dual-Core CPU BX80646I34130
Motherboard	GIGABYTE GA-Z87-D3HP LGA 1150 Intel Z87 ATX Motherboard
RAM	Crucial Ballistix Sport 16GB kit (8GBx2) DDR3-1600 1.5V 240-Pin 9-9-9-24 BLS2CP8G3D1609DS1S0
DVD ROM	none
PSU	Antec EarthWatts Green EA-380D Green 380W
Case	Cooler Master RC-600-KKN1-GP Gladiator 600 Mid Tower ATX Case with 140mm Fan (Black)
Case Fan	Rosewill ROCF-11003 Hyperborea 140mm
Case Fan HDs	Rosewill ROCF-11003 Hyperborea 140mm
UPS	APC BACK-UPS ES BE550G

Hard drives are now connected directly to the motherboard - no more HD cages or add-on controller card.  Cages were confusing to me when troubleshooting HD SATA problems. I still run two mirror pools and this works well for me so far.

Disk	Size (MB)	Description	Device model	Serial number	RPM	Pool
ada0	6TB	Seagate IronWolf NAS HD	ST6000VN0041-2EL11C	ZA17617E	7200	BMP
ada3	6TB	Western Digital Red	WDC WD60EFRX-68MYMN3	WD-WXN1H84576NR	5700
ada1	6TB	Western Digital Red	WDC WD60EFRX-68MYMN1	WD-WX21DC4D5XPZ	5700	vault
ada2	6TB	Western Digital Red	WDC WD60EFRX-68MYMN2	WD-WX11D841V6X0	5700

What's really new - my attempts to run virtual machines on this NAS box.  NAS4Free 11.1.0.4 has VirtualBox 5.2.8 with phpvirtualbox 5.2.0 as web GUI. More details in a separate post.
Relevant:
http://www.dotnetmafia.com/blogs/dotnettipoftheday/archive/2010/09/22/fix-high-guest-cpu-utilization-in-virtualbox-by-disabling-nested-paging.aspx
https://help.ubuntu.com/community/Ubuntu_as_Guest_OS