State of My Home LAN

Things settled here.  NAS4free proved to be fabulous.  Stable.  With regular updates steadily improving on visible cosmetics which gives me confidence in behind the scene work. Very convenient for use.  Therefore it is not surprising that I ran out space on the main pool I use to store video files.  I also need to add new storage to NAS and migrate photos and Lightroom catalog there.  My NAS box found its permanent place on a network shelf in the laundry room.  I love the case and the 3- and 4-drive racks are very convenient.  And, yes, they look just cool.
HTPC got a passive CPU cooler and the HD rack was removed to free space for it.  HTPC is still noisy though - 7k rpm HD and power supply can be easily heard.  But when music or movies are played it does not matter.  I doubt I will ever install SSD in this HTPC, if only to cut on noise.  Can I even migrate the existing Windows 7 install to SSD?  I remember reading that this NVidia disk controller does not handle SSD well.  I suspect the main culprit is power supply fan anyway.

NAS4free - Minor GUI Improvement

I love NAS4Free GUI.  It shows information I am interested in and is highly usable.  A minor change (removal of an outer frame below the toolbar) makes the GUI cleaner.  Here is an animated show with before/after examples.

Hardware, New Iteration

Here is my new NAS box:

• CPU: Intel Core 2 Duo 65nm E4300 Allendale 1.8GHz 800MHz FSB LGA 775 65W
• Motherboard: MSI G41M-P25 with Intel G41 video, Intel ICH7 south bridge with 4x SATA2 ports and Realtek 8111DL gigabit Ethernet controller.
• RAM: 2x4GB DDR3 1333mhz PC3-10666
• HBA:  I am playing with HDs connected to motherboard's ICH7.  I also have an IBM M1015 (with cables) re-flashed to LSI9211-IT.
• HDs:
    mirror of 2xSeagate Barracuda Green 2TB ST2000DL003,
    assorted drives for tests, all in this 4-bay and this 3-bay rack.  Fan controlled by this gadget.
• DVDROM: attached only for new installation, not enough external bays to have it full time.
• PSU: Antec EarthWatts Green EA-380D Green 380W
• Case: Cooler Master Gladiator 600
• UPS: APC BACK-UPS ES BE550G connected to USB port

Interestingly enough, this configuration refused to boot with both memory sticks installed.  Overclocking from 200MHz to 266MHz solves the problem.  I conclude that the memory is just too fast for such a slow CPU.  As a result CPU runs at 2.3GHz.  Not a big deal, given that SpeedStep throttles it down most of the time anyway.
Setup ssh.

NAS4Free, Take2, Success!

I fixed it!  That Samba crash I had while reading data from NAS - it is fixed!
I recreated mirror zpool, redid all CIFS/SMB settings, recreated shares, restarted the service and it just works!  Let me dig deeper to figure out what exactly had the effect!

Open Indiana, Take2

While nas4free is being solidified  - samba crash there takes all the wind out of my excitement - I am giving another try to (Samba-free!) OpenIndiana 151a.  I have very high hopes for kernel-based CIFS support. This time a new storage adapter (IBM M1015 re-flashed to LSI9211-IT) is plugged in but not connected - the cables are still on a slow boat from China.
As before, the default install pukes on X boot.  Choosing VESA helps.  More than that, this time it just boots in native 1280x1024!  Last time I spent hours trying to achieve this.  And now it just happens with no effort on my part.  This time I also have a pair of Barracuda Greens connected.  My plan is to create a mirrored zpool and check out file transfer speeds between NAS and Windows 7 htpc. I will take another shot at enabling SMART and SpeedStep!

Sanity Checks

Nas4free seems to be most suited to my requirements, but samba there just crashes on me.  In the meantime I am collecting network and hard drives throughput data on Windows.
HTPC runs Windows 7, NAS in this test runs Windows Vista. Both use Intel 82540EM PRO/1000 MT PCI NICs.
Rate of data copying between two internal SATA2 drives on HTPC: windows copy dialog shows 120MB/sec in the beginning of the copy process.  By the end it falls down to 87MB/sec.
Writing to NAS (to 300MB Seagate HD) is done at a rate of 43MB/sec.
Reading from NAS is done at a rate of 40MB/sec.
iperf bandwidth measurement between HTPC and another Windows PC is about 38MB/sec.  Enabling jumbo frames (or changing their size from 4k to 9k) has no effect at all.  I have no idea why the bandwidth is so low.  I ordered a CAT6 cable and will try one more time bandwidth tests with two windows PCs connected directly with no switch in between.
EDIT: later I determined that PCI NICs are the bottleneck!

Todo:

• install syslog on the LAN to collect debugging info, e.g.  syslog-win32
• Experiment with Wake-on-LAN - here is client which would wake NAS up.

While waiting for the production version of firmware to materialize and production HBA to show up on my door step I was pondering over the hard drive use strategy.  Should I go with RAIDZ?  Given that the NAS cage can hold up to 4 drivers, and I do not plan to expand, my options are limited to RAIDZ and MIRROR.  Here are two good articles on the subject.  Decided!  I will go with a mirror of two Barracuda Green 2TB drives.

NAS4Free 9

Install of NAS4Free 9.0.0.1.43 on to a flash stuck in PATA was uneventful. But unlike Solaris, OpenIndiana and FreeNAS the web GUI showed me exactly what I wanted to know - real time status of NAS appliance: CPU frequency and load, SMART status of hard drives.

Power management worked and CPU frequency was adjusting according to load! On Solaris and OpenIndiana smartctl never worked for me. On FreeNAS smartctl did work but powerd did not and GUI was lacking this info.
I imported a ZFS hard drive created on FreeNAS8 and jumped to network throughput testing. Writing to NAS (I copied a 5.08GB DVD from win7 PC) went at around 40MB/sec.
Reading from NAS begins but then fails. Windows shows "Network Error" dialog "There is a problem accessing \\nas\zfs".
dmesg tells me "pid 7239 (smbd), uid 0: exited on signal 6"
log.nmbd and log.smbd contain nothing but startup messages.
Houston, we have a problem.

I will try to re-create ZFS vdev and pool and to reproduce.
More from the GUI diagnostics log:

Mar 31 22:25:05 nas smbd[7102]: [2012/03/31 22:25:05.441422, 0] lib/fault.c:51(fault_report)
Mar 31 22:25:05 nas smbd[7102]: ===============================================================
Mar 31 22:25:05 nas smbd[7102]: [2012/03/31 22:25:05.441703, 0] lib/fault.c:52(fault_report)
Mar 31 22:25:05 nas smbd[7102]: INTERNAL ERROR: Signal 11 in pid 7102 (3.6.3)
Mar 31 22:25:05 nas smbd[7102]: Please read the Trouble-Shooting section of the Samba3-HOWTO
Mar 31 22:25:05 nas smbd[7102]: [2012/03/31 22:25:05.441749, 0] lib/fault.c:54(fault_report)
Mar 31 22:25:05 nas smbd[7102]:
Mar 31 22:25:05 nas smbd[7102]: From: http://www.samba.org/samba/docs/Samba3-HOWTO.pdf
Mar 31 22:25:05 nas smbd[7102]: [2012/03/31 22:25:05.441786, 0] lib/fault.c:55(fault_report)
Mar 31 22:25:05 nas kernel: pid 7102 (smbd), uid 0: exited on signal 6
Mar 31 22:25:05 nas smbd[7102]: ===============================================================
Mar 31 22:25:05 nas smbd[7102]: [2012/03/31 22:25:05.441813, 0] lib/util.c:1117(smb_panic)
Mar 31 22:25:05 nas smbd[7102]: PANIC (pid 7102): internal error
Mar 31 22:25:05 nas smbd[7102]: [2012/03/31 22:25:05.441883, 0] lib/util.c:1221(log_stack_trace)
Mar 31 22:25:05 nas smbd[7102]: BACKTRACE: 0 stack frames:
Mar 31 22:25:05 nas smbd[7102]: [2012/03/31 22:25:05.441976, 0] lib/fault.c:416(dump_core)
Mar 31 22:25:05 nas smbd[7102]: dumping core in /var/log/samba/cores/smbd
Mar 31 22:25:05 nas smbd[7102]:

Off to read Trouble-Shooting section of the Samba3-HOWTO

FreeNAS 8

I am impressed!  The installation lasted 10min 2sec.  FreeNAS 8.0.4 (FreeBSD 8.2, ZFS file system v4, ZFS storage pool v15) was installed on a 2GB flash card plugged into the PATA port. Reboot.  Reconfigured NIC for a static address.  Launched web GUI.  The GUI is kinda strange but totally usable.  Created a user.  SSHed into NAS to poke around. Left top running.  The rest was done solely through web GUI.
Imported NTFS volume, shared it through CIFS, measured network throughput by copying a 5.08GB DVD image.  NTFS is not really writable, but who cares about NTFS on NAS!
Created a UFS volume on a HD I used for OpenIndiana install - it was not recognized as ZFS and no option to import was offered.  Had to fiddle with CIFS service for the right permissions to become visible to Windows.  Measured network throughput. Pretty good!
Removed CIFS share and volume and created a simple ZFS volume forcing a 4KB sector.Again restarted CIFS to propagate the changes.  Measured network throughput.  Good!

FreeNAS 8.0.4 network (1Gbit, default frames) throughput as measured from a Windows 7 pc.  On the NAS side the following hard drive was used: WDC WD5000AAKS-00V1A0.

File System	Read (MB/s)	Write(MB/s)
NTFS	40	<1
UFS	75	35
ZFS	76	39

These are very solid results!  The biggest (and still unresolved) problem I had was in finding an SMTP server to accept reports e-mailed by NAS into my public mailbox.  So far none!
Update: resolved! Used google with application-specific password!
Yes, the versions of ZFS and even FreeBSD used in FreeNAS8 are not the latest and greatest.
There is no real time SMART status in web GUI. But smartctl works as expected which means you can hope that the GUI will catch up.
Overall it looks promising.  Good job, FreeNAS8 team!
I will check out FreeNAS 0.7.5.9496 tomorrow - look forward to FreeBSD 9.0-Release, ZFS storage pool v28, ZFS filesystem v5.
Scratch that! Instead I'll try NAS4Free 9.0.0.1.43
Improve benchmarking: use iperf.

OpenIndiana with Napp-it

napp-it site is a good source of information on Solaris flavors.  Decided to go with its advise and try OpenIndiana in hope that SMART information can be retrieved from the hard drives connected to motherboard.  This time used live CD.  It IS more user friendly than text install.  No problems with network setup this time!

Here are my notes:

• Still no SMART info.  Major bummer.  A storage appliance which can not tell the temperature of the hard drive or do SMART diagnostics is a joke.
• X kernel panics when intel video driver is used. So I am forced to use vesa driver instead.  Which does not support resolution higher than 800x600.  Which, in turn, makes some configuration GUI unusable.  Will try to follow these recommendations. 
  Tried it! Got nothing but kernel panic. So I am stuck with 800x600 resolution.
• Spent some time learning how to force boot into single user mode from GRUB prompt.
• disabled graphical login:
  svcadm disable svc:/application/graphical-login/gdm 
  Did not figure out how to start gnome desktop for those cases when GUI is desired, so later enabled GDM back.
• verified that CPU SpeedStep capability is recognized:
  
  

  alex@nas:/etc/X11# kstat -m cpu_info -s supported_frequencies_Hz
  module: cpu_info                        instance: 0
  name:   cpu_info0                       class:    misc
          supported_frequencies_Hz        1203000000:1803000000
  
  module: cpu_info                        instance: 1
  name:   cpu_info1                       class:    misc
          supported_frequencies_Hz        1203000000:1803000000
  

  
  Too bad this capability is never used and I always have:
  

  alex@nas:/etc/X11# kstat -m cpu_info -s current_clock_Hz
  module: cpu_info                        instance: 0
  name:   cpu_info0                       class:    misc
          current_clock_Hz                1803000000
  
  module: cpu_info                        instance: 1
  name:   cpu_info1                       class:    misc
          current_clock_Hz                1803000000
  

I am really disappointed with my inability to take advantage of SMART and feel this is a must have.  So I decided to jump the gun.  Investigated HBA controller options.  Done!  Ordered adapter and cables.
Finalized the plan for hard drives.  I will start with a mirror of 2x3.5" drives for performance testing and will take it from there.  OS will be stored on a mirror of 2x2.5" drives.  Got it!  Not to forget the cage!  I grew to like these!  So convenient for fast changes!  Looks like I can't move OS to a new smaller drive - another reinstall is in order.
Update: after trying FreeNAS, I decided to stick with it.  Or NAS4Free.  At least for now.  Booting from flash is soooo attractive.  So I cancelled an order for 2.5" drive and the cage.
Wooof.

Installing Solaris with Napp-it

Downloaded and burned text installer CD.  Text installation reminded me of good old slackware days.  Anyway.  On the first run the installer hung after the network settings page - I chose automatic config there.  Could it be the installer was looking for some LDAP/NIS server on my home LAN?  On the second run I chose to provide network settings manually and that helped.  I understand network layer went through substantial changes.

Reboot!  I have an ascetic login prompt.  Oh and sendmail complains it can't qualify my own domain.  Boy!  Some things never change
I type a single line to install napp-it, and after a sanity check (it confirms that Solaris 11 is supported!) off it goes to download stuff.  Lotsa stuff.  Now it loads gcc.  BTW, no progress bars.  These were so 20th century..
It worked!  I can now point my browser to nas and get to the napp-it GUI!  Also ssh works!  Set putty to use utf8 so that mc draws lines properly.
Did you notice how fast it reboots?  The fist time I thought something went wrong.  It does not go to BIOS!
Still for me to do:

• Solaris: set up e-mail delivery 
• Solaris: configure SSH to use public keys.
• Solaris: play with console setup - make sure mc can draw lines, maybe adjust # of cols/lines
• Solaris: verify that power management /speedstep works
• Solaris: verify that HDs SMART info is accessible
• Solaris: make rpool a mirror
• Solaris: CIFS server - is it configured or not?
• napp-it: verify HDs temps can be monitored

Software Options

I am excited about ZFS.  This entire project is about giving it a shot.  The promise of a reliable and easy to use file system is too big to pass.  I am excited to learn new things done in my neighborhood - it's not like new file systems appear that often.

FreeNAS

The first contender I considered was FreeNAS.  I love, love, love the idea of booting the system from flash and having a web-based administration GUI.  As a matter of fact, I made these my requirements.  Unfortunately I hear FreeNAS8 lacks in functionality and all versions of FreeNAS, by virtue of being based on FreeBSD, lag behind in the version of ZFS.  I also hear ZFS performance is lacking.  I do plan to give FreeNAS a shot!

Solaris 11 + napp-it

ZFS being native to Solaris/Illumian/NexentaCore/OpenIndiana I hope to find here the best features and performance.  Hope that napp-it will offer a functional and friendly GUI.  I understand Solaris has kernel-based CIFS server promising (unlike Samba) a good interoperability with Windows.

We shall see!

NAS Hardware

Here is an old desktop, once built for my parents to surf internet and look at their grandchildren pictures.

• CPU: Intel Core 2 Duo 65nm E4300 Allendale 1.8GHz 800MHz FSB LGA 775 65W
• Motherboard: MSI G31TM-P21 LGA 775 Intel G31 Micro ATX.  Onboard video, 4x SATA ports on ICH7.

It is now souped up:  RAM is maxed out to 8MB.  New hard drive cage is installed. It proved to be loud enough to do something about it.  Just disconnecting the cage fan results in HD temps raising to 50C within an hour, and that's with just 2 of them installed.  Placed an order for a fan controller - hope that will help.  The cage is also designed so that the connectors - all of them - are on the farther side so that after installation none of these are accessible in my case.  Hence I installed it upside down.

Rationale

At home the (only) "desktop" I use is an HTPC.  Noisy at that, because of the need to keep and cool large storage for media playback and editing.  I wonder whether something could be done.
Main requirements are performance, ease of use, reliability and ease of growth.
Copying between internal SATA HDs is being done at about 70-100MB/s.  This sets performance goal.

Reliability is now achieved by keeping multiple backup HDs.  The backup copies are frequently obsolete, but you can't beat the simplicity of sticking a bare HD into a docking station (the HTPC needs to be off) and launching a synctoy.  Growth is straightforward.  Every other year I buy a new HD, copy media to it, replace the media HD in the HTPC with a new one and relegate an old one to backup duties.
For a couple of years I was running a Linux box (I don't think term NAS was widely used then) with a software RAID5.  I verified a procedure (based on mdadm) to introduce new hard drives followed by volume expansion.  But boy, every time the thing would boot, watching it fschk'ing the drives was nerve wracking.  And write performance was underwhelming to say the least.