Mr. Petey’s FreeNAS Project Log

A shout out to the great gang that put together the Run BSD sticker drive. our donation went to the FreeBSD Foundation.

FreeNAS Server as it is currently

• SuperMicro X9SCL-F E3-1230 Xeon Server motherboard
• On-board video, no card
• 16GB ECC RAM
• Onboard SATA
  1 x SSD 119GB PNY, OWC 6G 112GB data errors, WD Green was pitched it caused painfully slow and had to boot 4 times for system to come up.
  1 x HGST 7.2k 3TB
  2 x Seagate IronWolf 5.9k 3TB
• HP H200 HBA SAS-SATA card – IT Mode – eBay
  4 x HGST 7.2k
• 2 x Seagate IronWolf 5.9k 3TB
• Gigabyte 650 Bronze
• FreeNAS 11.3- u5
• FreeNAS 11.3 -u4
• 3x 120mm Fractal Silent Series R2

What can it be?

Below is the the system that I have in place. The issue at hand is when transferring files from one of my Macs to the FreeNAS causes the server to crash, not just a little crash, the motherboard dies. When I push the power button it comes back to life from a cold boot and acts as nothing happened. The logs show nothing.

Both of the Macs I have been using in transferring files are wired with CAT-6 ethernet, motherboard ethernet ports are connected to a NetGear 5-port 10Gb, unmanaged switch, which the FreeNAS is also connected to via 1Gb onboard ethernet. One Mac is running 10.10 and the second 10.13.

I have just obtained a Windoze 10 box. I need this for one of my classes, so windoze is only in my house out of necessity. I have not tried transferring files from it to the NAS yet. It is on the list to do. It is further down because I don’t think the platform is going to make a difference. I could be wrong . . .

20 Oct Update – it lives

Well, after all of that it was a jumper, the JWD watch dog jumper. Once it was set to open, things started to work. I found this tidbit when I posted over on the ixsystems freenas forums. Thanks to everyone who helped track down my issue.

19 Oct Update – busy couple of days, no dice

The new fans are in and do make a difference. The CPU is cooler even during a load; but now it just crashes. Hangs, just sits there, and nothing is going on. as in I am not actively doing anything on the system.

I have a local monitor and keyboard connect so I can have top -u running to keep track of what is going on and as another way to keep an eye if the system is up or not. I am configuring in the web GUI.

SMB share is the primary. There are threads about issues with SMB and the Mac. Setup with fruit extension and a threshold increased (its listed somewhere in these notes). I did a 780MB copy, 1 file, all is good. Temps are good. Copy 230MB, 256MB, and 90MB directories, a bunch of files. Temps are good. Memory is good. CPU isn’t taxed. Here is goes, 2.3GB file, hangs on smbd, or that is the last thing I see.

OK, setup an AFP share, login via afp from my Mac Pro, and go through the same set of copying exercises. BOOM. It hangs on the smbd again. Drives? I have them split between the motherboard SATA and the HBA card. I have ports on the HBA card so lets put all the drives on the card and free up the motherboard so it only has the boot drive going to the motherboard SATA. I was pleasantly surprised that nothing had to be done to the pool. All of the drives and space was recognized and came right up.

Test copy same routine, BOOM, it hangs on afpd.

sigh . . .

A fresh download, dd to flash drive, boot, and fresh installation. I remove the media and reboot. I don’t do anything. I let it sit for over 3.5 hours. Now this is the longest is has ever stayed up. Time to setup a share and check it out. Create a user, all is good. Create an Apple AFP share, SMB is off. Ok, all is good. Lets log in. Yeah, but now with more than a little anxiety, just go for the gusto and upload a 2.3GB file.

BOOM – hangs on afpd.

How to test a CPU?

15 Oct Update

Well doesn’t it figure. The first set of fans had to be sent back as the wrong ones were sent. The wrong ones are sent back and a new set ordered. They are scheduled to be here Saturday. Crossing my fingers!

Lesson in case thermals 9 Oct Update

The AMD has been put back in service as my test rig again. While I am testing and waiting for parts I might as well get the config setup.

Wow, what a lesson on case thermals and a little redesign of my layout plus adding some fans. There are currently 2x 120mm fans, in-take and exhaust, on the CPU side, and 1x120mm exhaust fan on the drive side. They are controlled by a switch hardwired to the case with L, M, and H settings. I have it on H, which should be about 1200rmps.

While a nice case, it does seem that air intake is a little limited. There is probably about 25mm clearance on the bottom, a small section of the front panel, and the top cover. Lifting it up off the surface should help promote a little more air flow. The sides are solid and the back is open for exhaust ventilation. The front, bottom, and top have some type of restrictive mesh on them. Lesson one, static pressure. I have learned that fans with high static pressure are best for environments where air flow is restricted; i.e. mesh, radiators, heatsinks and the such. Plan a is for 2 Nortua 120mm HF-12S fans placed at the bottom, front, on each side. 3x Fractal Venturi HP-12 120mm fans are on order.

The way the motherboard is oriented and is laid out, the CPU is dumping heat up into the RAM. Which is probably further exacerbating my issue. A third fan will go almost above the process and exhaust up and out. This again will be a high static-pressure fan due to the restrictive mesh on the cover.

So the motherboard side will have:

• Exhaust Back – 1x 120mm Fractal Silent Series R2
• Exhaust Top – 1x 120mm Fractal Venturi HP-12
• Intake Lower Front – 1x 1Fractal Venturi HP-12
• Intake Upper Front – 1x 120mm Fractal Silent Series R2

And the drive side

• Exhaust Back – 1x 120mm Fractal Silent Series R2
• Intake Upper Front – 120mm Fractal Venturi HP-12

Its burning up in here 08 Oct Update

It appears that there is a thermal event taking out the board. Each there is a spike shortly thereafter the motherboard dies.

Some notes from today.

• It is now running 11.3-u5 – I don’t remember updating
• I was able to copy via AFP today
• 18GB .img file
• 19GB folder with 3200 files.
• 96GB folder with over 9K files and it died about 50GB in the transfer. There was a spike in the CPU temp and the system died. See above.
• Dedup is off.
• lz4 is on.
• Dies during long transfers
• Dies over a period of time > 8 hours.

I think I have to rethink the cooling in the box. So far it is only what came with the case, 3x 140mm fans. 2 on the motherboard side front and back, and 1 in back for the drives. I have 3x120mm in mind. The 140 is a second front and the 120 will go in the top, almost above the processor. I will have to look at the rats nest of wires in there too. I am sure they are not doing the air flow any favors. The motherboard is a used server board so yes it is going to be used to a pretty heavy airflow. HHHmmmmm, I have some fans that came out of a Dell server. I will have to dig one of those up and see if I can rig it up somehow.

Preamble: As many people I and my family have lots of files. Plus my wife is a professional photographer so there are a lot more files. It has come to the point where I need a large, reliable, fixable, storage device. For the longest time is has been a simple external, dual hard disk enclosures setup as mirrored drives. There are two other external hard disks. These are the backups. Once a week a local backup is done, this drive is kept on-site. The second drive is off-site. Once a month this is swapped with the local disk.

This is of course in theory. In reality it is more along the lines of local backup once a month, due to her having to use the system. And remote when I get the chance, but one is always off-site. Goal – scheduled backups to remote disk.

My wife has a lot of photos but doesn’t need that many at one time. There are quite a few that are kept for archive purposes as well. To date these are in the local disk loop. A disk on the network where they aren’t taking up her local space, but are still readily available. Goal – remote storage

With those two items established it was getting obvious that a Network Attached Storage was in line for this project. There are quite a few out there pick anywhere from two drives to however many you want. And the cost seems to go with them. I do have a budget to stay within; well this project started with that goal in mind, but it is getting stretched to the limit with the current issues.

A pre-made solution was out of the question due to the cost or quality of the build. I am creative and so started playing around with a couple of options. Being a MacMan, I figured I would take an old Mac Pro 5,1 with Mac OS X 10.13 with Server, stuff some drives in it, and run a software RAID. There are two posts if not more about why that solution went sideways and not used. Next . . .

I have an old Lenovo Thinkcentre AMD stuffed with 12GB of RAM and two, 2.5″ 500GB hard disks. I tossed on a fresh install of Debian. When it comes to linux Debian is what I know. I first installed it back in 1996 and have noodled with it off and on over the years; and more so lately having a couple of RPi’s around the network. Overall, I just couldn’t find a solution that I liked and wanted to administer over time.

Sidebar – I have been trying to pick a new desktop OS, Apple has finally worked my last nerve with the Mac OS, Linux or BSD. Linux has always seemed more that a little messy, like “Doc” Brown. Everyone has a distro for everything and so many of them overlap. I used Debian for my testing. The BSD family has always been tidy and solid. I used FreeBSD for the other side. In doing so I found TrueNAS/FreeNAS/CoreNAS. I was intrigued.

Above is about the furthest I have gone in really noodling around with hardware since I built my first computer back in 1988. So building up the NAS is kind of fun and a real pain in the ass as well. I thought I had things pretty well planned, but then even the best laid plans tend to go sideways.

I pulled down a copy FreeNAS 11.3-u3 and installed it on the little AMD with an external 4-drive SATA cage with 3x 2TB drives and started testing. I tell you what I was sold after the first couple of days playing around. I have scratched the surface with ZFS on my desktop, but this is a chance to really dive into it.

The interface is very nice to work with. It has two types of users inside and outside. The inside accounts are ones that come with the system such as root. The outside accounts are for admin and user accounts. AFP and SMB shares are fairly easy to setup. The basic config panels keep things simple to get something up and running quickly. The advanced options allows you to get a little more specific about the configurations, such as adding the fruit option on SMB Shares for MacOS clients. My storage needs are simple so setting up the pools was as well. All 6 drives are configured as one pool using raidz-2. This provides me with a little over 10TB of storage and allows for up to 2 drives to fail. I have a 4T on hand if things ever came down to that.

The hunt for hardware began. Here is the story with the hardware and its’ issue.

Mr. Petey’s FreeNAS Troubleshooting Sheet

X 1. The first crash –> One outside user, one smb share.

I can see the share, mount the share, start to copy to the share, and the server will hard crash. It just goes down. This is after transferring approximately 9GB of data using a SMB share.
The power button has to be pressed and powered back up, when it is the board/system starts to come back up again as if it were a cold boot.

The drives are still spinning.

The copy was for approx 12GB and over 5k files. It died at approx 9GB.

** set to debug mode
** there is nothing in any of the logs. They just stop and then start with the reboot. 

** Originally it was two SSD drives mirrored on the onboard SATA, but one or the other drive kept throwing up data errors. 
Paired it down to one drive.

Check all of the connections.
Maybe just a fluke. . .  

X 2. With just one boot SSD, fresh reinstall of FreeNAS. A new copy was downloaded and burned to CD.
One outside user, one smb share fruit enabled.

Copy was for same approx 12GB/5k and died at 9.5GB

** research into SMB and Mac OS. 
** add fruit option/setting
** SMB tuneable to net.inet,tcp.maxqueue<something>    changed from 100 to 14400

Same copy 12GB/5k, same hardcrash. 

** New outside user, new AFP share.
Same copy 12GB/5k, died approx 7GB
New copy - one 6.75GB file - copy fine. 
Second copy - one, 5.9GB file - died approx 3GB

** research possible causes random(?) crashes during file transfer; specific to freeNAS, drive setup, configuration, 
    A bug???
    How to collect more specific information?

X 3. Power supply
Originally it was a Gigabyte 450 80 Bronze, but after refining my math
a little bit, I was close to the limit on the 450. The PS can really provide about 360 and what I need is, one the high-end, 350; so I thought an upgrade might fix it issue. The larger supply also allows for the addition of drives. There are 3 more open slots. Once the 650 was installed a fresh installation of FreeNAS was placed on the SSD boot disk time to start the transfers again.

    Replaced with Gigabyte 650 80Bronze.

    SMB share- same as above
    Copy 12GB/5k 
    Hardcrash approximately 9GB

    Did Not fix the issue; but there is now power to spare.  

X 4. Memtest86 v4.3.7
Intel Xeon E3-1230 V2 @ 3.3GHz
L1 Cache – 64K
L2 Cache – 256K
L3 Cache – 8192K
Memory – 16GB (ECC RAM)
Time: 1:52:41
Iterations: 6
AdrsMode:64bit
Pass: 1
Errors: 0

It appears that the RAM is ok. 
It reported that 4 of the 8 cpus were used. 
**  still need to rule out processor, but they didn't    fail during the Memtest. 

*** Left house for work today 7 Oct, 1:45pm. At approximately 1:15pm the machine was booted as configured above. top was left running on a local shell. It did shutdown once before sometime during the night. Checking to see how long it stays up with nothing going on. News at 11, quite literaly.

Dead as a door nail when I got home. Huh?!? See updates posted above.

__ 4.5 Processor cooling solutions. From the notes above it seems to be a thermal issue that is taking out the processor/motherboard.

__ 5. Setup Windows 10 box and transfer the same files over to the FreeNAS. Could the issue be platform or protocol specific?

__ 6. HBA card only, remove SATA from motherboard, save the boot drive.
Move all data disks to the HBA card.
3 per channel/port/connector

__ 7. Remove HBA card and run motherboard SATA only.
only 6 SATA ports on motherboard.
Move all 6 drives to motherboard
Boot from USB flash drive – internal USB port on motherboard.

__ 8. Processor test – need to find one. Use benchmarks for test?

© 2020, wrightmac. All rights reserved.