No fluff, just TN (not even enough time to spell out "TrueNAS"). Hardware first, then ZFS, then TrueNAS config, and finally some frequently-asked questions.
In general, buy used when it makes sense to do so. Used and refurb HDDs aren't recommended. Used enterprise SSDs are probably okay. Be wary of sellers resetting SMART data!
A 4 core CPU is a good starting point but 6-8 cores are better if you can afford it. Run more cores (10-16) if you're supporting a lot of users and/or need high performance. Go multi-socket if you need even higher performance. Look for higher clock speeds versus more deeply-threaded CPUs. Used Xeons are stupid cheap, Intel Xeon Haswell or Ivy Bridge (v2 and v3) are great and run DDR3 (which is also stupid cheap). Ideally, aim for PCIe 3.0 support at minimum. If you want to use ZFS encryption, get a beefier CPU and make sure it has AES-NI support.
More RAM is better. 8GiB is fine but 16GiB is a better starting point for small systems. 32GiB is a good starting point for medium-sized systems and 64-128GiB for large systems. Run more RAM if you need higher performance, especially if your application re-uses data a lot (like VMs, media editing, large office file shares, etc). More lower-speed RAM is better than less higher-speed RAM. If you want to run a ton of RAM, consider a multi-socket system. Get ECC RAM if you can afford it and your motherboard and CPU support it.
Look for something compatible with your CPU and RAM and then try to find a board with IPMI. If you can find something with onboard 10GbE (ideally with an Intel network controller) you can save a PCIe slot. If you're buying used gear, try to get at least PCIe 3.0 support.
Shucking external drives will give the best bang for the buck. Check https://shucks.top/ and wait for a sale. Do not get SMR drives, make sure all your drives are CMR. 5400 RPM vs 7200 RPM won't make a huge performance difference. 7200 RPM drives use a lot more power and put out a lot more heat. Don't buy 10K and 15K drives (unless they're on a huge sale). SAS or SATA drives are fine (SATA usually a bit cheaper) but don't mix the two on the same expander chip. Whatever capacity drive you buy, plan to stick with that for any expansions. It's not easy to start a pool with 12TB drives and later switch it to 24TB drives.
Use one or two small SATA SSDs from a reputable manufacturer for boot drives. Do not use a USB thumb drive for boot. 120GiB is more than enough space for boot drives. Consider buying used enterprise SSDs. Mirroring boot drives isn't essential, you can easily back up your TrueNAS config and restore it later if a boot drive fails (no pool data will be lost).
Add an SSD read cache/L2ARC to your pool if you're supporting a lot of users (more than like 20 or 30) and performance is important. The cache drive should be as fast as possible, ideally NVMe. You don't need a ton, 1-2TiB is usually fine. You'll see diminishing returns as you increase cache size.
Add a log drive/SLOG if your application is running over iSCSI, NFS, or S3. If you're only using SMB, a SLOG is useless (unless you've set sync=always on your datasets). The SLOG should have very low write latency and very high NAND endurance. An Optane SSD is ideal. The SLOG doesn't need to be large, even 16GiB (not 160, just 16) is plenty. Make sure the SLOG SSD is power safe (i.e., it does not lose any data if power is suddenly cut).
You don't need to use a cache or log SSD on all-flash pools.
You'll need an HBA unless you have enough SATA ports on your motherboard to attach all your drives (pool drives, boot drive(s), and cache drives). SAS2 HBAs are fine for most applications. The IBM M1015 is very inexpensive on the used market. Make sure to reflash it to IT mode before using. Do not use a hardware RAID card to attach your drives unless it supports passthrough/HBA mode.
Get a name-brand card from Intel or Chelsio. Mellanox cards work too but don't have the same level of driver support. Avoid Realtek and Broadcom NICs. TrueNAS doesn't have extensive support for NIC offloading features so don't pay extra for cards that support this. TrueNAS does not have any RDMA support so you don't need an RDMA capable NIC. CORE and SCALE only support Ethernet-based protocols, no FCP or IB, so skip those fancy HBAs.
Just get something large enough to support all your drives. The Supermicro 846 is a great chassis and can be found on the used market for reasonable prices, just make sure you get the right backplane for your HBA setup. Norco also makes a 24 bay chassis but it's not as good. The Supermicro 847 has 36 bays but the rear 12 bays are very hard to cool. You may want to replace internal chassis fans to keep noise down. Larger fans will run quieter than smaller fans while moving the same amount of air.
Redundant PSUs are nice but not critical for home applications. They will come standard on most Supermicro chassis. If noise is an issue, you'll want to look for PSUs with the "-SQ" suffix in the model name.
Powering your TrueNAS system through a UPS is highly recommended. Make sure it can support the power draw of your whole system and leave room for growth. You don't need a ton of runtime, even a couple minutes is okay.
If a single vdev in a ZFS pool fails, the entire pool is lost and all your data is gone. Plan your pool layout accordingly.
Mirrored vdevs will give you the best random and sequential performance but the largest hit to usable capacity. Use RAIDZ2 on hard disks if capacity is more important than performance. 6-wide Z2 vdevs are a good middle-ground between performance and capacity, with 8- to 12-wide Z2 going more towards capacity. RAIDZ1 is risky on hard disks and is not recommended with larger drives and/or vdevs wider than 5. RAIDZ3 is overkill for almost all applications (RAID is not a substitute for backup!).
3-wide Z1 or 5-wide Z1 are good choices for SSD-based pools that need a balance of capacity and performance. Going wider than 5 disk vdevs is not recommended. Mirrored vdevs with SSDs will be faster but are almost always overkill and need a huge CPU to keep them fed.
If you have more than ~10 drives in your pool, try to fit a hot spare in your chassis. If you can't fit a hot spare, order an extra drive to keep on the shelf.
If you need to expand your pool, you'll do so by adding more vdevs of the same type with the same capacity drives. You can also replace all your existing drives with larger ones. You can not add single disks to expand RAIDZ vdevs.
Set recordsize=1MiB on datasets supporting large sequential workloads, leave default 128KiB recordsize on datasets and zvols supporting mixed workloads, and set a smaller recordsize on datasets and zvols supporting applications that only do small I/O like BitTorrent and databases. Set recordsize=16KiB on your BitTorrent download dataset and then move/copy the completed downloads to a different dataset when they're done.
If you don't need the atime tag, disable atime on your datasets. This is disabled by default on modern versions of TrueNAS.
Set sync=disabled for datasets supporting sync writes where data integrity is less critical than performance (like BitTorrent) and sync=always on datasets where data integrity is critical. If you set sync=always on any dataset in your pool, you should have a SLOG on that pool.
Configure a minimum of one snapshot task. Even if you're only keeping the snapshots for an hour or two, it will save your bacon at least once. Ideally, keep snapshots for a few days or even a week. If your data is critical, put a second system off-site and replicate to it. The replication target can be a "cheap-and-deep" configuration.
Ensure scrubs are scheduled on regular intervals. TrueNAS enables scrubs every 35 days. You should set your pool(s) to scrub at least every few months.
Leave compression enabled and set to lz4. In many cases, compression will actually improve performance. If you need a deep, slow archive, consider zstd-5 or -10. Leave dedup disable; it's a massive massive performance drain.
If you just need rock-solid storage and good performance, go with CORE. If you need HA functionality and/or professional software support, contact iXsystems about TrueNAS Enterprise. TN Enterprise is only available on iX-built appliances; you can not run Enterprise on your own hardware.
If you want your system to also host some applications in VMs or containers, SCALE will be a better choice. It's not as mature as CORE so you may run into some cosmetic bugs or missing features (let iX know if you do!) but the foundation is generally solid.
Clustered storage is in very early availability in SCALE. If you're interested in helping iX test it out, it would be greatly appreciated, but clustered storage should not be considered stable enough for important data or critical workloads. SMB and S3 are the only protocols that support clustering. A TrueCommand instance is required.
If you're not sure what protocol to use, go with SMB. You can connect to it from Windows, Mac, and Linux systems and it performs great. Create a dataset (taking care to set the appropriate options in the config), then create TrueNAS users (and groups) to connect to the share, then create the share itself, and finally set the ACLs on the share so your new user(s) can connect.
If you need low-latency storage for a VM backend (for example), use iSCSI instead. You can use NFS for a VM backend but the performance won't be as good.
Based on testing at iX, SMB performs better for Linux clients than NFS.
AFP and WebDAV should not be used unless it's absolutely necessary.
You should set a static IP for your TrueNAS. You can do this either through the TrueNAS UI or through your router. If you have multiple subnets in your network, you can run the management UI (and IPMI, if you have it) on one subnet and data on another. If you run iSCSI, it's highly recommended to isolate this traffic to its own subnet.
LACP is useful for network redundancy and performance but it will only scale throughput beyond a single link's speed if you have multiple simultaneous connections (usually from multiple users accessing the storage at once). If you have a single-user TrueNAS system, LACP will not provide a performance benefit.
Create some SMART test tasks for all your disks. Long tests should be run every month or two (make sure they don't overlap with a scrub) and short tests should be run every week or two. Long tests take several hours or even a day, short tests take just a few minutes. Run smartctl -a $disk_id as root in the CLI for full SMART details.
You can back up your data to the cloud if you have a subscription with a cloud vendor. Cloud sync in TrueNAS only works with file system datasets, not zvols.
Make sure you set up and test email alerts (or some other alert mechanism) so your system will ping you if there is an issue. You can tune how frequent the alerts are in the system settings.
Here are some answers to some of the most common TrueNAS-related questions.
These are changes you're allowed to do to a pool after it has been created:
These changes are NOT allowed after a pool has been created:
That last one is very important. If you mess up and add the wrong RAIDZ vdev configuration to a pool, the only way to undo it is to destroy the pool and recreate it.
These are changes you're allowed to do to a vdev after it has been created:
These changes are NOT allowed after a vdev has been created:
These are changes you're allowed to do to a dataset or zvol after it has been created:
zfs rename and might break existing shares)These changes are NOT allowed after a dataset or zvol has been created:
No, it's not as critical as some users might have you think. If your CPU and motherboard support it, it is recommended though. The "scrub of death" is a myth.
You can either add a vdev (keeping the same layout and disk size in the new vdev) or replace all your drives with larger drives. Try to use spare slots when replacing drives rather than removing active pool drives. You can not currently add a single drive to a RAIDZ vdev to expand.
Only if you're running NFS, iSCSI, or S3. You do not need a SLOG if you're only using SMB and you don't plan to change the sync settings.
No, this is no longer necessary. If the SLOG fails, you won't experience data loss unless the RAM happens to fail or power is lost at basically the same instant. On much older versions of ZFS, a failed SLOG would result in total pool failure. This is no longer the case.
Probably not unless you're supporting a large group of users or running a high performance workload with your system. You won't see any benefit to an L2ARC if you're running a small home media NAS.
Worst case, maybe ~15%. If you want to run encryption, get a slightly beefier CPU (make sure it has AES-NI support) to help offset this.
Yes, but it requires manual network configuration on both sides. You'll need to assign IP addresses on the same subnet to both the TrueNAS and to the NIC you're plugging into on your computer. It's much easier to run the TrueNAS through a switch.
No, access to TrueNAS storage over USB and Thunderbolt are not supported.
No, providing storage over direct attach SAS is not supported.
You might be able to get this to work but it hasn't been tested so it's not recommended. Configuration changes and mounts may not persist after a TrueNAS reboot.
If you mix different capacity drives in a single vdev, ZFS will treat all drives as if they were the smallest drive size you use. For example, if you use one 1TB drive and nine 20TB drives, ZFS will treat all ten drives as if they were 1TB each. You can have vdevs that each use different size drives (i.e., one vdev with all 4TB drives and another with all 6TB drives) but this will cause data imbalance as some vdevs will end up getting more I/O than others. For this reason, mixing vdev capacities is discouraged.
No, TrueNAS Enterprise can only run on iXsystems appliances. Contact iX for more information and pricing.