40-core ARM cluster using the NanoPC-T3
Building a high performance 5-node cluster for under £100 (£375 including five T3s)After the interest in my cluster of Raspberry Pi 3s in April, I was keen to try building clusters with some of the other excellent SBCs now on the market. FriendlyARM in China generously sent me 5 of their latest NanoPC-T3 64-bit ARM boards, each with an EIGHT core ARM A53 SoC running at 1.4GHz, gigabit Ethernet and 8GB of eMMC storage onboard.
The completed cluster measures 146.4 (w) × 151 (h) × 216mm (d) and weighs 1.6kg (5.6 × 5.9 × 8.3", 56oz), including the AC-DC power supply.
NanoPC-T3 vs Raspberry Pi 3
The NanoPC-T3 board is a considerable advance over the Raspberry Pi 3 both in terms of performance and features, but unfortunately this comes at nearly twice the price (at least for those of us in the UK):
| Model | NanoPC-T3 | Raspberry Pi 3 |
|---|---|---|
| SoC | 8-core ARM A53 1.4GHz | 4-core ARM A53 1.2GHz |
| Memory | 1GB DDR3 | 1GB DDR2 |
| Network | 1000Mbps | 100Mbps |
| WiFi | 802.11bgn | 802.11bgn |
| Bluetooth | 4.0 dual | 4.1 + BLE |
| Storage | 8GB eMMC onboard SD card | microSD card |
| USB spare | 2 fitted + 2 headers 1 microUSB | 4 fitted |
| Video | HDMI 1.4, LVDS, RGB-LCD, MIPI-DSI | HDMI, DSI |
| Camera ports | DVP, CSI | CSI |
| Audio | 3.5mm | 3.5mm |
| Microphone | onboard | no |
| Size | 100 × 60mm | 85 × 56mm |
| Power | 1.6 → 4.3W 2A max, DC 4/1.7mm | 1.2 → 2.1W 2.5A max, microUSB |
| Launched | Q2 2016 | Q1 2016 |
| Price (UK) | £59.021 | £33.59 |
1US$60 T3 + US$1.50 heatsink + $8 shipping + 20% VAT + 0% import duty = £59.02 | ||
3D design in SketchUp
I modified my original Raspberry Pi cluster design using the free version of SketchUp and built rough 3D templates of the NanoPC-T3’s, network switch, sockets, etc. I didn't bother to include ventilation slots/grids in the 3D model.
2D design in Inkscape
I used the free Inkscape application for 2D design, ready for exporting to the laser cutter. Each colour is a different pass of the laser, at different power/speed levels, so the green lines are cut first to make holes for ports/screws/ventilation, pink are extra cuts to help extract delicate parts, orange is text/lines that are etched and finally blue cuts the outside of each panel.
Download files for laser cutting on one 600×400×3mm sheet:
Design Changes from the Raspberry Pi 3 cluster
Although very similar to my cluster of Raspberry Pi 3s, I made a number of changes and improvements:- I kept the horizontal mounting rail design but the T3 has M3 holes which are easier to find parts for, rather than M2.5 on the Pi. And the holes are further apart because the overall board size is slightly bigger than the Pi. Screwing the plastic nuts onto horizontal rails is a bit tedious (and a tight squeeze to get past the ethernet port on each T3), and I'd like to 3D print a C-shaped ‘clip’ that holds the boards in place along each rail, or perhaps use tight elastic washers.
- External PSU vs Internal USB hub – the T3s are powered via 4mm/1.7mm DC plugs rather than the Pi’s microUSB, so it made sense to swap out the internal USB hub power supply for an AC-DC power ‘brick’ that sits outside the case. This gives more space inside the case (for the larger T3 boards, and a fan), and should help with dissipating the heat from the power supply. Each T3 can draw up to a maximum of 2A, but will actually be much less in this cluster, without extra USB and GPIO accessories.
- DC cable splitter. I was pleased to find a ready-made 5-way splitter cable (by MyVolts for Korg Volcas) with the less common 4mm/1.7mm DC plugs that the T3s use. However at 1.75m it is far too long to fit inside the case, and my utter bodge-job of shortening the cable was... well, best forgotten
. Splitter cable v2.0 was far more successful – a cheap 45cm 6-way splitter cable sold for CCTV cameras is around £2, and cutting off the (wrong size) DC plugs and soldering new right-angled 4mm/1.7mm DC plugs gives a neat result for less than half the price of the MyVolts cable. The 6-way cable also allows for a 5V connection to power the ethernet switch, saving on the cost of a separate cable. - Internal Case Fan vs External – I was sure that the high performance T3 boards would need active cooling, and so I stuck with an ultraquiet 92mm fan, but moved it inside the case, and to the back rather than on top.
- Case USB ports – while perfectly functional, I never really liked the look of the twin USB ports on my Pi cluster, so for the T3 cluster I sourced a more compact 2-port USB mount.
- Right-angle USB extenders – I didn’t realise until everything else was finished that the USB cables would stick up too far vertically to shut the case lid (!) so I had a last-minute purchase of two right-angle extenders so I could connect up the USB. A neater solution would be making/buying a single cable.
- No internal shelf vs Shelf – not needing a shelf to attach the USB hub to simplifies the design, and also means it can be cut from a single sheet of 600×400mm acrylic. Removing the shelf would have reduced the rigidity of the case, but screwing the horizontal mounting rails to the side panels keep it secure.
- Flat 25cm LAN cables vs Round 15cm – I loved the rainbow LAN cables in my RPi3 build, but it was a tight fit bending them in the case. These flat cables bend far more easily, and 25cm is enough extra length to reach the LAN ports on the top of the T3 board.
- Green LAN cables vs Boring grey – the green really brings out the blue of their eyesPCBs
... plus the FriendlyARM logo is blue+green. - Gigabit vs 10Gigabit-switch – the T3 network ports are each 1000Mbps (10× faster than the Pi), so using at least a 1000Mbps switch is a no-brainer really. A 10Gbps switch would eliminate any bottleneck (e.g., if the T3's were simultaneously saturating their links talking to the outside network), however these are still very expensive at £600+, and too large to fit inside this case. The new Asus XG-D2008 switch looks promising.
Power, temperature & cooling
At idle, the entire system of five T3s, network switch & 5V fan sips a mere 13W, and at 100% load it still only uses 31W in total. There is the possibility of further reducing the power requirements by disabling WiFi, Bluetooth and HDMI?Is a heatsink worth it? with double the number of cores, the T3 SoC can generate far more heat than the Pi3, so I'd suggest a heatsink is pretty much an essential purchase. Luckily FriendlyARM sell a substantial heatsink with thermal paste for only US$1.50 that securely clips onto the T3 board. It is much bigger than other aftermarket SBC heatsinks I've seen, and in my tests it reduced the SoC temperatures by a remarkable 10-35°C !
The power adaptor can supply up to 50W (2A per T3), which is more than enough for a couple of power-hungry external devices to be plugged into the USB ports. Using:
cat /sys/class/hwmon/hwmon0/device/temp_labelto measure the SoC core temperature, the cluster idles at 36°C (97°F) with cooling from the 5V fan.
At 100% load and the 5V fan, using:
sysbench --test=cpu --cpu-max-prime=200000 --num-threads=8 run &the SoC core temperatures reached a stable 68°C (154°F). If you don't provide at least some active cooling, the SoCs will get as high as 85°C and automatically throttle down their clock speed, to avoid overheating. They can safely run long-term at that temperature, but you don’t get maximum performance.
Exactly the same case design should work with NanoPC T2s and T1s which run cooler than the T3, and so might not need a fan to cool them. For cooling a single T3 you could use a much smaller fan, perhaps 40-60mm.
Silent cooling:
To cool down the cluster I fitted a single 92mm fan inside the rear of the case. I used a (effectively) silent fan recommended by Quietpc.com, the Nanoxia Deep Silence (£10.15). At 5V I have to get my ear within 50-75mm (2-3") to hear even the slightest whisper from the fan, and the supplied rubber gromets definitely do a good job of isolating the case from any small vibrations.
| description | heatsinks? | idle | 100% load | performance |
|---|---|---|---|---|
| Case, 12V fan, 1400 rpm | no | 39°C | 85°C | throttles |
| Case, 12V fan, 1400 rpm | yes | 30°C | 50°C | OK |
| Case, 7V fan, 800 rpm | yes | 33°C | 59°C | OK |
| Case, 5V fan, ? rpm | yes | 36°C | 68°C | OK |
Unlike PC motherboards, a T3 doesn't have a standard 3 or 4-pin fan header, but it is quite easy to power the fan using the GPIO 5V power rail instead. WARNING you can't run anything that needs a lot of current this way – however a low-power fan should be fine. You do need to make sure you have a quality power supply, and not too many power-hungry USB devices plugged into your T3, otherwise you'll get crashes & rebooting. The main GPIO pins are for data, but using the VDD_5V power rail on pin 29 should work in theory.
This should be enough to run the fan with 5V, or if you need more cooling, splice a step-up/boost converter in the middle to adjust the speed of the fan, by controlling its voltage between 5V and 12V.
Building the T3 Cluster
- Remove network switch case (2 small screws)
Attach the network switch to case base, using 4× 6mm brass spacers + 4× 14mm bolts + nuts. This only fits one way around - Cut off only 5 (of the 6) plugs on the DC power splitter cable, and solder on 5 new 4.0/1.7mm right angled plugs
Bolt DC socket port to case back
Cut off the combined end of the splitter cable, and solder onto the DC screw mount socket.![Building T3 Cluster case 2]()
- Screw 2 external LAN ports to inside of the case back (has “DC5V 10A” etched on the outside)
Fit the 92mm case fan to inside of the case back using the supplied rubber grommets. The fan logo should face inwards if you want the fan to blow cold air into the case. The rubber grommets might need to be pushed into place with a small screwdriver/pen – don’t pull too hard!
Clip case back to case base (marked C+D)
Plug external LAN into network switch ports 1+3 (no room to use 1+2)![Building T3 Cluster case 3]()
- Place the pads of heatsink compound to the SoC chip of each T3, and firmly clip on each heatsink
Slide 5 T3’s onto 4 threaded rods using 48 nuts to secure. This might be faster with the rod in an electric screwdriver? Leave 30mm space at left end, space each T3 25mm apart. The LAN+USB side of the T3’s point upwards and towards the front of the case
Attach case sides to T3 rods (EF+AB), secure with 8 metal nuts.![Building T3 Cluster case 4]()
- Fit the 3-pin fan plug into the GPIO pins of one of the T3’s, so that black (ground) connects to pin 30, red (power 5-12V) to pin 29, and yellow (fan speed) hangs unconnected over the PCB.
![Building T3 Cluster case 5]()
- Plug T3 LAN cables into network switch ports 4-8 (be very careful if removing these – it is easy to break tiny plastic clips on the switch ports)
Clip case sides to case bottom (E+F, A+B)![Building T3 Cluster case 6]()
- Route HDMI cable through case
Screw external twin USB to front case
Bolt external HDMI to front case, 2 nuts
Plug 2 external USBs + right-angle adaptors into any T3 (or 2 separate T3’s)
Plug T3 LAN cables into T3’s
Plug T3 DC power cables into T3’s
Plug HDMI cable + right-angle adaptor into any T3![Building T3 Cluster case 7]()
-
Clip case front to case bottom (G+H)
Clip case lid to case sides, front & back (M+N+O+P+Q+R+S+T)
Attach self-adhesive rubber feet to underside of base
Bill of materials
Most of these parts were sourced from individual sellers on Amazon or eBay, which of course racks up the postage charges. If there were enough demand, it would be cheaper to bulk buy the parts and have a kit with everything you need to build the cluster.
| Edimax ES-5800G V3 Gigabit Ethernet Switch | £18.45 |
| Flat 25cm Cat6 LAN cables (5 pack) | £6.80 |
| 5V 10A AC-DC external power supply, 5.5/2.5mm plug | £12.99 |
| 5.5/2.5mm chassis mount DC socket (1 from a 2 pack) | £1.95 |
| 6-way DC power splitter cable | £2.05 |
| DC plug right angled 4/1.7mm (5 pack) | £3.49 |
| M3 steel screws 8mm (4 from a 10 pack) | £1.10 |
| RJ45 male to female screw mount (2 pack) | £1.98 |
| M3 steel hex nuts (4 from a 5 pack) | £1.10 |
| M3 steel bolts 14mm (4 from a 5 pack) | £1.35 |
| M3 brass female standoff 6mm (4 from a 10 pack) | £0.99 |
| M3 steel threaded bar 150mm inc. nuts (4 from a 5 pack) | £5.50 |
| 0.5m HDMI male to female panel mount (inc. bolts) | £2.39 |
| Twin USB female socket to male cable (inc. screws) | £2.19 |
| USB right-angle male to female socket extender (2 pack) | £1.98 |
| M3 nylon hex nuts (48 from a 50 pack) | £3.45 |
| 3mm extruded clear perspex 600×400mm | £5.32 |
| Laser cutting charge | n/a |
| HDMI 270 degree adaptor | £0.99 |
| Nanoxia 92mm Deep Silence 1400rpm fan | £10.15 |
| Polyurethane rubber feet (4 from a 12 pack) | £2.99 |
| Subtotal inc P&P | £87.21 |
|---|---|
| NanoPC-T3 at US$60/each (5 pack)1 | £272.78 |
| NanoPC-T3 heat sink 5mm (5 pack) | £6.24 |
| Total inc P&P | £366.23 |
1The NanoPC-T3 is duty free to import into the UK, and only costs US$21 to ship 5 boards from China, but then there is UK VAT at 20%, bringing the total to £279.02.
The cluster of T3s (excluding the boards themselves) is slightly cheaper to build than the Pi3s: you don't need a USB hub, separate microUSB cables, SD cards for each board, etc. However it is a shame they used 4/1.7mm DC power connectors, rather than the far more common 5.5/2.5mm or 5.5/2.1mm sizes.
A cheaper high-performance option? the NanoPi Fire3 was announced by FriendlyARM after this T3 cluster was built, and looks like a great alternative for a low-cost, compact cluster. It has the same 8-core 1.4GHz SoC and gigabit Ethernet, but on a smaller board with fewer extra interfaces and nearly half the price – a 5-node cluster should give the same performance, but for a total of £250 instead of £366.
Show me the Benchmarks!!
I'm writing a separate article benchmarking this NanoPC-T3 cluster against my original Raspberry Pi 3 cluster, as well as detailed benchmarks comparing the individual boards. Watch this space...Clusters of other Single Board Computers
So far I’ve also built:- DIY 5 Node Cluster of Raspberry Pi 3s
- 40-core ARM cluster using the NanoPC-T3
- 5 Node Cluster of Orange Pi Plus 2e
- Bargain 5 Node Cluster of PINE A64+
- ClusterHAT with 4× Raspberry Pi Zero
- 96-core ARM supercomputer using the NanoPi-Fire3
The NanoPC-T3, Orange Pi Plus 2e and PINE A64+ clusters mostly share the same components, including the acrylic case panels – only the 2 side panels are unique because the boards are different sizes. I'm planning to revise my original Raspberry Pi cluster design so it will share most of these same case/components.
I’d like to build a small cluster of all the current crop of sub-$100 ARM SBCs, comparing the different features, and with detailed benchmarks. e.g., Odroid C2/XU4 and the Banana Pi M3. Please email me if you'd like to send boards for review.
Software to run on a cluster?
or... What is it for??Education, training, blah, blah... well personally I’m just running a Debian Linux provided by FriendlyARM on each T3 for now, and I’m going to experiment with things like load-balanced web/database servers.
Running Docker on ARM on each node looks like an excellent way of controlling the cluster.
Nick Smith, June 2016.
