SDDC in a Box – Part 1 (BOM)

SDDC in a Box – Part 1 (BOM)

Comments 25 comments

I’ve been in IT for over 20+ years and consider myself an old-school.  My career is built on the server, software, virtualization, and cloud and I still like to touch, feel, and tinker with hardware.  I love talking to my customers about the Software Defined Data Center and the building blocks; compute virtualization, network virtualization, and storage virtualization.  Software-defined storage, vSAN is such a simple and easy solution that sometimes it can be underwhelming when showing GUI from the vCenter. Really, there’s not much more other than turning “on” vSAN features on the cluster – at least from the customer demo perspective.  Yeah, yeah – I know. there are more than just “on” buttons on vSAN.

I wanted my old school customers to see the power of the Software Defined Data Center in a different physical form factor.  SDDC is not defined by four-walls, brick-and-mortar with racks of servers and switches.  SDDC represents software-driven enterprise-level data center capabilities and pooled resources that are detached from the underlying hardware, using virtualization technologies.  When the data center is abstracted as software, you can now automate and orchestrate to meet today’s DevOps demand.  Infrastructure as a Code anyone?

In a planning process, I wrote down requirements that would meet my needs.

  1. Able to carry 4 nodes SDDC on to the plane, meeting airline carry-on luggage requirements.
  2. 10Gbe Network connectivity
  3. Software Defined Network for  Networking services, security, and virtualization
  4. Self-contained (power, network, compute, storage, cooling)
  5. Low power consumption (my old home lab cost me over $300 per month in power)
  6. Low Noise! (It will be used in my office and I can’t have loud fan noise)
  7. Physical North-South traffic that handles NAT, WIFI, and routing capability

 

First, I search the popular blog sites to see what other home-lab-ers are doing.  TinkerTry and VirtuallyGhetto blogs were very helpful.  It was a toss-up between NUC and E200-8D.  Supermicro E200-8D offered 10GbE out of the box and vSphere 6.5 had the drivers for it.  It was an easy choice.  Being an ex-audio engineer, I know about road cases.  Audio engineers have been carrying around audio equipment since the age of… men?  Anyways, the rack mount points are the same exact spec as the server rack mounts.  SKB offered robust travel case that met the airline luggage sizing requirement in 4U form.

 

Bill of Materials:

  1. (4) Supermicro E200-8D
  2. (1) SKB 1SKB-R4S 4U Shallow Roto Rack
  3. (1) Buffalo BS-XP2012 10GbE Managed Switch (discontinued as Aug 2018)
  4. (1) TrippLite CAT6a Shielded 24-Port Patch Panel
  5. (1) StarTech 1U Rackmount 16 Outlet PDU (RKPW161915A)
  6. (16) 32GB DDR4-2400 LP ECC REG (M393A4k40BB1-CRC)
  7. (4) 1TB SSD Samsung 860 EVO
  8. (4) 250GB NVMe Samsung 970 EVO
  9. (1) SanDisk Ultra Fit USB 3.1 16GB Flash Drive [Pack of 5]
  10. (2) NavePoint 1U 19″ Vented Universal Rack Tray 8″ Deep, NO LIP [Pack of 2]
  11. (12) Noctua NF-A4x20 PWM quiet 40mm fan
  12. (3) Noctua 120mm SS02 Bearing Case Cooling Fan NF-S12A FLX
  13. (1) 12V 2A Power Supply Adapter
  14. (1) Apevia G-120MM Fan Grill  [Pack of 3]
  15. (1) 3U Panel for three 120mm Fans
  16. (2) CableCreation 1 Feet 18 AWG NEMA 5-15P to IEC320C13 Cable [Pack of 2]
  17. (1) CablesOnline 3 Feet Right-Angle AC Power Cord with 3-Conductor (PC-303R)
  18. (3) Monoprice SlimRun CAT6a Ethernet Patch Cable 3 Feet [Pack of 10]
  19. (1) GL.iNet GL-MT300N-V2 Mini Travel Router

Options:

  1. PCI-E NVMe 1U Adaptor for All-NVMe vSAN

Tools:

  1. Zip Ties
  2. Gorilla double sided industrial tape
  3. Zip tie downs
  4. Heavy-Duty velcros
  5. Glue gun

 

 

Edit: Updated Options (10/2/2018)

Next – Part 2 (Decision-Making Process)

25 thoughts on “SDDC in a Box – Part 1 (BOM)”

  1. Fantastic work David, it is really awesome to see this work, thank you for all your effort, and for your new blog, can’t wait to see what you publish next!

    Reply

  2. Looking forward to part 2! I’m interested in the benchmarks. I’ve considered doing something similar to this with NUCs. I’m curious as to why you went with the Super Micro over NUCs. I have not found any NUCs so far with Xeons.

    Reply

  4. Hi David, great work here. I’m impressed. Looking forward to get into more details about the build and software configuration.
    I have a question for you regarding the Noctua NF-A4x20 PWM quiet 40mm fan. I have 2 E200-8D that I find quite noisy (with their original fans) and I was planning to change the fan for the Noctua. However, speed and heat dissipation is very different with the Noctua in comparison to the original fans. What is your experience with them? Do you have any problems with the server CPU throttling down because of heat?

    Reply

    1. I had same concerns with slower speed Noctua Fans. I tested a couple of days at idle and running the CPU hot on the table at room temperature. I didn’t have any issues so I moved them into the travel case and installed three case exhaust fans for extra protection. In the Supermicro lights out management portal, I have it set to “High IO?” (2nd highest speed setting). My office doesn’t sound like a jet engine anymore.

      Reply

          1. OK Looks like things work out pretty well. I have changed all the fans for Noctua and temperature are a bit higher, but it oscillate around 65-72 for the CPU. And it is way more quiet. I love it!

            Thanks !

    1. The server comes with loud two fans. I replaced them with three Noctua fans. Noctua fans are thinner than the standard fan so you will need a glue gun to secure it. The case is tight and it won’t go anywhere. By itself on the table, it runs low at 60C idle and low 70C on high utilization. In the case, it gets a little warmer so I have three 120mm exhaust fans in the back. I still get the same low 60C and low 70C numbers.

      Reply

      1. I did not used any glue to secure the fan, you just need to expand the holes of the fan support in the server with a larger drill bit. Then make sure there is no small metal dust anywhere on the fan support before installing the Noctua fan with the provided screws. I used 2 screws per fan and install them only on on side of the support. It is very solid.

        Reply

  7. Outstanding work my friend! Thanks for sharing this with the community. You’ve just given me my next project. My wife is going to give me and by extension, you, the evil eye, but it will be well worth it. 😃

    Reply

  10. Thanks for venturing into this territory for us, David.

    I’ve got a single E200-8D and two of Paul’s SuperServers (SYS-5028D-TN4T-12C).

    The fact that the smaller E200 was louder than the bigger servers always bothered me. It’s been in the back of my mind to replace them, but I hadn’t done it yet. Paul’s post referencing your build out was the reminder I needed.

    The three factory fans have just been replaced with three of the Noctua A4x20’s. What a huge difference in noise reduction. It’s a no brainer mod for anyone who wants this unit on their desk or in their office. I bet you void warranties when doing this so beware.

    For the install, I used the screws that came with the Noctua fans. They are larger than the factory ones, but I managed to screw them in after I hot glued the fans to the mounting plate. (great idea on the hot glue, btw!)

    Here’s a shot of what they look like in the mounting plate: http://links.shdc.io/90650b276a20.

    The one on the left has some hot glue oozing out of the bottom. I was able to reduce the amount of glue on the others and the remain firmly attached.

    Here’s a pic comparing the depth (width) of the factory fans vs. the replacements: http://links.shdc.io/533a9cde27a9

    As a test, I installed the Low Noise adapter cable, thinking that would help further reduce any noise. However, I did a very informal test using my ears with the LNA on and off, and couldn’t tell the difference. I ended up removing them as it made for easier cable management by not having the additional 6 inches of cable in the small chassis of the E200-8D.

    I also sprang for the PCI-E NVMe 1U Adaptor and the 500GB version of the Samsung 970 EVO. Doubling the NVMe storage was worth the 20 or 30 bucks extra.

    I was able to install the NVMe drive to the adapter, and then installed the adapter into PCIE slot on the left side of the board. Just a bit of a warning for those who go with these particular adapter boards. When installed in the E200, the adapter board sits high in the slot, so when I put the cover with my 1TB SSD drive over the top of the assembly, it was really, really tight. I played with it and was able to get it all back together. I removed and re-seated it to be sure it was all the way in, and it was. The top of the adapter board sits a millimeter higher than the left edge so getting the plate back in was tight.

    Did you had a similar issue with the NVMe adapter being just a tad too wide/tall?

    No pictures of this at the moment as I wanted to get it back together and test it all out to make sure it came back up and it did.

    Here are some additional images:

    Screenshot of the 3 drives showing in the web client (500GB NVME, 1TB SSD m2, 2TB SSD SATA): http://links.shdc.io/2403f2ded178

    Image of Fan Speeds as reported by IPMI: http://links.shdc.io/5484b6ad451f

    System Sensors from the web client show all is well too: http://links.shdc.io/3a140b83bae7

    Again, thanks for leading the charge and helping me get off my rear end and tweak my setup.

    I’ll likely order 3 more of the E200-8D’s to make a kit similar to yours. I already have exhaust fans, patch panels and such from other projects so the money I save there will go maxing out the RAM for each of them.

    Reply

    1. Shawn,

      I am glad mods are working for you. Yeah, factory fans are waaaaaay too loud. You can adjust fan speed mode from the BIOS setting to make the fan spin faster if needed but it gets a little bit louder. I will make changes to the blog so people don’t have to wait for me to approve posts. 🙂

      David

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *