And indeed it was!

One of the products we sell at work is a caching platform, something that sells quite well into many of our African clients’ networks because transit is often ridiculously expensive and every Mbps saved is USD500~2000 you can use on something else. Traditionally we’ve been deploying on HP hardware, and as of earlier this week we have some SuperMicro equipment to try out for the platform. One 1U unit, and one 3U 8-blade unit. This post is about the latter of these two.

After racking the thing, and re-installing the blades (took them out to move the chassis. Side note: the clasp which holds the blades in is kinda crappy for self-locking. You need to wiggle it a bit to ensure the blade is properly in place). I started poking around on the systems. First issue I found is that the ethernet controllers are Intel 82580’s, which is not supported in the squeeze kernel we had on our PXEBoot server at the time (updated kernel which does have support is included in 6.0.4, or any version greater than 2.6.32-33). Now we were informed by our supplier ahead of time that there was one blade which was DOA and that they had a replacement on the way, so I got started on preparing the other systems in the meantime (as they would form a cache cluster). Doing this, I experienced some strange weirdness with the power sequence. Sometimes all the blades would power on, sometimes only the first 4 bays, sometimes only 3. Sometimes I could power 5 on, one off, another one, then attempt to reverse the power sequence of the last two but not succeed. A few more combinations like this were tried, including removing a unit far enough to disconnect it from its connector and then reseat it, but suffice it to say that it didn’t make sense.

At this point in time I’m left with the options of removing units from the bays (to eliminate PSU overload), and of changing the order of the units in the bays to see if that makes a difference (which one would not expect, but if all the possible options have been eliminated then whatever’s left is probably the answer). As I start removing the units one by one, I notice that there’s lint on the one blade’s connector. This hadn’t been there when I installed them, so I asked one of my coworkers to bring my torch over so that I could inspect the inside of the chassis. Turns out there’s a bit of lint hanging loose (about 6cm worth, presumably from the sleeving of one of the fans’ power connectors?) inside the chassis, and that it had somehow managed to get caught up in/on the connector. I remove the lint, and suddenly everything is working as expected.

Lessons learned:

• SuperMicro BMC units probably have a shared power control bus
• If you’re seeing weird things happening, maybe it’s lint!