Fri, 06 Feb 2026 07:13:30 -0600Stone Soupercomputer II

The Stone Soupuercomputer was an early Beowulf-class cluster assembled from whatever way laying-around.  It's name is a reference to the story about a soup made from a stone (and whatever else the village had available).  At the time, the primary motivation for creating this system was a lack of budget, but since then it's also been a way to make hardware that would otherwise be destined for the dump useful again.

I built a similar system out of scrap machines in the 90's when I was working as a consultant and it not only taught me a lot about high-performance computing, it also ran a few practical applications at speeds higher than any of the single high-end servers we had at the time.

Over the year's I've experimented with building various HPC clusters, perhaps the largest of my own was an 8-node, 32-core machine built out of discarded G5 ProLiant DL360's.

The downside to these scrap supercomputers is primarily the cost to operate them.  Older hardware tends to be less energy-efficient, which tends to add-up when you start building clusters.  For example, that DL320-based system could theoretically pull about 6KW running at full speed, and idled around 1KW.  When we were using this system to help find a vaccine for COVID-19, my utility bills doubled.

I have experimented with using ARM and other lower-power machines but that gets into the real of "new" hardware, which defeats the purpose of the Stone Soupercomputer.  Not only this, these devices are simply not as well-made as discarded workstations or enterprise equipment and while the old machines tend to have slower processors our outdated GPU's, etc. they are often equipped with good amounts of RAM and solid I/O subsystems, both key to high-performance applications.

After talking with my friend Matt (and the the fediverse) about the trade-offs of designing & building new laptops vs. reviving old ones, I started to think about ways to use old hardware in general, and specifically how I might make a cluster built from older, less energy-efficient hardware use less power.

Here's what I've come-up with so far:

Contemporary Linux distros provide a lot more in the way of power management than the old versions did.  Some energy-efficiency gains could be had simply by upgrading the operating system on these old machines and putting some work into tuning these settings.  There may be limits due to older machines having less hardware support for power management, but I think you could knock the numbers down a bit just by spending time tuning the software.

Coordinate software power management with workload scheduling

The way clusters like this are used is very different from how you use a personal computer or even a server.  They tend to run large, scheduled jobs that run for a long time.  To do this they employ cluster management software that keeps track of the overall system's capacity, health and schedules where the individual tasks that make-up the overall job will run.

Normally power management software has to "guess" when you will need the most power from your computer but on a supercomputing cluster this is usually well-known in advance.  Minimally we can assume that if the job scheduler doesn't have a job running, the system can essentially be "parked".  This knowlege could be used to put nodes in the cluster to sleep or even turn them completely off (if it's possible to wake them back up again from the scheduler).  Given that these jobs run for days or weeks, the small amount of latency associated with booting-up a node won't have a large impact on the overall time the job takes.  At the other end of the job, nodes can be powered-off as the work for the job winds-down, and there may be even more efficiency to be found in how the work is structured.

There are countless ways to optimize hardware utilization when it is an objective in designing HPC workloads.  Much of the software in this space (not only the jobs themselves but the infrastructure software) is designed for an always-on environment, so I think there's a lot of room for improvement here.

Building on the last section, what about hardware that can't turn itself off, or be turned back on by the scheduler? For this I'm imagining simple software-controlled outlets that can physically remove power from the nodes (or even other equiptment like switches) in the cluster.  Versions of this capable of handling 15A are very inexpensive and can be controlled via a simple REST interface.  Most bring the additional advantage of providing power useage data, so it would be possible to know exactly how much electricity is being used.

The scheduler could use these outlets to coordinate safely shutting-down equiptment with removing power altogether, allowing the cluster to idle-down to amost zero energy use.  Even machines capable of wake-on-lan can't reach zero.

Another option is to involve a standby power source that can be controlled by the scheduler.  Aside from preventing work from being lost during power outages, even an off-the-shelf Bluetti power station could be used to enhance power management.

For example, such a device could intelligently switch between running the cluster on grid power to battery power to "time shift" drawing power from the grid to times when demand is lower.  This not only reduces the cost of electricity but could also be used to draw power when renewable energy is most avaliable.

Another option is to add solar panels to the mix and power the cluster directly from the sun when it's available.

A third way that a device like this could be helpful is by allowing "oversubscription" of a lower-power circuit by the cluster.  For jobs that the scheduler knows won't last longer than the available standby power, the cluster could draw more amps from the battery than the circuit supplying it would allow.  For example, if the power station has a 10KWh capacity, it could deliver 2KW to the cluster for for a five hours job even if it's fed by a 15A (1800W) circuit.  The power station can do this because it can "backfill" the additional power from the energy stored in the battery, then replenish this after the job is complete.  The net energy consumed is the same but by allowing a lower-power circuit to be used the cluster can operate in more common environments and possibly from lower-energy sources like solar, wind, etc.

The last thing I'll mention is that net energy use could be lowered by coordinating the clusters activities with broader environmental effects related to heating and air conditioning.  The most obvious form of this is offsetting the need for heating a space by reclaiming the heat generated by the cluster.  To a degree this happens automatically as a thermostat will turn a furnace on less often if the cluster is heating the air in the same room.  A step further would design the system such that the heat from the cluster is more evenly distributed (perhaps vented into the intake side of a forced-air heating system) and ensuring that the thermostat is able to take this secondary heat generation into consideration.

A more advanced form of this might more directly integrate the HVAC with the cluster.  For example, consider a liquid-cooled system that places it's radiator in the HVAC system stream or maybe even integrates in a direct way with a heatpump system to re-use the existing plumbing.  It's not hard to imagine how the heat differentials produced in a predictable way by such a system might be used to offset energy use elsewhere in the broader system.

Another option is to move that heat outdoors, in climates where outdoor temperatures are low enough to make a difference.  The bluntest way to do this would be to move the cluster to a non-climate-controlled space outdoors in the winter, but it's not hard to imagine slightly more convenient ways to accomplish the same thing.

This is just a few things that came to mind when I started to reconsider using old hardware again.  I've used these things most of my life but at some point I stopped because I thought the ecological impact of increased energy use justified discarding the old equipment in favor of new hardware and new designs.  But now I'm reconsidering this, because there is simply a lot of this old stuff around, and while it is recycled to some degree, there's still a lot of waste.

This is compounded by the processes used to create new things.  A friend who worked at a scrap yard referred to it as an "above-ground mine".  She often mused about how wasteful it is in terms of both finance and life to dig metals out of the ground when all the metal we'd ever need has already been dug up, refined and is now sitting on the ground waiting to be used.

Maybe the same applies to computing hardware.  Maybe all the compute we'll ever really need has already been made, and we simply need to find ways to re-use it?  I think reviving The Stone Soupercomputer as a proof of concept is one way I can help test this theory.