I’m wondering what folks do to optimise the power efficiency of their Linux servers. I’ve never really got to the bottom of what is the best way to do this and with the current energy crisis its a pertinent topic.
I’m talking about home servers, so the availability requirements are not the same as in a corporate environment. There might be vast chunks of time during the day or night when they sit idle, and home users are more tolerant of a lag when accessing resources if it means lower energy bills.
Specifically I’ve been thinking about:
- allowing lower power states when idle
- spinning-down hdd’s when they’re not in use
- MAYBE letting machines sleep/hibernate
- setting schedules of times where you know demand will be low/zero and efficiency can be managed aggressively
- any other quick wins I’ve missed
It would be amazing if there was one tool or one guide that helps with all of that but thats never the case, is it 😅
Thoughts?
I split my loads (gigity) between the power hungry NAS and a passively cooled low power Proxmox host.
For me, most 24/7 activities are low CPU - like Home Assistant, so it needs to be there, but it doesn’t need to do anything.
Other VMs are ansible, uptime kuma, smokeping, etc… the most they use is RAM
Then the (relatively) more power hungry NAS powers up 3 times a day to syncthing everything, maybe upload a backup, and if no-one’s using Immich, etc. then it’ll power back off again.
The only other thing I have yet to downsize is my pfSense box (still a low powered device, but has fans…) and a Raspberry Pi I use for my Zigbee network.
Just yesterday I found this video talking about this very topic: https://www.youtube.com/watch?v=MucGkPUMjNo
Downsizing. If you don’t need to run or keep stuff, then you don’t need so many servers and storage. You may run stuff on cheap mini-pcs.
Sleep: In my experience sleep sucks, I’ve spent long hours planning around sleep in homelab, like: when do I restart, do updates, when do I upload backups. I have Pi and at some point I realised some actions on Pi need my sleeping NAS… so I dropped all the sleep and now it works 24/7.
I made a post about this a while back because I had some of the same concerns about power usage. (https://lemmy.world/post/41185625)
As most people are telling you, as they told me, ‘get better, more efficient equipment’, which is great advice if you are buying, but not so helpful if you’ve already bought. First thing I’d do is get a watt meter to get a baseline of what is being consumed. You can get watt meters from Amazon, or your preferred big box vendor, for cheap.
Once you have the baseline, then you can go about fiddling with Linux and your server, to find areas of excess power consumption. Please note tho, you’re not going to tame a $100 a month server down to $20. That’s just outside the realm of reality. However, every little bit helps. Go into your BIOS and check if there are power saving options there as well.
I know this will go against the grain, but I have a cron that powers off the server every evening before I retire. I am the only user of my services and I have no midnight, mass Linux ISO downloads going on, so I couldn’t justify having the server run for 10+ hours idle, consuming electricity. Electricity here is fairly cheap and I also have solar panels, but still, I think it’s worth the effort to be prudent with your resources.
In the end, the final tally was that I spend around $35 USD per month in electricity, which is far less than most people spend on a hobby.
Letting go of older inefficient hardware is no 1.
Why run a 200w server when a 30w mini PC or 10w pi can run a dozen containers
The mini PCs draw up to 30W, mine runs at an 8W average
I have never measured my pi consumption but ilo says my DL380 Gen8 idles at between 120w to 180w
The ipmi on my supermicro says it’s running 497w.
But that’s what 38 HDDs will do to ya. I used to spin them down when not in use (it’s mostly a Plex server) but it’s in my garage and over the winter I had some issues with the older HDDs not wanting to spin up in freezing temps. I’ll probably move it back to spinning them down over the summer.
Ooof, your power bill.
Cheaper than netflix, paramount plus, hulu, sling, fubo, Disney+, etc
Maybe not when talking about the hdd cost, though. It’s over 300tb of usable space. But at least a good amount of what I use has been wasted I’ve gotten for free.
There was that site a couple months back doing the rounds about calculating the roi on self hosting vs paying for streaming
My old Pi4 with SSD averaged around 7W, so only 1W lower than my mini PC, but performance and usability of my mini PC is far greater (and it comes with 1tb NVME and 16GB ram). I generally advice against using most SBCs these days unless you specifically need the pin I/O for something.
My Pi5 tops out at 15W.
Sadly this. I have a graveyard of nice server boards that I got cheap before realizing how power hungry they are.
For CPUs basically anything older than gen 6 intel is too power hungry (although be careful with Xeon and xeon derived cpus, that are sometimes older gens rebadged as gen 6).
This Jeff Geerling video from a couple of weeks back was an eye-opener.
Just because you might be able to find a cheap G5 Xserve server that runs at 200W when idle, doesn’t mean you should.
Mini PCs are even less usually, mine are around 2W idle which is less than my Pi! (i3-7100u CPUs)
spinning-down hdd’s when they’re not in use
I’ve just been through this recently. I decided to have my 2 backup HDDs spin down when not in use (99% of the time). I ran into an issue though where I needed them to wake up for SMART tests (which SMART didn’t trigger). Tried a few things that didn’t work so just set them to spin all the time. There’s about a 1-2w difference when they’re spinning all the time. So it’s just not something worth worrying about IMO (In the UK with high energy costs that comes out to 2.5 pence per day roughly).
Spinning up and down hard drives repeatedly drastically reduces their lifespan though. Once a day or so, fine, but if you set a 30 minute idle time or something and it spins them down a dozen times per day, you are putting acceleration forces on the drive many more times than intended.
If you have to buy a new HDD twice as often because you spin it down, any financial or environmental savings is instantly negated and in the end it is much, much worse in both respects.
My NAS powers up & down about 3 times a day. Drives are all fine & healthy and some have been in there for years.
I don’t disagree with your core point though…
If the drive just finished spinning down and then it’s triggered for a 1 byte file, spins down, repeat… yeah, that definitely needs sorting out.
Just the initial spin-up lag would do my head in.
But off & on ~ daily, yeah not a problem.
backup HDDs
They were spun down the majority of the time. I’m not an idiot.
There’s about a 1-2w difference
Yeah, you got to pick your battles.
Added solar and a battery. I don’t worry too much about the power use as a result.
Newer hardware that has lower idle consumption mostly. I’ve found there’s not much to do on a typical setup as far as software optimization, as most OS’s are already set up for pretty low power usage while idle.
HDD sleep can work if you don’t have anything accessing the drives, but with all the stuff running on my server there’s basically always some kind of activity going on so they never sleep. Less HDDs is the answer for me, I just have 2 large drives in a ZFS mirror.
My HP box with an i5-7500 idles around 15-20W which is decently low, but I also have 2 PCs with i3-7100u mobile chips that idle at 1-2W with 32GB of RAM and an NVMe SSD, which is wild.
Avoiding enterprise gear is key, it’s extremely power hungry.
My video server runs from 6 pm to 1 am. No need to run always.
I do this via cron. Shuts down the server while I’m asleep. I haven’t gotten around to it, but I want to issue a WOL from my standalone pFsense firewall to the server, and have it power up when I get out of bed.
Unfortunately, wol does not yet work for me.
I use rtcwake
When I built my NAS I intentionally bought the latest gen cpu, but kept it in to the 65W series with a GPU chip onboard. It’s an AMD Ryzen 5 7600 6-Core @ 3800 MHz. My coral usb does frigate and the integrated graphics chip does jellyfin just fine. I started with ssds, but half of them burned out pretty quick, so I replaced them with spinning rust. But, as-is it can run for an hour on my desktop grade UPS before it shuts down. My proxmox cluster is old laptops that mount an NFS drive from my NAS. So, yes, I took power efficiency into account.
I got a power-efficient mainboard and PSU. I think that’ll be the lion’s share. And I don’t have any unnecessary stuff like a GPU or extra stuff connected.
I ran
powertopand adopted the recommendations to set the various buses, peripherals and devices into powersave mode. That does a few Watts here and there. CPU of course is also allowed to save power when idle.And then I made the harddisks spin down after 40min of not being used. Or something like that. So they’ll automatically spin down at night and when I’m not using them. As spinning hdds consume quite a lot of power if you have multiple of them and compare it to the 15-20W or so the rest of the computer uses. The operating system is on a SSD.
Yes, ppl usually don’t really look at PSU efficiency curves.
Random example:
If you are gonna run on 20W most of the time you can calculate how much of a difference a certain PSU can make you (if you find a proper test that is).
That said, I do not recommend to use shitty PSUs, safety first, the components & layout need to be good (most of consumer PSUs are fairly garbage).
A bit like you wouldn’t use regular consumer HDDs.
(For most ez homelab cases every other component can be cheap consumer grade tho, exemptions ofc exist.)
As someone else here mentioned, powertop can significantly reduce consumption at idle, which helps.
In the world of servers, though, unless a device is only accessed once a day, it’s tough to get the HDD to spin down.
Do you have any idea what your hardware is actually pulling from the outlet? Maybe it’s not that bad after all?
Mine is pulling around 55W from the wall in its “normal” state. Meaning two 3.5" HDDs spun up, and a bunch of light services running. Which is squarely in “not great, not terrible” territory.
Apart from flipping the power saver switch on the mainboard I haven’t done anything to save power. I haven’t checked if that’s doing anything either. It’s a 3rd gen core i5 iirc, which isn’t great at idle power consumption, so maybe that switch is doing something…
I also haven’t had any luck with getting the drives to spin down reliably anyways, and afaik it’s better for them to just stay spinning so I haven’t bothered much to change that.
Logging power use by my server was one of the motivators to add homeassistant. That also showed me that specific containers use a (relative) ton of background power. Immich and authentik each raised power consumption by 2-3 watts, so I leave them down unless I have specific need.
Just as a reference, my NUC with 40+ containers, runs at around 3-4 W, not counting the 2,5" 5400 rpm HDD attached.
I had to start over anyways, so I choose SSD only. Pricier of course, but I don’t need a terrible lot of space anyways.
When I still had HDDs, they were usually used once a day for backups, and I spun them down after that.
Optimizing power profiles and C states makes a little difference, but planning with efficient hardware from the beginning is the most important thing. Don’t use your old gaming PC if you care for power efficiency.
Dell Thin Client type machine with a 2TB SSD of the essentials. Backs up once a week to my gaming desktop. Has enough horsepower for Homeassistant, Jellyfin, adblocking. Enough fast USB for two 2.5GBE. Upgraded to 16GB RAM before the crisis, but could make due with 8GB. Cheaper than a raspberry pi for only a few extra watts.
