Tuesday, May 14, 2024, 2:15 p.m. Central Ohio. The grid dipped twice — each blip under two seconds. It felt much longer.
I manage facilities procurement for a 47-person engineering consulting firm. In 2024, I directly controlled an annual facilities spend of roughly $182,000 — equipment, maintenance contracts, and spare parts included.
“Directly” meaning: before that afternoon, I thought our power protection was one of the areas I had a reasonable handle on.
Our data room ran an APC Smart-UPS SMT1500C. We had 14 APC Back-UPS 1500 desktop units scattered across three office floors. A 20kW natural-gas standby generator sat in the electrical room. A portable generator lived in the back garage for emergency use.
Our operations director used to call the setup “our triple redundancy.” I liked hearing that phrase.
The first dip was fast and smooth. The Smart-UPS caught it — nobody in the server room would have noticed. By the time I walked to the data corridor, several Back-UPS 1500 units were already screaming. That high-pitch beep. The one that makes people leave the building.
Six of the 14 had red battery fault lights. Those batteries were installed in February 2021. According to APC’s own guidance for the Back-UPS line, battery life typically runs 3 to 5 years. So by May 2024, we were at the tail end of that window with no room to maneuver.
I ran the math that week: replacing all six battery packs would run somewhere between $1,900 and $2,300, depending on whether I sourced through APC directly or a third-party channel. All fourteen ran closer to $5,000.
That should have been an easy decision. It wasn’t, because the same week, our ops director asked a different question: “Why don’t we just put in solar with a grid tie string inverter?”
Our energy consultant had been pitching it. One of those proposals that sounds smart on the first page — a grid tie string inverter system would cut daytime electricity costs and, the consultant said, “add resilience.”
I didn’t fully understand how a grid tie string inverter worked until I read the spec sheet properly. And what I found is something I now wish more facility managers knew: a grid tie string inverter, by design, must shut down the moment the grid goes offline. That’s a UL 1741 safety requirement — an anti-islanding function to protect line workers during an outage.
Meaning: in a grid-down event, a grid tie string inverter will not power your building. It saves on electricity bills. It does not provide outage protection.
The consultant wasn’t lying — he said “resilience,” which in his vocabulary meant something different (less stress on the grid, lower demand charges). But I heard “backup.” The conversation took ten minutes to unwind. Frustrating, but honestly my fault as much as his.
Then the second problem surfaced.
The generator didn’t start.
About ten seconds after the first blip, the standby generator should have been running under load. I looked out the data corridor window toward the electrical room. No exhaust smoke.
Our facilities tech pulled the spark plugs that afternoon. Not a gas supply issue, not a starter issue. The BPMR7A spark plugs were worn. The BPMR7A is one of the more common NGK plugs used in small and mid-size natural gas engines, and this one had been on our equipment for over three years without replacement.
BPMR7A plugs should really be swapped at 250 operating hours or annually, whichever comes first. Our outsourced maintenance contractor had told us in 2023: “plugs look fine, skip this cycle.” I believed them.
The most frustrating part of spare parts management: the same issues recurring despite clear documentation. You’d think the contractor knew the spec. They did. The spec just wasn’t profitable for them to follow.
We pulled the plugs and I asked our tech — an old-school guy who’s been doing this for twenty years — how he’d check spark plug wires while we were already in there. The method is simple, though I had no idea at the time. You disconnect the wire at the plug end, then measure resistance with a multimeter — usually in the thousands of ohms per foot range. If you get infinity or something way higher than spec, the wire is broken or the insulation is shot and it needs replacing.
Two of the four wires on our unit failed the test. So even with fresh plugs, the generator would have run rough — maybe started, maybe not, definitely running uneven. A $40 part sitting behind a $3,000 emergency service call.
I’m not 100% sure of the exact dollar figure on what that disruption cost us operationally. We lost about six hours of access to our file server and CAD workstation during a deadline week. Our engineering VP called it “annoying but not catastrophic.” Still, annoyance is a real cost when you’re a consulting firm billing by the hour.
There’s something satisfying about finally understanding how all the pieces of a piece of equipment interact. After the shock of the outage, I sat down and built an actual power reliability spreadsheet — every device, install date, expected service life, cost of replacement, cost of failure.
I want to say the numbers shocked me. They didn’t. What shocked me was how obvious the weak points were once I actually wrote them down.
Here’s how the total cost breakdown worked out:
- APC Back-UPS 1500 battery replacement: 14 units at roughly $130–$180 per pack, so $1,820–$2,520 total (based on our Q2 2024 quotes; verify current pricing). Recommended cadence: every 3 years, staggered by install date, not all at once when the red lights come on.
- APC SMT1500C Smart-UPS battery: That unit runs on a 3–5 year replacement cycle depending on ambient temperature. Ours was installed in 2022 — the next swap is due 2025 or 2026. Budget $350–$450 for the cartridge plus service.
- NGK BPMR7A spark plugs: $6–$10 each, four plugs per engine. Call it $30–$40 per cycle. The spark plug wires were $60–$120 for a set. In the context of a $182K facilities budget, this is invisible money. And yet it’s the single component that determined whether a 20kW generator would actually start.
- Grid tie string inverter: Do not buy this for backup. Buy it if your goal is reducing daytime electricity consumption. It has a genuine role, but it is not a UPS substitute, and it cannot power your building when the grid is down.
- Maintenance contract: We switched providers after the spark plug situation. The old vendor’s “skip this cycle” call wasn’t a technical judgment — it was a scheduling convenience. The new contract includes photo documentation of every replaced part. Costs about 12% more. Worth it.
If I remember correctly — and I might be misremembering the exact figure — our total unbudgeted spend on the outage aftermath was somewhere close to $4,800 across batteries, plugs, wires, and the emergency service call. That’s not catastrophic. But it was entirely avoidable, and it came out of a contingency line I’d earmarked for HVAC.
What I’d actually recommend to another procurement person:
First, build a battery replacement calendar for every UPS in your fleet. Don’t wait for red lights. APC’s own documentation gives a reasonable life expectancy, and you can shorten it if your server room runs warm. For every 10°C above normal operating temperature, battery service life roughly halves.
Second, put spark plugs and wires on a quarterly checklist for any engine-driven backup equipment. Change the BPMR7A plugs at least annually. Check the wires with a multimeter — it takes five minutes and catches failures that no visual inspection will reveal.
Third, before you consider any solar option, clarify your actual goal. If it’s lowering bills, a grid tie string inverter is a legitimate choice. If it’s keeping the lights on during an outage, it isn’t. Two different problems, two different budgets.
I’d argue the biggest lesson was this: the reliability of a backup system is set by its weakest component, not by the sum of its parts. My triple redundancy looked fine on paper. In practice, a $40 spark plug and a $2,000 battery replacement schedule were doing more work than any of the bigger-ticket items we’d prioritized.
There’s a version of this story where I sound smart and organized. That version isn’t true. What actually happened is: I got lucky that the outage came during business hours, that no one was hurt, and that nothing failed permanently. Then I wrote things down so the next one wouldn’t depend on luck.
Take this with a grain of salt — but if you’re managing any kind of facilities budget, go check your spark plug wires right now. It’s cheaper than the alternative.
