I got an email from APC PowerChute Business Edition that said simply “Battery Disconnected.” The connected UPS is an APC SmartUPS 1500 model DLA1500 (the Dell equivalent of an SUA1500). Of course the battery is not disconnected; the connection inside the unit is a very secure clip that can’t just wiggle loose.
Once I got on site, I found that the “Replace Battery” light was blinking. For some reason there was no corresponding event in the PowerChute event log.
Of greater concern was that the unit was almost too hot to touch. When I got the battery door open, I could see the batteries bulged somewhat and were very hot. The plastic pull tab broke off when I tried to pull out the batteries. I finally managed to hook a wrench around the top and pull them out. And get this: two hours after removing them, the batteries still felt hot! I guess I’m lucky I didn’t have flammable hydrogen gas escaping as this thread says can happen. I disconnected the two batteries from each other and left them outside overnight to cool.
Reviewing the PowerChute data log, I see that the temperature is usually around 30 Celcius (86 Fahrenheit), but starting at about 7:30am, it slowly climbed to 41 C (106 F). The temperature was 40 C when the unit reported “battery disconnected.” Is it a coincidence that 40 C is the temperature at which the fan turns on (KB article)? Perhaps some safety switch was tripped to disconnect the battery? The fuse between the batteries did not trip.
Here’s the event log, showing the “battery disconnected” event at 2:39pm:
And here’s the temperature graphed from 4:10am to 4:50pm:
How Hot Is Too Hot?
According to its specs, the DLA1500 is supposed to operate safely in an environment up to 40 C. The building where this unit is installed does not have air conditioning, but the high in San Diego that day was 71 F, so the room was well below 80 F (27 C). I understand that ventilation is important, and I’ll move the UPS to sit on top of the server tower rather than beside it, but clearly the overheating here was not due to a hot room but to a battery or controller failure.
What Should Have Happened
I am definitely concerned that this high-end UPS did not detect a failed battery or an overheating condition earlier. All I got was the “battery disconnected” warning; since the server was still up, I didn’t consider it an emergency. Fortunately, it was a convenient time, so I went on site within a couple hours and discovered the overheated unit.
Next time: carefully check the Internal UPS Temp column in the Data Log. If it has risen 10 C in one day, consider a shutdown and get on site.
I wondered why PowerChute hadn’t warned me about the overheating. It turns out that the default setting for “UPS Internal Temperature Threshold Exceeded” event is 70 C (158 F):
Wow! If the unit is almost too hot to touch at 40 C, 70 C seems way too high.
I don’t know how accurate these internal temperature sensors are. At another location, I have an APC SU1400NET that always reports about 40 C, but its case is hardly warm to the touch.
Maybe the solution is to check the normal operating temperature for a given UPS in the Data Log, then to set PowerChute’s UPS Internal Temperature Threshold Exceeded event to trip about 5 degrees above. So if 30 C is normal, set PowerChute to 35 C. Note that this is a full shutdown event by default, but that is certainly preferable to risking a fire in the server room.
Has anyone else seen issues like this? What does PowerChute report as the typical internal temperature of your SmartUPS?
Update November 23, 2013 I just noticed that there is a separate Warning event called UPS Internal Temperature Warning which can be configured to send an email. It makes sense to set that a few degrees lower than the Critical Threshold Exeeded event, which shuts down the machine. That will hopefully give some advance notice of a UPS on its way to overheating.
I have recently had the same problem, UPS too hot to touch, battery disconnected warning, batteries overheating, bulged out, terrible smell of hot plastic in the office alerted me when I arrived in the morning.
I was wondering if you ever tried replacement batteries or found the cause of the problem – I don’t know if I want to trust the unit again or invest in a new battery pack if it is going to be trashed by the UPS.
APC replaced that unit under warranty. Apparently the replacement was a refurb since it shows a manufacture date of 2008. I assume the battery was new. No issues since then.
I did move the unit off the floor (now sits on top of the tower server) and set the UPS Internal Temperature Threshold Exceeded event to shut down at 40 degrees Centigrade instead of 70. Currently it’s running at 22.5 C. The server itself shut down from overheating last summer, and the client installed a small room air conditioner.
I would definitely call APC. In my opinion, a bulging battery and overheated plastic may indicate a faulty unit, not to mention a fire hazard. A new battery might not solve it. Sometimes companies will replace equipment even past the warranty if there is a real hazard.
Hi Mark, really appreciate your prompt response.
I contacted APC this morning who referred it immediately to 2nd Tier support. They called me a few minutes ago (great service) and are going to replace the battery at their cost (battery was genuine APC 3 yrs old so just at the edge of normal working life). They were confident that the problem is only the battery and the UPS would be safe to use. They were not able to give me any real advice to prevent this in future other than operating the unit in a ‘not-too hot’ environment, approx 20C. There is no way to monitor the battery temp itself with the SUA1500i.
I will adjust the UPS Internal Temperature Threshold Exceeded event as you suggest.
I will also monitor the battery temp independently for a while with my digital thermometer to see what happens.
Thanks again for great advice – that one call to APC saved me £100 as well! Next time I am in San Diego I will buy you a beer!
Neil
Good to hear, Neil. If that model has a temperature probe, I would think you could see the current temp in the data log (last column of first screen shot above). Use that as a basis for setting the threshold. Also, under Data Log > Configure Log, set Data Log Entry Expiration to something like 1 month so you can go back after an event to see what happened.
Slightly similar issue … The “Replace Battery” light came on on my XS 1200 unit.
I got replacement batteries at Batteries Plus; the guy there assembled the battery pack.
I put them in the unit, and now the Replace Battery light is flashing. If I look at the unit in Power Chute, it claims there is no battery in the unit.
I’m thinking that the battery pack was not put together properly, or one of the wires in the joining unit came disconnected.
I’m inclined to take the battery pack apart and reassemble based on a YouTube video that explains how to do the replacement.
Any thoughts or experience before I do this, would be helpful.
First I would take the whole thing back to Batteries Plus and get them to double-check it. Have them show you how it’s wired and explain why it’s right. Or if they messed up, hopefully they will make good on it, which they’re unlikely to do if you have been pulling things apart first.
Failing that, I would test the batteries with a meter, as a pack, then individually. You could also test the voltage output of the UPS. Battery pack assembly for a UPS is not difficult, but it _is_ electricity and if you aren’t familiar with the basics (like, can you confirm you have two 12v batteries in series with a 24v output), or if you don’t have a meter, I’d find a friend to help. Even then, if it was wired wrong and plugged in, I don’t know what that might do to the battery chemistry.
You could certainly have a faulty UPS; as I said, I had to have mine replaced.
Thanks for the quick reply …
UPDATE: I took the whole unit into Batteries Plus. There was nothing coming out of the plug from the battery, and when they disassembled the pack and tested the batteries, neither was putting out full power. One more of the batteries they pulled from the shelf was iffy, but they came up with two that were good, and now the UPS is working fine.
Bert, glad to hear your Batteries Plus store stood behind its product!
I realize this is an old post, but I just experienced this issue with my APC SmartUPS 750. I got an email alerting me to low battery, but it was running on AC and had been for several weeks. I checked and the battery was at 3%. I then went to check on it, and it gave me a “battery disconnected” error, that then switched to a “new battery connected” question, and back to battery disconnected over and over, and the battery voltage jumped back and forth between 0 and 24 V. No advance warning to replace battery.
The unit was burning hot, so I powered it down, removed the safety disconnect plug in the back, and took the front off to allow the batteries to cool faster, and I also took out my thermal camera which showed a temperature of between 50 and 70°C on the batteries a good 40 minutes after power down. Battery acid had leaked out and formed a brown and white mess inside. Not impressed at all.
Not sure if the unit is good with new batteries or if it needs replacing. It’s 5 years old at this point and with it’s original batteries from factory.
@Erik, what does the APC log say about battery temps? I do have the impression that the weekly battery test is not always reliable. Five years is a good run for a set of batteries; it’d probably be safer to replace after three years.
See my 2/6/2013 comment above: “I would definitely call APC. In my opinion, a bulging battery and overheated plastic may indicate a faulty unit, not to mention a fire hazard. A new battery might not solve it. Sometimes companies will replace equipment even past the warranty if there is a real hazard.”
Also see Neil’s response on 2/7/2013.
The log on the device itself (accessed through the LCD) has nothing in it except a note about the last start of the unit. I don’t have APC software running since I’m running VMWare ESXi and a couple of Linux boxes and a Synology NAS. The UPS is connected to a raspberry pi which handles shutdown of all the other machines. I didn’t find another way to shut down the NAS last.
I find it a bit scary that the battery went back and forth between 0 and 24 volts and the UPS still tried to charge them, even though it periodically asked if a new battery had been installed to which it never received a yes.
Erik, what does APC support say?
How are you shutting down from the Pi? I’ve used apcupsd on Linux before; I would think it logs events? Not sure about temperature. That was on a box running a phone system called PiaF:
https://www.mcbsys.com/blog/2009/03/using-an-apc-ups-with-pbx-in-a-flash/
I actually have a UPS in a closet for a switch. Wondered about adding a Pi for monitoring…
According to Synology, you can shut down the NAS from a Smart-UPS 750. “The UPS can either connect to your Synology products via USB or network.” Don’t know if that’s compatible with your Pi controller though. https://www.synology.com/en-us/compatibility?search_by=category&category=upses&filter_tested_by=Synology&p=1
I have yet to contact them. As a regular non-business consumer using it at home I don’t have too high hopes for goodwill, but I’m going to contact them tomorrow to voice my concerns.
As far as the Pi, I’m using NUT. I tried apcupsd which has access to some more specific data from APC units, but it wasn’t compatible with the network ups support in my other devices. The Synology is monitoring the NUT server on the Pi, and will shut down after 10 minutes. My ESXi box will be shut down after 2 minutes over SSH by a script on the Pi. Then my networking equipment will keep running until the UPS is down to 30% at which time the Pi will shut down and tell the UPS to power down. This is enough to keep the network going for 2-3 hours.
The Pi works well, but I would recommend to put /var/tmp, /var/log and /tmp on ramdisks (tmpfs). Otherwise the micro SD will wear out pretty quickly. Remote syslog could also be useful as /var/log would be erased when it’s powered off using this method.
The problem with connecting it straight to the Synology is that it then can’t keep the UPS on for the network equipment. It will either have to shut it down immediately on power down, or not shut it down at all. On the other hand, the few times where there is a power failure on a neighborhood level my fiber connection will go down either way unfortunately. It is useful to be able to keep browsing the net when doing electrical maintenance in my home though.
I will write another comment when I get a response from APC.
The Smart-UPS is APC’s business line and should be “smart” no matter where it is installed. It’s perfectly valid for a prosumer to buy a business-class device because s/he needs business-class performance.
I promised an update, so here it is:
APC has shipped a factory new unit to me at no cost, and they want the old one back for investigation. Still not happy that the batteries failed without prior warning, but I’m very pleased with them taking responsibility for their products.
Good to hear! Thanks for posting the followup.
I only tried this on commercial-quality UPS systems like the Smart-UPS. Unlike el cheapo systems that expect to run only a few minutes as your PC shuts down, these have fans that can keep the UPS cool indefinitely.
Alina, if it wasn’t clear, this was in fact a SmartUPS and the fans did NOT keep it cool. The battery was bulging, very hot to the touch, and possibly a fire danger. The point of this article is to help people use the PowerChute software to get a warning when a “commercial-quality UPS” fails. As you can see in the previous comments, mine was not an isolated incident.
Hi Mark, thanks for sharing the story.
I’m wondering if APC have improved anything in this regard to avoid continuing to charge the batteries when they are already damaged?
Now in APC documentation I see the following sentence:
“Temperature-compensated battery charging. Prolongs battery life by regulating the charge voltage according to battery temperature.”
Still, I’m not sure if it’s something they added recently and if it’s a real hardware modification for their newer batches of SMT series UPS units or if it’s just marketing.
I’m looking for a solid sine-wave silent UPS for my workstation at home but I definitely wouldn’t want a “smart device” that is not smart enough to attempt to charge damaged batteries again and again, possibly causing a fire hazard. What if I’m not physically at home and cannot turn the UPS completely off? Will it go into the loop “attempt to charge – wow, it’s too hot, shut down – cool down – attempt to charge again – wow, it’s too hot…”
I believe you can configure a smart UPS to NOT turn back on after a power outage. The question to ask their tech support would be, even if automatic start is enabled for a power outage, if the unit shuts down due to an abnormal state (something other than a power outage), will it stay off?
Battery life is 3-6 years. Typical symptoms of a battery getting old and failing is reduced runtime, overheating and battery bulge. Overheating and battery bulge are a bummer but as long as the battery can be safely removed without damaging the UPS case, a replacement battery will usually resolve the issue. The UPS is NOT defective because this happened.
@John, the UPS was defective not because the battery went bad, but because it didn’t tell me that the battery was going bad. APC agreed and replaced it.