Sudden voltage drop on half of my powerwall; what just happened

H

harrisonpatm

Member
Joined
Jan 5, 2022
Messages
401
Pack is 14s Li-ion, made of tested laptop cells. Each parallel group is made up of 3ea 24p packs, added at different times. Been in service for almost 2 years, no major issues at all.

Last night i went to bed, pack was at 54.5v, .02-.03v cell deviation between pack (about 3.9v per cell), as per my normal usage. Overnight loads are 3d printer, cell chargers, fridge, modem, router, some lights, phone chargers. Not the whole house. Came home from work this morning at 8am, pack was around 52.5v, if I remember correctly. I started charging my motorcycle from my boost converter, which takes my pack voltage and boosts it up to 83v. I checked the powerwall for temperature, it was around 7-8 degrees celsius. I did not notice or check cell deviation at that time. After one hour, I checked again: my BMS had shut off discharging. Packs 1-5 had dropped voltage to 3.0-3.2. The rest were over 4.0v.

What the hell happened? Possibly overnight, but also possibly in the span of just 2 hours, packs 1-5 drained almost completely, while packs 6-14 seemed to have risen to almost full charge.
Screenshot_20240328-093925.jpg


I'm currently in emergency damage control mode, I've opened up my outdoor enclosure to access the cells directly, and I'm using a hobby charger and alligator clips to charge individual groups. No obvious damage to the cells. No cell fuses seem to be broken, though I haven't laid eyes on every single cell yet. No smoke, no smell of smoke, from shorts or whatnot. Not burnt wires or visable shorts. The enclosure is buttoned up tight in a small sealed enclosure, and if there was any sort of thermal event, I would have easily smelled it when I took off the front panel. No dice.

I will check back as I work on the issue from my end today. Any suggestions for issues to look for are appreciated.
 
Observation that might be a symptom: when I was in a hurry, I was using my hobby charger to first charge 1s by itself, @ 5-10 amps. When I did so, the BMS was measuring the input 10 amps. It shouldn't have, I was bypassing the BMS?

Then I switched to charging just 2s by itself. When I did so and initiated charging via the hobby charger, the BMS registered a 30-60 amp current draw, and 2s dropped voltage down to 2.8, so I stopped charging.

Now I'm currently using the hobby charger on a totally separate AC line to charge 1-5s as one group, as a 5s pack. This seems to be working.

But I still need to figure out why it happened in the first place.

1s-5s are currently taking current normally. The whole powerwall is now off and out of service while I figure this out. I have a smaller 16s LFP in use to run my router and internet in the meantime, and I have all but 2 of my panels turned off.
 
First random thought - maybe the BMS is not reporting correct info? I could imagine 1-5 discharging more than 6-14 (I guess) but I cannot imagine 6-14 going from 3.9v 'up to' 4.0v.... while the battery is providing load. This makes me suspicious of what the BMS is reporting.

Perhaps independently measure the packs with a voltmeter as a 1st step?
 
OffGridInTheCity said:
First random thought - maybe the BMS is not reporting correct info? I could imagine 1-5 discharging more than 6-14 (I guess) but I cannot imagine 6-14 going from 3.9v 'up to' 4.0v.... while the battery is providing load. This makes me suspicious of what the BMS is reporting.

Perhaps independently measure the packs with a voltmeter as a 1st step?
Click to expand...
Already done, BMS reported correct voltage. 1-5 went way down, 6-14 went up a little, confirmed by multimeter. So yeah it does seem like somehow, I have no idea how, 1-5 "charged" 6-14.

I'm totally at a loss. I got 1-5 back up to about 3.4v, with 6-14 still at 4.0v, and briefly reconnected the powerwall and charged my motorcycle again to see what would happen. It charged perfectly normally for several minutes, with me staring at cell voltages the whole time, and nothing abnormal happened. I stopped because I still need to rebalance the whole thing.

Currently still charging 1-5 with a hobby charger. They're still accepting 15 amps normally and getting back into balance with the rest of the pack.
 
So now I'm thinking that perhaps 1-5 voltage levels actually did droop and the BMS cut-off the overall battery.

So the next thought is.... was the load overwhelming the 1-5 cells causing *severe* voltage droop. Even though 1-5 packs have the same capacity as 6-14 packs - depending on the cell type, I could imagine a larger voltage drop under higher loads for 1-5 than 6-14.

Do you know the the peak load ( ma per cell ) during the meltdown? For example, even 1a/cell and certainly 2a/cell could potentially cause a big voltage drop on 'laptop type' 2nd hand cells on 1-5. If so, you can reproduce this under a controlled load to verify.
 
Last edited:
OffGridInTheCity said:
So now I'm thinking that perhaps 1-5 voltage levels actually did droop and the BMS cut-off the overall battery.

So the next thought is.... was the load overwhelming the 1-5 cells causing *severe* voltage droop. Even though 1-5 packs have the same capacity as 6-14 packs - depending on the cell type, I could imagine a larger voltage drop under higher loads for 1-5 than 6-14.

Do you know the the peak load ( ma per cell ) during the meltdown? For example, even 1a/cell and certainly 2a/cell could potentially cause a big voltage drop on 'laptop type' 2nd hand cells on 1-5. If so, you can reproduce this under a controlled load to verify.
Click to expand...
Not even close to 1a per cell. I was charging my motorcycle at the time, using 800w. @ 52v, that would be 15.5 amps. The pack is currently 72p, so thats 200ma per cell. Call it 300ma per cell, assuming there were a few other smaller loads.

In addition, I have currently been charging groups 1-5 with 15 amps, for the past 4.5 hours. That's 67ah. It brought the cells from 3.2v to 3.75v so far. My estimated capacity is 150ah. So... that all tracks.

The more I think about it, the more it seems like groups 1-5 somehow transferred capacity to 6-14. I have no freaking idea how. I've been inspecting the structure of the enclosure looking for screws, loose connections, stray wires. That's so much capacity in such a short time, I can't see how that could have happened. It's rather frustrating because the pack is behaving totally normally now, and if I don't know what happened I can't keep it from happening again.
 
I don't have the equipment, know-how or patience to do data logging. But damn, it's times like these I wish I had.
 
harrisonpatm said:
Not even close to 1a per cell. I was charging my motorcycle at the time, using 800w. @ 52v, that would be 15.5 amps. The pack is currently 72p, so thats 200ma per cell. Call it 300ma per cell, assuming there were a few other smaller loads.

In addition, I have currently been charging groups 1-5 with 15 amps, for the past 4.5 hours. That's 67ah. It brought the cells from 3.2v to 3.75v so far. My estimated capacity is 150ah. So... that all tracks.

The more I think about it, the more it seems like groups 1-5 somehow transferred capacity to 6-14. I have no freaking idea how. I've been inspecting the structure of the enclosure looking for screws, loose connections, stray wires. That's so much capacity in such a short time, I can't see how that could have happened. It's rather frustrating because the pack is behaving totally normally now, and if I don't know what happened I can't keep it from happening again.
Click to expand...
Yea.... I get it and the frustration. Not getting any ideas on 'how' this could happen - but clearly it happened!

OK, next idea is to add detailed monitoring so you have data to help clarify things if it happens again... I'm thinking voltage monitor each pack in series type of thing. Does you're BMS let you record pack voltages or can you extract them every 5sec or something like that?

Next idea if it happens again is to replace the 1-5 packs with other packs if you can. If it doesn't fail, then it points to those 5 packs specifically.

**I remember seeing a post or youtube on cells in series and that you can actually have cell(s) in series where a cell can go NEGATIVE with the rest positive but yet the overall battery will continue. It doesn't look like you went negative but went *severe* voltage drop.... (do I have this right?). The point I'm making is that packs in series can function extremely differently than the other packs - indicating that 1-5 are weak or bad packs?!? compared to 6-14. But the question as to how they are 'bad' isn't clear since you have paralleled cells... it's not like it's individual cells in series. It's hard to imaging all the parallel cells in the 1-5 packs going bad at once and not blow the fuse wire.
 
Last edited:
OffGridInTheCity said:
It's hard to imaging all the parallel cells in the 1-5 packs going bad at once and not blow the fuse wire.
Click to expand...
Exactly. I'm using the Battery Hookup fuse nickel sheets. So if something like that were to have happened, there would be smoke and hot metal inside my enclosure. There wasn't.

And if it's true that you can have a cell go negative with the rest positive. Sure, maybe. But for 15 packs (5 series, 3 packs in each)? All at the same time, all evenly?

And. They're balanced. Groups 1-5 all stayed balanced relative to each other. I'm almost done bringing them up, and all I had to do was hook the hobby charger's + and - to the first and last group. As I watched them charge for the last 6 hours, they all stayed balanced. Plus, groups 6-14 were also perfectly well balanced relative to each other the whole time, 4.0-4.02v the whole day.

Completely boggles my mind. Currently have it down to a 0.110v difference, so while I continue to separately charge groups 1-5 for the next few hours, I've reconnected my powerwall to all the panels and the normal loads it usually has, and I'll closely watch it tonight. So far no abnormal behavoir. I'm baffled.
 
Final comment - back to monitoring / getting data. Even thought the end result is 'severe', maybe something gradually pulled voltages down on 1-5 so the issue is more subtle (somehow) and you'll be able to see it happening with closer monitoring and give some clues.

Very interesting as a follower of you're work and I'm excited to see you figure it out and learn - but of course it's no fun for you. My sympathies!
 
Last edited:
OffGridInTheCity said:
Final comment - back to monitoring / getting data. Even thought the end result is 'severe', maybe something gradually pulled voltages down on 1-5 so the issue is more subtle (somehow) and you'll be able to see it happening with closer monitoring and give some clues.

Very interesting as a follower of you're work and I'm excited to see you figure it out and learn - but of course it's no fun for you. My sympathies!
Click to expand...
Thanks, i appreciate it. I am also just going to watch it like a hawk for the next few days. Luckily nothing serious happened, and the BMS did its job.
 
you could also look with a thermal camera during powerwall operation. I have a topdon which plugs into my mobile/cell phone. Brilliant for finding hotspots.

Also, the battery temperatures are low. My JK bms cuts off at 5 degrees. Just thinking nonsense out loud.
 
paulie said:
Also, the battery temperatures are low. My JK bms cuts off at 5 degrees. Just thinking nonsense out loud.
Click to expand...
The battery temps were indeed a bit low at that time, since I had to open up the case to access the cells directly. Normally my heaters keep the cells above 10 degrees, and they were indeed still above 10 degrees when I started. My cutoff is 4 degrees, so I was running it close to do this emergency balance, but I didn't really have a choice. I risked it because it was early in the morning and the outside temp was rising pretty fast.
 
I didn't think of temps... are 1-5 somehow affected by temps more than 6-14? When you say "4 degrees" I assume you mean 4C? 4C is within the functional range and should not cause extreme voltage drop.
 
OffGridInTheCity said:
I didn't think of temps... are 1-5 somehow affected by temps more than 6-14?
Click to expand...
Nope. Temperature did cross my mind as well: if both my heaters and the temp sensors on the BMS failed, it's conceivable that I had inadvertently charged the powerwall at subzero temps, which would indeed have led to rapid cell degradation. However, the heaters were still working when I cracked them open yesterday, and the BMS temp sensors were reading accurately. It's true they dropped to 5 degrees C by the time I took that screencap in my first post, due to exposure to outside air. But it hasn't been below 0 degrees C for the past several weeks in my location. So even if there was enclosure heating failure, they still would not have gotten so cold on their own just due to outside temp.

I do have the enclosure split into two separate sections, essentially sealed off from each other, with separate heaters. So if temp was an issues, or enclosure failure, it would have equally affected 1-7, or 8-14. Not just 1-5, leaving 6 and 7 unaffected like they were. So for these reasons, I ruled out temperature as an issue. And after one day, I can re-report that groups 1-5 held on to their voltage and capacity all night and are normally taking a charge today. I don't think temp was the culprit here. I certainly could be wrong, but it doesn't fit the evidence available.
OffGridInTheCity said:
When you say "4 degrees" I assume you mean 4C? 4C is within the functional range and should not cause extreme voltage drop.
Click to expand...
Correct, and I agree.
 
Few days later. I manually balanced 1-5 to within 50mv of the rest of the powerwall. I paid really close attention for several days. It stayed at around 50mv difference, still stays there. So, super weird. I still have no idea what happened.
 
harrisonpatm said:
Few days later. I manually balanced 1-5 to within 50mv of the rest of the powerwall. I paid really close attention for several days. It stayed at around 50mv difference, still stays there. So, super weird. I still have no idea what happened.
Click to expand...
Thanks for the update. Something happened, just not clear yet. Usually things eventually repeat... and when it does, perhaps you'll can get more data. On the bright side, sounds like the BMS (protections) worked. You have my sympathies!
 
OffGridInTheCity said:
Thanks for the update. Something happened, just not clear yet. Usually things eventually repeat... and when it does, perhaps you'll can get more data. On the bright side, sounds like the BMS (protections) worked. You have my sympathies!
Click to expand...
No, you have mine. So devastated to hear about your disaster. Really puts mine into perspective.
 
You must log in or register to reply here.
Back
Top