I was at work earlier this morning doing work inside of a network device. I spent about 5 hours working on one device in particular, but which is part of a pair (redundant devices). I'll call them device A (DA) and device B (DB). For the past several months, I have spent a lot of time working on these two devices. Much has happened, notably DA was replaced (entire chassis replacement) exactly one week ago.
While working on DB something on DA failed in a way that is beyond my current understanding of electricity.
Most of my work centered around DB in a non-direct way; I was moving fiber-optic cables around that connect to DB but do not directly attach. Only at the end did I physically and directly interact with DB. DA was not involved in this work at all, since it needs to be available during this time. At the very last stage, after at least 4 hours, I did something directly on DB by removing one of its components (a line card). While the card was half-way out, a fiber-optic cable brushed up against this card. I can remember seeing it touch a grounding strip that lines the card on both sides. As soon as this happened, I heard a loud electro-mechanical POP noise* and a power-supply on DA failed. Although I have not heard that particular sound before, I have several times heard the sound of a power circuit tripping and the circuit breaker activating. It was not like this, it had very much the sound of something permanently breaking to it. Troubleshooting the issue revealed the device reporting that power-supply (there are two) failing at the same time as the sound, that the circuit-breaker on the power circuit did not trip, and that the power-supply would not work after removing and re-inserting it. It was replaced with a different power-supply and all appears to be well.
I cannot remember which fiber exactly touched, because there were several nearby. There are some fibers (3) that connect DA to DB directly. Most of these fibers connect to other devices. This work occurred within a data center environment so the humidity is controlled, which prevents electro-static discharge (ESD). I had already been touching the device as I had to remove connectors on it and slide it out. It happened as the card was half-way out, and this fiber touched it; there might have been two but I clearly remember one.
I am at a loss to understand how this event on DB triggered a failure of a power-supply on DA.
When I return the part to the manufacturer, I will specifically request a failure determination - they test the unit to see what failed on it and provide additional details related to that. I will also inquire as to this specific occurrence; who knows, maybe this has happened before on their equipment. Maybe the level of sophistication regarding their power-circuit protection is much more advanced than I realize.
I really just don't know what to make of this. Even my manager, whom I had to call after this happened, said he didn't understand how that could've happened. He quickly joked that it might be karma, perhaps that escalation call I did not take over the weekend was coming back to haunt me. For some context,
here is a link to a picture of the device I was working on. I was working on one of these, while the one directly above it had some sort of electrical failure. I also will have to discuss this event to some degree in our weekly meeting today so that will be interesting.
* The sound: it's hard to describe, my best guess is it was some sort of circuit-breaker (or fuse) inside of the power-supply tripping (activating) and the abrupt interruption of the flow of around 600 watts coupled with the (designed) breaking of this component caused this noise.