40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  PSL  Not logged in ELOG logo
Message ID: 2312     Entry time: Sat Mar 23 21:50:40 2019
Author: anchal 
Type: Summary 
Category: Electronics Equipment 
Subject: Power tripping incident and follow-up 

This week on 19th March in the evening, I was working on replacing the GND connections of the power supply with thicker wires and checking out any AC rms voltage between different ground points to look for ground loops. During this, I found that the high voltage power supply for PMCs wasn't directly grounded with the rest of the power supplies. These are Kepco PCX 200-0.1 MAT power supplies. From here onwards, I'll tell about this incident chronologically:

Before the incident:

  • Pins 4 (Sensing -) and 5 (Output -) were not shorted with a shorting clip to keep voltage regulation good.
  • Pins 8 and 9 were shorted. These pins are for the external resistor for voltage control. At this point, my understanding from the manual was that front panel voltage knob overrides this external voltage control.
  • Pin 5 was GND terminal for two such power supplies connected in series and operated at 80V each, totaling to 160 V for the single-sided rail of PA85 in the PMC servo boards.

The incident:

  • I removed the shirt between PIN 8 and PIN 9 leaving it open and used this clip to short PIN 4 and PIN 5. I also connected PIN 4 to the rest of the GND of other power supplies which is GND for the rest of the lab.
  • Then to test this, I switched on all power supplies, leaving one pair of +-24V power supplies at the bottom. I left them as at this point already, +18 V supply used for FSS boxes showed me a trip (it became a current source limiting to 3 A with about 3V voltage). Also, the Kepco power supplies for PMC where I did the changes shot to maximum 200V (instead of set 80V).
  • I shut down everything and rewired everything as before.

My understanding of what happened here:

  • Since short between PIN 8 and PIN 9 were removed, the external voltage control of Kepco power supply got activated and rose to maximum 200V making rail of PA85 in PMC servo card +280 V (Maximum allowed +450V).
  • This allowed the PMC PZTs to be driven to upto 280V (Maximum allowed being 200V) if the control signal required it to. So far after all the tests, I came to the conclusion that this didn't happen and PZTs are healthy.
  • The +18V power supply simply tripped because the FSS boxes use 18V rails to power 24V rail circuits as well in absence of power at 24V rails (which I didn't switch on yet). So the lesson here is that always switch on 24V rails power supply first for the FSS boxes.
  • Finally, this is just a guess, the sudden rise of Kepco power supplies to 200V might have sent a surge of charge to rest of the GND. It doesn't really make sense, but something must have happened because I found RF amplifier of PMC reflection RFPD dead after all of this.

Current state:

  • I replaced the MAX4107 RF amplifier in SN020 RFPD (and actually after doing this twice), this RFPD became as before with same transfer function as measured in PSL:2247.
  • I noticed that PDH error signal for South FSS was too low. Later I found this was just because Andrew added a microwave amplifier in the RF output which mismatched the phase matching of FSS PDH.
  • There are more findings on the South reflection RFPD and FSS of South path. I'll report this in a future post with some numbers.
  • I have tested all FSS boxes and PMC Servo cards as well as EOM drivers. Everything else in the lab seems ok at this point. The lasers are locking nicely and following Slow PID as well.
  • Regarding the ground loop issue which was the start point of all this, everything is status quo, so it is also there.
ELOG V3.1.3-