40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Mon Nov 7 14:50:19 2011, Jenne, Update, Computers, ISCEX is having a bad day 
    Reply  Mon Nov 7 17:27:28 2011, jamie, Update, Computers, ISCEX IO chassis timing slave dead 
       Reply  Wed Nov 9 18:02:42 2011, jamie, Update, CDS, ISCEX front-end working again (for the moment...) 
Message ID: 5854     Entry time: Wed Nov 9 18:02:42 2011     In reply to: 5836
Author: jamie 
Type: Update 
Category: CDS 
Subject: ISCEX front-end working again (for the moment...) 

The c1iscex IO chassis seems to be working again, and the iscex front-end is running again.

However, I can't say that I actually fixed the problem.

Originally I thought the timing slave board had died by the fact that the front LED indicators next to the fiber IO were out.  I didn't initially consider this a power supply problem since there were other leds on the board that were lit.  I finally managed to track down Rolf to give downs the OK to pull the timing boards out of a spare IO chassis for us to use.  However, when I replaced the timing boards in the chassis with the new ones, they showed the exact same behavior.

I then checked the power to the timing boards, which comes off a 2-pin connector from the backplane board in the back of the IO chassis.  Apparently it's supposed to be 12V, but it was only showing ~2.75V.  Since it was showing the same behavior for both timing boards, I assumed that the issue was on the IO chassis backplane.

I (with the help of Todd Etzel) started pulling cards out of the IO chassis (while power cycling appropriately, of course) to see if that changed anything.  After pulling out both the ADC and DAC cards, the timing system then came up fine, with full power.  The weird part is that everything then stayed fine after we started plugging all the cards back in.  We eventually got back to the fully assembled configuration with everything working.  But, nothing was changed, other than just re-seating all the cards.

Clearly there's some sort of flaky connection on the IO chassis board.  Something is prone to shorting, or something, that overloads the power supply and causes the voltage supply to the timing card to drop.

All I can do at this point is keep an eye on it and go through another round of debugging if it happens again.

If it does happen again, I ask that everyone please not touch the IO chassis and let me look at it first.  I want to try to poke around before anyone giggles any cables so I can track down where the issue might be.

ELOG V3.1.3-