| ID |
Date |
Author |
Type |
Category |
Subject |
|
1335
|
Tue Feb 24 18:42:15 2009 |
pete | Update | Locking | mc board repair |
Peter, Yoichi
Last night:
| Quote: | However, when we measured the MCL loop gain with several different AO path gains, the loop shape did not change at all. This led us to suspect the AO path may not be connected. The cabling from the common mode board to the MC board seemed ok. We tested the signal flow in the MC board using a signal generator and an oscilloscope. Then we found that a signal injected to the IN2 (AO path) does not reach to the TP1A (right after the boost stages), though the signal is visible in the OUT2 (monitor BNC right after the initial amplifier (B-amp) for the AO path). The signal from IN1 (MC REFL) can be observed at TP1A. This means something is broken between the B-amp and the sum-amp in the AO path. We will check the MC board tomorrow.
|
Today we examined the MC board. With the extension board in place everything seemed fine. Without the extension board we could replicate the problem. Jiggling the IN2 jack caused the injected signal observed at TP1A to come and go. These jacks are unfortunately mounted directly on the board. We traced the problem to a resistor in this path (R30) which looked fishy. We soldered on a new 2K resistor with OWC and it fixed the problem. |
|
1336
|
Wed Feb 25 03:10:24 2009 |
Yoichi | Update | Locking | Locking status |
Rob, Yoichi, Kakeru, Kiwamu
Tonight, CARM -> MCL hand off was not stable. The MCF signal monotonically went up to +2V after CARM and MCL gain was turned down to zero.
This was repeatable and it only goes up (not down).
After a while, we found that putting sleep (~5sec) between the zeroing of CARM gain and MCL gain prevents this problem.
Handing off of CARM error signal from TR to PODC was also not robust.
It seems that the suitable gain changes every time.
tdsavg started to exit with errors. We rebooted fb40m.
When tdsavg returns an error, the cm_step script tries to write NaN into SPOB DC offset.
To prevent this, I put the tdsavg in a while loop which runs until tdsavg returns something other than NaN.
I was able to hand off to PODC several times, but could not proceed further because the IFO lost lock soon.
|
|
1337
|
Wed Feb 25 11:48:02 2009 |
Jenne | Update | PEM | Wiener filtering update - work on filtering some S5 DARM_CTRL data |
Quick update on my wiener filtering status:
Joe has been helping me get on the GRID, so I now have a grid certificate, and accounts on most/all of the clusters.
Joe also helped me get menkar to get S5 data so that I can do wiener filtering to the back-data.
I've been running the wiener filtering algorithm, and right now, it doesn't do anything to improve the DARM_CTRL data. I am confident that this is because something is funky in the wiener filtering algorithm somewhere. The indicator of this is that the wiener filtering calculation takes the same amount of time (~95 seconds) to calculate a filter for 64 seconds of data as for 1 hour of data (both for N = 2000 taps).
For reference, attached are my plots for the wiener filtering result for (1) 64 seconds of S5 data, and for (2) 3600 seconds of S5 data.
These plots were made using H1:DARM_CTRL as the signal to minimize, with 4 seismometers as the witness channels (EX_SEISX, EY_SEISY, LVEA_SEISX, LVEA_SEISY)
I'm working on figuring out what's going on with the filtering algorithm, and why it does work for C1:MC_L minimization, but does not work for H1:DARM_CTRL minimization.
|
| Attachment 1: h1_DARM_64s_4seis_25Feb09.png
|
|
| Attachment 2: h1_DARM_3600s_4seis_25Feb09.png
|
|
|
1339
|
Thu Feb 26 01:24:44 2009 |
Yoichi | Update | Computers | Martian wireless is back |
Today, a new wireless router arrived.
I configured and installed it. Now the martian wireless network is back.
I updated the wiki page about the wireless network.
http://lhocds.ligo-wa.caltech.edu:8000/40m/Network |
|
1341
|
Thu Feb 26 19:59:23 2009 |
Yoichi | Update | Locking | Daytime locking |
Osamu, Yoichi
We tried locking today from about 2PM.
It took about 1000sec on average to acquire the initial lock.
After the initial lock is achieved, the hand-off/ramp-up steps were reasonably robust, although the AS beam sometimes fluctuates a lot (not good for mental health).
Like last night, the IFO loses lock at around arm-power=8.
We measured the CARM AO path loop gain at arm-power=4. We used the SR785 connected to the A-excitation channel of the common mode board through my TFSR785.py script.
The first attachment is the transfer function measured right after the arm power was ramped up to 4.
The overall bandwidth of the CM servo is only 400Hz. Note that since this is the loop gain of only the AO path, the low frequency gain is eaten by the MCL path.
The second attachment is the same transfer function measured after the AO path gain was increased by 6dB.
It is evident that the AO path is working.
We increased both the AO path and MCL gain by 18dB. The third attachment is the AO path TF in this state.
We then increased the arm power but lost lock at arm-power=6. We should have checked the DARM loop too.
BTW, these plots are automatically generated when you use TFSR785.py for transfer function measurements.
I added -notickle option to c1_watch_dr_bang, since tickling seems to be not necessary during the daytime (actually the initial lock was easier with no tickling).
As the construction work in the next building is now calmed down, I think it is ok to do locking during the day time, though I still plan to come at night.
The improvement of my brain efficiency during the day time may compensate for the longer wait time for initial lock.
|
| Attachment 1: CM1.png
|
|
| Attachment 2: CM2.png
|
|
| Attachment 3: CM3.png
|
|
|
1343
|
Fri Feb 27 13:49:19 2009 |
rob | Update | Locking | thurs night |
Could not get past arm power of ~11 or so. I was suspicious of the transmon high-gain/low-gain PD handover, so I ran the matchTransMon scripts, but that did not help. I also removed the line in the cm_step script that increased the CM gain by 18dB at an arm power of 4. The gain of the CM servo will increase naturally as the power in the IFO builds up, so it may not be good to crank it right away. I tried several other CM gains, and watched the DARM loop, but still could not get past an arm power of ~10-11. I'm not sure what's wrong, but it may be that mysterious CM-servo/McWFS conspiracy, so we can try turning down the McWFS gain next time. |
|
1344
|
Mon Mar 2 03:57:44 2009 |
Yoichi | Update | Locking | Sunday night locking |
Tonight's locking started with a boot fest of the FE computers which were all red when I came in.
It also took me sometime to realize that C1:IOO-MC_F was returning always zero to tdsavg, causing the offloadMCF script to do nothing.
I fixed this by rebooting c1iovme and c1iool0.
Like Rob on the thursday night, I was only able to reach arm power around 10.
This time, I turned down the MC WFS gain to 0.02 (from 0.3).
I also checked gains of most of the loops (MICH, PRC, SRC, DARM, CARM-MCL, CARM-AO).
All the loops looked fine until the lock was lost suddenly. Also the spectrum of MC_F did not change as the arm power was ramped up.
Actually, I was able to reach arm power=10 only once because I spent a long time checking the loop gains and spectrum at fine steps of the arm power.
So it is quite possible that this loss of lock was just caused by a seismic kick. |
|
1346
|
Mon Mar 2 21:16:32 2009 |
Yoichi | Update | Locking | Low-gain High-gain PD switching may not be working well |
Osamu, Yoichi
This afternoon, I run the locking script while doing calculations for the upgrade.
The IFO lost lock even at lower arm powers (around 6) if it was operated for a while (~ 5min).
It seemed as if there were some intermittent glitches (seismic? laser?) causing the lock losses.
We also saw once the TRX and TRY signals saturated at around arm power = 11 when there was a large fluctuation in the arm power.
Osamu suggested that it looked like the high-gain to low-gain PD switching was not working.
I won't come tonight as I may have caught a cold, but if someone comes tonight, it is worth checking the PD switching. |
|
1347
|
Tue Mar 3 08:44:31 2009 |
steve | Update | PEM | air cond. maintenance today |
IFO room 104 air conditions will be shut down for maintenance today.
This should be finished by noon.
The temperature and particle count variation can be more than usual. |
|
1348
|
Tue Mar 3 10:39:07 2009 |
Alberto | Update | LSC | c1lsc discontinued functioning |
The c1lsc has been unstable since last night. Its status on the DAQ screen was oscillating from green to red every minute.
Yesterday, I power recycled it. That brought it back but the MC got unclocked and the autolocker could not get engaged. I think it's because the power recycling also turned c1iscaux2 off which occupies the same rack crate.
Killing the autolocker on op340 e restarting didn't work. So I rebooted also c1dcuepis and burt-restored almost all snapshot files. To do that, as usual, I had to edit the snapshot files of c1dcuepics to move the quotes from the last line.
After that I restarted the autolocker that time it worked.
This morning c1lsc was again in the same unstable status as yesterday. This time I just reset it (no power recycling) and then I restarted it. It worked and now everything seems to be fine. |
|
1350
|
Tue Mar 3 19:26:44 2009 |
Yoichi | Update | Locking | Low-gain High-gain PD switching may not be working well |
I checked the switching of the QPDX from high gain to low gain.
Switching happens as expected, but the low gain QPDX output was very low compared to QPDY.
Also the digital gain for the high gain TRX was not matched with the low gain one. So when the switching happens, there is a large jump in the TRX.
I also found that the offset values for the low gain QPDX were totally wrong. I adjusted it.
Then I removed a beam splitter in front of the QPDX to increase the power falling on it.
But still the low gain QPDX output is four times lower than the low gain QPDY.
I'm still working on it. So don't expect the switching to work correctly at this moment.
I'm planning to be back after the dinner.
| Quote: | Osamu, Yoichi
This afternoon, I run the locking script while doing calculations for the upgrade.
The IFO lost lock even at lower arm powers (around 6) if it was operated for a while (~ 5min).
It seemed as if there were some intermittent glitches (seismic? laser?) causing the lock losses.
We also saw once the TRX and TRY signals saturated at around arm power = 11 when there was a large fluctuation in the arm power.
Osamu suggested that it looked like the high-gain to low-gain PD switching was not working.
I won't come tonight as I may have caught a cold, but if someone comes tonight, it is worth checking the PD switching. |
|
|
1351
|
Tue Mar 3 23:59:26 2009 |
Yoichi | Update | Locking | Low-gain High-gain PD switching may not be working well |
| Quote: | I checked the switching of the QPDX from high gain to low gain.
Switching happens as expected, but the low gain QPDX output was very low compared to QPDY.
Also the digital gain for the high gain TRX was not matched with the low gain one. So when the switching happens, there is a large jump in the TRX.
I also found that the offset values for the low gain QPDX were totally wrong. I adjusted it.
Then I removed a beam splitter in front of the QPDX to increase the power falling on it.
But still the low gain QPDX output is four times lower than the low gain QPDY.
I'm still working on it. So don't expect the switching to work correctly at this moment.
I'm planning to be back after the dinner.
|
This sounds like the QPD whitening gain sliders may be stuck. The slider twiddling script should be run, or the sliders should be twiddled by hand. |
|
1352
|
Wed Mar 4 03:37:51 2009 |
Yoichi | Update | Locking | Low-gain High-gain PD switching may not be working well |
| Quote: |
This sounds like the QPD whitening gain sliders may be stuck. The slider twiddling script should be run, or the sliders should be twiddled by hand. |
Yes, it was indeed the whitening gain slider problem.
I moved them and the QPDX gain went up suddenly. I reinstalled the BS in front of the QPDX and adjusted the offsets, gains accordingly.
Now the high-gain to low-gain switching is fine. |
|
1353
|
Wed Mar 4 03:50:17 2009 |
Yoichi | Update | Locking | Still won't go above arm power 10 |
Yoichi, Kentaro
The IFO still loses lock at around arm power 10.
I attached time series of various error signals when losing lock.
In all of the three cases, both of the arm powers go up rapidly just before losing lock.
(The first attachment was taken before I fixed the QPDX switching, so you can see saturation in TRX.)
But PD1_DC (the DC power in the PRC) did not go up in the third case.
I should also check correlations with laser power, CARM length (OSEM signals), seismic noise etc. |
| Attachment 1: lockLoss1.png
|
|
| Attachment 2: lockLoss2.png
|
|
| Attachment 3: lockLoss3.png
|
|
|
1354
|
Wed Mar 4 12:38:07 2009 |
Alberto | Update | Computer Scripts / Programs | 3f locking simulations |
I simulated the REFL signals demodulated at the differential frequencies of the sidebands (f2-f1), at their summed frequencies (f2+f1). I also simulated their combination as in the Double Demodulation.
I repeated the simulation for:
- Old (current) 40m
- 40m Upgrade
- AdvLIGO
I'm attaching the results to this elog entry.
The plots show how the signal varies exploring the two-dimensional space of the demodulation frequencies (differential and sum).
Both the Upgrade and the Old40m's signals look anomalous since the zero-crossing point does not change with the demodulation phases.
I suspect there's is a problem with the optickle model of the 40m. |
| Attachment 1: 23_3f_40mOld_DDplots.pdf
|
|
| Attachment 2: 59_3f40m_Upgrade_DDplots.pdf
|
|
| Attachment 3: 20_3f_AdvL_DDplots.pdf
|
|
|
1355
|
Wed Mar 4 17:20:04 2009 |
josephb | Update | Cameras | Camera code upgrades |
I've updated the digital camera python code as well as changed the network topology.
At the moment, both cameras are connected to a small gigabit switch which only talks to Ottavia. This means all camera servers must be run on Ottavia, allow camera output is still UDP multicast so any machine capable of running gstreamer can pick up the images.
The server and client programs now have the ability to read a configuration file for the setup of the cameras. They default to pcameraSettings.ini, but this can default can be changed with a -c or --config option
For example, "serverV3.py --config pcam1.ini" will run the server using the pcam1.ini settings file. Similarly, "client.py --config pcam1.ini" will also take the IP settings from the config file so that it knows at which port and IP to listen.
These programs and .ini files have been placed in /cvs/cds/caltech/apps/linux64/python/pcamera/
I've updated the cshrc.40m aliases so that it uses the new configuration file options, so now pcam1 calls "client.py -c pcam1.ini" in the above directory.
So to start a client use pcam1 or pcam2 (for the 32223 camera in PSL looking at MC trans or 44026 looking at an analog moniter in the control room respectively). These can be run on Allegra, Rosalba or Ottavia at the moment.
To start a server, use pserv1 or pserv2. These *must* be run on Ottavia.
I've also added a -n or --no-gui option at Yoichi's request, one which just starts up and plays, with no graphical gui.
Lastly, I've made some changes to the base pcamerasrc.py file, which should make display more robust. After a failed transmission of an image from the camera to Ottavia, it should re-attempt up to 10 times before giving up. I'm hoping this will make it more robust against packet loss. The change in network topology has also helped this, allowing 640x480 to be transmitted on both cameras before tens of minutes before a packet loss causes a stop. |
|
1358
|
Thu Mar 5 00:06:32 2009 |
Kakeru, Rana | Update | IOO | WFS centering |
We found that the MC REFL image was no longer round and that the MCWFS DC quadrant spots were mostly
in one quadrant. So we re-centered the MCWFS beams in the following way:
1) We unlocked the MZ and adjusted the PZT voltage to keep the beam on the WFS from saturating.
2) Re-aligned the black hole beam dump to center its beam in its aperture.
3) centered the beam on the MCWFS optics and MCWFS QPD displays.
4) Relocked MC.
Below is the image of the IOO Strip tool. You can see that the MC REFL DC is now more flat. The
MC pointing has also been changed (see the MC TRANS HOR & VERT channels). The MC transmitted
light is also now more stable and higher.
We tried to center the QPD, and we found that there were a few hundred mV of dark offset for each
quadrant of QPD. We adjusted them with this scripts:
/cvs/cds/caltech/scripts/MC/WFS/McWFS_dc_offsets |
| Attachment 1: IOO_graph.jpg
|
|
|
1359
|
Thu Mar 5 01:09:29 2009 |
rana, alberto | Update | DMF | still not working |
We tried to run DMF on mafalda, but it didn't work. I tried to compile it using Rob's elog instructions.
On mafalda, I started matlab and ran the mdv_config.m to set up mDV. I tested that the seisBLRMS.m
script ran and correctly produced changes in the seisBLRMS strip tool. but when I tried to compile it I got:
>> mcc -v -m -R -nojvm seisBLRMS.m
Warning: Duplicate directory name: /cvs/cds/caltech/apps/linux/matlab/toolbox/local.
Compiler version: 4.6 (R2007a)
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/matlab/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/signal/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/control/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/filterdesign/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/shared/controllib/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/ident/mcc.enc
Warning: an error occurred while parsing class FilterDesignDialog.AbstractEditor:
Undefined function or variable 'DAStudio.Object'.
> In /cvs/cds/caltech/apps/linux/matlab/toolbox/shared/filterdesignlib/@FilterDesignDialog/@CoeffEditor/schema.p>schema at 9
Warning: an error occurred while parsing class FilterDesignDialog.CoeffEditor:
Invalid superclass handle.
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/fixedpoint/mcc.enc
terminate called after throwing an instance of 'ApplicationRedefinedException*'
Abort (core dumped)
"/cvs/cds/caltech/apps/linux/matlab/bin/mcc" -E "/tmp/fileRnU5Qj_31324": Aborted
??? Error executing mcc, return status = 134.
In the meantime, I've started up a green terminal on allegra which is ssh'd into megatron.
On megatron there is a regular matlab session which is running seisBLRMS.m as a matlab script
and the seis DMF channels are getting updated. |
|
1361
|
Thu Mar 5 05:07:09 2009 |
Yoichi | Update | Locking | Wednesday night locking |
Tonight, I was having a problem with the PO_DC hand-off.
It fails most of the time.
I increased the averaging time for the PD1_DC offset measurement.
I also wrote a script to match the gain of the transmission DC and the PO_DC signals.
This script (/cvs/cds/caltech/scripts/CM/matchPODCGain ) measures the gains of the old (TRX+TRY) and new (PO_DC) signals at 150Hz and returns the optimal value to be put into the input matrix.
cm_step script calls matchPODCGain to determine the matrix element value for the PO_DC signal.
Even with this script, the hand-off was still unreliable.
I checked the AO path loop gain just before the hand off. It looked normal.
Then I realized that the oscilloscope I hooked up to the PO_DC signal using a T-BNC may be introducing some noise into the channel.
So I removed it. Then the PO_DC hand off went well at least once.
The IFO still loses lock at around arm power 10.
I attached time series of the latest lock loss. The second attachment is a zoom of the first one.
This time, there is a glitch in the ETM feedback signals, which is also present in the DARM and CARM and error signals.
I saw this kind of glitches several times today. |
| Attachment 1: lockLoss5.png
|
|
| Attachment 2: lockLoss5-zoom.png
|
|
|
1364
|
Fri Mar 6 05:44:14 2009 |
Yoichi | Update | Locking | Locking distracted by the QPD whitenning problem again |
Tonight, I was able to ramp up the arm power to around 20. Then the DARM loop started to oscillate and the IFO lost lock in a few seconds.
I repeated this several times, then realized that the transmission QPDs were not working properly again due to the well known sticky slider problem.
I should have run slider_twiddle script. Since the DARM RF signal is normalized by the sqrt(TRX+TRY), it is reasonable that the DARM loop got unstable.
The fact that I was able to go up to arm power = 20 means there is nothing saturating below this power level.
|
|
1365
|
Fri Mar 6 15:23:39 2009 |
Yoichi | Update | Locking | Locking distracted by the QPD whitenning problem again |
By looking at the time series of DARM signal at the time of a lock loss, the oscillation frequency was about 3.5kHz (see the attm1 and its zoomed version attm2).
I will measure the DARM loop gain around this frequency next.
| Quote: | Tonight, I was able to ramp up the arm power to around 20. Then the DARM loop started to oscillate and the IFO lost lock in a few seconds.
I repeated this several times, then realized that the transmission QPDs were not working properly again due to the well known sticky slider problem.
I should have run slider_twiddle script. Since the DARM RF signal is normalized by the sqrt(TRX+TRY), it is reasonable that the DARM loop got unstable.
The fact that I was able to go up to arm power = 20 means there is nothing saturating below this power level.
|
|
| Attachment 1: lockLoss3.pdf
|
|
| Attachment 2: lockLoss3-zoom.pdf
|
|
|
1366
|
Fri Mar 6 18:14:58 2009 |
Yoichi | Update | Computers | awg not working |
Starting from this afternoon, the awg is not working.
I rebooted FE computers, c0daqawg as well as tpman and daqd processes on fb40m several times.
But the problem is still there.
I sent an email to Alex. |
|
1369
|
Sat Mar 7 16:50:25 2009 |
Yoichi | Update | Computers | Not even data retrieval working |
Now our digital system is really in trouble.
We can't even get data from tp channels.
I did another round of computer reboots, this time including the RFM bypass switch, c0daqctrl, c0dcu1 and fb40m itself.
But the problem still persists.
I guess there is nothing I can do until Alex comes in. |
|
1370
|
Sun Mar 8 23:09:26 2009 |
rana | Update | Computers | Not even data retrieval working |
Although getting the regular DAQ data works, we can't get any testpoints.
I tried restarting tpman several times; there's no inittab on fb40m for this so we should get Alex to set one up when he comes.
I also tried various power cycles and reboots: daqawg, daqctrl, etc. I also notice that Osamu's setup of new stuff is connected to
the same rack and power strips as all of our sensitive DAQ machines. We should find out if there was any hardware installed in the
last couple days; it would be easy to accidentally unplug or damage on of our fibers.
I moved the old tpman.log over to tpman.log.090308. It starts out with a header and then just lists when each TP is requested.
When restarting tpman it puts the following into the terminal:fb:controls>./tpman &
[1] 1037
fb:controls>VMIC RFM 5565 (0) found, mapped at 0x2868c90
VMIC RFM 5579 (1) found, mapped at 0x2868c90
Could not open 5565 reflective memory in /dev/daqd-rfm1
16 kHz system
Spawn testpoint manager
Channel list length for node 0 is 4168
Test point manager (31001001 / 1): node 0 which is OK?; its the same startup outputs that are in the old log file. It would be nice if there was not and error message about the RFM.
Requesting new testpoints via tdsdata, dtt, or the diag command line doesn't seem to work. tpman doesn't spit anything out although 'tp show 0'
does show that the TP is selected.
Once Alex fixes the 'tpman' issue, we should make sure to put an inittab or startup script in there so that tpman writes a log
file and also archives its old log files upon a restart. |
|
1371
|
Sun Mar 8 23:14:52 2009 |
rana | Update | Computer Scripts / Programs | tdsdata doesn't work |
Matt logged in and rebuilt the TDS stuff for us on Mafalda in /cvs/cds/caltech/apps/linux/tds_090304.
He says that he can't build his stuff on 64-bit because there's not a sanctioned 64-bit build of GDS yet.
This should have all the latest fixes in it. I tried using both the old and new code from allegra and they both are fine:
./tdsdata 16384 2 C1:IOO-MC_F > /users/rana/test.txt
I loaded the data I got with the above command and there were no data dropouts. Possibly the dropout problem is only
associated with testpoints and so we have to wait for the TP fix. |
|
1373
|
Mon Mar 9 11:09:33 2009 |
Alberto | Update | Computers | Re: Not even data retrieval working |
| Quote: | Although getting the regular DAQ data works, we can't get any testpoints.
I tried restarting tpman several times; there's no inittab on fb40m for this so we should get Alex to set one up when he comes.
I also tried various power cycles and reboots: daqawg, daqctrl, etc. I also notice that Osamu's setup of new stuff is connected to
the same rack and power strips as all of our sensitive DAQ machines. We should find out if there was any hardware installed in the
last couple days; it would be easy to accidentally unplug or damage on of our fibers.
I moved the old tpman.log over to tpman.log.090308. It starts out with a header and then just lists when each TP is requested.
When restarting tpman it puts the following into the terminal:fb:controls>./tpman &
[1] 1037
fb:controls>VMIC RFM 5565 (0) found, mapped at 0x2868c90
VMIC RFM 5579 (1) found, mapped at 0x2868c90
Could not open 5565 reflective memory in /dev/daqd-rfm1
16 kHz system
Spawn testpoint manager
Channel list length for node 0 is 4168
Test point manager (31001001 / 1): node 0 which is OK?; its the same startup outputs that are in the old log file. It would be nice if there was not and error message about the RFM.
Requesting new testpoints via tdsdata, dtt, or the diag command line doesn't seem to work. tpman doesn't spit anything out although 'tp show 0'
does show that the TP is selected.
Once Alex fixes the 'tpman' issue, we should make sure to put an inittab or startup script in there so that tpman writes a log
file and also archives its old log files upon a restart. |
Alex fixed the problem. It was caused by the awgtpman running on kami1.martian which conflicted with the tpman in fb0.
Killing awgtpman on kami1 allowed for the tpman on tp0 to work properly again.
If more test points are needed, Alex suggested to tune the GDS settings accordingly.
What this actually means, I still have to understand it. |
|
1374
|
Mon Mar 9 12:04:18 2009 |
Yoichi | Update | Computers | TPs and AWG are back |
I had to do one more reboot of tpman and daqd to get the TPs working.
I confirmed the alignment scripts run fine.
Now the oplevs of some optics are largely mis-centered. Alberto and I will center them after lunch. |
|
1375
|
Mon Mar 9 14:57:30 2009 |
Kakeru | Update | Computer Scripts / Programs | tdsdata doesn't work |
I tested new tdsdata and found it was working well.
I excited C1:SUS-ITMY_SUSPIT_EXC with tdssine, and get data from C1:LSC-TRY_OUT (testpoint) and C1:SUS- ITMY_OPLEV_PERROR (recorded point) with new and old tdsdata.
With old tdsdata (/cvs/cds/caltech/apps/linux/tds/bin/tdsdata), I found some jumps of datapoint, which is a same problem with before (Attachment 1).
With new tdsdata (/cvs/cds/caltech/apps/linux/tds_090304/bin/tdsdata), there looks to be no jumps (Attachment 2; taken about 10 minutes after Attachment 1).
The problem of old tdsdata looks to be remaining even for recordedpoints.
You should use /cvs/cds/caltech/apps/linux/tds_090304/bin/tdsdata.
|
Quote:
|
|
Matt logged in and rebuilt the TDS stuff for us on Mafalda in /cvs/cds/caltech/apps/linux/tds_090304.
He says that he can't build his stuff on 64-bit because there's not a sanctioned 64-bit build of GDS yet.
This should have all the latest fixes in it. I tried using both the old and new code from allegra and they both are fine:
./tdsdata 16384 2 C1:IOO-MC_F > /users/rana/test.txt
I loaded the data I got with the above command and there were no data dropouts. Possibly the dropout problem is only
associated with testpoints and so we have to wait for the TP fix.
|
|
| Attachment 1: oldtds.png
|
|
| Attachment 2: newtds.png
|
|
|
1376
|
Mon Mar 9 16:54:52 2009 |
Kakeru, Rana | Update | Computer Scripts / Programs | tdsdata doesn't work |
We confirmed that new tds(/cvs/cds/caltech/apps/linux/tds_090304/) works well on linux 64, and replaced it to /cvs/cds/caltech/apps/linux/tds/
The old /cvs/cds/caltech/apps/linux/tds is put in /cvs/cds/caltech/apps/linux/tds.bak |
|
1379
|
Mon Mar 9 19:33:10 2009 |
rana | Update | DMF | seisBLRMS in temp condition |
The seisBLRMS has been running on megatron via an open terminal ssh'd into there from allegra with matlab running. This
is because I couldn't get the compiled matlab functionality to work.
Even so, this running script has been dying lately because of some bogus 'NDS' error. So for today I
have set the NDS server for mDV on megatron to be fb40m:8088 instead of nodus.ligo.caltech.edu. If this seems to fix the problem
I will make this permanent by putting in a case statement to check whether or not the mDV'ing machine is a 40m-martian or not. |
| Attachment 1: Untitled.png
|
|
|
1380
|
Mon Mar 9 23:13:22 2009 |
Yoichi | Update | Computer Scripts / Programs | tdsdata doesn't work |
| Quote: |
|
We confirmed that new tds(/cvs/cds/caltech/apps/linux/tds_090304/) works well on linux 64, and replaced it to /cvs/cds/caltech/apps/linux/tds/
The old /cvs/cds/caltech/apps/linux/tds is put in /cvs/cds/caltech/apps/linux/tds.bak
|
The tdscntr.pl in the new tds was probably the one from LLO, which is actually the version I sent to Tobin. It had paths and channel names defined for the LLO. So I copied back my original 40m version. |
|
1382
|
Tue Mar 10 04:55:41 2009 |
Yoichi | Update | Locking | Locking: 3.7kHz large oscillation |
Yoichi, Jenne, Alberto,
As I reported on the last Thursday, there is a large oscillation in CARM and DARM error signals (attm1).
I put notch filters (3.75kHz, Q=10, 30dB) in the CARM and DARM loops. This let us go up to the arm power of more than 20 and stay there for a while.
The dashed curves in the attm1 are the spectra when the notches are off, and the solid curves are when the notches are used.
We could somewhat suppress the DARM peaks but not CARM.
Of course this is clearly not a good solution. We should find the cause of the oscillation and kill it.
Attm2 is the spectrum of the PO_DC signal flowing in the CM board measured by the SR785. More specifically, CH1 is TP1A and CH2 is TP2A of the CM board.
This was taken right after the AO path was engaged. At this stage, the AO path gain is very low. But you can already see a seed of the oscillation in the spectrum.
Attm3 shows the same spectra taken after the arm power is increased to 4 but before the PO_DC hand off. You can see large peaks around 3.75kHz.
After this, the peaks grow as the power goes up.
Attm4 is the loop gain of the AO path after the PO_DC hand off (arm power = 4).
Attm5 is the zoom of the same TF around 3.7kHz. Clearly there is something wrong at this frequency. We should check the CM board and the MC board as well as the SPOB PD.
One time I was able to go up to arm power = 27 or so. At this power level, the DARM loop started to oscillate, probably, around the UGF.
However because of the 3.7kHz problem, we can't stay at this power level long enough to make diagnostic measurements (like open loop TF).
We should tackle the 3.7kHz issue first. |
| Attachment 1: CARM_DARM_Spectra.pdf
|
|
| Attachment 2: PODC_Spe_AOPath_Engaged.png
|
|
| Attachment 3: PODC_Spe_before_PODC_handoff.png
|
|
| Attachment 4: AOGain3.png
|
|
| Attachment 5: AOGain2.png
|
|
|
1386
|
Wed Mar 11 14:51:01 2009 |
Kakeru, Joe, Rob | Update | IOO | MC alignment |
This morning, MC alignment was gone and MC wasn't lock.
We checked old value of pitch, yaw, and position offset of each MC mirror, and found they were jumped.
We don't know the reason of this jump, but we restore each offset value and MC backed to lock. |
|
1387
|
Wed Mar 11 16:41:22 2009 |
steve | Update | MOPA | spare NPRO power |
The spare M126N-1064-700, sn 5519 of Dec 2006 rebuilt NPRO's power output
measured 750mW at DC2.06A with Ohpir meter.
Alberto's controller unit 125/126-OPN-PS, sn516m was disconnected from lenght measurment NPRO on the AP table.
5519 NPRO was clamp to the optical table without heatsink and it was on for 15 minutes. |
|
1388
|
Wed Mar 11 16:53:48 2009 |
Yoichi | Update | Locking | Junks in around kHz |
Rana, Yoichi
Last night, we tried to find out the source of the kHz region peaks in the DARM and CARM error signals.
These peaks are also present in the error signal of the single arm locking by RF (both X and Y).
The attachment 1 shows spectra of MC_F and XARM error signal when XARM is locked by the POX PDH signal.
There is a sharp peak at 3.8kHz in MC_F. This peak was there in a reference spectrum taken on June 24 2008.
In the XARM error signal, there is also a broad peak around 3.8kHz. This peak moves between 3.75kHz and 3.8kHz from time to time.
(the brown curve was taken when the peak moved to 3.75kHz).
Also there is a notch like structure at 3.8kHz in the XARM error spectrum. Looks like the peak in the MC_F is creating a notch here, but
no idea why.
We tapped on the PSL table, the end chambers and the SPOB table and looked at the spectra to see if there is any change.
Rana also developed a cool Walkie-Talkie excitation technique, where he put one of the walkie-talkies on the PSL table by the MZ and yelled at the other one while looking at a DTT screen in the control room.
None of these had any effect on the XARM error, while MC_F responded to the disturbances.
We also turned on and off the steering mirror PZT closed loop buttons, moved the PMC, MZ and the ISS gain sliders and changed the MC gain, offset.
Nothing affected the XARM error.
Osamu found old spectra of the XARM signal (attm2). The legends say DARM but these are XARM signals.
Almost the same structures can be seen including the notch at 3.8kHz. Seems like it's been like this for long time.
We should check, RF-AM, MC coil dirivers, Piezo-Jena noise etc. |
| Attachment 1: MC_F-XARM.pdf
|
|
| Attachment 2: old-xarm.pdf
|
|
|
1389
|
Wed Mar 11 21:03:51 2009 |
Kakeru and Kiwamu | Update | IOO | PSL angle QPD |
Kakeru and Kiwamu
We placed a QPD on the PSL bench for PSL angle monitor.
| Quote: |
|
I checked a broken QPD, which was placed for PSL angle monitor, and finally I cocluded one segment of the quadrant diode was broken.
The broken segment has a offset voltage of -0.7V after 1st I-V amplifier. It means the diode segment has a current offset without any injection of light.
Tomorrow I will check a new QPD for replacement.
Kiwamu IZUMI
|
As we mentioned before, old QPD which used to be placed is broken.
And we put broken QPD into the "photodiodes" box under the soldering table.
|
|
1390
|
Wed Mar 11 22:57:48 2009 |
Yoichi | Update | Locking | Calibrated XARM error signal spectrum |
I did a rough calibration of the XARM error spectrum.
See the attached calibrated spectrum.
I started from this Rana's elog entry.
http://www.ldas-sw.ligo.caltech.edu/ilog/pub/ilog.cgi?group=40m&task=view&date_to_view=04/07/2005&anchor_to_scroll_to=2005:04:07:20:28:36-rana
I first injected a 20Hz sin signal into C1:SUS-ETMX_LSC_EXC and measured the response to the ETMX SUSPOS.
Using the calibration of the SUSPOS given in the above entry, I calibrated the ETMX coil actuation efficiency.
It was 3.4e-12 m/cnt @20Hz for C1:SUS-ETMX_LSC_EXC.
Then I locked the X-arm and injected a calibration peak at 20Hz.
From the ratio of the peaks in C1:SUS-ETMX_LSC_IN2 and C1:LSC-XARM_IN1, I calibrated the X-arm error signal to be 4.2e-13 m/cnt.
We have to also take into account the cavity pole of the arm, 1525Hz (the design value, may not be actual).
So I used the following calibration in the DTT:
G: 4.2e-13
P: 1525
Z:
Note that the attached spectrum shows the actual motion of the X-arm (or equivalent frequency noise) after suppressed by the feedback servo,
unlike conventional noise spectra showing "virtual" displacement which would have been induced in the absence of servos. |
| Attachment 1: XarmErrorSpeCalibrated.pdf
|
|
|
1391
|
Wed Mar 11 23:41:33 2009 |
Kakeru, Yoichi | Update | IOO | WFS centering |
We found the MC reflection was distorted . And WFC beam went to upward of QPD
We recentered WFC beam and these problems were fixed |
|
1393
|
Thu Mar 12 02:18:42 2009 |
Yoichi | Update | Locking | MC_I spectra (RF_AM) |
I took several spectra of MC_I signal (see attm1).
The blue curve is when the MC was locked. The green curve (RF_AM) shows the MC_I spectrum when the MC is unlocked and MC2 is mis-aligned,
so that no resonance should happen. The brown curve is when the PSL shutter was closed (dark noise).
There are some structures in the green curve but not at 3.8kHz.
The second attachment compares the MC_I spectrum (the same as the green one in the first attachment) with the Xarm error signal.
Of course these two spectra were taken at different times.
Some of the peaks in the X-arm error signal seem to be coming from the MC RF_AM. |
| Attachment 1: MC_I_Spe.png
|
|
| Attachment 2: MC_I-Xarm.png
|
|
|
1394
|
Thu Mar 12 15:57:53 2009 |
Yoichi | Update | IOO | MC drift is terrible |
Yoichi, Osamu,
Last night's locking work was totally interrupted by the sabotage by the MC.
First, after I measured the RF_AM, the MC alignment was somehow shifted largely and the MC did not lock to TEM00 mode.
I only mis-aligned MC2 to measure the RF_AM, but the MC reflection beam was also shifted (looking at the WFS QPD), that means MC1 was mis-aligned somehow.
Moreover, even when the MC is not locked, i.e. no feedback to the mirrors, the OSEM values of the MC mirrors (all of them) drift a lot in 10min scale.
I was totally puzzled. So I rebooted c1iovme and c1sosvme. Then this strange drift of the OSEM values stopped.
Even though, the MC tended to lose lock within ten minutes because the WFS QPDs were not centered.
We did several iterations of re-centering and finally the MC started to stay locked happily. The MC reflection beam was symmetric.
Then this morning when I came in (to be honest, afternoon), the MC reflection looked asymmetric again. The WFS QPDs were mis-centered again.
The attached files show an 8-hour trend of various MC related signals.
There was a half-degree temperature change starting from around 11AM. Corresponding to that, the IOO-QPD signals drifted indicating that the PSL beam pointing
was shifted. The MZ PZT signal shows a similar trend, so the beam pointing may have been shifted by the MZ (not sure).
The MC WFS, transmission QPD signals show the same trend.
This is too bad.
Right now, the PSL beam pointing is monitored by the QPDs detecting the transmitted beam through the first mirror of the periscope.
This means even if we can track the beam pointing drift with the QPDs, we can't correct the beam pointing using the periscope mirrors.
I don't want to touch the MZ mirrors for this purpose.
I propose to put a pick-off mirror after the second mirror of the periscope to send light to the IOO-QPDs. This way, we can use the periscope
mirrors to restore the beam pointing screwed up by the MZ.
|
| Attachment 1: MC_Drift-1.pdf
|
|
| Attachment 2: MC_Drift-2.pdf
|
|
|
1395
|
Thu Mar 12 18:44:02 2009 |
Yoichi | Update | PSL | MZ aligned |
The MC lost the alignment somehow this afternoon.
So I thought it was good time to touch the MZ because I had to align the MC using the periscope anyway.
I mainly touched the mirror with a PZT. The MZ reflection went down from 0.5 to 0.3. |
|
1396
|
Thu Mar 12 18:48:37 2009 |
Yoichi | Update | IOO | MC aligned but ... |
After the MZ alignment, I aligned the MC with the periscope mirrors.
It looked like the MC mis-alignment was mainly caused by the input beam change.
So I left the MC mirrors as they were to keep the output beam pointing.
However, after I finished the alignment, the MC output beam was too low on the Faraday.
Also the X-arm did not lock to TEM00 mode. So the MC mirrors must have also shifted to a weird alignment state.
I should have restored the MC mirror alignment to a good state using the OSEM DC signals.
Rana came in and restored the MC mirror alignment using the SUS drift mon.
He and Kakeru is now working on the periscope to align the beam into the MC. |
|
1397
|
Thu Mar 12 19:11:27 2009 |
rana | Update | IOO | MC drift is terrible |
Kakeru, Rana, Yoichi
We used the SUS DRIFT MON screen to set the MC biases such that the mirrors were returned to the old OSEM values.
To do this, we set the nominals and tolerances using the appropriate scripts in the mDV/extra/C1/ directory.
We then used the MC_ALIGN screen to set the angle bias sliders.
Then Kakeru and I went to the PSL table to the periscope magic and maximize the MC transmission. Kakeru seems to
have the careful Japanese alignment touch and I am hungry, so I am leaving him to optimize the power. After he
finishes he is going to align the beam to the WFS and turn the MC autolocker back on. The x-arm is locked on a
TEM00 mode so the MC alignment is maybe OK. |
|
1398
|
Thu Mar 12 20:59:04 2009 |
Kakeru | Update | IOO | MC drift is terrible |
After Rana went for his dinner, I aligned periscope to make the MC output 3.2 (Attachment 1).
After that, to align WFS, I unlocked the MC, unlocked the MZ and decrease the beam power to WFS QPD, and re-centerd WFC beam.
I restored MZ and MC lock.
I enabled MC autolocker, and change C1:IOO-WFS_Gain_Slider from 0 to 0.02 to lock WFS.
| Quote: | Kakeru, Rana, Yoichi
We used the SUS DRIFT MON screen to set the MC biases such that the mirrors were returned to the old OSEM values.
To do this, we set the nominals and tolerances using the appropriate scripts in the mDV/extra/C1/ directory.
We then used the MC_ALIGN screen to set the angle bias sliders.
Then Kakeru and I went to the PSL table to the periscope magic and maximize the MC transmission. Kakeru seems to
have the careful Japanese alignment touch and I am hungry, so I am leaving him to optimize the power. After he
finishes he is going to align the beam to the WFS and turn the MC autolocker back on. The x-arm is locked on a
TEM00 mode so the MC alignment is maybe OK. |
|
| Attachment 1: MCtrans090312.png
|
|
|
1399
|
Fri Mar 13 05:16:21 2009 |
Yoichi | Update | Locking | Locking update |
Yoichi, Osamu,
With adjustments of the loop gains during the CARM offset reduction, the IFO reaches arm_power = 25 sort of robustly unless the 3.8kHz oscillation rings up.
At arm_power = 25, the CARM and DARM start to oscillate at around 400Hz. Probably I need more gain tweaks.
Annoying thing is that the 3.8kHz oscillation sometimes rings up suddenly and kills the lock.
This can happen anywhere above arm_power = 6 or so.
Because of a strange structure in the CARM loop gain around 3.8kHz, we cannot increase the CARM UGF beyond 1kHz.
The attached plots are the AO path open loop transfer function (attm2 is the zoom of attm1) measured at arm_power = 13.
Tomorrow, I will lock the X-arm and measure the transfer function from the AO path input to the X-arm error signal to see
if there is the same structure at 3.8kHz (X-arm error signal has the 3.8kHz peak). |
| Attachment 1: AOTF2.png
|
|
| Attachment 2: AOTF2-zoom.png
|
|
|
1400
|
Fri Mar 13 19:26:09 2009 |
Yoichi | Update | DMF | seisBLRMS compiled |
I compiled seisBLRMS.
The tricks were the following:
(1) Don't add path in a deployed command.
It does not make sense to add paths in a compiled command because it may be moved to anywhere. Moreover, it can cause some weird side effects. Therefore, I enclosed the addpath part of mdv_config.m in a "if ~isdeployed ... end" clause to avoid adding paths when deployed. Instead of adding paths in the code, we have to add paths to necessary files with -I options at the compilation time. This way, mcc will add all the necessary files into the CTF archive.
(2) Add mex files to the CTF archive by -a options.
For some reason, mcc does not add necessary mex files into the CTF archive even though those files are called in the m-file which is being compiled. We have to add those files by -a options.
(3) NDS_GetData() is slow for nodus when compiled.
NDS_GetData(), which is called by get_data() stops for a few minutes when using nodus as an NDS server.
This problem does not happen when not compiled. I don't know the reason. To avoid this, I modified seisBLRMS.m so that when an environmental variable $NDS is defined, it will use an NDS server defined in this variable.
I wrote a Makefile to compile seisBLRMS. You can read the file to see the details of the tricks.
I also wrote a script start_seisBLRMS, which can be found in /cvs/cds/caltech/apps/DMF/compiled_matlab/seisblrms/. To start seisBLRMS, you can just call this script.
At this moment, seisBLRMS is running on megatron. Let's see if it continues to run without crashing.
|
Quote:
|
|
The seisBLRMS has been running on megatron via an open terminal ssh'd into there from allegra with matlab running. This
is because I couldn't get the compiled matlab functionality to work.
Even so, this running script has been dying lately because of some bogus 'NDS' error. So for today I
have set the NDS server for mDV on megatron to be fb40m:8088 instead of nodus.ligo.caltech.edu. If this seems to fix the problem
I will make this permanent by putting in a case statement to check whether or not the mDV'ing machine is a 40m-martian or not.
|
|
|
1401
|
Fri Mar 13 20:23:37 2009 |
Yoichi | Update | LSC | AO path transfer function with X-arm locked |
I measured the AO path transfer function while the X-arm is locked with the POX PDH signal.
The POX-I signal was already connected to the input 1 of the CM board. So I injected a signal from the EXC-B channel of the board and measured the transfer function from TP2B to TP1A. To open the loop, I disabled the switch befor the EXC-B.
The attached plot shows the measured transfer function.
There is a bump around 2kHz, which can also be seen in the AO path TF posted in elog:1399, but not the large structure at around 3.8kHz.
The 3.8kHz structure is probably created by the feedback. |
| Attachment 1: AOPath-Xarm.png
|
|
|
1402
|
Fri Mar 13 22:07:14 2009 |
Yoichi | Update | Locking | Calibrated XARM error signal spectrum |
Of course I made a mistake.
I put a pole at 1525Hz whereas it should have been a zero.
The correct calibration factor is:
G: 4.2e-13
P:
Z: 1525
I attached a revised spectrum.
| Quote: | I did a rough calibration of the XARM error spectrum.
See the attached calibrated spectrum.
I started from this Rana's elog entry.
http://www.ldas-sw.ligo.caltech.edu/ilog/pub/ilog.cgi?group=40m&task=view&date_to_view=04/07/2005&anchor_to_scroll_to=2005:04:07:20:28:36-rana
I first injected a 20Hz sin signal into C1:SUS-ETMX_LSC_EXC and measured the response to the ETMX SUSPOS.
Using the calibration of the SUSPOS given in the above entry, I calibrated the ETMX coil actuation efficiency.
It was 3.4e-12 m/cnt @20Hz for C1:SUS-ETMX_LSC_EXC.
Then I locked the X-arm and injected a calibration peak at 20Hz.
From the ratio of the peaks in C1:SUS-ETMX_LSC_IN2 and C1:LSC-XARM_IN1, I calibrated the X-arm error signal to be 4.2e-13 m/cnt.
We have to also take into account the cavity pole of the arm, 1525Hz (the design value, may not be actual).
So I used the following calibration in the DTT:
G: 4.2e-13
P: 1525
Z:
Note that the attached spectrum shows the actual motion of the X-arm (or equivalent frequency noise) after suppressed by the feedback servo,
unlike conventional noise spectra showing "virtual" displacement which would have been induced in the absence of servos. |
|
| Attachment 1: XarmErrorSpeCalibrated.pdf
|
|
|
1403
|
Sat Mar 14 22:53:12 2009 |
Kakeru | Update | oplevs | arm cavity oplev calibration |
I finished a calibration of optical levers.
To calibrate oplevs, I locked appropriate cavity and tilted a mirror.
A cavity with tilted mirror decrease its arm power. So I can know how much the tilt is.
For calibration of ITMX and ETMX, I locked X arm and measured TRX.
For ETMX, ETMY and BS, I locked Y arm and measured TRY
For PRM, I locked PRC and measured SPOB
For SRM, I locked SRC and measured REFL166
I used, for example, C1:SUS-ITMX_OPLEV_PERROR as an oplev signal.
The calibration factors for each mirror is below. The attachment is figures of my fitting.
I used modified equation for ITM calibration from my last calibration, so the value become small around 30%.
ITMX Pitch: 142 microrad/counts
ITMX Yaw: 145 microrad/counts
ITMY Pitch: 257 microrad/counts
ITMY Yaw: 206 microrad/counts
ETMX Pitch: 318 microrad/counts
ETMX Yaw: 291 microrad/counts
ETMY Pitch: 309 microrad/counts
ETMY Yaw: 299 microrad/counts
BS Pitch: 70.9 microrad/counts
BS Yaw: 96.3 microrad/counts
PRM Pitch: 78.5 microrad/counts
PRM Yaw: 79.9 microrad/counts
SRM Pitch: 191 microrad/counts
SRM Yaw: 146 microrad/counts
It looks strange that ITMY, BS and SRM has different value. I think this is a fitting problem.
These data have some asymmetry and cause these 20%-30% difference.
Actually, PRM Yaw has a little asymmetry but the value doesn't differ from Pitch.
This means that this calibration factor potentially has below 30% error.
(These data are the most fine data. I think we must adjust Y arm yaw alignment. The beam spot of ETMY looks too low!)
For SRM, I couldn't get fine data because it was very sensitive to tilt and easily lose its lock.
When I tuned cavity enough, The data become almost flat, so I used detuned cavity.
It is also strange that ITMX and ITMY is different. I guess that this is caused by the difference of the QPD input. The sum of QPD is around 10000 for ITMX and around 4500 for ITMY.
The difference between BS or PRM and SRM is same, I guess. The sum of QPD input for BS and SRM is around 1500, but for SRM, it is around 10000.
I will write more detailed document and upload it with my calibration code. |
| Attachment 1: oplev.pdf
|
|
|
1404
|
Sun Mar 15 21:50:29 2009 |
Kakeru, Kiwamu, Osamu | Update | Computers | Some computers are rebooted |
We found c1lsc, c1iscex, c1iscey, c1susvme, c1asc and c1sosvme are dead.
We turned off all watchdogs and turned off all lock of suspensions.
Then, I tried to reboot these machines from terminal, but I couldn't login to all of these machines.
So, we turned off and on key switches of these machines physically, and login to them to run startup scripts.
Then we turned on all watchdogs and restored all IFO.
Now they look like they are working fine.
|