ID |
Date |
Author |
Type |
Category |
Subject |
7350
|
Thu Sep 6 16:46:44 2012 |
Jenne | Update | SUS | BS aligned, target removed |
Q and I aligned the BS such that we were hitting the center of ETMX. The ETMX cage does not have OSEM setscrew holes on the front, so it is not possible to put the targets that Steve made on this optic. So, I put the freestanding ruler in front of the optic, with the edge of the ruler at the center (as viewed from above) of the optic. Then Eric steered the BS until we were hitting the 5.5" mark, and roughly half of the beam was obscured by the ruler.
We then aligned ITMX such that the prompt reflection was colinear with the incoming beam.
I checked the 2 spots through the BS, heading to the AS port. (2 spots since MICH hasn't been locked / finely aligned yet). They were being clipped on the 2nd output PZT. I adjusted the knobs of the first output PZT to center the spots on the 2nd PZT. Note that the output PZTs' power is still off, and has been off for some unknown length of time. I had found them off when prepping for the vent a week or two ago. So the current alignment depends on them staying off. We don't really need them on until we're ready to employ our OMC.
The beams now look nicely unclipped on the AS camera, and we're aligning MICH. |
806
|
Wed Aug 6 22:19:07 2008 |
Yoichi | Update | SUS | BS alignment |
Koji, Yoichi
We realized that we did not pay attention to the BS alignment while working on the alignment of the ITMX today. Because we were injecting the ALM laser (absolute length measurement laser) from the AS port, the ITMX alignment depends on the BS alignment.
The BS optical lever was not centered and the sum was about 2000cnt, which is low compared, for example, to the SRM oplev.
So we were not sure if the BS was in a good alignment or not.
So we decided to move the BS to center the QPD.
In doing so, we also moved the ITMX so that we do not lose the ALM laser beam coming back to the AS port.
When the BS oplev was centered, the sum of the QPD was still about 2000. So it was not far off centered.
After the tweaking, we were able to see some interference between the light reflected by the ITMY and ITMX at the AS port (actually this is the bright port for the ALM laser). By tweaking the ITMY, we were able to see Michelson fringes at the AS port.
If we believe the ALM laser alignment is still good after the vent, the ITMX, ITMY, BS and SRM should be now in a good alignment condition.
The OSEM values for the ITMX, BS, SRM seem to be ok (0.9+/-0.2). The ITMY LL is a bit low (~ 0.45). |
7392
|
Fri Sep 14 21:03:02 2012 |
Den | Update | PEM | BS and AS tables |
I've measured seismic and acoustic noise on BS and AS tables. It seems that horizontal motion of BS table is ~1.5-2 times more then AS table in the frequency range 5-50 Hz.
Edit by Den: this was POI table, not BS!


|
7395
|
Sat Sep 15 20:27:54 2012 |
Den | Update | PEM | BS and AS tables |
Quote: |
I've measured seismic and acoustic noise on BS and AS tables. It seems that horizontal motion of BS table is ~1.5-2 times more then AS table in the frequency range 5-50 Hz.
Edit by Den: this was POI table, not BS!
|
This seismic measurement is for BS and AS tables.

|
7396
|
Mon Sep 17 07:57:58 2012 |
rana | Update | PEM | BS and AS tables for REFL change |
This was in response to my suggestion to move the REFL beam path to the table containing the BS/PRM Oplevs. From this seismic data it is clear that the BS table is no worse than the AS table, so we should plan to make the layout change during the next vent. |
6903
|
Mon Jul 2 18:27:25 2012 |
yuta | Update | General | BS and ITMX chambers closed |
[Koji, Steve, Jamie, Jenne, Yuta]
We opened BS and ITMX chambers, took lots of photos, and closed them with heavy doors.
I turned off high voltage power supplies for PZTs and blocked PSL beam. We are ready for the pumping tomorrow.
Important photos we took:
- positions of green optics at BS chamber, which was moved on the vent on Aug 2011
- positions of PZT mirrors and cable connectors at BS chamber, which will be replaced with tip-tilts on the next vent
- arrow on PR2 pointing HR (it was correct)
- tried to take photos of clipping IR beam at BS OSEM holder from ITMX chamber
We also took bunch of other photos.
Beam dump needed at BS chamber:
We also checked some un-dumped beams at BS chamber. We need dumps;
- behind MMT1, for unwanted transmitted beam
- behind IPPOSSM3, for unwanted transmitted beam (IPPOSSM3 is the last mirror in BS chamber for IPPOS) |
7329
|
Fri Aug 31 17:26:54 2012 |
Steve | Update | General | BS camera |
Steve and Eric
Placed pick up mirror for BS face and PRM back. I will ask Jamie to clamp it.
There will be an other camera set up to view the face of PRM |
7772
|
Sat Dec 1 00:24:37 2012 |
Den, Ayaka | Update | Alignment | BS chamber |
Today at 11:13 AM the stack of invacuum BS table was kicked and IFO misaligned. We adjusted PZT2 voltage by ~20 V in yaw such that IPPOS was restored. Then we could lock arms.

|
7773
|
Sat Dec 1 13:56:38 2012 |
rana | Update | Alignment | BS chamber |
Whoever was working around the BS chamber at 11 AM on Friday should admit it now and take the punishment.
For those of you who like to do work on the interferometer without reporting it in the elog because you think that what you did doesn't affect anything, this is your example of how our time can be wasted by such laziness. |
7784
|
Tue Dec 4 18:53:50 2012 |
Ayaka | Update | Alignment | BS chamber |
BS chamber seemed to be kicked again around 10:00 am today.
I moved PZT mainly in YAW and locked both arms. I adjusted the beam to be almost on the center of both ETM by sights.

|
3105
|
Wed Jun 23 12:52:35 2010 |
kiwamu | Update | Photos | BS chamber before cleaning up |
|
2800
|
Tue Apr 13 20:02:02 2010 |
Koji | Update | SUS | BS chamber opened, PRM/SRM SOS removed from the table |
Bob, Steve, and Koji
We opened North heavy door of the BS chamber in the afternoon.
In the evening, Koji worked on the PRM/SRM removal.
- Cleaned up the OPLEV mirrors to create some spaces near the door.
- Clamped PRM/SRM.
- Removed OSEMs. Made a record of the OSEMs. The record is on the wiki (http://lhocds.ligo-wa.caltech.edu:8000/40m/Upgrade_09/Suspensions)
- Found the SOSs are quite easy to remove from the table as they are shorter than the MOSs.
- Put a new Al sheet on a wagon. Put the SOSs on it. Wrapped them by the Al foils.
- Carried it to the clean room. They are on the right flow bench. Confirmed the wires are still fine.
- Closed up the chamber putting a light door. |
3332
|
Fri Jul 30 12:46:38 2010 |
Koji | Configuration | 40m Upgrading | BS chamber status |
|
16574
|
Tue Jan 11 14:21:53 2022 |
Paco | Update | Electronics | BS feedthroughs and in-vac cables installed |
[Paco, Yehonathan, Chub]
The BS chamber 10" flange with 4 DSUB-25 feedthroughs has been installed with the cables connected at the in-vac side. This is the second of two flanges, and includes 4 cables ordered vertically in stacks of 2 & 2 for [[LO2-1, LO2-2, PR3-1, PR3-2]] respectively. |
8257
|
Fri Mar 8 12:57:57 2013 |
Annalisa | Update | ABSL | BS installed on ITMY table |
Sendhil and I installed the S polarized BS on the ITMY table to steer the NPRO beam through the AR wedge and align it to the POY beam.
We took a shutter from the BSPRM table (which was not used) and a beam dump from the AS table (which was used by the auxiliary laser already removed and installed on the ITMY).
To do: do better alignment of the NPRO beam, maybe installing some iris after the BS and before the AS wedge, phase lock the two beams. |
8258
|
Fri Mar 8 13:42:35 2013 |
Jenne | Update | ABSL | BS installed on ITMY table |
Re: POY beam reduction.
We are able to lock the Yarm with the beam / gain as it is. I had thought we might need to increase the DC gain in the whitening board by a factor of 2, but so far it's fine. |
10852
|
Mon Jan 5 12:42:09 2015 |
ericq | Update | SUS | BS misbehaving |
The BS was showing some excess motion. I think I've fixed it. Order of operations:
- The DC PIT bias from previous ASS runs was at ~500, I zeroed this and aligned the BS to be centered on its oplev QPD with DC alignment sliders
- I squished the gold box cables. This changed the alignment slightly, and brought the UR voltage back to a normal value. Excess motion still existed
- I found that the the
C1:SUS-BS_LRSEN filter had HOLD OUTPUT enabled. I turned it off. All seems well.
I'm not sure how this might have gotten switched on... |
7714
|
Thu Nov 15 02:18:24 2012 |
Den | Update | Modern Control | BS oplev |
I've applied LQR feedback technique to BS oplev in pitch. I think the most inconvenient thing in using LQR controller is the amount of additional states created during cost function shaping. It requires 1 filter bank for each state. To avoid this I wrote state estimation code so all states are calculated inside one function.
On the plots below cost function and oplev feedback controller performance are shown.

|
11199
|
Fri Apr 3 14:57:38 2015 |
manasa | Update | SUS | BS oplev |
The BS oplev has been misbehaving and kicking the optic from time to time since noon. The kicks are not strong enough to trip the watchdogs (current watchdog max counts for the sensors is 135).
I took a look at the spectrum of the BS oplev error in pit and yaw with both loops enabled while the optic was stable. There is nothing alarmingly big except for some additional noise above 4Hz.
I have turned the BS oplev servo OFF for now. |
11200
|
Fri Apr 3 15:15:55 2015 |
Steve | Update | SUS | BS oplev |
I saw this kicking before
Quote: |
The BS oplev has been misbehaving and kicking the optic from time to time since noon. The kicks are not strong enough to trip the watchdogs (current watchdog max counts for the sensors is 135).
I took a look at the spectrum of the BS oplev error in pit and yaw with both loops enabled while the optic was stable. There is nothing alarmingly big except for some additional noise above 4Hz.
I have turned the BS oplev servo OFF for now.
|
|
11201
|
Fri Apr 3 19:35:14 2015 |
Jenne | Update | SUS | BS oplev centered |
I think that this happens when the beam gets too close to the edge of the QPD. We see this regularly in the ETMs, if they've been kicked a bit, but not enough to trip the watchdogs. I think it might be the step/impulse response of the RES3.3 filter, which rings for almost 20 seconds.
Anyhow, I've just recentered the BS oplev. It was at -21urad in pitch, and had more than 400 counts on the top two quadrants, but only about 100 counts on the bottom two. Now it's around 300 counts on all 4 quadrants.
As a totally unrelated aside, I have installed texlive on Donatella, so that I could run pdflatex. |
4976
|
Fri Jul 15 16:14:00 2011 |
steve | Update | SUS | BS oplev error signal spectra |
|
11100
|
Thu Mar 5 10:32:58 2015 |
Steve | Update | SUS | BS oplev servo turned off |
The BS oplev servo was kicking up the BS. It was turned off |
4947
|
Wed Jul 6 16:44:37 2011 |
steve, kiwamu | Update | SUS | BS oplev spectra |
Healthy BS oplev |
4963
|
Tue Jul 12 17:30:24 2011 |
steve, | Update | SUS | BS oplev spectra |
I repeated the BS oplev spectrum today and I do not understand why it does look different. I did it as Kiwamu describes it in entry#4948 The oplev servo was left ON! |
4966
|
Thu Jul 14 09:38:50 2011 |
steve, | Update | SUS | BS oplev spectra |
Quote: |
I repeated the BS oplev spectrum today and I do not understand why it does look different. I did it as Kiwamu describes it in entry#4948 The oplev servo was left ON!
|
It is working today! Finally I repeated the BS spectra, that we did with Kiwamu last week |
5622
|
Wed Oct 5 17:08:49 2011 |
steve | Update | SUS | BS oplev spectra |
Kiwamu and Steve,
The He/Ne oplev shows no coherece so relative intensity noise is not limiting factor for the oplev servo |
14108
|
Fri Jul 27 10:48:57 2018 |
Steve | Update | SUS | BS oplev window |
Yesterday I inspected this BS oplev viewport. The heavy connector tube was shorting to table so It was moved back towards the chamber. The connection is air tight with kapton tape temporarly.
The beam paths are well centered. The viewport is dusty on the inside.
The motivation was to improve the oplev noise. |
831
|
Wed Aug 13 17:00:59 2008 |
steve | Configuration | SUS | BS sat amp removed |
The PRM sat amp is broken. Ben is working on it.
The BS sat amp was removed from the BS sus and it is used with the PRM in
order to damp it for wire stand-off alignment. |
226
|
Mon Jan 7 09:01:39 2008 |
steve | Update | SUS | BS sus damping restored |
The BS sus damping was lost at 8am Sunday morning. |
8119
|
Wed Feb 20 19:48:16 2013 |
yuta | Update | Alignment | BS table oplev re-arranged |
[Sendhil, Yuta]
After aligning IFO and putting the access connector on, we also centered IPANG/IPPOS and all oplevs (except SRM).
To avoid clipping of PRM/BS oplevs, we re-arranged oplev steering mirrors on BS table.
What we did:
1. Checked IPANG comes out unclipped after putting on the access connector.
2. Centered IPANG on its QPD.
3. Checked oplevs beams for ITMX/ITMY centered on in-vac mirrors, and centered them on their QPDs.
4. Checked IPPOS beam is centered on the mirrors inside BS chamber, and centered IPPOS on its QPD.
5. Tweaked oplev mirrors on BS chamber to make PRM/BS oplev beam unclipped and centered on mirrors, and centered them on their QPDs. To avoid clipping of oplev beams in BS table, we re-arranged oplev steering mirrors on BS table (outside the vaccum).
Current status:
QPD values, IFO_ALIGN/MC_ALIGN screens, OSEM values attached.
- IR incident beam and IFO aligned
- X/Y end green coming out to PSL table (in higher order modes)
- IPANG/IPPOS available
- All oplevs available
- AS/REFL/POP cameras ready
- access connector, ETMX/ETMY heavy doors on
- ITMX/ITMX/BS heavy doors are not on
- AS/REFL/POP PDs not centered
- POX/POY/TRX/TRY not aligned
- AS beam coming out of the OMC chamber low by ~ 1 beam diameter (my bad)
Tomorrow:
- Align AS/REFL/POP PD and lock PRMI
- Take pictures of ITMX/ITMY/BS stacks
- Put heavy doors on ITMX/ITMY/BS chambers
- Start pumping down |
8120
|
Wed Feb 20 19:58:59 2013 |
rana | Update | Alignment | BS table oplev re-arranged |
Please confirm the SRM OL beam is not too bad and also find where the mis-aligned SRM puts its beam. WE want to be sure that there is not too much unwanted scattering from SRM into the PRFPMI. |
8915
|
Wed Jul 24 10:35:41 2013 |
Steve | Update | VAC | BS, ITMY doors are removed |
Quote: |
We will open the BS and ITMY doors first thing tomorrow morning. I plan to try to be in around 9 am. The first order of business will be to flip the folding mirrors that are not currently flipped (SR2, SR3, PR3).
|
Jenne, Annalisa & Steve |
8914
|
Tue Jul 23 22:55:13 2013 |
Jenne | Update | VAC | BS, ITMY doors to be opened in the morning |
We will open the BS and ITMY doors first thing tomorrow morning. I plan to try to be in around 9 am. The first order of business will be to flip the folding mirrors that are not currently flipped (SR2, SR3, PR3). |
8393
|
Tue Apr 2 18:19:30 2013 |
Jenne | Update | SUS | BS, PRM oplev servos improved |
[Gabriele, Jenne]
We have implemented 4Hz resonant gains for both PRM and BS yaw. The filter was already in place for PRM Yaw, so we just turned it on, but we also copied the filter over to BS Yaw. We also changed the 3.3Hz res gain and the ELP for the PRM servo to match the BS servo, since after implementing the 4Hz gain, PRM was still much noisier than BS. Now the 2 servos match, and PRM is a little quieter. We hope that tonight's locking might be a little more stable after this work.


|
14345
|
Tue Dec 11 18:20:59 2018 |
gautam | Update | Optical Levers | BS/PRM HeNe is dead |
I found that the BS/PRM OL SUM channels were reading close to 0. So I went to the optical table, and found that there was no beam from the HeNe. I tried power-cycling the controller, there was no effect. From the trend data, it looks like there was a slow decay over ~400000 seconds (~ 5 days) and then an abrupt shutoff. This is not ideal, because we would have liked to use the Oplevs as a DC alignment reference during the vent I plan to use the AS camera to recover some sort of good Michelson alignment, and then if we want to, we can switch out the HeNe.
*How can I export PDF from NDscope? |
14404
|
Fri Jan 18 12:52:07 2019 |
gautam | Update | Optical Levers | BS/PRM Oplev HeNe replaced |
I replaced the BS/PRM Oplev HeNe with one of the heads from the SP table where Steve was setting up the OL RIN/pointing noise experiment. The old one was dead. The new one outputs 3.2 mW of power, I've labelled it with this number, serial number and date of replacement. The beam comes out of the vacuum chamber for both the BS and PRM, and the RIN spectra (Attachment #1) look alright. The calibration into urad and loop gains possibly have to be tweaked. Since the beam comes out of vacuum, I say that we shouldn't open the BS/PRM chamber for this vent - we don't have a proper plan for the in-air layout yet, so we can add this to the list of to-dos for the next vent.
I think we are down to our last spare HeNe head in the lab - @Chub, please look into ordering some more, the ITMX HeNe is going to need replacement soon. |
1563
|
Fri May 8 04:46:01 2009 |
rana, yoichi | Summary | oplevs | BS/PRM/SRM table bad! |
We went to center the oplevs because they were far off and found that (as usual) the numbers changed
a little after we carefully centered the oplevs and came back to the control room.
To see if the table was on something soft, we tried pushing the table: no significant effect with ~10 pounds of static force.
With ~10 pounds of vertical force, however, we saw a large change: ~0.25 Oplev units. This corresponds to
~20-30 microradians of apparent optic pitch.
In the time series below you can see the effects:
2.5 s: lid replaced on table after centering.
2.5 - 11 s: various force tests on table
11 s: pre-bias by aligning beams to +0.25 in pitch and then add lid.
So there's some kind of gooey behavior in the table. It takes ~1 s to
settle after we put the lid on. Putting the laptops on the table also
has a similar effect. Please do not put anything on this table lid. |
12722
|
Mon Jan 16 18:54:01 2017 |
rana | Update | SUS | BS: whitening re-engaged |
Found that the BS whitening was off. Gautam says that "it has always been that way" and "there's nothing in the elog about this" and "I have no special relationship with Putin".
I looked at DV and DTT while turning the OSEM whitening back on. As expected, the sensor noise improved by 10x above 10 Hz. The time series shows no problems - its just less fuzzy now.
All OSEM spectra after the switch show on upper panel of plot. Lower panel shows comparison of BS UL before/after. To rotate the DTT PDF landscape output I typed this:
pdftk BS-white.pdf cat 1N output BSwhite.pdf
"if you see something, do something" |
6085
|
Thu Dec 8 00:18:38 2011 |
rana | Summary | Computer Scripts / Programs | BURT restore problems / issues in snapshot scripts |
I tried to run the seismic StripTool tonight, which seems liek a simple task. But instead I fell through the rabbit hole again...
The seismic.stp couldn't be run from zita/op340m because zita doesn't have EPICS or MEDM and because the op340m version of StripTool cannot load the new file format in which rossa/pianosa save their files.
Once I got it running by sshing in to rossa, I noticed that the BLRMS trends didn't make any sense. Correctly guessed that this was because all of the BP and LP filters were off. Why? Because of bad BURT, that's why.
As it turns out (if you look through the autoburt logs), several of our FE machines haven't been correctly saving snapshots because of some channel count mismatch between old and new SNAP files. So we are not getting the settings restored correctly for these systems when they get booted. Probably, someone has got to suss out the source of the BURT snap messages; it may be that we have to rehash the snap process after any substantial model rebuild.
For c1pemepics, I did a manual restore from the time when Mirko last ELOG'd that BLRMS was trending OK (Nov 23 @ 3 AM).
Seems like we should also get some kind of auto email warning if the BURTs fail in this way. Otherwise, we'll lose a lot of work if it goes on for weeks.
Bottom line: fix the autoburt so that it doesn't fail after model rebuilds. |
9454
|
Tue Dec 10 17:27:47 2013 |
Jenne | Update | Treasure | Baby oplev LQR designed loop |
I *finally* figured out how to bend Matlab to my will, and close a very simple oplev loop using LQR technology.
This is super, super simple, but it's a step in the right direction. There is no noise, just a simple pendulum with a resonant frequency of 0.75Hz, and a Q of 10. The LQR tries to keep the position of the pendulum at a minimum, and does not care at all about the velocity. You cannot (with Matlab's LQR, at least that I can find) make it care "0" about the control output, so it cares about the control output a factor of 1e-4 as much as the position.
Here are some plots:
The first plot has the open loop system (just the pendulum, no control at all), as well as the closed loop system (the pendulum under control).

Plot 2 is the open loop gain of the controller that the LQR designed.

Plots 3 and 4 are the step and impulse responses of the open loop (pendulum with no control), and closed loop (pendulum with feedback) systems.


The conclusion from the plots (if this were an interesting system) is that it doesn't take much to damp an ideal pendulum. The real conclusion here is that I think I now know how to use the Matlab LQR function, and can make a loop with some noise now. |
8180
|
Wed Feb 27 02:52:40 2013 |
Jenne | Omnistructure | SAFETY | Back door unlocked |
Did someone unlock the back door by the (unofficial) bike storage lately? Out of habit, I checked the door behind me while about to leave, and it is unlocked.
Please recall that if you leave through that door, it should automatically lock behind you (if it was locked already), however if you unlock and enter through the back door, it stays unlocked until someone locks it again.
(Obviously, I'm locking the door before I actually go). |
3133
|
Tue Jun 29 11:48:17 2010 |
Jenne | Configuration | SAFETY | Back in LASER HAZARD mode. |
[Steve, Kiwamu, Jenne]
The 40m is now back in Laser Hazard mode. Safety glasses are required for entry into the LVEA / IFO room. |
3135
|
Tue Jun 29 14:16:35 2010 |
Koji | Configuration | SAFETY | Back in LASER HAZARD mode. |
The insects and the laser trouble... Strange coincidences with LHO surprised me, but now I have been relieved.
Quote: |
[Steve, Kiwamu, Jenne]
The 40m is now back in Laser Hazard mode. Safety glasses are required for entry into the LVEA / IFO room.
|
|
11463
|
Thu Jul 30 03:19:24 2015 |
ericq | Update | LSC | Back towards PRFPMI |
The refreshed ALS didn't look so bad today (elog forthcoming), so I decided to give PRFPMI locking a shot tonight. I was able to hold the PRMI while swinging through resonsance, but transitions to RF signals failed. Demod angles / whitening gains/ etc. etc. all need to be rechecked
Some little things here and there that got cleaned up...
- The PRM oplev beam was being blocked. Why? I removed the block. Should recheck OLTF/spot size on QPD.
- ALS -> CARM, DARM signs changed, maybe because I've used the delayed beat as the RF input on the demod board, whereas I imagine it may have been the LO in the beatbox. No big deal.
- REFL165 whitening gain and input matrix updated. Should recheck demod angles.
- PRMI triggering settings weren't being set in the script. It's important to include arm transmission signals, since POP2F signals can momentarily dive when swinging through resonance.
- I should revisit phase tracker UGF normalization. I/Q amplitudes are varying quite a bit from lock to lock.
- PRC angular feedforward disabled for now; need to remeasure the witness/target data with DC coupled ITM oplevs
- I think there has been a little bit of MC servo tweaking since our last locks, may need to recheck AO TF / gains.
|
4043
|
Fri Dec 10 12:55:27 2010 |
Jenne | Update | Computers | Backup should be running successfully now |
[Joe, Jenne]
The nightly backup of the frames and the /cvs/cds directories is back up and running. We are free again to do crazy stuff at will, and it will all be saved for eternity. |
16317
|
Wed Sep 8 19:06:14 2021 |
Koji | Update | General | Backup situation |
Tega mentioned in the meeting that it could be safer to separate some of nodus's functions from the martian file system.
That's an interesting thought. The summary pages and other web services are linked to the user dir. This has high traffic and can cause the issure of the internal network once we crash the disk.
Or if the internal system is crashed, we still want to use elogs as the source of the recovery info. Also currently we have no backup of the elog. This is dangerous.
We can save some of the risks by adding two identical 2TB disks to nodus to accomodate svn/elog/web and their daily backup.
host |
file system or contents |
condition |
note |
nodus |
root |
none or unknown |
|
nodus |
home (svn, elog) |
none |
|
nodus |
web (incl summary pages) |
backed up |
linked to /cvs/cds |
chiara |
root |
maybe |
need to check with Jon/Anchal |
chiara |
/home/cds |
local copy |
The backup disk is smaller than the main disk. |
chiara |
/home/cds |
remote copy - stalled |
we used to have, but stalled since 2017/11/17 |
fb1 |
root |
maybe |
need to check with Jon/Anchal |
fb1 |
frame |
rsync |
pulled from LDAS according to Tega |
|
|
|
|
|
11051
|
Thu Feb 19 15:45:43 2015 |
ericq | Update | CDS | Bad CDS behavior |
At about 10AM, the C1LSC frontend stopped reporting any EPICS information. The arms were locked at the time, and remained so for some hours, until I noticed the totally whited-out MEDM screens. The machine would respond to pings, but did not respond to ssh, so we had to manually reboot.
Soon thereafter, we had a global 15min EPICS freeze, and have been in a weird state ever since. Epics has come back (and frozen again), but the fast frontends are still wonky, even when EPICS is not frozen. Intermittantly, the status blinkers and GPS time EPICS values will freeze for multiple seconds at a time, sporadically updating. Looking at a StripTool trace of an IOPs GPS time value shows a line with smooth portions for about 30 seconds, about 2 minutes apart. Between this is totally jagged step function behavior. C1LSC needed to be power cycled again; trying to restart the models is tough, because the EPICS slowdown makes it hard to hit the BURT button, as is needed for the model to start without crashing.
The DAQ network switch, and martian switch inside were power cycled, to little effect. I'm not sure how to diagnose network issues with the frontends. Using iperf, I am able to show hundreds of Mbit/s bandwidth betweem the control room machines and the frontends, but their EPICS is still totally wonky.
What can we do???  |
11052
|
Thu Feb 19 23:23:52 2015 |
Chris | Update | CDS | Bad CDS behavior |
The frontends have some paths NFS-mounted from fb. fb is on the ragged edge of being I/O bound. I'd suggest moving those mounts to chiara. I tried increasing the number of NFS threads on fb (undoing the configuration change I'd previously made here) and it seems to help with EPICS smoothness -- although there are still occasional temporal anomalies in the time channels. The daqd flakiness (which was what led me to throttle NFS on fb in the first place) may now recur as well.
Quote: |
At about 10AM, the C1LSC frontend stopped reporting any EPICS information. The arms were locked at the time, and remained so for some hours, until I noticed the totally whited-out MEDM screens. The machine would respond to pings, but did not respond to ssh, so we had to manually reboot.
Soon thereafter, we had a global 15min EPICS freeze, and have been in a weird state ever since. Epics has come back (and frozen again), but the fast frontends are still wonky, even when EPICS is not frozen. Intermittantly, the status blinkers and GPS time EPICS values will freeze for multiple seconds at a time, sporadically updating. Looking at a StripTool trace of an IOPs GPS time value shows a line with smooth portions for about 30 seconds, about 2 minutes apart. Between this is totally jagged step function behavior. C1LSC needed to be power cycled again; trying to restart the models is tough, because the EPICS slowdown makes it hard to hit the BURT button, as is needed for the model to start without crashing.
The DAQ network switch, and martian switch inside were power cycled, to little effect. I'm not sure how to diagnose network issues with the frontends. Using iperf, I am able to show hundreds of Mbit/s bandwidth betweem the control room machines and the frontends, but their EPICS is still totally wonky.
What can we do??? 
|
|
11053
|
Fri Feb 20 12:08:10 2015 |
ericq | Update | CDS | Bad CDS behavior |
I've been able to get all models running. Most optics are damped, but I'm having trouble with the ITMs, BS and PRM. |