After running into more problems with the upgrade, I eventually decided to abort todays upgrade attempt, and revert back to where we were this morning (RTS 2.1). I'll try to follow this with a fuller report explaining what problems I encountered when attempting the upgrade.
However, when Alex and I were trying to figure out what was going wrong in the upgrade, it appears that the fb symmetricom card lost the ability to sync with the GPS receiver. When the symmeticom module is loaded, dmesg shows the following:
[ 285.591880] Symmetricom GPS card on bus 6; device 0
[ 285.591887] PIC BASE 2 address = fc1ff800
[ 285.591924] Remapped 0x17e2800
[ 285.591932] Current time 947125171s 94264us 800ns
[ 285.591940] Current time 947125171s 94272us 600ns
[ 285.591947] Current time 947125171s 94280us 200ns
[ 285.591955] Current time 947125171s 94287us 700ns
[ 285.591963] Current time 947125171s 94295us 800ns
[ 285.591970] Current time 947125171s 94303us 300ns
[ 285.591978] Current time 947125171s 94310us 800ns
[ 285.591985] Current time 947125171s 94318us 300ns
[ 285.591993] Current time 947125171s 94325us 800ns
[ 285.592001] Current time 947125171s 94333us 900ns
[ 285.592005] Flywheeling, unlocked...
Because of this, the daqd doesn't get the proper timing signal, and consequently is out of sync with the timing from the models.
It's completely unclear what caused this to happen. The card seemed to be working all day today, then Alex and I were trying to debug some other(maybe?) timing issues and the symmetricom card all of a sudden stopped syncing to the GPS. We tried rebooting the frame builder and even tried pulling all the power to the machine, but it never came back up. We checked the GPS signal itself and to the extend that we know what that signal is supposed to look like it looked ok.
I speculate that this is also the cause of the problems were were seeing earlier in the week. Maybe the symmetricom card has just been acting flaky, and something we did pushed it over the edge.
Anyway, we will try to replace it tomorrow, but Alex is skeptical that we have a replacement of this same card. There may be a newer Spectracom card we can use, but there may be problems using it on the old sun hardware that the fb is currently running on. We'll see.
In the mean time, the daqd is running rogue, off of it's own timing. Surprisingly all of the models are currently showing 0x0 status, which means no problems. It doesn't seem to be recording any data, though. Hopefully we'll get it all sorted out tomorrow.
I will be attempting (again) to upgrade the RTS, including the RCG and the daqd, to version 2.4 today. The RTS will be offline until further notice.
Dataviewer is recovered after heavy new year party.
Communication between the front end models and the framebuilder has been restored. I'm not sure exactly what the issue was, but rebuilding the framebuilder daqd executable and restarting seems to have fixed the issue.
I suspect that the problem might have had to do with how I left things after the last attempt to upgrade to RCG 2.4. Maybe the daqd that was running was linked against some library that I accidentally moved after starting the daqd process. It would have kept running fine as was, but if the process died and was attempted to be started again, it's broken linking might have kept it from running correctly. I don't have any other explanation.
It turns out this was not (best I can tell) related to the new year time sync issues that wer seen at the sites.
The dynamic power normalization system has been modified such that the normalization happen after the LSC input matrix.
It turned out that the power normalization need a modification.
I will work on it tomorrow and it will take approximately 2 hours to finish the modification.
I moved the following old and obsolete MEDM directories to an archive /cvs/cds/rtcds/caltech/c1/medm
None of these models were present in /cvs/cds/rtcds/caltech/c1/target
Remember that we don't use /cvs/cds/rtcds anymore. Everything should be in /opt/rtcds now.
Is there a reason the framebuilder status light is red for all the front ends?
Also, I reenabled PRM watchdog.
Apparently there is a bug in the timing cards having to do with the new year roll-over that is causing front-end problems. From Rolf:
For systems using the Spectracom IRIG-B cards for timing information, the code did not properly roll over the time for
2012 (still thinks it is 2011 and get reports from DAQ of timing errors (0x4000)). I have made a temporary fix for this
in the controller.c code in branch-2.3, branch-2.4 and release 2.3.1.
I was going to check to see if the 40m is suffering from this. I'll be over to see if that's the problem.
The problem is the same as yesterday.
Here is a plan in my mind and these are basically the details of the gantt chart (#6143):
I checked on the two single-axis shakers that are present at the 40m that Steve pointed out:
Neither of these meet the force requirement of 2.04 kN peak.
Time to lower your expectations!
Do we really need 40 microns at 500 Hz? Or perhaps should there be a frequency dependent displacement requirement?
I don't know what exactly is going on, but MC became flaky and it's been frequently unlocked.
I have turned off the MC WFS servo to check if the WFSs are doing something bad. But it still tends to be unlocked without the WFS servo.
Right now it doesn't stay locked for more than 10 min.
I updated the design requirements for the Stewart platform. I weighed the unloaded "dirty" SOS that was sitting on the workbench in the control room; its mass is 11 kg. Steve suggested that the OSEMs (not installed on this model) would add another 0.5 kg. From the specs in the final SOS design document, LIGO-T970135, I added 0.25 kg for the optic itself; I am therefore taking the total payload mass to be 11.75 kg. (Now, the upper stage of the Stewart platform itself will likely add a nontrivial amount, but I am not worrying about this yet.)
I have e-mailed Janeen Romie to obtain the actual center of mass and principal moments of inertia of the platform. I also cooked up a simple scheme to measure both quantities, should this information not be available. It would involve rigidly mounting the dirty SOS to a rigid bar hung from a pivot. By translating the mount point in two dimensions and measuring the period of the pendulum, I ought to be able to find the center of mass and moments of inertia by multilinear regression. However, this elaborate scheme is not necessary to just compute some ballpark figures; it could wait until a later stage in the design. For the time being, I just rescaled the moment of inertia proportional to the increase in mass, such that the torque-to-force ratio is unchanged.
As such, the design requirements are now
From these assumptions, the revised actuator requirements and dimensions are:
See the attached PDF document.
It appears that the actuator that I had originally nominated, PI's model P-225.80, would very nearly meet the actuator force requirement. Steve also pointed out the following single-axis shakers that are already in use in the 40m:
I want to find out if either of these would meet the present need, but I'm waiting on a response from the manufacturer to get access to the data sheets.
I found the PSL laser has been off for four hours. Nobody seemed to know why.
I just turned it on and it is now providing about 10% lower power compared with one before the shutdown.
Let's keep the eyes on the power if it can recover as the housing gets warm.
Concept of Power Normalization
Now a power normalization is doable for the LSC error signals.
It is working fine, but at some point we may want to have some kind of a saturation filter or limiter to avoid dividing a signal by a small number.
(How to set the normalization)
The high frequency noise, which has been a dominant noise above 30 Hz in the Y arm ALS (#6133), decreased by a factor of 5.
This reduction was done by increasing the modulation depth at the Y end PDH locking. Now the noise floor at 100 Hz went to 0.2 pm/sqrtHz.
However the noise source is not yet identified and hence it needs a further investigation.
(Increasing the modulation depth)
I added an ALS feedback path on the MC2 suspension and this path will enable us to stabilise the MC length using the ALS scheme.
Leaving a note on the ALS feedback before I forget:
The MC2 suspension needs to have an input for the ALS feedback in the realtime model like ETMs.
I have realigned the steering mirrors for PMC because the transmitted light had been at ~ 0.741
After the alignment it went back to ~ 0.850.
Some new screens have been made for the new multiple-LOCKIN system running on the LSC realtime controller.
The medm screens are not so pretty because I didn't spend so long time for it, but it is fine for doing some actual measurements with those new screens.
So the basic works for installing the multiple-LOCKIN are done.
The attached figure is a screen shot of the LOCKIN overview window.
As usual most of the components shown in the screen are clickable and one can go to deeper levels by clicking them.
How to set the thresholds
The below is a screenshot of the trigger matrix screen. The thresholds column is pointed by a big white arrow.
Of course, DO NOT set the upper threshold value to be smaller than that of the lower threshold. Otherwise it won't correctly work.
Also if you want to have the usual trigger rather than the Schmitt trigger, simply put the upper and lower thresholds at the same values.
The picture below is a screen shot of the LSC real time model, zoomed in the new LOCKIN part.
The LOCKIN module consists of three big components:
I have edited c1scx.ini by hand in order to acquire some green locking related channels.
Somehow c1sus.ini, c1mcs.ini, c1scx.ini and c1scy.ini are not accessible via the daqconfig script.
As far as I remember it had been accessible via daqconfig a week ago when I edited c1scy.ini.
Anyway I had to edit it by hand. They need to be fixed at some point
Often people say "I don't use conlog because its real slow". Its a little like not driving because your car has no gas.
I looked into what's going on with conlog. No one has fixed its channel list in ~1 year so it didn't make much sense. Also since Oct 1 of this year, it expired the leap seconds epoch and has been waiting for someone to look at the log file and update the list of leap seconds.
There are a bunch of bad channels which are screwing up various tools (DV, DTT, etc.):
Examples: C1:LSC-Subsystem_NPRO_SW1, C1:-DOF2PD_MTRX_0_0_SW1, C1:BAD-BAR_CRAZY_2_RSET, C1:C1L-DOF2PD_MTRX_3_14_SW2, C1:DUB-SEIS_GUR2_Z_LIMIT, etc.
To fix up some of these issues, I have deleted several MEDM directories which I thought were old (there are several extras left from Aidan's Green time). I also have put a bunch of exclude variables into the conlog 'scan_adls' script to prevent it from adding some of the new worthless channels. Finally, I have started this command
../bin/strip_out_channels '.*STAT.*','.*_ALIVE.*','C1:PEM.*','.*_Name.*','C1:UCT.*','C1:MCP.*','C1:SP.*','C1:DU.*','C1:RF.*','C1:NIO.*','C1:TST.*','C1:SUP.*','C1:X.*','C1:FEC.*','.*_LFSERVO.*','.*FSS_SLOWDC.*','C1:LSC-LA_MTRX_21','C1:LSC-PD.*OFFSET','C1:LSC-ETM.*OFFSET' conlog*.log
which should strip lots of the excess conlog data out of the conlog directory. The only downside is that its setting all of the timestamps of the .log files to today instead of the historical times but I don't think we'll care about this too much. Hopefully it will speed things up to have less than 450 GB of conlog files...
update: 12 hours later, its still running and has removed ~100 GB so far. It will probably take the rest of the weekend to finish.
After I did a fine alignment of the X green beam path on the PSL table, the X arm beat-note was also obtained.
Here is a picture of the latest setup. The blue lines represent S-polarizing green beams.
During I was working on the PSL table HEPA was at 80 %, and after the work I brought it to 20 %.
I set the limitters of the MC WFS servos not to inject the feedback more than 2000.
The residual of the WFS shakes the MC SUSs at every lock loss.
Assumptions on the parameter estimations
I made the first trial of locking a Power-recycled single arm.
Here is the Gantt chart we discussed in the 40m meeting today.
Based on the discussions we had, I applied a little bit of corrections on the chart but the main stream remains the same.
While discussing the suspension hysteresis measurements, Koji, Kiwamu, and I realized that the suspension wire standoff is aluminum, whereas the standoff for the LIGO LOS are using quartz.
Using a soft aluminum standoff is bad. The movement of the suspension will slowly wear the groove and produce opportunities for mechanical upconversion and hysteresis.
In fact, the wire standoff as well as the clamping block on the top should be made of sapphire or ruby to prevent any such wearing issues. Steve is hot on the case.
This is NOT a work in the main stream,
but it gives us some prospects towards the full lock and perhaps some useful thoughts.
Lock Acquisition Steps
Actual Time Series
Below is a plot of the actual lock acquisition sequence in time series.
Scripting of the single arm automated lock script is 80% done.
The remaining 20 % is not something immediately needed and I start decreasing the priority on the Y arm ALS.
3/4 " thick colored acrylic material will be used in this air tight design. Surgical tubing for o-ring. We may have to put an o-ring into the bottom to have it really air tight.
The top drawing is not ready. It will have handle and industrial grade L-handle lock pin to hold cover down. There will be 2 one inch od post in the midle of the table to hold the cover and lock the ball pin.
I'm waiting for your inputs, so I can send this preliminary design out for quote.
I have been looking at the swings in the RAMmon channels since the heater was reengaged, to compare them to the data from beforehand (with and without the foam box). With the large grains of salt that I will list after, it appears that the EOM temperature controller does in fact reduce the amplitude of the swings by a measurable factor.
It seems impractical to try and rope off essentially 3 straight days where nothign major can be done to the IFO just to take RAM data. Instead, I think we should figure out a way to mimic the diurnal temperature swings on ~hour timescales. The EOM can temperature follows PSL-FSS_RMTEMP almost exactly and with a very short delay, so we can probably even accomplish this by stepping the lab A/C temperature. If this won't work, we can use an incandescent lamp or something similar to heat up the area around the EOM by a noticeable amount.
I'll try to come up with a good way to do this so that we can get some reliable data...
Online filter diverges. I did offline simulations with current c-code. Offline filter also diverges, even in the simplest case
witness = randn(1e6, 1); target = witness + 0.01*randn(1e6, 1);
I tried to create a new implementation of FXLMS algorithm as a c code. Then with this c code I did offline filtering with MCL and GUR signals and compared the error signals depending on the length of the filter.
One can see the code at the svn
adaptOnline - start here and choose algorithm
adaptive_filtering - Matlab implementation of AF
current_version.c - current version of the Filter (Matt's)
fxlms_filter.c - new version of the FXLMS filter
oaf.c - agent between Matlab and C (edited Matt's file)
Data samples can be found at nodus /users/den/wiener_filtering/data
Another hysteresis test has begun at 1:50 PT, Dec/19.
It will finish after 3 or 4 hours. During the measurement the PSL mechanical shutter will be kept closed.
I have recentered the oplev beams, including BS, ITMs and ETMs.
The measurement finished at ~ 21:50 PT.
A new test started from 16:05 PT, Dec 18th and takes a couple of hours to finish the measurement.
Do not touch the suspensions until further notice.
Here are the latest plots that I have obtained from the Friday night:
The residual motion in the arm displacements reached 70 pm in rms.
Koji has modified the script for the hysteresis measurement.
The hysteresis test has been aborted.
The test was from: 2011-12-17 09:48 to 11:49 (UTC).
This corresponds to the period from 2011-12-17 01:48 to 3:49 (PST).
Do you guys have timestamps for when you started/ended the test? I have been trying to take some long-term RAM data but keep getting foiled by stuff (this test, RTS upgrade, switching of RAMmon channels, etc.)
To test it, we are shaking all of the suspension biases +/-1.0 with a script.
All of the suspensions have accumulated unexpectedly big DC biases of about 5 from their nominal points.
All of the suspensions have accumulated unexpectedly big DC biases of about 5 from their nominal points.
We wonder if the flakiness of MC2 comes from the suspension or not.
The script is here:
For this test, we closed the mechanical shutter before the MC.
Also some amount of misalignment is anticipated.
Don't be surprised if you see nothing is working when you come to the lab in the weekend.
The 60 Hz line noise has gone away.
About Noise Budget
How to improve it ?
Unfortunately, after working on it all day, I had to abort the upgrade and revert the system back to yesterday's state.
I think I got most of the upgrade working, but for some reason I could never get the new models to talk to the framebuilder. Unfortunately, since the upgrade procedure isn't document anywhere, it was really a fly by the seat of my pants thing. I got some help from Joe, which got me through one road block, but I ultimately got stumped.
I'll try to post a longer log later about what exactly I went through.
In any event, the system is back to the state is was in yesterday, and everything seems to be working.