The up and down scripts accessible from the OAF (still C1:ASS-TOP) screen are now totally functional and awesome. They are under the blue ! button. The up script can either be for the Seismometers, or the Accelerometers at this time. The only difference between these 2 is which burt restore file they look at: the seismometer version puts all 7 seismometer channels in the PEM selecting matrix, while the accelerometer version puts the 6 accelerometer channels in that matrix. Both scripts also turn on HP_1mHz filters in the ERR_EMPH filter bank and all of the witness filter banks, and the AA32 and AI32 filters in ERR_EMPH, CORR and PEM filter banks. This makes all of the starting filters the same between the witness paths and the error path.
If you want to use a different combination of sensors, run one of the up scripts, then change the PEM matrix by hand.
The down script disables the output to the optics, and resets the adapted filter coefficients. DO NOT use this script if you're trying to "pause" the filter to take some nice long averages.
The EQ did not change the input beam pointing. All back to normal, except MC2 wachdogs tripped again.
Round 3 for the day of MC2 watchdogs tripping.
As per Matt's instructions in his OAF document (elog 395) in the Tuning section, Sanjit and I took a transfer function measurement from the output of the OAF system, to the input. i.e. we're trying to measure what happens out in the real world when we push on MC1, and how that is fed back to the input of our filter as MC_L. The game plan is to measure this transfer function, and read off the phase at the nyquist frequency, and use this value to calculate the appropriate sample-and-hold delay to be used in the OAF. The downsample rate for the OAF is 32, so that we're running at 64Hz instead of the 2048Hz of the front-end. Thus, our Nyquist frequency is 32Hz.
Phase@Nyquist * ------------------------
In the attached figure we do a swept sine from CORR_EXC to ERR_EMPH_OUT to determine the transfer function. Here, we turn off all of the filters in both the CORR and EXC banks, because those are already matched/taken into account in the PEM filter banks.
Using the cursor on DTT, we find that the phase at 29.85Hz is -228.8deg, and at 37.06Hz is -246.0deg. Extrapolating, this means that at 32Hz, we expect about -234deg phase. Using our handy-dandy formula, this means that we should try a delay of 41 or 42 (41.6 is between these two...)
We'll give this a shot!
Phase@Nyquist * ------------------------ = Delay
As Rana pointed out to me last night, I was using continuous phase, which is not good. When using regular phase, I find: (29.85Hz, 131.216deg), (37.06Hz, 113.963deg), so extrapolating gives (32Hz, 126.07deg). Plugging this in to our handy-dandy formula, we get a delay of 22.4, so we should try both 22 and 23.
I spent part of the afternoon cleaning up the area next to the Mode Cleaner where we keep all of our RF stuff: Attenuators, BNC/SMA/LEMO adapters, Mini-Circuits items, and all sorts of other things which are useful while looking at our electronics/RF stuff.
We got another set of "Lyon" drawers, which aided in the organization process....Bob ordered 2, so we now have a 'spare' drawer set if anyone can think of something else to organize (unless this was premeditated for optics or something else?).
As you can see in the picture, (1) it's no longer a total disaster over there, and (2) some of the drawers have sub-divisions to make it faster and easier to find what you're looking for. Please help out by putting things away in their proper place, and adding more labels or dividers to the drawers if there's something else which needs a 'spot'.
Here's a plot of the spectra of the seismometers and MCL. The coherence shows which axes are aligned right now: MC1_X is coherent with GUR_NS which means that its mis-oriented.
I've now swapped the "MC1" cables: so the old "NS" now goes into EW and the old EW now goes into NS. VERT is unchanged.
Also fixed the channel names - the Guralp previously named MC1 is now GUR1 and the other one is GUR2. Also no more EW, NS, & VERT. Its all XYZ.
DAQD restarted with the new channel names.
I spiffed up the order of the cables / sensors plugged into the PEM ADCU. Now all of the seismometers are labeled as Rana left them, and the 2 Guralp's have their sets of 3 channels next to eachother in channel-number-land. None of the accelerometer names/cabling have changed recently. In the table, Cable-label refers to the physical tag tied to the end of the cables plugged into the ADCU...they are meant to be descriptive of what seismometer channels they are hooked up to, and then the names change to something useful for us when they come into the DAQ system. Also, the labels of input channels on the ASS_TOP_PEM screen have been updated accordingly.
The c1ass computer, which is now used for the OAF system, has many remnants from the days when it was actually used as an ASS. These PIT and YAW filter banks and other things were taking up a lot of unnecessary space, so I deleted them in the ass.mdl file. These files are all backed up, so we can always revert back to an older version when we want some Alignment Stabilization again someday. I then did a make ass, following the instructions on the 40m Wiki -> Computers and Scripts -> Simulink to Front-End Code page. Rana moved some things around, most notably all of the things (like the ASS screens) which were only in ...../users/alex/.... are now in ....../caltech/cds/advLigo/..... . This required a few restarts of the c1ass machine (after a couple different versions of the simulink diagram....one to make sure we knew how to do it, and then again actually deleting the unused portions).
The big lesson of the night was that there are 2 signal paths for the PEM channels. As is shown in Figure 3 in the mevans document, the PEM channels get the matching filters when they go to the adaptation algorithm, but when they go to the FIR filter, they do not get the matching filters. This is implemented by taking the output of the giant PEM matrix, and having a duplicate of each of the channels "selected for adaptation", one which gets filtered through the PEM_N_ADPT banks, and one which goes straight (in code-land) to the FIR filter. So, it seems like all the filters which we had been including in the input side of the matrix for matching purposes need to be put in the output side. One of the AA32 filters needs to stay in the input side, for actual anti imaging of the PEM channels, then we put the AA32 and AI32 which are for matching the ERR_EMPH and CORR filter banks up in the PEM_N_ADAPT banks. Rana and I made these filters, and they are now turned on appropriately with the OAF down script (so that all the filters are ready and waiting for the OAF to be turned on).
A little success with getting the 3Hz peak reduced, but not a lot beyond that. Tomorrow I'll put the accelerometers back where they used to be to see if they help out at all.
Last night while noise hunting, Rana found that the MC2 trans beam has been left for an unknown length of time just hitting the side of the black enclosure box. Today I put a brand-spankin'-new razor dump on the MC2 table, to dump the beam.
The set of 6 accelerometers which were semi-randomly placed underneath the MC2 tank are now back into 2 separate sets of 3 - one set at MC2 and one set at MC1. The channel names once again reflect reality, i.e. MC1_Y is actually under the MC1 tank, and aligned with the y direction. Also, the Guralp under MC1 was moved a little bit to the left, because Sanjit wanted to put the accelerometers where the seismometer had been.
Steve, Kiwamu, and Koji
We went through the PSL table to make sure any strong beam did not hit the wall.
We found that the reflection of Stephanie's OSA returned its path down to the beamsplitter.
This BS reflect that beam to the wall. That was fixed.
The surprising was that the relatively strong beam (~1mW?) went through the steering mirror
just before the PMC. We put thorlabs razor blades. I am still thinking what this indicates...
because the beam had been blocked if it was such from long time before.
During the work we found some stray optics such as a cube BS, a flipper mirror, and so on.
We can see them in the photo as those enclosed with yellow circles.
One of the beams was obtained from the reflection of the ND filter (...almost illeagal), and
was even hittting a metal fixture for the BS cube.
If someone uses these components for useful purposes,
please let me(Koji) know. Otherwise, they are removed next week.
The other thing we found was the bright scatter from the EOM for the PMC.
As this scatter is so blight, I am going to align it.
These components are from when Rana and I used the StochMon PD to do the RFAM tuning, documented in elog 1926. This was a very handy measurement, but I'm not sure if whether or not we need it often enough to keep the optics there.
There is some craziness going on with the delay in the PEM path for the OAF. We plot the difference between the C1:PEM-SEIS_GUR1_X and C1:ASS-TOP_PEM_10. These are physically the same channel, plugged into the PEM ADCU, and then the signal is used as a regular PEM channel, and is also sent to the ASS computer and used there for the OAF system. As you can see in the blue trace on the bottom plot, there is a huge amount of delay, and it's very noisy. We also plot the _GUR2_X / ASS-TOP_PEM_2 pair (red), and it has a similar amount of delay, but it is not nearly as fuzzy and noisy. For comparison, we plot the SUS-MC2_MCL (which is identical to IOO-MC_L) and ASS-TOP_ERR_MCL pair (green), and they don't have any big overall delay problems, so it's not totally a problem with the signals getting to the ASS computer.
This problem was present during/after all of the following attempts to fix it:
* The sample rate on the ASS computer is 2048. The PEM channels were being acquired the ADCU at 512. We changed the ADCU sampling rate to 2048 to match.
* We soft rebooted the ASS computer, in case it was a timing problem.
* Doing a "sudo shutdown -r now" while logged in as controls.
We might also try resetting/power cycling c0dcu in the morning. Alex has been emailed to help us try to figure this out.
In other news, the time delay that we measure from the plot gives us 180degrees in ~210Hz. This corresponds to a little more than 2msec of delay, with the C1:ASS version lagging behind the C1:PEM version. (2 samples at 840Hz) Converting to the 2048 sampling rate, we have a delay of 4.8, so 5 front-end cycles. Since Rana measured this morning that the delay indicated by the transfer function is 10 cycles, and this delay shows that the ASS lags the actual seismometer signal by 5 cycles, we should subtract this 5 from the 10 from the transfer function, giving us a final sample-and-hold delay of 5. Coincidentally(?), 5 is the delay that was found in the C1:ASS-TOP screen, after it's one year of dormancy. The point of the delay feature in the code is to help match the delay in the two signal paths: the PEM path and the output path of the filter. Since the output has a lag of 10, and the PEM path has a lag of 5, to make them match, we artificially put in a delay of 5.
Alex came in a week ago Friday to help figure this timing problem out, and some progress was made, although there's more to be done.
Here are the (meager) notes that I took while he was working:
we can rename the tpchn_C1_new back to tpchn_C1, but the _new one works right now, so why change it?
need to find dcuDma.c source code...this is (?) what sends the PEM channels over to ASS. Found: source code is dcu.c, th
en the binary is dcuDma.o Trying to recompile/remake dcuDma to make everything (maybe) good again.
Possibility: maybe having so many channels written to the RFM takes too long? shouldn't be a problem, but maybe it is. I
n the startup.cmd (or similar?) change the number of ISC modules to 1, instead of 2, since we only have one physical board
to plug BNCs into, even though we have 2 isc boards. c0dcu1 rebooted fine with the one isc board. now can't get ass tes
tpoints to try the DTT timing measurement again. rebooting fb40m to see if that helps. fb40m is back, but we still don't
have ASS testpoints. Alex had to leave suddenly, so maybe more later.
Also, next possibility is that c0dcu and c1ass are not synched together properly....we should look at the timing of the AS
After these adventures, the noisy trace in the timing delay (in the plot in elog 2066) has become quiet, as shown below (The blue trace, which was noisy in 2066 is now hiding behind the red trace). However, the overall timing delay problem still exists, and we don't quite understand it. Alex and I are meeting tomorrow morning at the 40m to try and suss this out. Our first plan of attack is to look at the ASS code, to see if it puts any weird delays in.
[Alex, Jenne, Sanjit]
Alex came to the 40m today, and did several awesome things in OAF-land.
We discovered that there is, in fact, an ADC board connected to the ASS machine. The tricky bit is that it only has a ribbon cable connector, so before we can use this ADC, we need to figure out how to make a breakout board/cable/something to connect the seismometer/accelerometer/microphone BNCs to this little board. This is the same little board that connects the timing slave to the ASS machine. For good or for ill, the timing slave is connected to this board via clip-doodles. Potentially we can connect an ADC tester board to this board, and go from seismometer BNCs to clipdoodles to the tester board, but I'm not in love with the idea of utilizing clipdoodles as a semi-permanent solution until the upgrade. I emailed Ben to see if he has a better idea, or (better yet) some spare hardware now that's the same as we'll use after the upgrade. If we can use this ADC, it may solve our timing problem which is caused by the 110B ADC used by the PEM computer. Alex showed Sanjit and I how to connect the ASS's ADC card to the simulink diagram, when we're ready for that.
We also poked around in the code, and it seems that we can now save and restore OAF coefficients at will. I added buttons to the OAF (ASS) screen, and Alex made it so the OAF coefficients are saved in RFM shared memory whenever you click the "save coeffs" button, and are restored when you click the "restore coeffs" button. The buttons are the same as the 'Reset' button which has been there for a long time, so they seem to maybe have a similar problem in that you have to hold the button for a while in order for the code to realize that the button has been depressed. We couldn't fix this easily, because it looks like our SimuLink cds stuff is a little out of date. Some day (before/when Joe and Peter make new screens for the new 40m), we need to update these things. Alex was concerned that it might take a while to do this, if the update broke some of the blocks that we're currently using. Also, Sanjit and I now need to check that the coefficient-saving is going as planned. When I have DTT open, and the OAF running, I see a certain shape to the signal which is sent to MC1 to correct for the seismic motion. This shape includes at least several peaks at resonant frequencies that exist in our stacks/suspensions. I can then save the coefficients, reset the active filter, and then restore the coefficients. When I do this while watching DTT, it seems as though the general shape of the filter is restored, but none of the detailed features are. The reason for this is still under investigation.
The code-modifications involved a few iterations of 'remaking the ass'.
Does anyone know what the channels plugged in to the PEM ADCU, channels 5,6,7,8 are? They aren't listed in the C1ADCU_PEM.ini file which tells the channel list/dataviewer/everything about all the rest of the signals which are plugged into that ADCU, so I'm not sure if they are used at all, or if they're holdovers from some previous time. The cables are not labeled in a way that makes clear what they are. Thanks!
Sanjit has been working today on trying to get the OAF coefficients to save properly. Alex got us most of the way, but right now it's looking like the filter that is being saved is totally constant (all the values are the same). We're poking around trying to figure out why this is.
Also, we're starting again (as we should have been for the last week or so since Alex came in to help us) to check in the TOP_XFCODE whenever we make changes to it, and when we recompile the front end code.
Zach found the HEPA switch on the PSL table OFF. He turned them on.
The current update on the Chameleon video switch is: no progress.
I connected the old laptop that Rob/Steve acquired via RS-232 serial to the back of the video switch. I'm using P2, the same serial port that the C1AUX computer was connected to just in case there's something good about P2 vs. P1.
I used HyperTerminal to (try to) talk to the switch. Settings were: COM1, bits per second = 9600, data bits = 8, parity = none, stop bits = 1, flow control = none. I can successfully send/get back responses to the basic commands, I (inquiry as to the type of equipment), and H (help - spits out the list of acceptable commands). But when I try to do an actual command to do some video switching, everything hangs. The front panel's rolling display (which just echos the company name) stops, then starts up again after ~20sec. The hyperterminal display doesn't change. I get neither the "DONE" answerback, which would indicate that the command executed successfully, nor do I get the "ERROR" answerback, which would indicate that something is wrong. It just hangs. If I disconnect, and restart the connection, and instead of trying a real command, but instead just send 'blahblahblah', then it will answerback 'ERROR' the first time, and then if I try to send another garbage message, everything hangs again. So, I can sort of talk to the video switch, but I can't make it do anything yet.
I'm leaving the laptop connected instead of C1AUX, since the video EPICS screen doesn't work anyway for now. If you want to start up the connection, either input the settings quoted above, or open "40m Video", which should have these connection settings saved in HyperTerminal.
The cables labeled "Gur2" which were connected to channels 2,3,4 of the PEM-ADCU have been moved to the PCIX ADC which is connected directly to the ASS machine. This means that until I (a) put the cables back or (b) figure out how to route channels from the ASS ADC to the RFM, we won't be able to use these channels for environmental monitoring, nor will they be saved.
The Gur2_X, Gur2_Y, Gur2_Z channels are now connected to the 2nd, 3rd and 4th ADC channels respectively, on the ASS ADC (the first channel / TP1 is ADC0_0, which is the 1pps signal.). The sketchy thing about the setup is the connection between the cables and the new ADC board. The PCIX card is connected to the ASS via a white ribbon cable, and the board is just sitting on top of the computer box. The 1pps (which has been hooked up for a long time) goes into the board via clip-doodles. The regular channels have a SCSI cable connector, so I used a SCSI cable to connect up the ADC tester board, and connected my seismometer inputs to this tester board via more clip doodles. Clearly this is a sketchy solution, and not okay for more than a day or so. But we'll see how it goes.
I'm going to, on the SimuLink Diagram, change the input source of these channels, from the RFMN to the ADC. Then we'll see if that fixes our timing problem, and magically makes everything relating to the OAF work, and subtract huge amounts of noise.
The on/off switch for the drill press is broken. Replacement parts should be here tomorrow.
The Ranger seismometer has been moved to ~the middle of the Mode Cleaner tube, and it's orientation has been changed to horizontal (using all of the locking/mass centering procedures). This is similar in orientation to the way things were back in the day when Rana and Matt had the OAF running nicely.
elog was acting up again (not running), so I restarted it.
And again. This makes 4 times since lunchtime yesterday....something bad is up.
Drill press is all better now. A spare switch is in the top drawer with the drill bits.
To estimate the noise floor of the Ranger, Rana and I locked the mass on the seismometer, so there will be no (aka minimal) signal from the motion of the ground in the pickup coil. We should be seeing primarily the noise of the readout electronics. We also put the Ranger on top of one of the foam lids from the Seismometer Isolation Boxes to further isolate from ground motion (this didn't change the signal noticeably).
In this plot, Green is the locked-mass-on-foam noise floor, blue is the regular spectra, with the SR560 AC coupled, and the red is the regular seismic spectra with the SR560 DC coupled. There doesn't seem to be a noticeable difference between blue and red (the spectra were taken at different times of day, with the red taken at night, when we generally expect things to be quieter). I'm leaving the SR560 DC coupled. (Rana had found it earlier this afternoon GND coupled....not sure yet why).
Also, we're not sure that the green curve is true readout noise, vs. how much of it is specifically due to the fact that the mass is locked down. Especially around a few hundred Hz, the green curve is much higher than the other 2, and at a few tens of Hz there is some weird peak action. However, this will be okay as a first-run noise estimate for the Ranger's noise floor.
The question at hand is: Do we need to redo any of the Ranger's readout electronics (i.e. replace the SR560 with a Pomona Box circuit) to lower the noise floor, or is it okay as-is?
Mostly a note to self: I have put terminators on the ADC inputs which are usually the PEM-SEIS-GUR2_(XYZ) channels. Since these 3 signals are currently going into the ASS ADC, these PEM ADC inputs are open, and have predefined channel names. I'll collect the data and put it as the ADC noise level in my nifty plot which will show the noise limits of all things which affect Wiener Filtering.
The attached plot shows the spectra of the 3 Z axes of the 3 seismometers we have (this data is from ~20Aug2009, when the Ranger was in the Z orientation) in Magenta, Cyan and Green, and the noise of each of the sensors in Red, Blue and Black. The noise curves were extracted from the spectra using the Huddle Test / 3 Corner Hat method. The Blue and Black traces which are just a few points are estimates of the noise from other spectra. The Blue points come from the Guralp Spec Sheet, and the Black comes from the noise test that Rana and I did the other day with the Ranger (elog 2223).
I'm not really happy with the black spectra - it looks way too high. I'm still investigating to see if this is a problem with my calibration/method....
I've opened the AP table and I'm working on it.
I re-aligned the Faraday on the AP table. I also aligned the beam to the periscope on the PSL and all the other optics along the beam path. Now I have a nice NPRO beam at the PLL which overlaps with the PSL beam. The alignment has to be further improved becasue I see no beat yet.
I wonder if the all the tinkering on the PSL laser done recently to revive the power have changed the PSL NPRO temperature and so its frequency. That could also explain why the beat doesn't show up at the same temperature of the NPRO as I used to operate it. Although I scanned the NPRO temperature +/- 2 deg and didn't see the beat. So maybe the misalignment is the casue.
Not feeling very well right now. I need to go home for a while.
AP table closed at the moment.
NPRO shutter closed
We definitely changed the PSL NPRO temp while fiddling around, trying to increase the laser power. I think it's noted in the elog both times that it's happened in the last few months (once when Rana, Koji and I worked on it, and then again when it was just Koji), but we opened up the side of the MOPA box so that we could get at (and change) the potentiometer which adjusts the NPRO temp. So you may have to search around for a while.
I've placed some notes pertaining to what Koji and I have learned today about getting the RCG code working on the 40m wiki at:
We're still trying to fix the tst system, as the moment its reporting an invalid number of daq channels and during daq initialization it fails. (This from the /cvs/cds/caltech/target/c1tst/log.txt file).
Dmass tells me that you have to record at least one channel. ie at least one channel in your .ini file must be set to acquire, otherwise the DAQ will flip out. It seems to be unhappy when you're not asking it to do things.
Sanjit and I are trying to put names to some signals which exist in SimuLink land, but which don't (yet) exist in EPICS land. The deelio is that for each of the chosen SEIS signals in the ASS_TOP_PEM screen, the signal is split. One part of the signal is used to decide how the adaptive filter should look, and the other part is actually used when doing the on-line subtraction. Previously only the part of the signal which is used to decide on the Adaptive Filter could be seen on the screens, and had names.
Before touching anything on the Simulink ASS.mdl, I did an svn check in, which put things at revision 36639.
To try to make the desired signals exist, I put cdsFilt boxes (to create filter modules for each of these signals), and gave each of them a name (kind of like the Neverending Story....once they have a name, they'll exist). My new names are C1:ASS-TOP_PEM_#_APPLY, which correspond to the previously-existing C1:ASS-TOP_PEM_#_ADPT (these are the ones that are along the top of the ASS_TOP_PEM matrix screen). This version of the simulink model was checked in, and the svn is now at revision 36640.
We then did some "make clean", "make ass" and "make install-ass" action, and burt restored c1assepics, but nothing seems to be happening. The screen doesn't have white boxes all over the place, and we didn't get any errors when we did the makes, and I'm sure we burt restored correctly (made sure the ASS GDS screen had a 1 in the lower left box etc), but all the values on the screen are still zero.
When we ran the ass front end in terminal on the c1ass machine, we did see an error: "Invalid chan num found 2 = 30624" "DAQ init failed -- exiting". I think this means that we need to have told some file somewhere that I was going to be adding 8 new channels. (maybe an .ini file?) Hopefully the Joe & Peter team can help us out with this, since they've been doing this kind of thing for the new system.
Moral of the story is, the new (non-working) simulink file has been svn checked in as revision 36640, and we're reverting to revision 36639, which was before I touched anything today.
As part of the fb40m restart procedure (Sanjit and I were restarting it to add some new channels so they can be read by the OAF model), I checked up on how the backup has been going. Unfortunately the answer is: not well.
Alan imparted to me all the wisdom of frame builder backups on September 28th of this year. Except for the first 2 days of something having gone wrong (which was fixed at that time), the backup script hasn't thrown any errors, and thus hasn't sent any whiny emails to me. This is seen by opening up /caltech/scripts/backup/rsync.backup.cumlog , and noticing that after October 1, 2009, all of the 'errorcodes' have been zero, i.e. no error (as opposed to 'errorcode 2' when the backup fails).
However, when you ssh to the backup server to see what .gwf files exist, the last one is at gps time 941803200, which is Nov 9 2009, 11:59:45 UTC. So, I'm not sure why no errors have been thrown, but also no backups have happened. Looking at the rsync.backup.log file, it says 'Host Key Verification Failed'. This seems like something which isn't changing the errcode, but should be, so that it can send me an email when things aren't up to snuff. On Nov 10th (the first day the backup didn't do any backing-up), there was a lot of Megatron action, and some adding of StochMon channels. If the fb was restarted for either of these things, and the backup script wasn't started, then it should have had an error, and sent me an email. Since any time the frame builder's backup script hasn't been started properly it should send an email, I'm going to go ahead and blame whoever wrote the scripts, rather than the Joe/Pete/Alberto team.
Since our new raid disk is ~28 days of local storage, we won't have lost anything on the backup server as long as the backup works tonight (or sometime in the next few days), because the backup is an rsync, so it copies anything which it hasn't already copied. Since the fb got restarted just now, hopefully whatever funny business (maybe with the .agent files???) will be gone, and the backup will work properly.
I'll check in with the frame builder again tomorrow, to make sure that it's all good.
We would like several channels from the OAF/ASS screen to be saved to frames, so that we can use the channels for our OAF model. In theory, this should involve uncommenting the desired channels in the .ini file (.../caltech/chans/daq/C1ASS.ini), and restart the frame builder. Since this .ini file was generated a long time ago, and things have been changed since then, the chnnums in the .ini file and the corresponding .par file don't match up. We need to go through the .par file (/cvs/cds/gds/param/tpchn_c3.par), and look up the chnnums for our channels, and copy those numbers into the .ini file. Figuring out what was going on involved many fb40m restarts, but on the last one of the night, I restarted the backup script, so it should (hopefully) run tonight, and get all of the frames that we've been missing.
Notes to self:
* When adding channels to other front ends, the end of the process is to click the blue button on the C0DAQ_DETAIL screen next to your computer. C1ASS isn't on that screen. Instead, in the C1ASS_GDS screen, click DAQ Reload.
* The channel names for the Test Points and the .ini files must be different. That's why there's a '_2048' suffix at the end of every channel in our .ini file.
* tpchn_C1 is all of the old-style system test points. tpchn_C2 is the C1OMC, and tpchn_C3 is for the C1ASS testpoints.
* When uncommenting channels in the C1ASS.ini file, make sure acquire is set to 1 for every channel we want saved. The default in this .ini file is set to acquire = 0.
With Koji's help, I measured the length of the Mode Cleaner.
The new modulation frequencies (as quoted on the Marconi front panels) are:
The Frequency Counter readback is 165980584.101 Hz (a 4Hz difference). All of the Marconi's front-panel frequencies read ###.##### MHz Ext, and the Frequency standard has it's "locked" light illuminated, and the 1pps input light blinking, so I think everything is still nicely locked to the frequency standard, and the frequency standard is locked to the GPS.
While changing the marconi's, I accidentally touched the MC's 29.5 MHz marconi. It is set back to the nominal value (according to Kiwamu's rack photos) of 29.485MHz. But the phase might be sketchy, although hopefully this doesn't matter since we don't do a double demodulation with it.
I also ran the scripts in the wiki page: How To/Diagonalize DRMI Length Control to set the DD Phases.
All is well again in the world of backups. We are now up to date as of ~midnight last night.
Backup Fail. At least this time however, it threw the appropriate error code, and sent me an email saying that it was unhappy. Alan said he was going to check in with Stuart regarding the confusion with the ssh-agent. (The other day, when I did a ps -ef | grep agent, there were ~5 ssh-agents running, which could have been then cause of the unsuccessful backups without telling me that they failed. The main symptom is that when I first restart all of the ssh-agent stuff, according to the directions in the Restart fb40m Procedures, I can do a test ssh over to ldas-cit, to see what frames are there. If I log out of the frame builder and log back in, then I can no longer ssh to ldas-cit without a password. This shouldn't happen....the ssh-agent is supposed to authenticate the connection so no passwords are necessary.)
I'm going to restart the backup script again, and we'll see how it goes over the long weekend.
Just felt a big "kerplunk" type ground-shaking, presumably from all the antics next door. MC2's watchdog tripped as a result. The watchdog has been reenabled.
When Alberto was parting the Red Sea this morning, and turning it green, he noticed that the wireless had gone sketchy.
When I checked it out, the ethernet light was definitely blinking, indicating that it was getting signal. So this was not the usual case of bad cable/connector which is a known problem for our wireless (one of these days we should probably relay that ethernet cable....but not today). After power cycling and replugging the ethernet cable, the light for the 2.4GHz wireless was blinking, but the 5GHz wasn't. Since the wireless still wasn't working, I checked the advanced configuration settings, as described by Yoichi's wiki page: 40m Network Page
The settings had the 5GHz disabled, while Yoichi's screenshots of his settings showed it enabled. Immediately after enabling the 5GHz, I was able to use the laptop at Alberto's length measurement setup to get online. I don't know how the 5GHz got disabled, unless that happened during the power cycle (which I doubt, since no other settings were lost), but it's all better now.
I found the FEsync light on the OMC GDS screen red. I power cycled C1OMC, and restarted the front end code and the tpman. I assume this is a remnant of the bootfest of the morning/weekend, and the omc just got forgotten earlier today.
Came back from dinner to find the Mach Zehnder unlocked. The poor IFO is kind of having a crappy day (computers, MZ, and I think the Mode Cleaner alignment might be bad too).
The Ranger was left in a place where it could be bumped during next week's activities (near the crawl-space to access the inside of the "L" of the IFO on the Yarm). It has been moved a meter or so to a safer place.
Also, so that Steve can replace the battery in the SR560 that is used for the Ranger, I swapped it out with one of the ones which already has a new, charged battery. All of the settings are identical. For posterity, I took a pic of the front panel before unplugging the old SR560.
We aligned the full IFO, and centered all of the oplevs and the IP_POS and IP_ANG QPDs. During alignment of the oplevs, the oplev servos were disabled.
Koji updated all of the screenshots of 10 suspension screens. I took a screenshot (attached) of the oplev screen and the QPD screen, since they don't have snapshot buttons.
We ran into some trouble while aligning the IFO. We tried running the regular alignment scripts from the IFO_CONFIGURE screen, but the scripts kept failing, and reporting "Data Receiving Error". We ended up aligning everything by hand, and then did some investigating of the c1lsc problem. With our hand alignment we got TRX to a little above 1, and TRY to almost .9 . SPOB got to ~1200 in PRM mode, and REFL166Q got high while in DRM (I don't remember the number). We also saw a momentary lock of the full initerferometer: On the camera view we saw that Yarm locked by itself momentarily, and at that same time TRX was above 0.5 - so both arms were locked simultaneously. We accepted this alignment as "good", and aligned all of the oplevs and QPDs.
It seems that C1LSC's front end code runs fine, and that it sees the RFM network, and the RFM sees it, but when we start running the front end code, the ethernet connection goes away. That is, we can ping or ssh c1lsc, but once the front end code starts, those functions no longer work. During these investigations, We once pushed the physical reset button on c1lsc, and once keyed the whole crate. We also did a couple rounds of hitting the reset button on the DAQ_RFMnetwork screen.
Taking a cue from entry 2346, I immediately went for the nuclear option and powered off fb40m. Someone will probably need to restart the backup script.
Backup script restarted.
[Koji, Jenne, Alberto, Steve, Bob]
ETMX has been drag wiped.
Around 2:45pm, after the main IFO volume had come up to atmospheric pressure, we removed both doors to the ETMX chamber. Regular procedures (wiping of O-rings with a dry, lint-free cloth, covering them with the light O-ring covers, etc.) were followed. Koji took several photos of the optic, and the rest of the ETMX chamber before anything was touched. These will be posted to the 40m Picasa page. Steve and Koji then deionized the optic.
Koji removed the bottom front earthquake stop, and clamped the optic with the remaining earthquake stops.
The clean syringes were prepared: These are all glass and metal (nothing else) medical syringes. The size used was 100microliters. Earlier today, we had prepared our solvents in small little beakers which had been baked over the weekend. Brand new glass bottles of Acetone and Isopropyl Alcohol were opened, and poured into the small beakers. To make sure we have enough, we have 3 ~10ml beakers of each Acetone and Isopropyl.
We started with Acetone. The syringe was filled completely with acetone, then squirted onto a kimwipe. This was repeated ~twice, to ensure the syringe was well rinsed. Then the syringe was filled a little past the 100 microliter mark. Koji held a piece of lens cleaning paper to ETMX and used an allen wrench underneath the optic to help guide the paper, and keep it near the optic (of course, the only thing in actual contact with the optic was the lens paper). In one smooth shot, the plunger of the syringe was pressed all the way down. (This is a bit tricky, especially when the syringe is totally full. You have to squeeze it so the plunger moves fairly quickly down the barrel of the syringe to get a good arc of liquid. The goal is to shoot all of the solvent to the same place on the lens paper, so that it makes a little circle of wetness on the paper which covers the coated part of the optic. The amount of solvent used should be balanced between having too little, so that the paper is dry by the time it has been wiped all the way down, and too much such that there is still a residue of liquid on the optic after the paper has been removed.) The target was to hit the optic just above the center mark (the oplev was on, so I went for just above the red oplev dot). Immediately after applying the liquid onto the paper, Koji slowly and smoothly pulled down on the lens paper until it came off of the bottom of the optic. The acetone was repeated, for a total of 2 acetone wipes. Because acetone evaporates very quickly, more acetone is used than isopropyl. The optimal amount turned out to be ~115 microliters of acetone. It is hard to say exactly how much I had on the second wipe, because the syringe is not marked past 100 microliters. On the first wipe, with about 105 microliters, the lens paper was too dry at the bottom of the optic.
We then switched to Isopropyl. A new syringe was used, and again we rinsed it by filling it completely with isopropyl, and emptying it onto a kimwipe. This was repeated at least twice. We followed the same procedure for applying liquid to the optic and wiping the optic with the lens paper. On the first try with isopropyl, we used 100 microliters, since that was the preferred amount for acetone. Since isopropyl evaporates much slower than acetone, this was determined to be too much liquid. On the second isopropyl wipe, I filled the syringe to 50 microliters, which was just about perfect. The isopropyl wiping was done a total of 2 times.
After wiping, we replaced the front bottom earthquake stop, and released the optic from the other earthquake stops' clamping. The OSEM values were checked against the values from the screenshots taken yesterday afternoon, and were found to be consistent. Koji took more photos, all of which will be placed on the 40m Picasa page.
We visually inspected the optic, and we couldn't see anything on the optical surface of the mirror. Koji said that he saw a few particulates on some horizontal surfaces in the chamber. Since the optic seemed (at least to the level of human vision without a strong, focused light) to be free of particulates on the optical surface to start with, the suspense will have to remain until we button down, pump down, and try to lock the IFO to determine our new finesse, to see if the wiping helped any substantial amount.
We replaced the regular, heavy door on the inner side of the ETMX chamber (the side closer to the CES building), and put only a light door on the outer side of the chamber (the side closer to the regular walkway down the arm). We will look at the spectra of the OSEMS tomorrow, to confirm that none of the magnets are stuck.
We commence at ~9am tomorrow with ETMY.
* The LED lights are awesome. It's easy to use several lights to get lots of brightness (more than we've had in the past), and the chamber doesn't get hot.
* We should get larger syringes for the acetone for the large optics. It's challenging to smoothly operate the plunger of the syringe while it's so far out. We should get 200 microliter syringes, so that for the acetone we only fill them about half way. It was noticeably easier to apply the isopropyl when the syringe only had 50 microliters.
* It may be helpful to have a strong, focused optical light to inspect the surface of the mirror. Rana says that Garilynn might have such an optical fiber light that we could borrow.
Opening of ETMY has been put on hold to deal with the computer situation. Currently all front end computers are down. The DAQ AWGs are flashing green, but everything else is red (fb40m is also green). Anyhow, we'll deal with this, and open ETMY as soon as we can.
The computers take priority because we need them to tell us how the optics are doing while we're in the chambers, fitzing around. We need to be sure we're not overly kicking up the suspensions.
Jenne, Kiwamu, Koji, Alberto, Steve, Bob
ITMX was wiped without having to move it.
After 'practice' this morning on ETMY, Kiwamu and I successfully wiped ITMX by leaning into the chamber to get at the front face.
Most notable (other than the not moving it) was that inspection with the fiber light before touching showed many very small particles on the coated part of the optic (this is versus ETMY, where we saw very few, but larger particles). The after-wiping fiber light inspection showed many, many fewer particles on the optical surface. I have high hopes for lower optical loss here!
All the front ends are back up.
I found all the front-ends, except for C1SUSVME1 and C0DCU1 down this morning. DAQAWG shows up green on the C0DAQ_DETAIL screen but it is on a "bad" satus.
I'll go for a big boot fest.
I burtrestored all the snapshots to Dec 9 2009 at 18:00.
The frame builder was power cycled during the morning bootfest. I have restarted the backup script once more.
[Everybody: Alberto, Kiwamu, Joe, Koji, Steve, Bob, Jenne]
The last heavy door was put on after lunch. We're now ready to pump.
last 20 days - including the pounding from next door
[Jenne, Kiwamu, Koji]
We got the IFO back up and running! After all of our aligning, we even managed to get both arms locked simultaneously. Basically, we are awesome.
This morning, we did the following:
* Turned on the PZT High voltages for both the steering mirrors and the OMC. (For the steering mirrors, turn on the power, then hit "close loop" on each. For the OMC, hit Output ON/OFF).
* Looked at the PZT strain gauges, to confirm that the PZTs came back to where they had been. (Look at the snapshot of C1ASC_PZT_Al)
* Locked all components of the PSL (This had already been done.)
* Removed beam dump which was blocking the PSL, and opened the PSL mechanical shutter. Light into the IFO!
* Locked the Mode Cleaner. The auto-locker handled this with no problem.
* Confirm that light is going through the Faraday. (Look at the TV sitting on top of MC13 tank...it shows the Faraday, and we're hitting the input of the Faraday pretty much dead-on).
* Look at IP_ANG and IP_POS. Adjust the steering mirrors slightly to zero the X&Y readings on IP_ANG. This did not change the PZTs by very much, so that's good.
* Align all of the Core Optics to their OpLev positions.
* On the IFO_Align screen, save these positions.
* Run the IFO_Configure scripts, in the usual order. (Xarm, Yarm, PRM, DRM). Save the appropriate optics' positions after running the alignment scripts. We ended up running each alignment script twice, because there was some residual misalignment after the first iteration, which we could see in the signal as viewed on DataViewer (Either TRX, TRY, or SPOB, for those respective DoFs).
* Restore Full IFO.
* Watch the beauty of both arms and the central cavity snapping together all by themselves! In the attached screenshot, notice that TRX and TRY are both ~0.5, and SPOB and AS166Q are high. Yay!
* The wiping may have helped. While aligning X and Y separately, TRX got as high as ~1.08, and TRY got as high as 0.98 This seems to be a little bit higher than it was previously.
* Since everything locked up in pretty short order, and the free swinging spectra (as measured by Kiwamu in elog 2405) looks good, we didn't break anything while we were in the chambers last week. Excellent.
* We are now ready for a finesse measurement to tell us more quantitatively how we did with the wiping last week.
I ran the armLoss script for both Xarm and Yarm. The results are confidential, pending the completion of Alberto's cavity pole/finesse measurement due to the 'bet' as to what the new losses are after the drag wiping.
If you're the kind of person who likes to look at their Chrismas presents, the log files with the results are in the usual place for this script: /scripts/LSC/loss-ARM-GPStime.log (loss-Y-944865071.log and loss-X-944865946.log)