I tested the amplifier with the Agilent network analyzer and measured 19.5 dB of gain between 29 and 30 mHz. The phase only changed by 1 degree over this same 1 MHz span. Since everything seems to be in order I'll hook it up this afternoon, unless there are any objections.
A few minutes back, I glanced up at the control room StripTool and noticed that the MCREFL PD DC level had gone up from ~0 to ~0.7, even though the PSL shutter was closed. This seemed bizzare to me. Strangely, simply cycling the shutter returned the value to the expected value of 0. I wonder if this is just a CDS problem to do with c1iool0 or c1psl? (both seem to be responding to telnet though...)
Since things look to be back to normal, I am going to start with my characterization of the various TFs in the IMC FSS loop...
It is more accurate to model the physical frequency noises at various places.
cf. See also 40m ALS paper or Shigeo Nagano PDH thesis on https://wiki-40m.ligo.caltech.edu/40m_Library
- The output 4 should be "Laser frequency"
- Seismic path should be excluded from the summing node
- The output after the PMC: "Laser frequency after the PMC"
- "Laser frequency after the PMC" is compared (diffed) with the output 1 "mirror motion in Hz"
- The comparator output goes to the cav pole, the PD, and the PDH gain: This is the output named "PDH Error"
- Tap a new path from "Laser frequency after the PMC" and multiply with the cav pole (C_IMC)
- Tap a new path from "Mirror motion" and multiply with the cavity high pass (s C_IMC/omega)
- Add these two: This is the output named "Frequency noise transmitted by IMC"
I've edited Rana's Simulink model to reflect the current IMC servo topology (to the best of my understanding). I've tried to use Transfer Function blocks wherever possible so that we can just put in the appropriate zpk model in the script that will linearize the whole loop. I've also omitted the FSS SLOW loop for now.
I've been looking through some old elogs and it looks like there have been several modifications to both the MC servo board (D040180) and the TT FSS Box (D040105). I think it is easiest just to measure these TFs since the IMC is still down, so I will set about doing that today. There is also a Pomona Box between the broadband EOM and the output of the TT FSS box, which is meant to sum in the modulation for PMC locking, about which I have not yet found anything on the elog.
So the next steps are:
If anyone sees something wrong with this topology, please let me know so that I can make the required changes.
Had to reboot c1psl, c1susaux, c1auxex, c1auxey and c1iscaux today. PMC has been relocked. ITMX didn't get stuck. According to this thread, there have been two instances in the last 10 days in which c1psl and c1susaux have failed. Since we seem to be doing this often lately, I've made a little script that uses the netcat utility to check which slow machines respond to telnet, it is located at /opt/rtcds/caltech/c1/scripts/cds/testSlowMachines.bash.
The script can be executed by ./testSlowMachines.bash.
The bottom 5 cable connections from Sat-Amp to Whittering Filter at 1X5 were clamped today.
Since the "stablizer box" doesn't really need to stabilize, it just needs to amplify, I decided to replace it with an off the shelf amplifier we already had, ZHL-2. I worked on getting it set up today, but didn't connect anything so that people have a chance to give some feedback.
So, I think the remaining thing to do is to connect the splitter to ASC out and to the line to the EOM, the +24V supply to the amplifier, and the 29.5 MHz input to the attenuator. I wanted to wait on this to get confiration that the setup is OK. Eventually we can put all of this in a box.
Also, I noticed that in the clear cabinet with the Sorensens next to this rack, the +24 V unit is not supplying any voltage and has a red light that says "OVP."
No, not confused on that point. We just will not be testing OS versions at the 40m or running multiple OS's on our workstations. As I've said before, we will only move to so-called 'reference' systems once they've been in use for a long time.
Ubuntu16 is not to my knowledge used for any CDS system anywhere. I'm not sure how you expect to have better support for that. There are no pre-compiled packages of any kind available for Ubuntu16. Good luck, you big smelly doofuses. Nyah, nyah, nyah.
True - its an issue. Koji and I are updating zita into Ubuntu16 LTS. If it looks like its OK with various tools we'll swap over the others into it. Until then I figure we're best off turning allegra back into Ubuntu12 to avoid a repeat of this kind of conflict. Once the workstations in the LLO control room are running smoothly on a new OS for a year, we can transfer into that. I don't think any of us wants to be the CDS beta tester for DV or DTT.
Just to be clear, since there seems to be some confusion, the SVN issue has nothing to do with Debian vs. Ubuntu. SVN made non-backwards compatible changes to their working copy data format that breaks newer checkouts with older clients. You will run into the exact same problem with newer Ubuntu versions.
I recommend the 40m start moving towards the reference operating systems (Debian 8 or SL7) as that's where CDS is moving. By moving to newer Ubuntu versions you're moving away from CDS support, not towards it.
I was able to bring back svn 1.6 formatting to /cvs/cds/caltech/chans by doing the following on nodus:
svn co https://nodus.ligo.caltech.edu:30889/svn/trunk/chans ./
rm -rf ../chans/.svn
mv ./.svn ../chans/
Note that I used the http address for the repository. The svn repository doesn't live at file:///cvs/cds/caltech/svn anymore; all of our checkouts (e.g. in the scripts directory) use http to get the one true repo location, regardless of where it lives on nodus' filesystem. (I suppose we could also use https://nodus.martian:30889/svn to stick to the local network, but I don't think we're that limited by the caltech network speed)
Presumably, at some point we will want to introduce a newer operating system into the 40m, as ubuntu 12.04 hits end-of-life in April 2017. Ubuntu 16.04 includes svn 1.8, so we'll also hit this issue if we choose that OS.
Aside from the svn issues, this directory (/cvs/cds/caltech/chans) only contains pre-2010 channels. Filters and DAQ ini files currently live in /opt/rtcds/caltech/c1/chans, which is not under version control. It's also not clear to me why summary page configurations should be kept in this /cvs/cds place.
In working on automatic DARM loop design, we have this code:
the things in there like mkCost*, etc. have examples of the cost functions that are used. It may be useful to look at those and then make a similar cost function calculation for the MCL/MCF loop.
More testing of fb1 today. DAQ DOWN UNTIL FURTHER NOTICE.
Testing Wednesday did not resolve anything, but Jonathan Hanks is helping.
Rana motivated me to take a step back and reframe the objectives and approach for this project, so I am collecting some thoughts here on my understanding of it. As I write this, some things still remain unclear to me, so I am leaving these as questions here for me to think about...
and come up with the best loop that meets all our rquirements? What constitutes the "best" loop? How do we weight the relative importance of our various requirements?
For the specific problem of making the MCL feedback loop better, the approach I have in mind right now is the following:
My immediate goal is to have the Simulink model updated.
Thoughts/comments on the above will be appreciated...
Re-aligned the beam going into the PMC today around 5 PM. I noticed that its all in pitch and since I moved both of the mirrors by the same amount it is essentially a vertical translation.
I wonder if the PMC is just moving up and down due to thermal expansion in the mount? How else would we get a pure vertical translation? Need to remember next time if the beam goes up or down, and by how many knob turns, and see how it correlates to the lab temperature.
and the song remains the same...
the version of SVN on these workstations is ahead of the one on the other workstations so now we can't do 'svn up' on any of the Ubuntu12 machines. One allegra and optimus I get this error:
controls@allegra|GWsummaries> svn up
svn: E180001: Unable to connect to a repository at URL 'file:///cvs/cds/caltech/svn/trunk/GWsummaries'
svn: E180001: Unable to open an ra_local session to URL
svn: E180001: Unable to open repository 'file:///cvs/cds/caltech/svn/trunk/GWsummaries'
Someone installed "Debian" on allegra. Why? Dataviewer doesn't work on there. Is there some advantage to making this thing have a different OS than the others? Any objections to going back to Ubuntu12?
My elog negligence punchcard is getting pretty full... It's pretty much for the same reason as using Debian for optimus; much of the workstation software is getting packaged for Debian, which could offload our need for setting things up in a custom 40m way. Hacking the debian-focused software.ligo.org repos into Ubuntu has caused me headaches in the past. Allegra wasn't being used often, so I figured it was a good test bed for trying things out.
The dataviewer issue was dataviewer's inability to pull the `fb` out of `fb:8088` in the NDSSERVER env variable. I made a quick fix for it in the dataviewer launching script, but there is probably a better way to do it.
I'm not sure if its possible to downgrade our chans repo back to the old one, but I highly recommend that no one do 'svn upgrade' in any of our repos until we remove all of the Debian installs in the 40m lab or hire a full-time sysadmin.
I had noticed something wonky with the microphone, but neglected to elog it. I had tested it after installation by playing a sine wave from my laptop and looking at the signal on the PSL table, it worked fine. But you can see in the attached minute trend plot that the signal characteristics changed abruptly ~half a day after installation, and never quite recovered.\
I don't know if anyone looked at the time series (not trend) or spectrum of the Microphone after installation, but it looks bad and featureless to me. Is the Microphone broken?
This shows the spectrum from early this morning and again from tonight. You can see that it is bi-stable in its noise properties. This thing is busted; we're now removing it from the PSL so that it doesn' light it self on fire.
This AmScope microscope would have 3.5x-180x magnification, calibratable measurement function, 5MP picture and good working distance to work on printed circuit boards.
FYI this issue has still not been solved, but the pages are working because I got the software running on an
alternative headnode (pcdev2). This may cause unexpected behavior (or not).
> LDAS has not recovered from maintenance causing the pages to remain unavailable until further notice.
> > System-wide CIT LDAS cluster maintenance may cause disruptions to summary pages today.
I made some of the changes. Gautam and I will finish tomorrow.
While I was soldering the sharpest tip of the soldering iron (the one whose power supply shows the temperature) stopped working and I switched to a different one. Not sure how to fix this.
Do we want to replace all of the removed ERA's with 50 Ohm resistors, or just the one along the spare output path? I shorted one of them with a piece of wire and left all the others open.
I couldn't get one of the attenuators off (AT1, at beginning of ASC path). In trying I messed up the solder pad. Part of the connecting trace on the PCB board is exposed so we should be able to fix it.
I think this then allows us to have the low noise OCXO signals everywhere with enough oomph.
Here's what I'm planning to do to the RF AM stabilizer box. I'm going to take out several of the components along the path to the EOM (comments in green), including the dead ERA-4 and ERA-5 amplifiers, the variable attenuator which is controlled by a switch that can't be accessed outside the box, and the feedback path from the daughter board servo. I'm arranging things so that the output of the HELA-10 does not exceed the maximum output power.
I wasn't quite as sure what to do about the path to the ASC box (comments in blue). I talked with Gautam and he said this gets split equally between several singals, one of which goes to the LO of the demod board which expects -10 dBm and currently gets -12 dBm (can go up to -8 by turning switch). So maybe we don't actually want the signal to be anywhere near +27 dBm at the output. The plans for the box are here, it looks like +27 in will end up with +10 at each output, which is way more than what's currently coming out. But maybe this needs to be increased to match the other path?
Also we haven't measured the actual response of the variable attenuator U4 for various switch positions; it's the same model as the one I'm removing from the EOM path and that one had slightly different behavior for different switch positions than what the spec sheet says. Same goes for the HELA-10 units along this path: what is their actual gain? So perhaps these should be measured and then a single attenuator should be chosen to get the right output signal level. Alternatively it could just be left alone, if it is at an OK level right now. Advice on what to do here would be appreciated.
I'll work on the EOM path tonight and wait for feedback on the rest of it.
EDIT: Gautam pointed out that there's some insertion loss from the components I'll be removing that hasn't been accounted for. Also the plans have been updated to reflect that I'm replacing AT5 with a 1dB attenuator (from 6 dB).
For a comparison: OMC ELOG 238
[rana, gautam, lydia]
Today we looked at the schematics for the RF AM stabilizer box and decided that there were an unnecessary amount of attenuators and amplifiers cancelling each other out and adding noise. At the end of the path are 2 HELA-10D amplifiers which we guessed based on the plots for the B version would have an acceptable amount of compression if the output of the second one is ~27dBm. This means the input to the first one should be a few dBm. This should be achieved with as simple a path as possible.
This begged the question, do we need the amplitude to be stabilized at all? Maybe it's good enough already when it comes into this box from the RF distribution box. So I tried to measure the AM noise of the 29.5 MHz signal that usually goes into the AM stabilizer:
It seems like I'm getting mostly noise from the SR560. Maybe it would be better to use an SR785 to take data instead of DAQ, and then skip the SR560? At low frequencies it seems like the AM noise measurement may be actually meaningful. In any case, if the actual AM noise from the crystal is lower than any of these other noise sources, it means we probably don't need to stabilize the amplitude with a servo, which means we can simplify the AM stabilizer board considerably to just amplify what it gets to 27 dBm.
I made a crude sketch for how Lydia and I envision the connector situation on the back of the vme crates to be solved. Essentially the side panels of each crate extend about 2" (52 mm) beyond the edge of the DIN connectors. This is plenty of space for a simple PCB board. The connector of choice is D-Sub. We can split the 64 used pins into 2x 37 D-Sub OR (2x25 pin + 1x15pin). The former has fewer cables, but a few excess unused leads. A quick google search showed me that it is much cheaper to get twisted pair cables for 15 and 25 pin D-Subs. From what I remember, the used pins on the DIN connectors are concentrated on the low numbers end and the high numbers end, so might not need the 'middle' connector in many cases if we decide to break it up into three. I have to check this with Lydia though.
The D-Sub connectors would be panel mounted, for which we need a narrow panel piece with dsub cutouts. We can run horizontal struts across the vme crate from side panel to side panel. This way the force upon cable (dis)connection is mostly on the panel which is attached to the struts which are attached to the crate. This will also prevent gravitational sag or cable strain from pulling on the DIN connection, and we can use twisted pair cables with backshell, screws, and strain reliefs.
I was lookng into getting started with the PCB when Altium complained that the license is expired and to renew it. This is a relatively simple board layout so some free software out there is probably enough.
I've added the schematic of the RF AM stabilization board to the 40m PSL document tree, after having created a new DCC document for our 40m edits. Pictures of the board before and after modification will also be uploaded here...
I've been suggesting that there may be something wonky with the Seismic Rainbow Striptool on the wall for the last couple of weeks. Here are a few things that were verified today.
A novice was learning at the feet of Master Daqd. At the end of the lesson he looked through his notes and said, “Master, I have a few questions. May I ask them?”
Master Daqd nodded.
"Do we record minute trends of our data?"
"Yes, we record raw minute trends in /frames/trend/minute_raw"
"I see. Do we back up minute trends?"
"Yes, we back up all frames present in /frames/trend/minute"
"Wait, this means we are not recording our current trends! What is the reason for the existence of seperate minute and minute_raw trends?
“The knowledge you seek can be answered only by the gods.”
"Can we resume recording the minute trends?"
Master Daqd nodded, turned, and threw himself off the railing, falling to his death on the rocks below.
Upon seeing this, the novice was enlightened. He proceeded to investigate how to convert raw minute trends to minute trends so that historical records could be preserved, and precisely when Master Daqd started throwing himself off the mountain when asked to record minute trends.
> What is the probe situation? Ought to use a high impedance FET probe to measure this or else the scope would load the circuit.
We did indeed use the active probe, with the 100:1 attenuator in place. The values Lydia has quoted have 40dB added to account for this.
> What kind of HELA are the HELA amplifiers? Please a link to the data sheet if you can find it. I wonder what the gain and NF are at 30 MHz. I think the HELA-10D should be a good variant
The HELA is marked as HELA-10. It doesn't have the '+' suffix but according to the datasheet, it seems like it is just not RoHS compliant. It isn't indicated which of the varieties (A-D) is used either on the schematic or the IC, only B and D are 50ohms. For all of them, the typical gain is 11-12dB, and NF of 3.5dB.
I think this cron job is running on NODUS (our gateway) instead of our scripts machine:
*/1 * * * * /opt/rtcds/caltech/c1/scripts/Admin/n2Check.sh >> /opt/rtcds/caltech/c1/scripts/Admin/n2Check.log 2>&1
Based on Jenne's chiara disk usage monitoring script, I made a script that checks the N2 pressure, which will send an email to myself, Jenne, Rana, Koji, and Steve, should the pressure fall below 60psi. I also updated the chiara disk checking script to work on the new Nodus setup. I tested the two, only emailing myself, and they appear to work as expected.
The scripts are committed to the svn. Nodus' crontab now includes these two scripts, as well as the crontab backup script. (It occurs to me that the crontab backup script could be a little smarter, only backing it up if a change is made, but the archive is only a few MB, so it's probably not so important...)
moreover this script has a 90MB log file full of not finding its channel
I wish this script was in python instead of BASH and I wish it would run on megatron instead of nodus (why can't megatron send us email too?) and I wish that this log file would get wiped out once in awhile. Currently its been spitting out errors since at least a month ago:
Tue Jan 31 14:10:02 PST 2017 : N2 Pressure:
Channel connect timed out: 'C1:Vac-N2pres' not found.
(standard_in) 1: syntax error
We looked at the RF AM stabilizer box to see if we could find out 1) Why the output power is so low, and 2) Why it can't be changed with the DC input "MOD CONT IN." Details to follow, attached is the annotated schematic from DCC document D000037.
We are not returning the box tonight so the PSL shutter remains closed.
We pulled out the RF AM stabilization box from the 1X2 rack. PSL shutter was closed, marconi output, RF distribution box and RF AM stabilization box were turned off in that order. We had to remove the 4 rack nut screws on the RF distribution box because of the stiff cables which prevented the RF AM stabilization box extraction. I've left the marconi output and the RF distribution boxes off, and have terminated all open SMA connections with 50 ohm terminators just in case. Rack nuts for RF distribution box have been removed, it is currently sitting on a metal plate that is itself screwed onto the rack. I deemed this a stable enough ledge for the box to sit on in the short run, while we debug the RF AM stabilization box. We will work on the debugging and re-install the box as soon as we are done...
I just aborted the fb1 test and reverted everything to the nominal configuration. Everything looks to be operating nominally. Front ends are mostly green except for c1rfm and c1asx which are currently not being acquired by the DAQ, and an unknown IPC error with c1daf. Please let me know if any unusual problems are encountered.
The behavior of daqd on fb1 with the latest release (3.2.1) was not improved. After turning on the full pipe it was back to crashing every 10 minutes or so when the full and second trend frames were being written out. lame. back to the drawing board...
I'm not sure if this is related, but since today morning, I've noticed that the data concentrator errors have returned. Looking at daqd.log, there is a 1 second timing mismatch error that is being generated. Usually, manually running ntpdate on the front ends fixes this problem, but it did not work today.
If this problem started before ~4pm on Friday then it's probably unrelated, since I didn't start any of these tests until after that. If unexplained problem persist then we can try shutting of the fb1 daqd and see if that helps.
Some more details of our investigation:
We set out to measure the 29.5 MHz power going to the EOM today but decided to start by looking at the output of the RF AM stabilizer box first. We wanted to measure the AM noise with a mixer, so we needed to know the power it was giving. We looked at the ouput that goes to the power combiner on the PSL table and found it was putting out only -2.0 dBm (~0.5 Vpp)! This was measured by taking a spectrum with the AG4395 and confirmed by looking on a scope.
To find out if this could be adjusted, we found an old MEDM screen (/opt/rtcds/caltech/c1/medm/c1lsc/master/C1LSC_RFADJUST.adl) and moved the 29.5 MHz EOM Mod Index Adjust slider while measuring the voltage coming in to the MOD CONTOL connection on the front of the AM stabilizer box. Moving the slider from 0 to 10 changes the input voltage linearly from -10 V to 10 V measured with a DMM at the cross-connects as we couldn't find an appropriate adapter for the LEMO cable. The 29.5 MHz modulation only appeared for slider values between 0 and 5, after which it abruptly shuts off. However, changing the slider value between 0 and 5 (Voltage from -10 to 0) does not change the amplitude of the output.
This seems like a problem; further investigation into the AM stabilizer box is neccessary. This DCC document outlines how to test the box, but we can't find a schematic. Since we don't have any mixers that can handle signals as small as -2 dBm, we gave up trying to measure the AM noise and will attempt to measure that and the reflection power from the EOM + resonant circuit once this problem has been diagnosed and fixed.
GV: After some digging, I found the schematic for the RF AM stabilization box (updated wiki and added it to the 40m document tree). According to it, there should be up to +22dBm of RF AM stabilized output to the EOM available, though we measured -2dBm yesterday, and could not vary this level by adjusting the EPICS voltage value. Neglecting losses in the cabling and the power combiner on the PSL, this translates to a paltry 0.178Vrms*0.6*8mard/Vrms ~ 0.85 mrad of modulation depth (gain at 29.5 MHz of the triple resonant circuit taken from this elog)... I think we need to pull this 1U chassis out and debug more thoroughly...
The coil and PD BLRMS are useful tools in identifying when glitches occur in the PD readout, I thought it would be good to install them for ITMY, ETMX and SRM (since I plan to switch the MC3 satellite box, which we suspect to be problematic, with the SRM one). For this purpose, I had to install some IPC SHMEM blocks in C1SUS and recompile. 24 IPC channels were added to pipe the coil, PD and Oplev signals from C1SUS to C1PEM - the recompilation went smoothly, and it doesn't look like the model computation time has increased significantly or that the model is any closer to timing out.
However, I was unable to install the BLRMS blocks in C1PEM, as when I tried to compile the model with BLRMS for these extra 24 channels, I got a compilation error saying that I have exceeded the maximum allowed 499 testpoints per channel. Is there any workaround to this? It would be possible to create a custom BLRMS block that doesn't have all those testpoints, maybe this is the way to go? Especially if we want to install these channels for all our SOS optics, and also replace the current Seismic BLRMS with this scheme for consistency?
GV edit: I have implemented this scheme - after backing up the original BLRMS_2k part, I made a new one with no testpoints and only EPICS readouts. Doing so allowed me to recompile c1pem without any issues, the CPU time seems to have gone up by 3us from ~55us to 58us. So the BLRMS data record is only available at 16Hz, since there are no DQ channels in the BRLMS block - do we want these in any case? Let's see how this does over the weekend...
"Why does the word wrapping not work in our browsers with ELOG?" I sometimes wonder. Some of the elogs are fine, but often the 40m one has the text run off the page.
I found that this is due to people uploading HUGE images. If you need to do this, just use the shrink feature in the elog compose window so that we only have to see the thumbnail at first. Otherwise your 12 MP images will make it hard to read everyone else's entries.
Just FYI I'm running a test of updated daqd code on fb1.
fb1 has it's own fiber to the daq network switch, so nothing had to be modified to do this test. This *should* not affect anything in the rest of the system, but as we all know these are famous last words.... If something is going haywire, and you can't get in touch with me and can't figure what else to do, you can just log on to fb1 and shut it down. It's not writing any data to any of the network filesystems.
The daqd code under test is from the latest advLigoRTS 3.2.1 tag, which has daqd stability fixes that will hopefully address the problems we were seeing last time I tried this upgrade. We'll see...
I'm going to let it run over the weekend, and will check in periodically.
Rebooted c1iscaux, c1auxex and c1auxey which were all not reponding to telnet. The watchdogs for the ETMs were turned off and then I keyed all 3 crates. All slow machines are reponding to telnet now. Both green lasers locked to the arms so I didn't do any burt restore.
Very nice! I got excited.
The 3 pieces of Sapphire v-groove test cuts are back. They look good. The suspension wire 0.0017" ( 43 micron ) is show on some of the pictures.
It was raised at the Wednesday meeting that I did not check the RF pickup levels while measuring the RF error signal levels into the Demod board. So I closed the PSL shutter, and re-did the measurement with the same measurement scheme. The detailed power levels (with no light incident on the WFS, so all RF pickup) is reported in the table below.
These numbers can be subtracted from the corresponding columns in the previous elog to get a more accurate estimate of the true RF error signal levels. Note that the abnormal behaviour of Quadrant #2 on both WFS demod boards persists.
Just collecting some links from my elog searching today here for easy reference later.
I couldn't find any details of the actual measurement technique, though perhaps I just didn't look for the right keywords. But Koji's suggestion of measuring powers with the bi-directional coupler before the triple resonant circuit (but after the power combiner) should be straightforward.