c1iool0 was down again. Rather than key the crate, this time I just pushed the reset button on the front and it came back.
As move towards the wonderfulness of AcroMag, we also have to buy a computer to handle all of these IOCs. Let's install the new c1iool0 over by the SUS computer.
To install the replacement amplifier, I did the following:
Still to be done:
Lydia finished up installing the new RF amplifier, and will elog the details of the installation.
I wanted to try and measure the IMC OLG to compare against my Simulink model. So I went about performing a few checks. Summary of my findings:
TBC tomorrow, I'm leaving the PSL shutter closed and the RF source off for tonight...
Here are the details as promised.
Attachment #1: Updated simulink model. Since I haven't actually run this model, all the TF blocks are annotated "???", but I will post an updated version once I have run the model (and fix some of the questionable aesthetic choices)
Attachment #2: Measured and fitted transfer functions from the "IN1" input (where the demodulated MC REFL goes) to the "SERVO" output of the MC servo board (to FSS box). As mentioned in my previous elog, I had to put in a pole (fitted to be at ~2MHz, called pole 9 in the plot) in order to get good agreement between fit an measurement up to 10MHz. I didn't bother fitting all the high frequency features. Both gain sliders on the MEDM screen ("IN1 Gain" and "VCO gain") were set to 0dB for this measurement, while the super boosts were all OFF.
Attachment #3: Measured and fitted transfer function from "TEST 1 IN" to "FAST OUT" of the FSS box. Both gains on the FSS MEDM screen ("Common gain adjust" and "fast gain adjust") were set to 0dB for this measurement. I didn't need any ad-hoc poles and zeros for this fit (i.e. I can map all the fitted poles and zeros to the schematic), but the fit starts to deviate from the measurement just below 1 MHz.. perhaps I need to add a zero above 1MHz, but I can't see why from the schematic...
Attachment #4: Measured TF from "TEST 1 IN" to "PC OUT" on the FSS box. MEDM gains were once again 0dB. I can't get a good fit to this, mainly because I can't decipher the poles and zeros for this path from the schematic (there are actually deviations from the schematic posted on the 40m DCC page in terms of component values, I will try and correct whatever I notice) . I'll work on this...
Attachment #5: Data files + .fil files used to fit the data with LISO
Data + plots + fits + updated schematics to follow...
Most of the model has come together, I am not too far from matching the modelled OLG to the measured OLG. So I will now start thinking about designing the controller for the MCL part (there are a couple of TFs that have to be measured for this path).
I'd like to fix a few things at 1X1 when we plug in the new amplifier for the 29.5MHz modulation signal.
Steve has ordered rolls of pre-twisted wire to run from 1X1 to the PSL table, so that part can be handled later.
But at 1X1, we need to tap new paths from +/- 24V to the DIN connectors. I think it's probably fine to turn off the two Sorensens, do the wiring, and then turn them back on, but is there any procedure for how this should be done?
http://www.amscope.com/3-5x-180x-boom-stand-trinocular-zoom-stereo-microscope-with-144-led-ring-light-and-10mp-camera.html will be ordered today.
The actual unit we are getting has lockable zoom for better repeatability after calibration: SM-3NTPZZ-144
Quick summary elog, details to follow. I did the following:
The measurements I have look reasonable. But I had a hard time trying to look at the schematic and determine what is the appropriate number and locations of poles/zeros with which to fit the measured transfer function. Koji and I spent some time trying to go through the MC Servo board schematic, but looks like the version uploaded on the 40m DCC tree doesn't have changes made to it reflected (we compared to pictures on the 40m google photos page and saw a number of component values were different). Since the deviation between fit and measurement only occurs above 1MHz (while using poles/zeros inferred from the schematic), we decided against pulling out the servo board and investigating further - but this should be done at the next opportunity. I've marked the changes we caught on a schematic and will upload it to the 40m DCC page, and we can update this when we get the chance.
So it remains to fit the other two measured TFs, and add them to the Simulink model. Then the only unknown will be the PDH discriminant, which we anyway want to characterize given that we will soon have much more modulation.
The ITMX oplev beam is clipping. It will be corected with locked arm
After fighting with Altium for what seems like an eternity I have finished putting my vision of the vme crate backplane adapter board into an electronic format. It is dimensioned to fill the back space of the crate exactly. The connectors are panel mount and the PCB attaches to the connectors with screws, such that the whole thing will be mechanically much more stable than the current configuration. A mounting bracket will attach to horizontal struts that need to be installed in the crates, mechanical drawings to follow.
I set everything up and connected it as shown on the block diagram attached to the previous entry, with the exception of the DC power. This is becuase there is no place open to connect to on the DIN rail where the DC power is distributed, so the +24V power will have to be shut off to the other equipment in 1X1 before we can connect the amplifier. (The amplifier is in 1X2, but the DC power distribution was more accessible in 1X1.) I also added 3 new +24 V clips with fuses despite needing only one, so next time we need to connect something new it's not such a hassle.
The RF distribution box where the 29.5 MHz signal originates should not be turned on until the amplifer has DC power. Since we may have a power interruption tomorrow, the plan is to wait until things are shut down in preparation, and then shut off anyhting else necessary before connecting the new clips on the rail to the existing ones.
received this note: at 4:11pm Tuesday, Feb 7, 2017
Date: Wednesday, February 8, 2017
Time: 7:30 AM – 8:30 AM
Contact: Rick Rodriguez x-2576
Pasadena Water and Power (PWP) will be performing a switching operation of the
Caltech Electrical Distribution System that is expected to be transparent to Caltech,
but could result in a minor power anomaly that might affect very sensitive equipment.
IMPACT: Negligible impact......?
There may be temporary power interruption tomorrow!
PS:we did not see any effect
I tested the amplifier with the Agilent network analyzer and measured 19.5 dB of gain between 29 and 30 mHz. The phase only changed by 1 degree over this same 1 MHz span. Since everything seems to be in order I'll hook it up this afternoon, unless there are any objections.
A few minutes back, I glanced up at the control room StripTool and noticed that the MCREFL PD DC level had gone up from ~0 to ~0.7, even though the PSL shutter was closed. This seemed bizzare to me. Strangely, simply cycling the shutter returned the value to the expected value of 0. I wonder if this is just a CDS problem to do with c1iool0 or c1psl? (both seem to be responding to telnet though...)
Since things look to be back to normal, I am going to start with my characterization of the various TFs in the IMC FSS loop...
It is more accurate to model the physical frequency noises at various places.
cf. See also 40m ALS paper or Shigeo Nagano PDH thesis on https://wiki-40m.ligo.caltech.edu/40m_Library
- The output 4 should be "Laser frequency"
- Seismic path should be excluded from the summing node
- The output after the PMC: "Laser frequency after the PMC"
- "Laser frequency after the PMC" is compared (diffed) with the output 1 "mirror motion in Hz"
- The comparator output goes to the cav pole, the PD, and the PDH gain: This is the output named "PDH Error"
- Tap a new path from "Laser frequency after the PMC" and multiply with the cav pole (C_IMC)
- Tap a new path from "Mirror motion" and multiply with the cavity high pass (s C_IMC/omega)
- Add these two: This is the output named "Frequency noise transmitted by IMC"
I've edited Rana's Simulink model to reflect the current IMC servo topology (to the best of my understanding). I've tried to use Transfer Function blocks wherever possible so that we can just put in the appropriate zpk model in the script that will linearize the whole loop. I've also omitted the FSS SLOW loop for now.
I've been looking through some old elogs and it looks like there have been several modifications to both the MC servo board (D040180) and the TT FSS Box (D040105). I think it is easiest just to measure these TFs since the IMC is still down, so I will set about doing that today. There is also a Pomona Box between the broadband EOM and the output of the TT FSS box, which is meant to sum in the modulation for PMC locking, about which I have not yet found anything on the elog.
So the next steps are:
If anyone sees something wrong with this topology, please let me know so that I can make the required changes.
Had to reboot c1psl, c1susaux, c1auxex, c1auxey and c1iscaux today. PMC has been relocked. ITMX didn't get stuck. According to this thread, there have been two instances in the last 10 days in which c1psl and c1susaux have failed. Since we seem to be doing this often lately, I've made a little script that uses the netcat utility to check which slow machines respond to telnet, it is located at /opt/rtcds/caltech/c1/scripts/cds/testSlowMachines.bash.
The script can be executed by ./testSlowMachines.bash.
The bottom 5 cable connections from Sat-Amp to Whittering Filter at 1X5 were clamped today.
Since the "stablizer box" doesn't really need to stabilize, it just needs to amplify, I decided to replace it with an off the shelf amplifier we already had, ZHL-2. I worked on getting it set up today, but didn't connect anything so that people have a chance to give some feedback.
So, I think the remaining thing to do is to connect the splitter to ASC out and to the line to the EOM, the +24V supply to the amplifier, and the 29.5 MHz input to the attenuator. I wanted to wait on this to get confiration that the setup is OK. Eventually we can put all of this in a box.
Also, I noticed that in the clear cabinet with the Sorensens next to this rack, the +24 V unit is not supplying any voltage and has a red light that says "OVP."
No, not confused on that point. We just will not be testing OS versions at the 40m or running multiple OS's on our workstations. As I've said before, we will only move to so-called 'reference' systems once they've been in use for a long time.
Ubuntu16 is not to my knowledge used for any CDS system anywhere. I'm not sure how you expect to have better support for that. There are no pre-compiled packages of any kind available for Ubuntu16. Good luck, you big smelly doofuses. Nyah, nyah, nyah.
True - its an issue. Koji and I are updating zita into Ubuntu16 LTS. If it looks like its OK with various tools we'll swap over the others into it. Until then I figure we're best off turning allegra back into Ubuntu12 to avoid a repeat of this kind of conflict. Once the workstations in the LLO control room are running smoothly on a new OS for a year, we can transfer into that. I don't think any of us wants to be the CDS beta tester for DV or DTT.
Just to be clear, since there seems to be some confusion, the SVN issue has nothing to do with Debian vs. Ubuntu. SVN made non-backwards compatible changes to their working copy data format that breaks newer checkouts with older clients. You will run into the exact same problem with newer Ubuntu versions.
I recommend the 40m start moving towards the reference operating systems (Debian 8 or SL7) as that's where CDS is moving. By moving to newer Ubuntu versions you're moving away from CDS support, not towards it.
I was able to bring back svn 1.6 formatting to /cvs/cds/caltech/chans by doing the following on nodus:
svn co https://nodus.ligo.caltech.edu:30889/svn/trunk/chans ./
rm -rf ../chans/.svn
mv ./.svn ../chans/
Note that I used the http address for the repository. The svn repository doesn't live at file:///cvs/cds/caltech/svn anymore; all of our checkouts (e.g. in the scripts directory) use http to get the one true repo location, regardless of where it lives on nodus' filesystem. (I suppose we could also use https://nodus.martian:30889/svn to stick to the local network, but I don't think we're that limited by the caltech network speed)
Presumably, at some point we will want to introduce a newer operating system into the 40m, as ubuntu 12.04 hits end-of-life in April 2017. Ubuntu 16.04 includes svn 1.8, so we'll also hit this issue if we choose that OS.
Aside from the svn issues, this directory (/cvs/cds/caltech/chans) only contains pre-2010 channels. Filters and DAQ ini files currently live in /opt/rtcds/caltech/c1/chans, which is not under version control. It's also not clear to me why summary page configurations should be kept in this /cvs/cds place.
In working on automatic DARM loop design, we have this code:
the things in there like mkCost*, etc. have examples of the cost functions that are used. It may be useful to look at those and then make a similar cost function calculation for the MCL/MCF loop.
More testing of fb1 today. DAQ DOWN UNTIL FURTHER NOTICE.
Testing Wednesday did not resolve anything, but Jonathan Hanks is helping.
Rana motivated me to take a step back and reframe the objectives and approach for this project, so I am collecting some thoughts here on my understanding of it. As I write this, some things still remain unclear to me, so I am leaving these as questions here for me to think about...
and come up with the best loop that meets all our rquirements? What constitutes the "best" loop? How do we weight the relative importance of our various requirements?
For the specific problem of making the MCL feedback loop better, the approach I have in mind right now is the following:
My immediate goal is to have the Simulink model updated.
Thoughts/comments on the above will be appreciated...
Re-aligned the beam going into the PMC today around 5 PM. I noticed that its all in pitch and since I moved both of the mirrors by the same amount it is essentially a vertical translation.
I wonder if the PMC is just moving up and down due to thermal expansion in the mount? How else would we get a pure vertical translation? Need to remember next time if the beam goes up or down, and by how many knob turns, and see how it correlates to the lab temperature.
and the song remains the same...
the version of SVN on these workstations is ahead of the one on the other workstations so now we can't do 'svn up' on any of the Ubuntu12 machines. One allegra and optimus I get this error:
controls@allegra|GWsummaries> svn up
svn: E180001: Unable to connect to a repository at URL 'file:///cvs/cds/caltech/svn/trunk/GWsummaries'
svn: E180001: Unable to open an ra_local session to URL
svn: E180001: Unable to open repository 'file:///cvs/cds/caltech/svn/trunk/GWsummaries'
Someone installed "Debian" on allegra. Why? Dataviewer doesn't work on there. Is there some advantage to making this thing have a different OS than the others? Any objections to going back to Ubuntu12?
My elog negligence punchcard is getting pretty full... It's pretty much for the same reason as using Debian for optimus; much of the workstation software is getting packaged for Debian, which could offload our need for setting things up in a custom 40m way. Hacking the debian-focused software.ligo.org repos into Ubuntu has caused me headaches in the past. Allegra wasn't being used often, so I figured it was a good test bed for trying things out.
The dataviewer issue was dataviewer's inability to pull the `fb` out of `fb:8088` in the NDSSERVER env variable. I made a quick fix for it in the dataviewer launching script, but there is probably a better way to do it.
I'm not sure if its possible to downgrade our chans repo back to the old one, but I highly recommend that no one do 'svn upgrade' in any of our repos until we remove all of the Debian installs in the 40m lab or hire a full-time sysadmin.
I had noticed something wonky with the microphone, but neglected to elog it. I had tested it after installation by playing a sine wave from my laptop and looking at the signal on the PSL table, it worked fine. But you can see in the attached minute trend plot that the signal characteristics changed abruptly ~half a day after installation, and never quite recovered.\
I don't know if anyone looked at the time series (not trend) or spectrum of the Microphone after installation, but it looks bad and featureless to me. Is the Microphone broken?
This shows the spectrum from early this morning and again from tonight. You can see that it is bi-stable in its noise properties. This thing is busted; we're now removing it from the PSL so that it doesn' light it self on fire.
This AmScope microscope would have 3.5x-180x magnification, calibratable measurement function, 5MP picture and good working distance to work on printed circuit boards.
FYI this issue has still not been solved, but the pages are working because I got the software running on an
alternative headnode (pcdev2). This may cause unexpected behavior (or not).
> LDAS has not recovered from maintenance causing the pages to remain unavailable until further notice.
> > System-wide CIT LDAS cluster maintenance may cause disruptions to summary pages today.
I made some of the changes. Gautam and I will finish tomorrow.
While I was soldering the sharpest tip of the soldering iron (the one whose power supply shows the temperature) stopped working and I switched to a different one. Not sure how to fix this.
Do we want to replace all of the removed ERA's with 50 Ohm resistors, or just the one along the spare output path? I shorted one of them with a piece of wire and left all the others open.
I couldn't get one of the attenuators off (AT1, at beginning of ASC path). In trying I messed up the solder pad. Part of the connecting trace on the PCB board is exposed so we should be able to fix it.
I think this then allows us to have the low noise OCXO signals everywhere with enough oomph.
Here's what I'm planning to do to the RF AM stabilizer box. I'm going to take out several of the components along the path to the EOM (comments in green), including the dead ERA-4 and ERA-5 amplifiers, the variable attenuator which is controlled by a switch that can't be accessed outside the box, and the feedback path from the daughter board servo. I'm arranging things so that the output of the HELA-10 does not exceed the maximum output power.
I wasn't quite as sure what to do about the path to the ASC box (comments in blue). I talked with Gautam and he said this gets split equally between several singals, one of which goes to the LO of the demod board which expects -10 dBm and currently gets -12 dBm (can go up to -8 by turning switch). So maybe we don't actually want the signal to be anywhere near +27 dBm at the output. The plans for the box are here, it looks like +27 in will end up with +10 at each output, which is way more than what's currently coming out. But maybe this needs to be increased to match the other path?
Also we haven't measured the actual response of the variable attenuator U4 for various switch positions; it's the same model as the one I'm removing from the EOM path and that one had slightly different behavior for different switch positions than what the spec sheet says. Same goes for the HELA-10 units along this path: what is their actual gain? So perhaps these should be measured and then a single attenuator should be chosen to get the right output signal level. Alternatively it could just be left alone, if it is at an OK level right now. Advice on what to do here would be appreciated.
I'll work on the EOM path tonight and wait for feedback on the rest of it.
EDIT: Gautam pointed out that there's some insertion loss from the components I'll be removing that hasn't been accounted for. Also the plans have been updated to reflect that I'm replacing AT5 with a 1dB attenuator (from 6 dB).
For a comparison: OMC ELOG 238
[rana, gautam, lydia]
Today we looked at the schematics for the RF AM stabilizer box and decided that there were an unnecessary amount of attenuators and amplifiers cancelling each other out and adding noise. At the end of the path are 2 HELA-10D amplifiers which we guessed based on the plots for the B version would have an acceptable amount of compression if the output of the second one is ~27dBm. This means the input to the first one should be a few dBm. This should be achieved with as simple a path as possible.
This begged the question, do we need the amplitude to be stabilized at all? Maybe it's good enough already when it comes into this box from the RF distribution box. So I tried to measure the AM noise of the 29.5 MHz signal that usually goes into the AM stabilizer:
It seems like I'm getting mostly noise from the SR560. Maybe it would be better to use an SR785 to take data instead of DAQ, and then skip the SR560? At low frequencies it seems like the AM noise measurement may be actually meaningful. In any case, if the actual AM noise from the crystal is lower than any of these other noise sources, it means we probably don't need to stabilize the amplitude with a servo, which means we can simplify the AM stabilizer board considerably to just amplify what it gets to 27 dBm.
I made a crude sketch for how Lydia and I envision the connector situation on the back of the vme crates to be solved. Essentially the side panels of each crate extend about 2" (52 mm) beyond the edge of the DIN connectors. This is plenty of space for a simple PCB board. The connector of choice is D-Sub. We can split the 64 used pins into 2x 37 D-Sub OR (2x25 pin + 1x15pin). The former has fewer cables, but a few excess unused leads. A quick google search showed me that it is much cheaper to get twisted pair cables for 15 and 25 pin D-Subs. From what I remember, the used pins on the DIN connectors are concentrated on the low numbers end and the high numbers end, so might not need the 'middle' connector in many cases if we decide to break it up into three. I have to check this with Lydia though.
The D-Sub connectors would be panel mounted, for which we need a narrow panel piece with dsub cutouts. We can run horizontal struts across the vme crate from side panel to side panel. This way the force upon cable (dis)connection is mostly on the panel which is attached to the struts which are attached to the crate. This will also prevent gravitational sag or cable strain from pulling on the DIN connection, and we can use twisted pair cables with backshell, screws, and strain reliefs.
I was lookng into getting started with the PCB when Altium complained that the license is expired and to renew it. This is a relatively simple board layout so some free software out there is probably enough.
I've added the schematic of the RF AM stabilization board to the 40m PSL document tree, after having created a new DCC document for our 40m edits. Pictures of the board before and after modification will also be uploaded here...
I've been suggesting that there may be something wonky with the Seismic Rainbow Striptool on the wall for the last couple of weeks. Here are a few things that were verified today.
A novice was learning at the feet of Master Daqd. At the end of the lesson he looked through his notes and said, “Master, I have a few questions. May I ask them?”
Master Daqd nodded.
"Do we record minute trends of our data?"
"Yes, we record raw minute trends in /frames/trend/minute_raw"
"I see. Do we back up minute trends?"
"Yes, we back up all frames present in /frames/trend/minute"
"Wait, this means we are not recording our current trends! What is the reason for the existence of seperate minute and minute_raw trends?
“The knowledge you seek can be answered only by the gods.”
"Can we resume recording the minute trends?"
Master Daqd nodded, turned, and threw himself off the railing, falling to his death on the rocks below.
Upon seeing this, the novice was enlightened. He proceeded to investigate how to convert raw minute trends to minute trends so that historical records could be preserved, and precisely when Master Daqd started throwing himself off the mountain when asked to record minute trends.
> What is the probe situation? Ought to use a high impedance FET probe to measure this or else the scope would load the circuit.
We did indeed use the active probe, with the 100:1 attenuator in place. The values Lydia has quoted have 40dB added to account for this.
> What kind of HELA are the HELA amplifiers? Please a link to the data sheet if you can find it. I wonder what the gain and NF are at 30 MHz. I think the HELA-10D should be a good variant
The HELA is marked as HELA-10. It doesn't have the '+' suffix but according to the datasheet, it seems like it is just not RoHS compliant. It isn't indicated which of the varieties (A-D) is used either on the schematic or the IC, only B and D are 50ohms. For all of them, the typical gain is 11-12dB, and NF of 3.5dB.
I think this cron job is running on NODUS (our gateway) instead of our scripts machine:
*/1 * * * * /opt/rtcds/caltech/c1/scripts/Admin/n2Check.sh >> /opt/rtcds/caltech/c1/scripts/Admin/n2Check.log 2>&1
Based on Jenne's chiara disk usage monitoring script, I made a script that checks the N2 pressure, which will send an email to myself, Jenne, Rana, Koji, and Steve, should the pressure fall below 60psi. I also updated the chiara disk checking script to work on the new Nodus setup. I tested the two, only emailing myself, and they appear to work as expected.
The scripts are committed to the svn. Nodus' crontab now includes these two scripts, as well as the crontab backup script. (It occurs to me that the crontab backup script could be a little smarter, only backing it up if a change is made, but the archive is only a few MB, so it's probably not so important...)
moreover this script has a 90MB log file full of not finding its channel
I wish this script was in python instead of BASH and I wish it would run on megatron instead of nodus (why can't megatron send us email too?) and I wish that this log file would get wiped out once in awhile. Currently its been spitting out errors since at least a month ago:
Tue Jan 31 14:10:02 PST 2017 : N2 Pressure:
Channel connect timed out: 'C1:Vac-N2pres' not found.
(standard_in) 1: syntax error
We looked at the RF AM stabilizer box to see if we could find out 1) Why the output power is so low, and 2) Why it can't be changed with the DC input "MOD CONT IN." Details to follow, attached is the annotated schematic from DCC document D000037.
We are not returning the box tonight so the PSL shutter remains closed.
We pulled out the RF AM stabilization box from the 1X2 rack. PSL shutter was closed, marconi output, RF distribution box and RF AM stabilization box were turned off in that order. We had to remove the 4 rack nut screws on the RF distribution box because of the stiff cables which prevented the RF AM stabilization box extraction. I've left the marconi output and the RF distribution boxes off, and have terminated all open SMA connections with 50 ohm terminators just in case. Rack nuts for RF distribution box have been removed, it is currently sitting on a metal plate that is itself screwed onto the rack. I deemed this a stable enough ledge for the box to sit on in the short run, while we debug the RF AM stabilization box. We will work on the debugging and re-install the box as soon as we are done...
I just aborted the fb1 test and reverted everything to the nominal configuration. Everything looks to be operating nominally. Front ends are mostly green except for c1rfm and c1asx which are currently not being acquired by the DAQ, and an unknown IPC error with c1daf. Please let me know if any unusual problems are encountered.
The behavior of daqd on fb1 with the latest release (3.2.1) was not improved. After turning on the full pipe it was back to crashing every 10 minutes or so when the full and second trend frames were being written out. lame. back to the drawing board...
I'm not sure if this is related, but since today morning, I've noticed that the data concentrator errors have returned. Looking at daqd.log, there is a 1 second timing mismatch error that is being generated. Usually, manually running ntpdate on the front ends fixes this problem, but it did not work today.
If this problem started before ~4pm on Friday then it's probably unrelated, since I didn't start any of these tests until after that. If unexplained problem persist then we can try shutting of the fb1 daqd and see if that helps.