I think this is because /cvs/cds is getting too big. lsblk reveals:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 465.8G 0 disk
├─sda1 8:1 0 446.9G 0 part /
├─sda2 8:2 0 1K 0 part
└─sda5 8:5 0 18.9G 0 part [SWAP]
sdb 8:16 0 2.7T 0 disk
└─sdb1 8:17 0 2T 0 part /home/cds
sr0 11:0 1 1024M 0 rom
sdc 8:32 0 1.8T 0 disk
└─sdc1 8:33 0 1.8T 0 part /media/40mBackup
sdd 8:48 0 1.8T 0 disk
└─sdd1 8:49 0 1.8T 0 part
I believe one of sdc or sdd is connected via SATA while the other is an external USB drive. Maybe we have to get bigger backup disks, but this may be a huge pain to setup as it will involve taking chiara down. Actually, now that I check the backup log, seems like backup is executing successfully - not sure if this is due to my unelogged mounting of sdc (using sudo mount /dev/sdc1 /media/40mBackup) last week, or if this is some LDAS backup. But in any case, seems undesirable that sdb1 is larger than sdc1 or sdd1.
2018-06-06 07:00:01,086 INFO Updating backup image of /cvs/cds
2018-06-06 07:00:01,086 ERROR External drive not mounted!!!
2018-06-07 07:00:01,147 INFO Updating backup image of /cvs/cds
2018-06-07 07:00:01,147 ERROR External drive not mounted!!!
2018-06-08 07:00:01,244 INFO Updating backup image of /cvs/cds
2018-06-08 08:23:32,939 INFO Backup rsync job ran successfully, transferred 316870 files.
2018-06-09 07:00:01,465 INFO Updating backup image of /cvs/cds
2018-06-09 07:12:11,865 INFO Backup rsync job ran successfully, transferred 1926 files.
2018-06-10 07:00:01,842 INFO Updating backup image of /cvs/cds
2018-06-10 07:12:28,931 INFO Backup rsync job ran successfully, transferred 1656 files.
2018-06-11 07:00:01,294 INFO Updating backup image of /cvs/cds
2018-06-11 07:06:14,748 INFO Backup rsync job ran successfully, transferred 1664 files.
2018-06-12 07:00:02,081 INFO Updating backup image of /cvs/cds
2018-06-12 07:07:36,775 INFO Backup rsync job ran successfully, transferred 1870 files.
2018-06-13 07:00:02,194 INFO Updating backup image of /cvs/cds
2018-06-13 07:08:37,356 INFO Backup rsync job ran successfully, transferred 1818 files.
2018-06-14 07:00:01,753 INFO Updating backup image of /cvs/cds
2018-06-14 07:01:43,270 INFO Backup rsync job ran successfully, transferred 1744 files.
Local backup on chiara seems not working since Nov 19, 2017.
2017-11-18 07:00:01,504 INFO Updating backup image of /cvs/cds
2017-11-18 07:03:00,113 INFO Backup rsync job ran successfully, transferred 1954 files.
2017-11-19 07:00:02,564 INFO Updating backup image of /cvs/cds
2017-11-19 07:00:02,592 ERROR External drive not mounted!!!
I finally analyzed the sensing measurement I ran on Tuesday evening. Sensing responses for the DRMI DOFs seems consistent with what I measured in October 2017, although the relative phasing of the DoFs in the sensing PDs has changed significantly. For what it's worth, my Finesse simulation is here.
All optics have been re-aligned. Jon/Johannes will elog about the work today.
Using the numbers from the sensing measurement, I calibrated the measured in-loop MICH spectrum from Tuesday night into free-running displacement noise. For convenience, I used the noise-budgeting utilities to make this plot, but I omitted all the technical noise curves as the coupling has probably changed and I did not measure these. The overall noise seems ~x3 higher everywhere from the best I had last year, but this is hardly surprising as I haven't optimized anything for low noise recently. To summarize:
I will do a more thorough careful characterization and add in the technical noises in the coming days. The dominant uncertainty in the sensing matrix measurement, and hence this free-running noise spectrum, is that I haven't calibrated the actuators in a while.
I did the measurement with the BeatMouth open today. Main changes:
So neglecting asymmetry in the branching ratio of the fiber beamsplitter, the asymmetry between the test PD optical path and the reference PD optical path is a single fiber mating sleeve in the former vs a collimator in the latter. In order to recover the expected number of 409 V/W for the Menlo PDs, we have to argue that the optical loss in the test PD path (fiber mating sleeve) are ~3x higher than in the NF1611 path (free space coupler). But at least the X and Y PDs show identical responses now. The error I made in the previously attached plot was that I was using the 20dB coupled output for the X PD measurement .
Revised conclusion: The measured optoelectronic response of the Menlo PDs at 10s of MHz, of ~130 V/W, is completely consistent with the numbers I reported in this elog. So rogue polarization is no longer the culprit for the discrepancy between expected and measured RF beatnote power, it was just that the expectation, based on Menlo PD specs, were not accurate.#2 of the linked elog seems to be the most likely, although "broken" should actually be "not matching spec".
While killing time b/w measurements, I looked on the ITMY optical table and found that the NF1611 I mentioned in this elog still exists. It is fiber coupled. Could be a better substitute as a Reference PD for this particular measurement.
I will repeat the measurement tomorrow by eliminating some un-necessary patch fiber cables, and also calibrating out the cable delays.
[Jon, Gautam, Johannes]
We did the following today:
This measurement seems like a fine candidate to trial the idea of looking for the FSRs (and in general, cavity resonances) of the PRC in the phase of the measured TFs, rather than the amplitude.
Yesterday, I moved the following optics:
After moving these components around a bit, I locked them down once I was happy that the beam was pretty well centered on both of them, and also on AS110 and AS55 (measured using O'scope with single bounce from one ITM, other optics misaligned).
The beam was close to clipping on the lens mentioned in #1, probably because this wasn't checked when the 90-10 BS was installed for the AUX laser. Furthermore, I believe we are losing more than 10% of the light due to this BS. The ASDC (which is derived from AS55 PD) level is down at ~110cts as the Michelson is fringing, while it used to be ~200 cts. I will update with a power measurement shortly. But I think we should move ahead with the plan to combine the beam into the IFO's AS mode as discussed at the meeting last week.
Unrelated to this work, but c1psl and c1iscaux were keyed.
ASDC has something weird going on with it - my main goal yesterday was to calibrate the actuators of ITMX, ITMY and BS using the Michelson. But with the Michelson locked on a dark fringe, the ASDC level changed by up to 50 counts seemingly randomly (bright fringe was ~1000 cts, I had upped the whitening gain to +21dB), even though the CCD remained clearly dark throughout. Not sure if the problem is in the readout electronics or in the PD itself.
The actuator (pendulum) gains for the Beam Splitter and the two ITMs were measured to be:
BS: 9.54 +/- 0.05 nm/ct [100 ohm series resistor in coil driver board]
ITMX: 2.44 +/- 0.01 nm/ct [400 ohm series resistor in coil driver board]
ITMY: 2.44 +/- 0.02 nm/ct [400 ohm series resistor in coil driver board]
Counts here refers to DAC counts at the output of the coil filter banks (as opposed to counts at the LSC servo output). The dominant (systematic) uncertainty (which isn't quoted here) in this measurement is the determination of the peak-to-peak swing of the dark port sensor, AS55_Q. I estimate this error to be ~1ct/33cts = 3%. These values are surprisingly consistent with one another once we take into account the series resistance.
The last time this was done, we used ASDC to do the measurement. But I don't know what signal conditioning ASDC undergoes (in PD or in readout electronics). In any case, in my early trials yesterday, ASDC was behaving unpredictably. So I decided to do redo the measurement.
[Attachment #1]- Flowchart describing the calibration procedure.
[Attachment #2] - AS55_Q output while the Michelson was freeswinging. I had first aligned the ITMs using ASS. The peak-to-peak value of this corresponds to . So we know AS55_Q in terms of cts/m of MICH displacement.
[Attachment #3] - Magnitudes of transfer function from moving one of the MICH optics, to the now calibrated AS55_Q. Fits are to a shape , with being the fitted parameter. Coherence during the measurement is also plotted.
 - http://www.phys.ufl.edu/~bernard/papers/CQG20_S903.pdf
POP QPD checkout:
Per discussion today eve, barring objections, I will do the following tomorrow morning:
I finished the re-soldering work today, and have measured the coil driver noise pre-Mods and post-Mods. Analysis tomorrow. I am holding off on re-installing the board tonight as it is likely we will have to tune all the loops to make them work with the reduced range. So ETMX will remain de-commissioned until tomorrow.
I decided to take a quick look at the data. Changes made to the ETMX coil driver board:
I also took the chance to check the integrity of the LM6321 ICs. In the past, a large DC offset on the output pin of these has been indicative of a faulty IC. But I checked all the ICs with a DMM, and saw no anomalies.
Measurement condition was that (i) the Fast input was terminated to ground via 50ohm, (ii) the Bias input was shorted to ground. SR785 was used with G=100 Busby preamp (in which Steve installed new batteries today, for someone had left it on for who knows how long) for making the measurement. The voltage measurement was made at the D-Sub connector on the front panel which would be connected to the Sat. Box, with the coil driver not connected to anything downstream.
Summary of results:
[Attachment #1] - Noise measurement out to 800 Hz. The noise only seems to agree with the LISO model above 300 Hz. Not sure if the low-frequency excess is real or a measurement artefact. Tomorrow, I plan to make an LPF pomona box to filter out the HF pickup and see if the low-frequency characteristics change at all. Need to think about what this corner freq. needs to be. In any case, such a device is probably required to do measurements inside the VEA.
[Attachment #2] - Noise measurement for full SR785 span. The 19.5 kHz harmonics are visible. I have a theory about the origin of these, need to do a couple of more tests to confirm and will make a separate log.
[Attachment #3] - zip of LISO file used for modeling coil driver. I don't have the ASCII art in this, so need to double check to make sure I haven't connected some wrong nodes, but I think it's correct.
Measurements seem to be consistent with LISO model predictions.
*Note: Curves labelled "LISO model ..." are really quad sum of liso pred + busby box noise.
My main finding tonight is: With the increased series resistance (400 ohm ---> 2.25 kohm), LISO modeling tells me that even though the series resistance (Johnson noise) used to dominate the voltage noise at the output to the OSEM, the voltage noise of the LT1125 in the bias path now dominates. Since we are planning to re-design the entire bias path anyways, I am not too worried about this for the moment.
I will upload more details + photos + data + schematic + LISO model breakdown tomorrow to a DCC page.
gautam noon 21 June 2018: I was looking at the wrong LISO breakdown curves. So the input stage Op27 voltage noise used to dominate. Now the Bias path LT1125 voltage noise dominates. None of the conclusions are affected... I've uploaded the corrected plots and LISO file here now.
I took this opportunity of EX downtime to change the supply voltage for the AA unit (4-pin LEMO front panel) in 1X9 from +/-5V to +/-15V. Inside the AA board are INA134 and DRV135 ICs, which are rated to work at +/-18V. In the previous state, the inputs would saturate if driven with a 2.5Vpp sine wave from a DS345 func. gen. After the change, I was able to drive the full range of the DS345 (10Vpp), and there was no saturation seen. This AA chassis is only used for the OSEM signals and also some ALS signals. Shadow sensor levels and spectra are consistent before and after the change. The main motivation was to not saturate the Green PDH Reflection signal in the digital readout. The steps we took were:
Initial tests look promising. Local damping works and I even locked the X arm using POX, although I did it in a fake way by simply inserting a x5.625 (=2.25 kohm / 400 ohm) gain in the coil driver filter banks. I will now tune the individual loop gains to account for the reduced actuation range.
Now I have changed the loop gains for local damping loops, Oplev loops, and POX locking loop to account for the reduced actuation range. The dither alignment servo (X arm ASS) has not been re-commissioned yet...
pianosa has been upgraded to SL7. I've made a controls user account, added it to sudoers, did the network config, and mounted /cvs/cds using /etc/fstab. Other capabilities are being slowly added, but it may be a while before this workstation has all the kinks ironed out. For now, I'm going to follow the instructions on this wiki to try and get the usual LSC stuff working.
MEDM, EPICS and dataviewer seem to work, but diaggui still doesn't work (it doesn't work on Rossa either, same problem as reported here, does a fix exist?). So looks like only donatella can run diaggui for now. I had to disable the systemd firewall per the instructions page in order to get EPICS to work. Also, there is no MATLAB installed on this machine yet. sshd has been enabled.
Seems like DTT also works now. The trick seems to be to run sudo /usr/bin/diaggui instead of just diaggui. So this is indicative of some conflict between the yum installed gds and the relic gds from our shared drive. I also have to manually change the NDS settings each time, probably there's a way to set all of this up in a more smooth way but I don't know what it is. awggui still doesn't get the correct channels, not sure where I can change the settings to fix that.
I think if the magnet fell off, we would see high DC signal, and not 0 as we do now. I suspect satellite box or PD readout board/cabling. I am looking into this, tester box is connected to ITMY sat. box for now. I will restore the suspension later in the evening.
Suspension has now been restored. With combination of multimeter, octopus cable and tester box, the problem is consistent with being in the readout board in 1X5/1X6 or the cable routing the signals there from the sat. box.
We may lost the UL magnet or LED
For a series resistance of 4.5 kohm, we are suffering from the noise-gain amplified voltage noise of the Op27 (2*3.2nV/rtHz), and the Johnson noise of the two 1 kohm input and feedback resistances. As a result, the current noise is ~2.7 pA/rtHz, instead of the 1.9 pA/rtHz we expect from just the Johnson noise of the series resistance. For the present EX coil driver configuration of 2.25 kohm, the Op27 voltage noise is actually the dominant noise source. Since we are modeling small amounts (<1dB) of measurable squeezing, such factors are important I think.
[Attachment #1] --- Sketch of the fast signal path in the coil driver board, with resistors labelled as in the following LISO model plots. Note that as long as the resistance of the coil itself << the series resistance of the coil driver fast and slow paths, we can just add their individual current noise contributions, hence why I have chosen to model only this section of the overall network.
[Attachment #2] --- Noise breakdown per LISO model with top 5 noises for choice of Rseries = 2.25 kohm. The Johnson noise contributions of Rin and Rf exactly overlap, making the color of the resulting line a bit confusing, due to the unfortunate order of the matplotlib default color cycler. I don't want to make a custom plot, so I am leaving it like this.
[Attachment #3] --- Noise breakdown per LISO model with top 5 noises for choice of Rseries = 4.5 kohm. Same comments about color of trace representing Johnson noise of Rin and Rf.
Possible mitigation strategies:
I've chosen to ignore the noise contribution of the high current buffer IC that is inside the feedback loop. Actually, it may be interesting to compare the noise measurements (on the electronics bench) of the circuit as drawn in Attachment #1, without and with the high current buffer, to see if there is any difference.
This study also informs about what level of electronics noise is tolerable from the De-Whitening stage (aim for ~factor of 5 below the Rseries Johnson noise).
Finally, in doing this model, I understand that the observation the voltage noise of the coil driver apparently decreased after increasing the series resistance, as reported in my previous elog. This is due to the network formed by the fast and slow paths (during the measurement, the series resistance in the slow path makes a voltage divider to ground), and is consistent with LISO modeling. If we really want to measure the noise of the fast path alone, we will have to isolate it by removing the series resistance of the slow bias path.
Comment about LISO breakdown plots: for the OpAmp noises, the index "0" corresponds to the Voltage noise, "1" and "2" correspond to the current noise from the "+" and "-" inputs of the OpAmp respectively. In future plots, I'll re-parse these...
I wanted to investigate my coil driver noise measurement technique under more controlled circumstances, so I spent yesterday setting up various configurations on a breadboard in the control room. The overall topology was as sketched in Attachment #1 of the previous elog, except for #4 below. Summary of configurations tried (series resistance was 4.5k ohm in all cases):
Attachment #1: Picture of the breadboard setup.
Attachment #2: Noise measurements (input shorted to ground) with 1 Hz linewidth from DC to 4 kHz.
Attachment #3: Noise measurements for full SR785 span.
Attachment #4: Apparent coupling due to PSRR.
Attachment #5: Comparison of low frequency noise with and without the LM6321 part of the fast DAC path implemented.
All SR785 measurements were made with input range fixed at -42dBVpk, input AC coupled and "Floating", with a Hanning window.
I've been thinking about what we need to do to the de-whitening boards for the ITMs and ETMs, in order to have low noise actuators. Noting down what I have so far, so that people can comment / point out things I've overlooked.
Attachment #1: Block diagram schematic of the de-whitened signal path on D000183 as it currently exists. I've omitted the unity gain buffer stage at the output, though this is important for noise considerations.
Some considerations, in rough order of priority:
I will experiment with a few different shapes and investigate noise and de-whitened digital signal levels based on these considerations. At the very least, I guess we should remove the x3 gain on the ETM boards, they have already been bypassed for the ITMs.
I moved the N2 check script and the disk usage checking script from the (sudo) crontab of nodus to the controls user crontab on megatron .
For the upcoming vent, we'd like to rotate the SOS towers to correct for the large YAW bias voltages used for DC alignment of the ITMs and ETMs. We could then use a larger series resistance in the DC bias path, and hence, reduce the actuation noise on the TMs.
Today, I used the calibrated Oplev error signals to estimate what angular correction is needed. I disabled the Oplev loops, and drove a ~0.1 Hz sine wave to the EPICS channel for the DC yaw bias. Then I looked at the peak-to-peak Oplev error signal, which should be in urad, and calibrated the slider counts to urad of yaw alignment, since I know the pk-to-pk counts of the sine wave I was driving. With this calibration, I know how much DC Yaw actuation (in mrad) is being supplied by the DC bias. I also know the directions the ETMs need to be rotated, I want to double check the ITMs because of the multiple steering mirrors in vacuum for the Oplev path. I will post a marked up diagram later.
Steve is going to come up with a strategy to realize this rotation - we would like to rotate the tower through an axis passing through the CoM of the suspended optic in the vertical direction. I want to test out whatever approach we come up with on the spare cage before touching the actual towers.
Here are the numbers. I've not posted any error analysis, but the way I'm thinking about it, we'd do some in air locking so that we have the cavity axis as a reference and we'd use some fine alignment adjust (with the DC bias voltages at 0) until we are happy with the DC alignment. Then hopefully things change by so little during the pumpdown that we only need small corrections with the bias voltages.
Oplev error signal readback
PRM watchdog was tripped around 7:15am PT today morning. I restored it.
Aaron and I are going to do the checkout of the OMC electronics outside vacuum today. At some point, we will also want to run a c1omc model to integrate with rtcds. Barring objections, I will set up this model on one of the spare cores on the physical machine c1ioo tomorrow.
We only anticipate opening up the IOO chamber and the EY chamber.
Vent preparation: see here.
@SV, we are ready to vent tomorrow. Aaron is supposed to show up ~830am to assist.
After getting the go ahead from Steve, I removed the physical beam block on the PSL table, sent the beam into the IFO, and re-aligned the MC to lock at low power. I've also revived my low power autolocker (running on megatron), seems to work okay though the gains may not be optimal, but it seems to do the job for now. Nominal transmission when well aligned at low power is ~1200cts. I briefly checked Y arm alignment with the green, seems okay, but didn't try locking the Y arm yet. All doors are still on, and I'm closing the PSL shutter again while Keerthana and Sandrine are working near the AS table.
For the heater setup on EY table, I EQ-stopped ETMY. Only the face EQ stops (3 on HR face, 2 on AR face) were engaged. The EY Oplev HeNe was also shutdown during this procedure.
Per Steve's instructions, we did the following:
I implemented this today. For now, the LSC output matrix is set to actuate on MC2 for Y arm locking. As expected, the transmission was much more stable, and the PLL control signal RMS was also reduced by factor of ~3. MC2 control signal does pick up a large (~2000 cts) DC component over a few hours, so we need to relieve this periodically.
Now that we have a workable ASS for the Y arm as well, we should be able to have more confidence in returning to the same beam spot position on the ETM and staying there during a scan using this technique.
The main improvement to be trialled next in the scanning is to improve the speed of scanning. As things stand, my script takes ~2.5 seconds per datapoint. If we can cut this in half, that'd be huge. On Wednesday night, we were extraordinarily lucky to avoid MC3 glitching, EPICS/slow machine failures, and GPIB freezes. Today, the latter reared its head. Fortunately, since I'm dumping data to file for each datapoint, this means we at least have data till the GPIB freeze.
For future measurements, we should consider locking the IMC length to the arm cavity - this would eliminate such alignment drifts, and maybe also make the PLL control signal RMS smaller.
Not related to this work: Terra, Sandrine, Keerthana and I cleaned up the lab a bit today, and made better cable labels. Aaron and I have to clean up the OMC area a bit. Huge thanks to Steve for taking care of our mess elsewhere in the lab!
We did a quick check of this board today. Main takeaways:
With the correct , we expect 0V from the DAC to result in 0 actuation on the mirror, assuming that an equal 75V goes to 2 PZTs mounted diametrically opposite on the optic. Hopefully, this means we have sufficient range to scan the input pointing into the OMC and get some sort of signal in the REFL signal (while length PZT is being scanned) which indicates a resonance.
We plan to carve out some IFO time for this work next week.
I was thinking a little more about the way we are training the network for the current topology - because the network has no recurrent layers, I guess it has no memory of past samples, and so it doesn't have any sense of the temporal axis. In fact, Keras by default shuffles the training data you give it randomly so the time ordering is lost. So the training amounts to requiring the network to identify the center of the Gaussian beam and output that. So in the training dataset, all we need is good (spatial) coverage of the area in which the spot is most likely to move? Or is the idea to develop some tools to generate video with spot motion close to that on the ETM in lock, so that we can use it with a network topology that has memory?
This looks like good progress. Instead of fixed sines or random noise, you should generate now a time series for the motion which is random noise but with a power spectrum similar to what we see for the ETM pitch motion in lock. You can use inverse FFT to get the time series from the open loop OL spectra (being careful about edge effects)
After this work, I've been having some trouble getting data with Python NDS. Eventually, I figured out that the nds connection request has to be pointed at '22.214.171.124' (the address of the NAT router which faces the outside world), port 31200 (it used to work with 'nds40.ligo.caltech.edu' or '126.96.36.199'). So the following snippet in python allows a connection to be opened. Offline access of frame data via NDS2 now seems possible.
So far, ssh (22), web services (30889), and elog (8081, 8080) were tested. We also need to test megatron NDS port forwarding and rsync for nodus, too.Finally I turned off the firewall rules of shorewall on nodus as it is no longer necessary.
Kevin and I saw some weird IMC / PEM BLRMS behaviour today - see Attached screenshot. Not sure what was happening with the IMC, but MCtrans was oscillating at ~3Hz for a good 20 minutes or so. I just killed the lock, and restarted MCautolocker on megatron. There was a strange feature in the 3-10Hz BLRMS around that time as well. All seems back to normal now...
When I came in this morning:
Checking status of slow machines, it looked like c1sus, c1aux, and c1iscaux needed reboots, which I did. Still PMC would not lock. So I did a burtrestore, and then PMC was locked. But there seemed to be waaaaay to much motion of MCREFL, so I checked the suspension. The shadow sensor EPICS channels are reporting ~10,000 cts, while they used to be ~1000cts. No unusual red flags on the CDS side. Everything looked nominal when I briefly came in at 6:30pm PT yesterday, not sure if anything was done with the IFO last night.
Pending further investigation, I'm leaving all watchdogs shutdown and the PSL shutter closed.
A quick look at the Sorensens in 1X6 revealed that the +/- 20V DC power supplies were current overloaded (see Attachment #1). So I set those two units to zero until we figure out what's going on. Possibly something is shorted inside the ITMX satellite box and a fuse is blown somewhere. I'll look into it more once Steve is back.
[koji, steve, gautam]
We debugged this in the following way:
So for now, the power cable to the box is disconnected on the back end. We have to pull it out and debug it at some point.
Apart from this, megatron was un-sshable so I had to hard reboot it, and restart the MCautolocker, FSSslowPy and nds2 processes on it. I also restarted the modbusIOC processes for the PSL channels on c1auxex (for which the physical Acromag units sit in 1X5 and hence were affected by our work), mainly so that the FSS_RMTEMP channel worked again. Now, IMC autolocker is working fine, arms are locked (we can recover TRX and TRY~1.0), and everything seems to be back to a nominal state. Phew.
After this work, we recovered the nominal RTCDS state. The main points were:
Some stuff that is not working as usual:
I made a model change in c1x03 (the IOP model on c1ioo) to add a DAC part. The model compiled, installed and started correctly, and looking at dmesg on c1ioo, it recognises the DAC card as what it is. Next step is to use a core on c1ioo for a c1omc model, and actually try driving some signals.
Note that the only change made to the c1ioo expansion chassis was that a DAC card was installed into the PCIe bus. The adaptor card which allows interfacing the DAC card to an AI board was already in the expansion chassis, presumably from whenever the DAC was removed from this machine.
*I think I forgot to restart optimus after this work...
The main motivation behind adding a DAC card in c1ioo was to setup an RTCDS model for the OMC. Attachment #1 shows the new look CDS overview screen. Here is what I did.
Mostly, I followed instructions from when I setup the model for the EX green PZTs.
The model is just a toy for now (CDS parameters, ADC block and 2 CDS filter modules). I leave it to Aaron to actually populate it, check functionality etc. The path to the model is /opt/rtcds/caltech/c1/userapps/release/isc/c1/models/c1omc.mdl. I am listing the parameters set on the CDS_PARAMETERS block:
Building and installing model:
Once the model was installed, I logged into c1ioo, and built and installed the models using the usual rtcds make and rtcds install instructions. Before starting the model, I edited /diskless/root.jessie/etc/rtsystab to allow c1omc to be run on c1ioo. Using sudo cset set, I verified that CPU #6 is no longer listed (if I understand correctly, the RTCDS system takes over the core).
To reflect all this on the MEDM CDS OVERVIEW screen, I just edited the screen.
Finally, I followed the instructions here to get the channels into frames and make all the indicators green. Went into fb and restarted the daqd processes. All looks good . I'm going to leave the model running overnight to investigate stability. I forgot to svn commit the model tonight, will do it tomorrow.
The testing plan (at least initially) is to install the AA and AI boards from the OMC rack in 1X1/1X2. Then we will have short SCSI cables running from the ADC/DAC to these. The actual HV driving stages will remain in the OMC rack (NE corner of AS table).
@Steve, can we get 10 Male-Female D9 cables so that we can run them from 1X1/1X2 to the OMC rack?
Unrelated to this work: There were 2 crashes of the models on c1lsc, one ~6pm and one right now ~1030pm. The restart script brought everything back gracefully ...
I walked down to the X end and found that the entire AUX laser electronics rack isn't getting any power. There was no elog about this.
I couldn't find any free points in the power strip where I think all this stuff was plugged in so I'm going to hold off on resurrecting this until tomorrow when I'll work with Steve.
The X arm green does not stay locked to the cavity - the alignment looks fine, and the green flashes are strong, but the lock does not hold. This shouldn't be directly connected to anything we did today since the Green PDH servo is entirely analog.
Actually, c1lsc had crashed again sometime last night so I had to reboot everything this morning. I used the reboot script again, but I increased the sleep time between trying to start up the models again so that I could walk into the VEA and power cycle the c1lsc expansion chassis, as this kind of frequent model crash has been fixed by doing so in the past. Sure enough, there have been no issues since I rebooted everything at ~1030 in the morning.
The c1omc model itself has been stable as well, though of course, there is nothing in there at the moment. I may do a check of the newly installed DAC tomorrow just to see that we can put out a sine wave.
Steve has ordered the D-sub cabling that will allow us to route signals between AA/AI boards in 1X1/1X2 to the HV PZT electronics in the OMC rack. Things look setup for a measurement next week. Aaron will post a block diagram + photoz of what box goes where in the electronics racks.
Steve and I restored the power to the EX AUX electronics rack. The power strip on the lowest shelf of the AUX rack now goes to another power strip laid out vertically along the NW corner of 1X9. The EX green locks to the arm just fine now.
The idea we are going with to push the coil driver noise contribution down is to simply increase the series resistance between the coil driver board output and the OSEM coil. But there are two paths, one for fast actuation and one that provides a DC current for global alignment. I think the simplest way to reduce the noise contribution of the latter, while preserving reasonable actuation range, is to implement a precision DC high-voltage source. A candidate that I pulled off an LT application note is shown in Attachment #1.
If all this seems reasonable, I'd like to prototype this circuit and test it with ETMX, which already has the high series resistance for the fast path. So I will ask Steve to order the OpAmp and transistors.
The wall StripTool indicated that the IMC wasn't too happy when I came in today. Specifically:
The last time this happened, it was due to the Sorensens not spitting out the correct voltages. This time, there were no indications on the Sorensens that anything was funky. So I just disabled the MCautolocker and figured I'd debug later in the evening.
However, around 5pm, the shadow sensor values looked nominal again, and when I re-enabled the local damping, the MC REFL spot suggested that the local damping was working just fine. I re-enabled the MCautolocker, MC re-locked almost immediately. To re-iterate, I did nothing to the electronics inside the VEA. Anyways, this enabled us to work on the X arm ASS (next elog).
After I effected the series resistance change for ETMX, the X arm ASS didn't work (i.e. IR transmission would degrade if the servo was run). Today, we succeeded in recovering a functional ASS servo .
So both arms have working dither alignment servos now. But remember that the Y arm ASS gains have been set for locking the Y arm with MC2 as the actuator, not ETMY.
We then tried to maximize GTRX using the PZT mirrors, but were only successful in reaching a maximum of 0.41. The value I remember from before the vent was 0.5, and indeed, with the IR alignment not quite optimized before we began this work, I saw GTRX of 0.48. But the IR dither servo signals indicate that the cavity axis may have shifted (spot position on the ITM, which is uncontrolled, seems to have drifred significantly, the Pitch signal doesn't stay on the StripTool scale anymore). So we may have to double check that the transmitted beam isn't falling off the GTRX DC PD.
Since the lab-wide computer shutdown last Wednesday, all the realtime models running on c1lsc have been flaky. The error is always the same:
[58477.149254] c1cal: ADC TIMEOUT 0 10963 19 11027
[58477.149254] c1daf: ADC TIMEOUT 0 10963 19 11027
[58477.149254] c1ass: ADC TIMEOUT 0 10963 19 11027
[58477.149254] c1oaf: ADC TIMEOUT 0 10963 19 11027
[58477.149254] c1lsc: ADC TIMEOUT 0 10963 19 11027
[58478.148001] c1x04: timeout 0 1000000
[58479.148017] c1x04: timeout 1 1000000
[58479.148017] c1x04: exiting from fe_code()
This has happened at least 4 times since Wednesday. The reboot script makes recovery easier, but doing it once in 2 days is getting annoying, especially since we are running many things (e.g. ASS) in custom configurations which have to be reloaded each time. I wonder why the problem persists even though I've power-cycled the expansion chassis? I want to try and do some IFO characterization today so I'm going to run the reboot script again but I'll get in touch with J Hanks to see if he has any insight (I don't think there are any logfiles on the FEs anyways that I'll wipe out by doing a reboot). I wonder if this problem is connected to DuoTone? But if so, why is c1lsc the only FE with this problem? c1sus also does not have the DuoTone system set up correctly...
The last time this happened, the problem apparently fixed itself so I still don't have any insight as to what is causing the problem in the first place . Maybe I'll try disabling c1oaf since that's the configuration we've been running in for a few weeks.
Independent from the problems the vertex machine has been having (I think, unless it's something happening over the shared memory network), I noticed on Friday that the ETMX watchdog was tripped. Today, once again, the ETMX watchdog was tripped. There is no evidence of any abnormal seismic activity around that time, and anyways, none of the other watchdogs tripped. Attachment #1 shows that this happened ~838am PT today morning. Attachment #2 shows the 2k sensor data around the time of the trip. If the latter is to be believed, there was a big impulse in the UL shadow sensor signal which may have triggered the trip. I'll squish cables and see if that helps - Steve and I did work at the EX electronics rack (1X9) on Friday but this problem precedes our working there...
OK, how about this:
The question still remains of how to combine the fast and bias paths in this proposed scheme. I think the following approach works for prototyping at least:
In the longer term, perhaps the Satellite Box revamp can accommodate a bias voltage summation connector.
Bah! Too complex.
I have neglected many practical concerns. Some things that come to mind:
I spent most of today fighting various CDS errors.
Let's see how stable this configuration is. Onto some locking now...
Stability was short-lived it seems. When I came in this morning, all models on c1lsc were dead already, and now c1sus is also dead (Attachment #1). Moreover, MC1 shadow sensors failed for a brief period again this afternoon (Attachment #2). I'm going to wait for some CDS experts to take a look at this since any fix I effect seems to be short-lived. For the MC1 shadow sensors, I wonder if the Trillium box (and associated Sorensen) failure somehow damaged the MC1 shadow sensor/coil driver electronics.