We unstuck ETMX by shaking the stack. Most effective was to apply large periodic human sized force to the north STACIS mounts.
At first, we noticed that the face OSEMs showed nearly zero variation.
We tried unsticking it through the usual ways of putting large excitations through AWG into the pit/yaw/side DOFs. This produced only ~0.2 microns of motion as seen by the OSEMs.
After the stack shake, we used the IFO ALIGN sliders to get the oplev beam back on the QPD.
The ETMX sensor trends observed before and after the earthquake are attached.
** plots deleted; SOMEONE, tried to take raster images and turn them into PDF as if this would somehow satisfy our vetor graphics requirement. Boo. lpots must be actual vector graphics PDF
In fact, ETMX was not stuck until the M7.1 EQ today. After that it got stuck, but during the after shocks, all the OSEMs occasionally showed full swing of the light levels. So I believe the magnets are OK.
- ITM unstuck now
- IMC briefly locked at TEM00
A series of aftershocks came. I could unstick ITMX by turning on the damping during one of the aftershocks.
Between the aftershocks, MC1~3 were aligned to the previous dof values. This allowed the IMC flashing. Once I got the lock of a low order TEM mode, it was easy to recover the alignment to have a weak TEM00.
Now at least temporarily the full alignment of the IMC was recovered.
Koji came to the lab to align the IMC/IFO, but found the mirrors are dancing around. Kruthi told me that there was M7.1 EQ at Ridgecrest. Looks like there are aftershocks of this EQ going on. So we need to wait for an hour to start the alignment work.
ITMX and ETMX are stuck.
The quoted elog has figures which indicate that the network did not learn (train or generalize) on the used data. This is a scary thing as (in my experience) it indicates that something is fundamentally wrong with either the data or model and learning will not happen despite how hyperparameters are tuned. To check this, I ran the training experiment for nearly 25 hyperparameter settings (results here)with the old data and was able to successfully overfit the data. Why is this progress? Well, we know that we are on the right track and the task is to reduce overfitting. Whether, that will happen through more hyperparameter tuning, data collection or augmentation remains to be seen. See attachments for more details.
Why is the fit so perfect at the start and bad later? Well, that's because the first 90% of the test data is the training data I overfit to and the latter the validation data that the network has not generalized well to.
And finally, a network is trained!
Result summary (TLDR :-P) : No memory was used. Model trained. Results were garbage. Will tune hyperparameters now. Code pushed to github.
More details of the experiment:
What I did:
What I saw
What I think is going wrong-
Well, what now?
I got to speak to Gabriele about the project today and he suggested that if I am using Rana's memory based approach, then I had better be careful to ensure that the network does not falsely learn to predict a sinusoid at all points in time and that if I use the frame wise approach I try to somehow incorporate the fact that certain magnitudes and frequencies of motion are simply not physically possible. Something that Rana and Gautam emphasized as well.
So Cal Earthquake. All suspension watchdogs tripped.
Tried to recover the OSEM damping.
=> The watchdogs for all suspensions except for ITMX were restored. ITMX seems to be stuck. No further action by me for now.
There was a magnitude 6.6 earthquake just a few minutes ago. I am attaching photographs of the monitor feeds for reference here. Is there a standard protocol to be followed in this situation? I'm looking through the wiki now.
Further, the IMC seems to be misaligned and is not locking! As Koji has let me know, I really hope this is not too serious and can be fixed easily.
Tried collecting data today. Was unable to keep the camera_server code running for any length of time as it threw segfaults. Will take a shot again tomorrow.
The GigE is focused now (judged by eye) and I have closed the lid. I'm attaching a picture of the MC2 beam spot, captured using GigE at an exposure time of 400µs.
What was the solution to resolving the flaky video streaming during the alignment process????
-> I think, the issue was with either the poor wireless network conection or the GigE-PoE ethernet cable.
Turns out, focusing the GigE is actually a bit tricky. With pylon, everytime I change the exposure or the focus, I'm running into the error I had mentioned earlier in one of my elogs; so I tried using the python scripts to interact with the GigE. But whenever I try to change the focal plane distance by rotating the lens coupler, the ethernet cable connection becomes loose and the camera server needs to be relaunched every now and then. Also, everytime we want to change the distance between the lenses, the telescope needs to be dismantled and refocused again. I'll try to come up with a better telescope design for this.
Yesterday, I had focused the GigE using a low exposure time and small aperture of iris, to make sure that we are actually seeing a sharp image of the beam spot. I'm attaching a picture of the beam spot I had clicked while focusing it, unfortunately, I forgot to take a picture after I had focused it completely. I'm also attaching a picture of the final setup for future reference.
Yesterday night, Rana asked me to lock the MC2. I figured that the PSL shutter was closed; I just opened it and was able to see the beam spot on the analog camera screen.
A question was raised as to how much passive filtering we benefit from if we pick off the local oscillator beam for BHD from the PRC. I did some simplified modeling of this. For the expected range of arm cavity round trip losses (20-50 ppm), I think that the 40m CARM pole will be between 75-85 Hz. The corresponding recycling gain will be 40-50, with the current PRM. I assumed 1000 ppm loss inside the PRC. The net result is that, assuming the single pole coupled cavity response, we will get ~8-9 dB of filtering at ~200 Hz of the intensity noise of the input laser field to the interferometer if we pick the LO beam off from the PRC (e.g. PR2 transmission), instead of picking it off before.
The next questions are: (i) can we do a sufficiently good job of achieving the required RIN stability on the LO field for BHD without relying on the passive filtering action of the PRC? and (ii) is the benefit of the PRC filtering ruined in the process of routing the LO field from wherever the pickoff happens to the BHD setup?
The latest in my fling with the PMC. Though PMC trans is back to nominal levels (~0.713 V), we'd still like to understand the PMC noise.
Last time, I took some spectra with the RF probe (Agilent 41800A). I had already measured the PDH error signal by sweeping the PZT at ~1 Hz. The notebook I used for analysis has been updated in /users/aaron/analysis/PDH_calibrate.ipynb. The analysis was the following:
There are a few problems remaining:
In preparation for the ASS debugging, I decided to check out the beam path on the EY table. In order to be able to do this, I had to setup the POY locking to trigger on AS110 instead of TRY (as is usual for this kind of debugging). Then I could poke an IR card in the beam path without destroying the lock.
There are two irides in the beam path immediately between the vacuum window and the harmonic separator that splits off the IR and green beams. I found that the beam was in fact getting clipped on both of them. It was also somewhat off center on a 2" beamsplitter that sends half of the light to the QPD (currently decommissioned). The purpose of these irides are (I think) to eliminate some ghost reflections of the green beam and also the Oplev beam. I opened up the irides until I felt that there wasn't any more clipping of the IR beam, but the appropriate ghost beams were still getting caught.
I also re-aligned the beam onto the TRY Thorlabs PD so as to better center it on the active area. In summary, the result of this work was that the TRY level went from ~0.6 to ~0.93. There may still be some scope for optimizing this - I tried running the Y-arm ASS scripts, and already, the loops don't run away any more. I'll do the systematic analysis of the servo anyways. But given that the IMC Trans lev el used to be ~15,500 counts and is now ~14,500 counts, I think ~7% drop in TRY level is in line with what we "expect" (assuming the pre-power-degradation TRY level was 1.000).
Note that these irides were installed (I think) by Yuki, and so cannot explain the ASS anomalies of July 2018 (i.e. it does not exonerate in-vacuum clipping of the beam, as Koji had already verified that the in-air path was clean back then).
Since the work earlier this morning, the fast c1sus model has crashed ~5 times. Tried rebooting vertex FEs using the reboot script a few times, but the problem is persisting. I'm opting to do the full hard reboot of the 3 vertex FEs to resolve this problem.
Judging by Attachment #1, the processes have been stable overnight.
We crossed off another couple of bullets today.
It took me ~1 hour to realize that c1susaux requries the running of sudo /sbin/ifup eth0 to be run in order to see the martian network - why???
I don't understand the exact chain of causation, but during this work, the fast c1sus model crashed. I had to go through a few iterations of the scripted vertex machine rebooting, but things seem to be back in a normal state now, see Attachment #2. Should probably run the IFO test suite to make sure everything is a-okay, but for now, I am able to lock the IMC so I'm moving on.
The main task remaining here is to take new pictures of everything and upload to the wiki. Also, need to update the Sorensen labels to reflect their current values, some of them are outdated.
Just finished a raw version of the autolocker!! Tested it once and was able to achieve lock! This is a python version of the code at /opt/rtcds/caltech/c1/scripts/PSL/PMC/AutoLock.sh.
The current code lives in my users directory. Gautam asked me to put the completed autolocker at /opt/rtcds/caltech/c1/scripts/PSL/PMC/ and that I needn't necessarily put it on git. However, I had previously added it to my Non-linear control repo. Not sure if I should take it off? The current script still lacks some checks like those that enable it to stop after a certain time of attempting to lock or those that handle interrupt signals. Will do that in some time.
P.S. As Koji says, Victory! :-P
P.P.S. Rana pointed out that this is not the objective and what we actually wanna do is run a search over the parameter space of the locking process. I will document my ideas about this process as soon as I do a little more reading. He also said that it would not do to have command line arguments as the main source from which parameters are procured and that .yml files ought to be used instead. I will make that change asap.
I've begun working on this. Steps to complete:
As Rana asked me to in the last meeting, I dug through the elogs to determine what had become of the previous autolockers. I stumbled upon this elog by Rana from before Gautam cleaned up the medm screen. Out of curiosity, I ran the autolocker script using the instructions in Rana's elog. I did this a total of 5 times and could lock the PMC 3 times fairly quickly. I attempted to decipher the details of the code but did not make much headway owing to my unfamiliarity with the language. From what I could make out from the medm screen while the autolocker was running, it appeared to be the same method as that in this elog. I will take a look at it again tomorrow. However, I intend to spend most of tomorrow working on preprocessing the data, developing the CNN script and then the simulation.
To practise the dither alignment servo tuning, I decided to make the ASX system work again (mainly because it has fewer DoFs and so I thought it'd be easier to manage). Setup is: dither PZT mirrors on EX table-->demodulate green transmission at the dither frequencies-->Servo the error signals to 0 by an integrator.
The adjusted demod phases, servo gains were saved to the .snap file which gets called when we run the "DITHER ON" script. Also updated the StripTool template.
I plan to repeat similar characterization on the IR dither alignment servos. I think the tuning of the ASS settings can be done independently of figuring out the mystery of why the TRY level is so low.
Today, I read a lot more about BRDF and modelling but could not make much headway regarding the implementation in the simulation. I've stopped for now and I'll take a crack at it tomorrow again.
Yesterday, Rana asked me to look at Hiro Yamamoto's docs on the DCC to improve the simulation. I'm performing a first pass (=> Just skimming through to see if they're relevant, I will go through them more carefully soon!) and putting up stuff here for future reference. @Kruthi's help much appreciated!
Made changes as discussed in this issue. Further, I need to add signal handling capabilities to the code. I belive Jon has pointed me to some code. I will take a look at that ASAP.
> For channels corresponding to continuous values (such as say exposure time or the like) changes to abs(1+current_value)
Why abs? Is the current_value is like -5.4321 (for example for the alignment slider), this returns +4.4321 and the suspension will suffer from huge motion (well it will be returned to the original value soon though).
Wrote the script. It currently lives at /users/milind/NonlinearControl/milind/unstick/unstick.py. Also pushed it to the repo here. It does the following:
I tried print statements instead of actually writing to the channels as Gautam asked me to do that with supervision. Is this good enough?
a useful piece of code that we should ask one of this summer's SURFs to write:
this will solve the sticky slider problem and do it in a systematic way. We can run it from the command line: e.g. 'unsticky.py c1psl c1ioo c1lsc'
Aaron complained to me earlier that the PMC could not be locked. Turned out to be a classic sticky slider problem,
I wanted to try out the unstick.py script on c1aux but kept running into timeout errors. I was also confronted by a blank GigE screen. Further, couldn't telnet into c1aux using telnet c1aux as described here. Therefore, I went in and keyed the c1aux crate (1Y1).
we are thinking of doing a sprucing up of the input mode cleaner WFS (sensors + electronics + feedback loops)
Finding the gain of the Photodiode: The three-position rotary switch of the photodiode being used (PDA520) wasn't working, so I determined its gain by making a comparative measurement between ophir power meter and the photodiode. The photodiode has a responsitivity of 0.34 A/W at 1064 nm (obtained from the responsitivity curve given in the spec sheet using a curve digitizing software). Using the following equation, I determined the gain setting, which turned out to be 20dB.
Setup: Here a 1050nm (closest we have to 1064nm) LED is used as the light source instead of a laser to eliminate the effects caused by coherence of a laser source, which might affect our radiometric calibration. The LED is placed in a box with a hole of diameter 5mm (aperture angle = 40 degrees approx.). Suitable lenses are used to focus the light onto a white paper, which is fixed at an arbitrary angle and serves as a Lambertian scatterer. To make a comparative measurement between the photodiode (PDA520) and GigE, we need to account for their different sensor areas, 8.8mm (aperture diameter) and 3.7mm x 2.8 mm respectively . This can be done by either using an iris with a common aperture so that both the photodiode and GigE receive same amount of light , or by calculating the power incident on GigE using the ratio of sensor areas and power incident on the photodiode (here we are using the fact that power scattered by Lambertian scatterer per unit solid angle is constant).
Calibration of GigE 152 unit: I took around 50 images, starting with an exposure time of 2000 in steps of 2000, using the exposure_variation.py code. But the code doesn't allow us to take images with an exposure time greater than 100 ms, so I took few more images at higher exposures manually. From each image I subtracted a dark image (not in the sense of usual CCD calibration, but just an image with same exposure time and no LED light). These dark images do the job of usual dark frame + bias frame and also account for stray lights. A plot of pixel sum vs exposure time is attached. From a linear fit for the unsaturated region, I obtained the slope and calculated the calibration factor.
Result: CF = 1.91x 10^-16 W-sec/counts Update: I had used a wrong value for the area of photodiode. On using 61.36 mm^2 as the area, I got 2.04 x 10^-15 W-sec/counts.
I'll put the uncertainities soon. I'm also attaching the GigE spectral response curve for future reference.
Experiment file: train.py
After a more systematic alignment effort, I was able to get the spots better centered on the optics (judged by eye from the analog camera views). TRY ~0.7, TRX~1.15. The X-arm dither alignment system seems to work out-of-the-box with the existing settings, I was able to run it and maximize the X-arm transmission.
Other work: I also cleaned up the area around MC2 a litte - laptop from on top of the vacuum chamber was removed and a rogue ethernet cable was also removed. The resulted in some misalignment of the IMC, which I corrected by manual alignment. Now the IMC is locked again with nominal transmission levels.
On the PSL table, I re-routed the RF output from the BeatMouth to the regular IR-ALS electronics chain (it was hijacked for PLL investigations). At EX, I disconnected the cable running from the LB1005 to the EX NPRO laser PZT (again was being used for PLL locking), and re-connected the output from the Green uPDH box to allow for some ALS tests to be done. I could then lock the EX green beam to the X-arm, and achieved GTRY ~ 0.35 using the ASX system. More to follow on ALS tests later today.
Now that the IMC is remaining locked for extended periods of time, the next problem to attack is the ASS dither alignment system. For a start, I decided to try and get the POX and POY locking working again, as we have not fully recovered the interferometer alignment after the most recent pumpdown. I spent a couple of hours tweaking the alignment of the arm cavity mirrors, BS, and TTs to try and recover the maximum possible TRX and TRY - however, my best efforts only yielded TRX~0.8, TRY~0.75. Moreover, the beam axis is such that the spot is significantly off in YAW on both ETMs, as evidenced by the camera views (also true but less obvious on the ITMs). However, trying to bring the beam back to the center of the optics yields TRY and TRX values lower than the above reported maxima. The EX green beam is currently unavailable to verify the arm cavity alignment because of my hijacking the EX NPROs PZT control for PLL investigations, but with the Y arm, I'm able to lock a TEM00 mode. Probably just needs more careful systematic alignment, but I'm not pursuing this tonight.
For the beam spot position tracking, I am wondering if there is any benefit to going for a wider field of view and getting the OSEMs in the frame? It may provide some "anchor points" against which whatever algorithm can calibrate the spot position against. But there are also several point scatterers visible in the current view, and perhaps the Gaussiam beam profile moving over them and tracking the scattered intensity from these point scatterers serves the same function? I don't know of a good solution to have a "switchable" field of view configuration in the already cramped camera enclosure though.
Also, I think it may be useful to have a cron job take a picture of MC2 and archive it (once a week? or daily?) to have some long term diagnostic of how the scattered light received by the camera changes over several months.
The GigE is focused now and I have closed the lid. I'm attaching a picture of the MC2 beam spot, captured using GigE at an exposure time of 400µs
After helping Aaron key the crate and do a burt restore, I realized that it would probably be best to record the steps that Koji showed me to do a burt restore as a reference for (anyone) the future
Commands (in terminal):
Also Gautam explained today that the sticky slider problem is a hardware issue. That it basically means that the signal (voltage output for instance) that you request from the medm screen is not what the hardware delivers. Twice now, we have got around that with a burtrestore. My understanding of a burt restore is that it is a restoration of values from a certain time to the EPICS channels. Therefore, I don't understand why a restoration at the software level should fix how the hardware responds? Why does this happen?
The PMC was locking again after Gautam's steps above. However, after I added the directional coupler between the mixer I and the servo card (coupled to the Agilent analyzer), the PMC was again not locking, except occasionally with gain of -10 dB.
I removed the coupler (so the mixer I goes directly to the PMC servo card, as Gautam had it), and the PMC was still not locking. While checking connections, I noticed that one of the SMA cables between the LO and the mixer was not even finger tight, so I tightened them to approximately the right torque with a non-torque wrench.
This did not lead to the PMC locking, so Millind helped me key the c1psl VME crate. I burt restored the latest snapshot. Now, the PMC locks up until gain of -5. I try burt restoring the previous snapshot, which was from when the PMC was locking, and now it locks. Adding in the directional coupler again leads to the PMC not locking, though this time removing the coupler restores the normal behavior. I also tried using the coupler with the coupling port connected to a 50 Ohm terminator, and this configuration also did not lock.
I had been using a ZFDC-20-5-S+ (0.1-2000 MHz) with SMA ports and SMA-to-BNC on the input and output ports (since the mixer has BNC connectors). To reduce the number of potentially flaky connections, I am trying the ZFDC-20-4 (1-1000 MHz) that I found with BNC ports. The PMC still doesn't lock.
To get some spectrum, I've connected the PMC servo card's 'mixer out' to the Agilent's A channel, and collected a spectra from [10 Hz, 75 kHz], [75 kHz, 750 kHz], and [750 kHz, 2 MHz].
Wed Jun 26 15:23:37 2019
After the lab cleaning, I added a BNC T on the mixer I port, so now the configuration is:
Mixer I -> BNC T
-> PMC Servo card FP1TEST
-> directional coupler -> coupled to the spectrum analyzer, out port is terminated with 50 Ohms.
I thought maybe the issue was that the TF from in->out on the directional coupler is not what I expect (and Gautam suggested the in-out port might block DC), but the PMC still does not lock in the above configuration, in which the coupler is not between the mixer and the servo board--so only reflections from the coupler should matter, I think.
However, even when I plug the mixer directly into the servo board, the PMC is not locking (again) with gain above -8 dB or so. I did a burt restore again, and this fixed the problem. I wasn't sure why this burt restore is working, because all I am changing is the DC output adjust voltage and the gain, and switching on/off FP1TEST. However, I observed that after running the PMC autolocking script, observing that the autolocker did not achieve lock as it swept through resonance, and cancelling the autolocker, the PMC again cannot be locked for high gains. When I let the autolocker complete, this doesn't happen, so probably I'm just not letting some channel return to its nominal value after being changed by the autolocker.
Now after another burt restore, I'm avoiding using the autolocker and am still having trouble locking with the BNC T + directional coupler configuration above. However, now I'm noticing that the PZT control mon is always railed, as long as FP1TEST is in the loop (and independent of the output adjust voltage). I try returning to the 'baseline' configuration (mixer -> PMC servo card directly), and the PMC locks but with only 0.68 V transmission (was >0.7 V before).
Per Gautam's earlier suggestion, I switched to using the Agilent 41800A probe instead of the directional coupler. I was able to lock the OMC with this probe on a BNC T coming out of the mixer (transmission is 0.71 V). I recorded the spectra of the PMC servo board's "Mixer Out" channel, and the mixer's I as seen by the probe. I recorded spectra from 10 Hz to 100 MHz. The soft linked netgpibdata folder I had in my users directory is no longer soft linked--presumably intentional so I don't tamper with it?
I'm a bit skeptical that I've used the probe correctly, so I'm checking out the manual.
Indeed, I needed to pull back the sheath; I also noticed that the GPIB script I've been using doesn't save the data from both channels when I take a spectrum in dual mode, so I'm taking the spectra again one at a time (lights are on, IMC is locked).
I discussed this with Gautam and he asked me to come up with a list of signals that I would need for my use and then design the data acquisition task at a high level before proceeding. I'm working on that right now. We came up with a very elementary sketch of what the script will do-
Tomorrow I will try and prepare a dummy script for this before the meeting at noon. Gautam asked me to familiarize myself with the awg, cdsutils (I have already used ezca before) to write the script. This will also help me do the following two tasks-
Upcoming work (in the order of priority):
I am pushing the code that I wrote for
to the GigEcamera repository.
Gautam also asked me to look at Jigyasa's report and elog 13443 to come up with the specs of a machine that would accomodate a dedicated camera server.
Aaron complained to me earlier that the PMC could not be locked. Turned out to be a classic sticky slider problem, I keyed the c1psl VME crate, and did the usual burtrestore trick. After that, I could immediately lock the PMC and IMC with the nominal gain settings.
I also looked at the wiring at the rack. An SLP250 was installed at the mixer IF output, in parallel with a 50 ohm terminator to ground. I removed this, because as Aaron pointed out, the PMC servo card "FP1 TEST" input is already 50 ohm, and has two cascaded LC filter stages immediately after to filter out the 2f component, so the extra low-pass filtering is superfluous (in any case, 250 MHz is much too high a cutoff to be using for cutting out the 2f component which will be at ~70 MHz).
Finally, in the last ~2 weeks, we have been running with the PMC servo gain of +17 (as opposed to +18 from before). The old gain is too high, as noted by Milind. But the MEDM field for this gain goes RED unless the gain is +18. I adjusted the value of the C1:PSL-STAT_PMC_NOM_GAIN channel to +17, so that this is no longer the case. I also edited the PMC MEDM screen to get rid of my comment that the "SLOW ADC IS DEAD" for the PMC TRANS field, since I have now hooked up the PMC trans photodiode to our temporary Acromag box.
In the previous meeting, Koji pointed out (once again) that I should determine if the displacement values and frames are synchronized before training a network. Pooja did the following last time. Koji also suggested that I first predict the motion (a series of x and y coordintates) and then slide resulting plots around until I get the best match for the original motion. This is however not possible with a neural network based approach as the network learns exactly what you show it and therefore it will learn any mismatch between the labels and the frames and predict exactly that. Therefore I came up with what Koji described as "hacky" method to achieve the same using the opencv work described previously in this elog (the only addition being the application of a mask to block out the OSEMs and work only with the beam spot) .
Hacky technique to sync frames and labels:
[Koji, Milind - 21/06/2019]
Worked further on this. I skimmed through a few resources to look for details of what pre-processing can be done. Here (am planning to convert all these resources, particularly those I come across for GANs into either a README on the repo or a Wiki soon) are some of the useful things I found during today's reading. The work I skimmed through today mostly pointed to the use of a median filter for pre-processing, if any is to be done. I am presently using the Sequential() API in Keras to set up the neural network. I will train it tomorrow.
aI went to repeat these measurements using the mixer out channel from the servo box, and with a slower sweep for the PDH calibration.
I had trouble getting the PDH signal, here are some notes:
attachment 1 is the configuration of the PMC screen when I was trying to get some PDH signal; I did move the DC output adjust to 0V, but found that this led to the output being railed; this makes sense, the op amp at U9 has a negative bias at GND.
Rana came by and gave me some tips.
We finally got the PDH signal again, and I recorded the PDH signal while driving with the following settings on the Siglent function generator.
I tried getting a spectrum using the coupler, the mixer mon is seeing a DC offset though and causing the PZT to rail. Will try to understand why, but in the meantime removing the coupler (still no LP filter) lets us lock the PMC again.
RXA: Kruthi thinks all of our subsequent IMC locking problems are Aaron's fault (she was quick to give him up as soon as the thumb screws were tightened...)
This afternoon, Gautam helped me setup Giada to access the GigE installed for MC2. Unlike Paola, which was being used earlier, Giada has a better battery life and doesn't shutdown when the charger is unplugged. Gautam configured Giada to enable its wireless connection to Martian, just like Koji had configured Paola (https://nodus.ligo.caltech.edu:8081/40m/14672). We also rerouted the ethernet cable we were using with the PoE adaptor from Netgear Switch in 1x2 to 1x6.
Attachment #1 - In loop error spectra, measured as Koji posted end of last week.
Attachment #2 - OLTF of the IMC loop.
Attachment #3 - Photo courtesy Koji showing the bank of BNC connectors used for these measurements.
Clearly, these measurements were taken in a time when the IMC was "well behaved". How to characterize what's happening when this isn't the case?
Over the last 24 hours, the IMC autolocker was able to keep the MC locked ~60% of the time. This is not particularly good, but is an improvement on ~2 weeks ago when the IMC couldn't be locked.
There are two periods, which I've indicated by vertical cursors, between which the autolocker was doing something strange - usually this kind of trend is caused by one or more of the VME crates being unresponsive and the autolocker gets stuck, but I confirmed that both c1psl and c1iool0 are telnet-able. So I conclude that the stability and reliability of the IMC loop is still not as good as it used to be.
Note also that while the PC drive RMS level mostly hovers around 1 V, there are several excursions above that level. This in itself isn't a new phenomenon. I will do some more characterization by measuring the in-loop error signal spectrum and maybe the OLTF of the IMC locking loop.
Let' see how stable or otherwise the config is. I must've jiggled some poor cable connection back into a good spot while working on the PSL table? Or the NPRO decided to be less noisy on Sunday.
Note that I have removed an SR785, an oscilloscope, some SRS instruments from the PSL and PMC last night.
But they (and RF Network Analyzer) were not there when the problem started.
We should record the IMC error (at test point monitor) too? If the IMC locks on Monday too, I'll do it.
After typing up the elog, I decided to try locking the IMC again - now it locks again with the "OLD" gain settings. I tested it ~5 times, the autolocker brings the lock back and the PC drive levels are normal. IMC transmission and MC REFL DC light levels in lock are normal. The PC Drive RMS voltage is <1V. What's more, there is no longer any evidence of 60 Hz line harmonics any more in the PMC diagnostics channels. Compare attachment #1 to this elog.
I undid the changes Koji made to the autolocker gains, and am trying the old settings again. Let' see how stable or otherwise the config is. I must've jiggled some poor cable connection back into a good spot while working on the PSL table?
Anyway, this helps Kruthi and Milind.
Over the last few days, I've been doing some (complementary) measurements to what Aaron and Koji have been looking at. The motivation was to identify if the problems we are seeing are optical (i.e. imprinted on the PSL light) or electronic. My findings:
Attachment #1: Time domain look at PMC Refl and Trans signals under various operating conditions. During this work, I took the chance to remove ~4 BNC T connectors that were connected on the PMC TRANS photodiode (Thorlabs). Now, there is one cable going to the Acromag ADC, and one going to the Oscilloscope used to monitor these signals. Any further T-ing can be done at the oscilloscope.
Attachment #2: RIN measurement of the NPRO light. I opted to place a Thorlabs PDA55 in the IR ALS pickoff light path. This is before the light sees the PMC. A DC block was inserted between the PDA55 and the AG4395 used to make the measurement. DC level of the PD output was 3.1 V into high-Z and I used half this value to normalize the measurement made by the 50-ohm input AG4395 into RIN units. The measurement was made with the PZT and slow temperature controls to the NPRO connected/disconnected, but I saw no significant difference.
Attachment #3: Frequency noise measurement via PLL. This shows the loop transfer funtion for the PLL. Some details of the setup:
Attachment #4: Frequency noise measurement via PLL. This shows the frequency noise. I've overlaid the expected frequency noise between 2 free-running NPROs, model used is in the text box in the plot. There isn't strong evidence of excess high frequency noise in this measurement. The fact that the "LB 1005 input terminated" trace is below all the others supports the hypothesis that I'm measuring real frequency noise. The bump around a few kHz could indicate some gain peaking?
However, I'm unable to find good agreement between the measured frequency noise using the error point and the control point. For the former, I used the PLL discriminant mentioned above of 400 mV/rad, and undid the loop suppression, and for the latter I used a PZT discriminant of 1.7 MHz/V. However, there is still a constant scale difference between these two traces. So I'm doing something wrong?
I've not disturbed the PLL setup in case anyone else wants to repeat these measurements, but I have restored the normal electrical connections to the PSL PZT and temperature control.
Some other activity:
The IMC REFL error signal was measured to compare it with the other spectra (if we have).
The blue curve is the in-loop IMC error and the red is the dark noise. So they are not an apple-to-apple comparison. But the red noise is going to be suppressed by the loop, and still the red is below blue. This means that the blue curve is the measured noise rather than the readout noise.
We suspect that the current issue is the PC drive saturation (as usual). Does this indicate that the laser freq noise is actually increased?
Another suspect was that the degradation of the LO level. We used to have the issue of slowly dying ERA-5 (ERA-5SM indeed). The RF levels on the demod board were measured using an active probe.
The LO input: 0dBm, ERA-5 input: -2.7dBm and -2.1dBm for I and Q. I found that the outputs of the ERA-5SM were +10.5dBm and +10.6dBm.
This lead me to replace the chips but the situation was not changed. Then I realized that the LO levels should have been measured with the load replaced from the mixers to a 50Ohm load. Somehow these mixers lower the apparent LO levels. So I decided to say this is OK.
I think a Faraday rotator rotates the polarizations in a same way for both forward and backward beam, and it's not like in this figure.
And the transmission through multiple faradays will also be a big issue.
Here are the results from the fit. Data can be found on nodus in /users/aaron/40m/data/PMC/190617/. I've put a jupyter notebook with the analysis in /users/aaron/40m/analysis/PDH_calibrate.ipynb (might be some filename issues due to different directory structure on my laptop).
Here's a summary of the current measurement. I'll be referencing the diagram for the PMC servo card.
In the figures below, I obesrved that for fast (100Hz) drives, the PDH error signal had a pi/2 phase shift relative to the triangle wave, which means even though the resonance appears near the turnaround of the triangle, it is actually occuring near the center of the range.
There are several problems with this data:
Begun setting up an environment (as mentioned before, on my local machine) and scripts to run experiments with Convolutional networks for beam tracking. All code has been pushed to this folder in the GigEcamera repository. I am presently looking for pre-processing techniques for the video which go beyond the usual "Crop the images! Normalize pixel values! Convert to Grayscale!".
I made a script (/users/aaron/40m/GPIB/tds_dump.py) that grabs data from a Tektronix scope and packages into a pickled dict with the following structure:
I made a python notebook that does the following:
The fit in step (5) is still looking quite bad, despite the fitted values being close to the expected. Since we really just want a calibrated spectrum, I'll instead fit a line to the linear portion of the PDH error signal for the carrier and both sidebands, then determine the scaling from that.
Milind pointed out that all boxes on the medm screens were white. I didn't have diagnostics from the medm screens, so I started following the troubleshooting steps on the restart procedures page.
It seemed like maybe a frontend problem. I tried telnet-ing into several of the fe, and wasn't able to access c1psl. The section on c1psl mentions that if this machine crashes, the screens will go white and the crate needs to be turned off and on. Millind did this.
Now, most of the status lights are restored (screenshot).
Milind: I did a burtrestore following this and locked the PMC following the steps described in this elog.