Yesterday, Rana asked me to look at Hiro Yamamoto's docs on the DCC to improve the simulation. I'm performing a first pass (=> Just skimming through to see if they're relevant, I will go through them more carefully soon!) and putting up stuff here for future reference. @Kruthi's help much appreciated!
The PMC was locking again after Gautam's steps above. However, after I added the directional coupler between the mixer I and the servo card (coupled to the Agilent analyzer), the PMC was again not locking, except occasionally with gain of -10 dB.
I removed the coupler (so the mixer I goes directly to the PMC servo card, as Gautam had it), and the PMC was still not locking. While checking connections, I noticed that one of the SMA cables between the LO and the mixer was not even finger tight, so I tightened them to approximately the right torque with a non-torque wrench.
This did not lead to the PMC locking, so Millind helped me key the c1psl VME crate. I burt restored the latest snapshot. Now, the PMC locks up until gain of -5. I try burt restoring the previous snapshot, which was from when the PMC was locking, and now it locks. Adding in the directional coupler again leads to the PMC not locking, though this time removing the coupler restores the normal behavior. I also tried using the coupler with the coupling port connected to a 50 Ohm terminator, and this configuration also did not lock.
I had been using a ZFDC-20-5-S+ (0.1-2000 MHz) with SMA ports and SMA-to-BNC on the input and output ports (since the mixer has BNC connectors). To reduce the number of potentially flaky connections, I am trying the ZFDC-20-4 (1-1000 MHz) that I found with BNC ports. The PMC still doesn't lock.
To get some spectrum, I've connected the PMC servo card's 'mixer out' to the Agilent's A channel, and collected a spectra from [10 Hz, 75 kHz], [75 kHz, 750 kHz], and [750 kHz, 2 MHz].
Wed Jun 26 15:23:37 2019
After the lab cleaning, I added a BNC T on the mixer I port, so now the configuration is:
Mixer I -> BNC T
-> PMC Servo card FP1TEST
-> directional coupler -> coupled to the spectrum analyzer, out port is terminated with 50 Ohms.
I thought maybe the issue was that the TF from in->out on the directional coupler is not what I expect (and Gautam suggested the in-out port might block DC), but the PMC still does not lock in the above configuration, in which the coupler is not between the mixer and the servo board--so only reflections from the coupler should matter, I think.
However, even when I plug the mixer directly into the servo board, the PMC is not locking (again) with gain above -8 dB or so. I did a burt restore again, and this fixed the problem. I wasn't sure why this burt restore is working, because all I am changing is the DC output adjust voltage and the gain, and switching on/off FP1TEST. However, I observed that after running the PMC autolocking script, observing that the autolocker did not achieve lock as it swept through resonance, and cancelling the autolocker, the PMC again cannot be locked for high gains. When I let the autolocker complete, this doesn't happen, so probably I'm just not letting some channel return to its nominal value after being changed by the autolocker.
Now after another burt restore, I'm avoiding using the autolocker and am still having trouble locking with the BNC T + directional coupler configuration above. However, now I'm noticing that the PZT control mon is always railed, as long as FP1TEST is in the loop (and independent of the output adjust voltage). I try returning to the 'baseline' configuration (mixer -> PMC servo card directly), and the PMC locks but with only 0.68 V transmission (was >0.7 V before).
Per Gautam's earlier suggestion, I switched to using the Agilent 41800A probe instead of the directional coupler. I was able to lock the OMC with this probe on a BNC T coming out of the mixer (transmission is 0.71 V). I recorded the spectra of the PMC servo board's "Mixer Out" channel, and the mixer's I as seen by the probe. I recorded spectra from 10 Hz to 100 MHz. The soft linked netgpibdata folder I had in my users directory is no longer soft linked--presumably intentional so I don't tamper with it?
I'm a bit skeptical that I've used the probe correctly, so I'm checking out the manual.
Indeed, I needed to pull back the sheath; I also noticed that the GPIB script I've been using doesn't save the data from both channels when I take a spectrum in dual mode, so I'm taking the spectra again one at a time (lights are on, IMC is locked).
After helping Aaron key the crate and do a burt restore, I realized that it would probably be best to record the steps that Koji showed me to do a burt restore as a reference for (anyone) the future
Commands (in terminal):
Also Gautam explained today that the sticky slider problem is a hardware issue. That it basically means that the signal (voltage output for instance) that you request from the medm screen is not what the hardware delivers. Twice now, we have got around that with a burtrestore. My understanding of a burt restore is that it is a restoration of values from a certain time to the EPICS channels. Therefore, I don't understand why a restoration at the software level should fix how the hardware responds? Why does this happen?
a useful piece of code that we should ask one of this summer's SURFs to write:
this will solve the sticky slider problem and do it in a systematic way. We can run it from the command line: e.g. 'unsticky.py c1psl c1ioo c1lsc'
Aaron complained to me earlier that the PMC could not be locked. Turned out to be a classic sticky slider problem,
The GigE is focused now (judged by eye) and I have closed the lid. I'm attaching a picture of the MC2 beam spot, captured using GigE at an exposure time of 400µs.
What was the solution to resolving the flaky video streaming during the alignment process????
-> I think, the issue was with either the poor wireless network conection or the GigE-PoE ethernet cable.
Turns out, focusing the GigE is actually a bit tricky. With pylon, everytime I change the exposure or the focus, I'm running into the error I had mentioned earlier in one of my elogs; so I tried using the python scripts to interact with the GigE. But whenever I try to change the focal plane distance by rotating the lens coupler, the ethernet cable connection becomes loose and the camera server needs to be relaunched every now and then. Also, everytime we want to change the distance between the lenses, the telescope needs to be dismantled and refocused again. I'll try to come up with a better telescope design for this.
Yesterday, I had focused the GigE using a low exposure time and small aperture of iris, to make sure that we are actually seeing a sharp image of the beam spot. I'm attaching a picture of the beam spot I had clicked while focusing it, unfortunately, I forgot to take a picture after I had focused it completely. I'm also attaching a picture of the final setup for future reference.
Yesterday night, Rana asked me to lock the MC2. I figured that the PSL shutter was closed; I just opened it and was able to see the beam spot on the analog camera screen.
For the beam spot position tracking, I am wondering if there is any benefit to going for a wider field of view and getting the OSEMs in the frame? It may provide some "anchor points" against which whatever algorithm can calibrate the spot position against. But there are also several point scatterers visible in the current view, and perhaps the Gaussiam beam profile moving over them and tracking the scattered intensity from these point scatterers serves the same function? I don't know of a good solution to have a "switchable" field of view configuration in the already cramped camera enclosure though.
Also, I think it may be useful to have a cron job take a picture of MC2 and archive it (once a week? or daily?) to have some long term diagnostic of how the scattered light received by the camera changes over several months.
The GigE is focused now and I have closed the lid. I'm attaching a picture of the MC2 beam spot, captured using GigE at an exposure time of 400µs
Now that the IMC is remaining locked for extended periods of time, the next problem to attack is the ASS dither alignment system. For a start, I decided to try and get the POX and POY locking working again, as we have not fully recovered the interferometer alignment after the most recent pumpdown. I spent a couple of hours tweaking the alignment of the arm cavity mirrors, BS, and TTs to try and recover the maximum possible TRX and TRY - however, my best efforts only yielded TRX~0.8, TRY~0.75. Moreover, the beam axis is such that the spot is significantly off in YAW on both ETMs, as evidenced by the camera views (also true but less obvious on the ITMs). However, trying to bring the beam back to the center of the optics yields TRY and TRX values lower than the above reported maxima. The EX green beam is currently unavailable to verify the arm cavity alignment because of my hijacking the EX NPROs PZT control for PLL investigations, but with the Y arm, I'm able to lock a TEM00 mode. Probably just needs more careful systematic alignment, but I'm not pursuing this tonight.
After a more systematic alignment effort, I was able to get the spots better centered on the optics (judged by eye from the analog camera views). TRY ~0.7, TRX~1.15. The X-arm dither alignment system seems to work out-of-the-box with the existing settings, I was able to run it and maximize the X-arm transmission.
Other work: I also cleaned up the area around MC2 a litte - laptop from on top of the vacuum chamber was removed and a rogue ethernet cable was also removed. The resulted in some misalignment of the IMC, which I corrected by manual alignment. Now the IMC is locked again with nominal transmission levels.
On the PSL table, I re-routed the RF output from the BeatMouth to the regular IR-ALS electronics chain (it was hijacked for PLL investigations). At EX, I disconnected the cable running from the LB1005 to the EX NPRO laser PZT (again was being used for PLL locking), and re-connected the output from the Green uPDH box to allow for some ALS tests to be done. I could then lock the EX green beam to the X-arm, and achieved GTRY ~ 0.35 using the ASX system. More to follow on ALS tests later today.
And finally, a network is trained!
Result summary (TLDR :-P) : No memory was used. Model trained. Results were garbage. Will tune hyperparameters now. Code pushed to github.
More details of the experiment:
What I did:
What I saw
What I think is going wrong-
Well, what now?
I got to speak to Gabriele about the project today and he suggested that if I am using Rana's memory based approach, then I had better be careful to ensure that the network does not falsely learn to predict a sinusoid at all points in time and that if I use the frame wise approach I try to somehow incorporate the fact that certain magnitudes and frequencies of motion are simply not physically possible. Something that Rana and Gautam emphasized as well.
Experiment file: train.py
Finding the gain of the Photodiode: The three-position rotary switch of the photodiode being used (PDA520) wasn't working, so I determined its gain by making a comparative measurement between ophir power meter and the photodiode. The photodiode has a responsitivity of 0.34 A/W at 1064 nm (obtained from the responsitivity curve given in the spec sheet using a curve digitizing software). Using the following equation, I determined the gain setting, which turned out to be 20dB.
Setup: Here a 1050nm (closest we have to 1064nm) LED is used as the light source instead of a laser to eliminate the effects caused by coherence of a laser source, which might affect our radiometric calibration. The LED is placed in a box with a hole of diameter 5mm (aperture angle = 40 degrees approx.). Suitable lenses are used to focus the light onto a white paper, which is fixed at an arbitrary angle and serves as a Lambertian scatterer. To make a comparative measurement between the photodiode (PDA520) and GigE, we need to account for their different sensor areas, 8.8mm (aperture diameter) and 3.7mm x 2.8 mm respectively . This can be done by either using an iris with a common aperture so that both the photodiode and GigE receive same amount of light , or by calculating the power incident on GigE using the ratio of sensor areas and power incident on the photodiode (here we are using the fact that power scattered by Lambertian scatterer per unit solid angle is constant).
Calibration of GigE 152 unit: I took around 50 images, starting with an exposure time of 2000 in steps of 2000, using the exposure_variation.py code. But the code doesn't allow us to take images with an exposure time greater than 100 ms, so I took few more images at higher exposures manually. From each image I subtracted a dark image (not in the sense of usual CCD calibration, but just an image with same exposure time and no LED light). These dark images do the job of usual dark frame + bias frame and also account for stray lights. A plot of pixel sum vs exposure time is attached. From a linear fit for the unsaturated region, I obtained the slope and calculated the calibration factor.
Result: CF = 1.91x 10^-16 W-sec/counts Update: I had used a wrong value for the area of photodiode. On using 61.36 mm^2 as the area, I got 2.04 x 10^-15 W-sec/counts.
I'll put the uncertainities soon. I'm also attaching the GigE spectral response curve for future reference.
we are thinking of doing a sprucing up of the input mode cleaner WFS (sensors + electronics + feedback loops)
I wanted to try out the unstick.py script on c1aux but kept running into timeout errors. I was also confronted by a blank GigE screen. Further, couldn't telnet into c1aux using telnet c1aux as described here. Therefore, I went in and keyed the c1aux crate (1Y1).
Wrote the script. It currently lives at /users/milind/NonlinearControl/milind/unstick/unstick.py. Also pushed it to the repo here. It does the following:
I tried print statements instead of actually writing to the channels as Gautam asked me to do that with supervision. Is this good enough?
> For channels corresponding to continuous values (such as say exposure time or the like) changes to abs(1+current_value)
Why abs? Is the current_value is like -5.4321 (for example for the alignment slider), this returns +4.4321 and the suspension will suffer from huge motion (well it will be returned to the original value soon though).
Made changes as discussed in this issue. Further, I need to add signal handling capabilities to the code. I belive Jon has pointed me to some code. I will take a look at that ASAP.
Today, I read a lot more about BRDF and modelling but could not make much headway regarding the implementation in the simulation. I've stopped for now and I'll take a crack at it tomorrow again.
I've begun working on this. Steps to complete:
As Rana asked me to in the last meeting, I dug through the elogs to determine what had become of the previous autolockers. I stumbled upon this elog by Rana from before Gautam cleaned up the medm screen. Out of curiosity, I ran the autolocker script using the instructions in Rana's elog. I did this a total of 5 times and could lock the PMC 3 times fairly quickly. I attempted to decipher the details of the code but did not make much headway owing to my unfamiliarity with the language. From what I could make out from the medm screen while the autolocker was running, it appeared to be the same method as that in this elog. I will take a look at it again tomorrow. However, I intend to spend most of tomorrow working on preprocessing the data, developing the CNN script and then the simulation.
To practise the dither alignment servo tuning, I decided to make the ASX system work again (mainly because it has fewer DoFs and so I thought it'd be easier to manage). Setup is: dither PZT mirrors on EX table-->demodulate green transmission at the dither frequencies-->Servo the error signals to 0 by an integrator.
The adjusted demod phases, servo gains were saved to the .snap file which gets called when we run the "DITHER ON" script. Also updated the StripTool template.
I plan to repeat similar characterization on the IR dither alignment servos. I think the tuning of the ASS settings can be done independently of figuring out the mystery of why the TRY level is so low.
Just finished a raw version of the autolocker!! Tested it once and was able to achieve lock! This is a python version of the code at /opt/rtcds/caltech/c1/scripts/PSL/PMC/AutoLock.sh.
The current code lives in my users directory. Gautam asked me to put the completed autolocker at /opt/rtcds/caltech/c1/scripts/PSL/PMC/ and that I needn't necessarily put it on git. However, I had previously added it to my Non-linear control repo. Not sure if I should take it off? The current script still lacks some checks like those that enable it to stop after a certain time of attempting to lock or those that handle interrupt signals. Will do that in some time.
P.S. As Koji says, Victory! :-P
P.P.S. Rana pointed out that this is not the objective and what we actually wanna do is run a search over the parameter space of the locking process. I will document my ideas about this process as soon as I do a little more reading. He also said that it would not do to have command line arguments as the main source from which parameters are procured and that .yml files ought to be used instead. I will make that change asap.
We crossed off another couple of bullets today.
It took me ~1 hour to realize that c1susaux requries the running of sudo /sbin/ifup eth0 to be run in order to see the martian network - why???
I don't understand the exact chain of causation, but during this work, the fast c1sus model crashed. I had to go through a few iterations of the scripted vertex machine rebooting, but things seem to be back in a normal state now, see Attachment #2. Should probably run the IFO test suite to make sure everything is a-okay, but for now, I am able to lock the IMC so I'm moving on.
The main task remaining here is to take new pictures of everything and upload to the wiki. Also, need to update the Sorensen labels to reflect their current values, some of them are outdated.
Since the work earlier this morning, the fast c1sus model has crashed ~5 times. Tried rebooting vertex FEs using the reboot script a few times, but the problem is persisting. I'm opting to do the full hard reboot of the 3 vertex FEs to resolve this problem.
Judging by Attachment #1, the processes have been stable overnight.
In preparation for the ASS debugging, I decided to check out the beam path on the EY table. In order to be able to do this, I had to setup the POY locking to trigger on AS110 instead of TRY (as is usual for this kind of debugging). Then I could poke an IR card in the beam path without destroying the lock.
There are two irides in the beam path immediately between the vacuum window and the harmonic separator that splits off the IR and green beams. I found that the beam was in fact getting clipped on both of them. It was also somewhat off center on a 2" beamsplitter that sends half of the light to the QPD (currently decommissioned). The purpose of these irides are (I think) to eliminate some ghost reflections of the green beam and also the Oplev beam. I opened up the irides until I felt that there wasn't any more clipping of the IR beam, but the appropriate ghost beams were still getting caught.
I also re-aligned the beam onto the TRY Thorlabs PD so as to better center it on the active area. In summary, the result of this work was that the TRY level went from ~0.6 to ~0.93. There may still be some scope for optimizing this - I tried running the Y-arm ASS scripts, and already, the loops don't run away any more. I'll do the systematic analysis of the servo anyways. But given that the IMC Trans lev el used to be ~15,500 counts and is now ~14,500 counts, I think ~7% drop in TRY level is in line with what we "expect" (assuming the pre-power-degradation TRY level was 1.000).
Note that these irides were installed (I think) by Yuki, and so cannot explain the ASS anomalies of July 2018 (i.e. it does not exonerate in-vacuum clipping of the beam, as Koji had already verified that the in-air path was clean back then).
The latest in my fling with the PMC. Though PMC trans is back to nominal levels (~0.713 V), we'd still like to understand the PMC noise.
Last time, I took some spectra with the RF probe (Agilent 41800A). I had already measured the PDH error signal by sweeping the PZT at ~1 Hz. The notebook I used for analysis has been updated in /users/aaron/analysis/PDH_calibrate.ipynb. The analysis was the following:
There are a few problems remaining:
A question was raised as to how much passive filtering we benefit from if we pick off the local oscillator beam for BHD from the PRC. I did some simplified modeling of this. For the expected range of arm cavity round trip losses (20-50 ppm), I think that the 40m CARM pole will be between 75-85 Hz. The corresponding recycling gain will be 40-50, with the current PRM. I assumed 1000 ppm loss inside the PRC. The net result is that, assuming the single pole coupled cavity response, we will get ~8-9 dB of filtering at ~200 Hz of the intensity noise of the input laser field to the interferometer if we pick the LO beam off from the PRC (e.g. PR2 transmission), instead of picking it off before.
The next questions are: (i) can we do a sufficiently good job of achieving the required RIN stability on the LO field for BHD without relying on the passive filtering action of the PRC? and (ii) is the benefit of the PRC filtering ruined in the process of routing the LO field from wherever the pickoff happens to the BHD setup?
Tried collecting data today. Was unable to keep the camera_server code running for any length of time as it threw segfaults. Will take a shot again tomorrow.
There was a magnitude 6.6 earthquake just a few minutes ago. I am attaching photographs of the monitor feeds for reference here. Is there a standard protocol to be followed in this situation? I'm looking through the wiki now.
Further, the IMC seems to be misaligned and is not locking! As Koji has let me know, I really hope this is not too serious and can be fixed easily.
The quoted elog has figures which indicate that the network did not learn (train or generalize) on the used data. This is a scary thing as (in my experience) it indicates that something is fundamentally wrong with either the data or model and learning will not happen despite how hyperparameters are tuned. To check this, I ran the training experiment for nearly 25 hyperparameter settings (results here)with the old data and was able to successfully overfit the data. Why is this progress? Well, we know that we are on the right track and the task is to reduce overfitting. Whether, that will happen through more hyperparameter tuning, data collection or augmentation remains to be seen. See attachments for more details.
Why is the fit so perfect at the start and bad later? Well, that's because the first 90% of the test data is the training data I overfit to and the latter the validation data that the network has not generalized well to.
Koji came to the lab to align the IMC/IFO, but found the mirrors are dancing around. Kruthi told me that there was M7.1 EQ at Ridgecrest. Looks like there are aftershocks of this EQ going on. So we need to wait for an hour to start the alignment work.
ITMX and ETMX are stuck.
- ITM unstuck now
- IMC briefly locked at TEM00
A series of aftershocks came. I could unstick ITMX by turning on the damping during one of the aftershocks.
Between the aftershocks, MC1~3 were aligned to the previous dof values. This allowed the IMC flashing. Once I got the lock of a low order TEM mode, it was easy to recover the alignment to have a weak TEM00.
Now at least temporarily the full alignment of the IMC was recovered.
In fact, ETMX was not stuck until the M7.1 EQ today. After that it got stuck, but during the after shocks, all the OSEMs occasionally showed full swing of the light levels. So I believe the magnets are OK.
I modified the autolocker code I wrote to read from a .yaml configuration file instead of commandline arguements (that option still exists if one wishes to override what the .yaml file contains). I have pushed the code to github. I started reading about MCMC and will put up details of the remaining part of the work ASAP.
P.P.S. He also said that it would not do to have command line arguments as the main source from which parameters are procured and that .yml files ought to be used instead. I will make that change asap.
The beam splitter (BS1-1064-33-2037-45S) that is currently being used has an antireflection coating on the second surface and a wedge of less than 5 arcmin; yet it leads to ghosting as shown in the figure attached (courtesy: Thorlabs). I'm also attaching its spec sheet I dug up on internet for future reference.
I came across pellicle beamsplitters, that are primarily used to eliminate ghost images. Pellicle beamsplitters have a few microns thick nitrocellulose layer and superimpose the secondary reflection on the first one. Thus the ghost image is eliminated.
Should we go ahead and order them? (https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=898
I came across a paper (see reference) where they have used DAOPHOT, an astronomical software tool developed by NOAO, to study the point scatterers in LIGO test masses using images of varying exposure times. I'm going through the paper now. I think using this we can analyze the MC2 images and make some interesting observations.
Reference: L.Glover et al., Optical scattering measurements and implications on thermal noise in Gravitational Wave detectors test-mass coatings Physics Letters A. 382. (2018)
After the two earthquakes, I collected some data by dithering the optic and recording the QPD readings. Today, I set up scripts to process the data and then train networks on this data. I have pushed all the code to github. I attempted to train a bunch of networks on the new data to test if the code was alright but realised quickly that, training on my local machine is not feasilble at all as training for 10 epochs took roughly 6 minutes. Therefore, I have placed a request for access to the cluster and am waiting for a reply. I will now set up a bunch of experiments to tune hyperparameters for this data and see what the results are.
Trainng networks with memory
I set up a network to handle input volumes (stacks of frames) instead of individual frames. It still uses 2D convolution and not 3D convolution. I am currently training on the new data. However, I was curious to see if it would provide any improved performance over the results I put up in the previous elog. After a bit of hyperparameter tuning, I did get some decent results which I have attached below. However, this is for Pooja's old data which makes them, ah, not so relevant. Also, this testing isn't truly representative because the test data isn't entirely new to the network. I am going to train this network on the new data now with the following objectives (in the following steps):
I hope this looks alright? Rana also suggested I try LSTMs today. I'll maybe code it up tomorrow. What I have in mind- A conv layer encoder, flatten, followed by an LSTM layer (why not plain RNNs? well LSTMs handle vanishing gradients, so why the hassle).
you have to use a BS with a larger wedge angle (5 arcmin ~ 1 mrad) so that the beams don't overlap on the camera
Today, Gautam keyed the C1PSL crate and we got to test my unstick.py code. It seems to be working fine. Remarks:
Following this, we tested my PMC autolocker code. The code ran for about a minute before achieveing lock. Remarks:
The Y-arm ASS was tuned to be in a workable state. Basically, I followed Koji's recipe.
The SNR of the dither lines in the TRY and YARM control signals were checked - Attachment #1. The dither frequencies are marked with vertical dashed lines (can't figure out how to add 4 cursors in DTT so there's two in each row for a total of 4). A couple of days ago, when I was doing some preliminary checks, I found that the oscillator at 24.91 Hz caused a broadband increase in the TRY noise between DC and ~100 Hz. But today I saw no evidence of such behaviour. So I decided against changing the frequency.
The linearity of the demodulated error signals around the quadratic maxima of the TRY level was checked. I did not, however, investigate in detail the frequency-dependent offset Koji has reported in his elog.
After this work, the TRY level is at 0.95. This is commensurate with the MC trans level being lower by ~7% relative to July 2018. Furthermore, the ASS servo is able to return to TRY~0.95 with a time-constant of ~5 seconds in response to misalignment of the cavity optics. After I investigate the X-arm ASS, I will reset the normalization for TRX and TRY.
Update 645pm: In the spirit of general IFO recovery, I re-centered the ITM and ETM oplev spots, and also the IR beam on the IPPOS QPD to mark the new input pointing alignment (the spot is slightly lower on the AS camera than what I remember). I then tweaked the XARM transmission to maximize it, and re-set the TransMon normalization. I edited the normalization script to comment out the normalizing of the TransMon QPD gains as the QPDs are in some kind of indeterminate state now. Attachment #2 shows the current status, you can also see the normalization being reset. LSC mode disabled for overnight.
Once the XARM ASS is also checked out, I propose moving back to locking the DRMI / PRFPMI configs.
Last documented replacement in Nov 2018, so ~7 months, which I believe is par for the course. I am disconnecting its power supply cable.
There was no green light even though the EX NPRO was on. I checked the doubling oven temperature controller and found that its power cable was loose on the rear. I reconnected it, and now there is green light again.
I received access today. After some incredible hassle, I was able to set up my repository and code on the remote system. Following this, Gautam wrote to Gabriele to ask him about which GPUs to use and if there was a previously set up environment I could directly use. Gabriele suggested that I use pcdev2 / pcdev3 / pcdev11 as they have good gpus. He also said that I could use source ~gabriele.vajente/virtualenv/bin/activate to use a virtualenv with tensorflow, numpy etc. preinstalled. However, I could not get that working, Therefore I created my own virtual environment with the necessary tensorflow, keras, scipy, numpy etc. libraries and suitable versions. On ssh-ing into the cluster, it can be activated using source /home/millind.vaddiraju/beamtrack/bin/activate. How do I know everything works? Well, I trained a network on it! With the new data. Attached (see attachment #1) is the prediction data for completely new test data. Yeah, its not great, but I got to observe the time it takes for the network to train for 50 epochs-
Therefore, I will carry out all training only on this machine from now.
Note to self:
Steps to repeat what you did are:
I attempted to train a bunch of networks on the new data to test if the code was alright but realised quickly that, training on my local machine is not feasilble at all as training for 10 epochs took roughly 6 minutes. Therefore, I have placed a request for access to the cluster and am waiting for a reply. I will now set up a bunch of experiments to tune hyperparameters for this data and see what the results are.
Arnaud and I moved one of the two spare TT suspensions from the south clean cabinet to the bake lab clean room. The main purpose was to inspect the contents of the packaging. According to the label, this suspension was cleaned to Class A standards, so we tried to be clean while handling it (frocks, gloves, masks etc). We found that the foil wrapping contained one suspension cage, with what looked like all the parts in a semi-assembled state. There were no OSEMs or electronics together with the suspension cage. Pictures were taken and uploaded to gPhoto. Arnaud is going to plan his tests, so in the meantime, this unit has been stored in Cabinet #6 in the bake lab cleanroom.
In fact the projector is still working. The lamp timer showed ~8200hrs. I just reset the timer, but not sure it was the cause of the shutdown. I also set the fan mode to be "High Altitude" to help cooling.
We noticed that the PRM watchdog was tripping frequently. This is a period of enhanced seismic activity. The reason PRM in particular trips often is because the SIDE OSEM has 5x increased transimpedance. We implemented a workaround by modifying the watchdog tripping condition to scale the SD channel RMS by a factor of 0.2 (relative to the UL and LL channels). We restarted the modbus process on c1susaux and tested that the new logic works. Here is the relevant snippet of code:
# PRM Side is special, see elog 14745
field(DESC,"Tests whether RMS too high")
field(INPA,"C1:SUS-PRM_ULPD_VAR NPP NMS")
field(INPB,"C1:SUS-PRM_PD_MAX_VAR NPP NMS")
field(INPC,"C1:SUS-PRM_LLPD_VAR NPP NMS")
field(INPD,"C1:SUS-PRM_SDPD_VAR NPP NMS")
The db file has a note about this as well so that future debuggers aren't mystified by a factor of 0.2.
I trained a bunch (around 25 or so - to tune hyperparameters) of networks today. They were all CNNs. They all produced garbage. I also looked at lstm networks with CNN encoders (see this very useful link) and gave some thought to what kind of architecture we want to use and how to go about programming it (in Keras, will use tensorflow if I feel like I need more control). I will code it up tomorrow after some thought and discussion. I am not sure if abandoning CNNs is the right thing to do or if I should continue probing this with more architectures and tuning attempts. Any thoughts?
Right now, after speaking to Stuart (ldas_admin) I've decided on coding up the LSTM thing and then running that on one machine while probing the CNN thing on another.
Update on 10 July, 2019: I'm attaching all the results of training here in case anyone is interested in the future.
I heard a popping sound in the control room; the projector lightbulb has blown out.
Arnaud has taken 1 TT suspension from the 40m clean lab to Downs for modal testing. Estimated time of return is tomorrow evening.
All suspension watchdogs were tripped ~90mins ago. I restored the damping. IMC is locked.
ITMX was stuck. I set it free. But notice that the UL Sensor RMS is higher than the other 4? I thought ITMY UL was problematic, but maybe ITMX has also failed, or maybe it's coincidence? Something for IFOtest to figure out I guess. I don't think there is a cable switch between ITMX/ITMY as when I move the ITMX actuators, the ITMX sensors respond and I can also see the optic moving on the camera.
Took me a while to figure out what's going on because we don't have the seis BLRMS - i moved the usual projector striptool traces to the TV screen for better diagnostic ability.
Update 16 July 1515: Even though the RMS is computed from the slow readback channels, for diagnosis, I looked at the spectra of the fast PD monitoring channels (i.e. *_SENSOR_*) for ITMX - looks like the increased UL RMS is coming from enhanced BR-mode coupling and not of any issues with the whitening switching (which seems to work as advertised, see Attachment #3, where the LL traces are meant to be representative of LL, LR, SD and UR channels).
Optical chopper borrowed from CryoLab to 40m
On Friday, I took images for different power outputs of LED. I calculated the calibration factor as explained in my previous elog (plots attached).
Power incident on photodiode (W)
To estimate the uncertainity, I assumed an error of at most 20mV (due to stray lights or difference in orientation of GigE and photodiode) for the photodiode reading. Using the uncertainity in slope from the linear fit, I expect an uncertainity of maximum 4%. Note: I haven't accounted for the error in the responsivity value of the photodiode.
Johannes had reported CF as 0.0858E-15 W-sec/counts for 12 bit images, with measured a laser source. This value and the one I got are off by a factor of 25. Difference in the pixel formats and effect of coherence of the light used might be the possible reasons.
On Friday, Koji helped me find various components required for the scatterometer setup. Like he suggested, I'll first set it up on the SP table and try it out with an usual mirror. Later on, once I know it's working, I'll move the setup to the flow bench near the south arm and measure the BRDF of a spare end test mass.