Here is a rundown of my work so far on RL controllers for the puck:

Environment: I implemented an environment that simulates the puck's temperature, according to our crude fits of the step response and pid controller of the puck in foam. Currently, this environment assumes constant ambient temperature and doesn't simualte the delays between the applied heat and the measured temperature change. However, one can easily adjust this environment by just changing the underlying Tdot equation. The environment works in steps of 10s and executes 3600*5 steps (for a total simulated time of 50h). The reward function is L2: at every timestep, it adds -(T_puck - T_ref)^2 to the total reward. The reward is only disclosed to the agent on the last timstep, so far this seemed to have work better, perhaps because the reward you get at any timestep is heavily dependent on the previous state (in 10s you can't raise the temperature enough if you are 5 degrees away from T_ref, so the agent gets penalized again).

Agents: I have made the first implementations for two agents: DQN and PPO, that could be used to controller the puck's temperature. Currently, these implementations are rough and require hyperparameter tuning and bigger networks.

• PPO: this agent works on continuous environemnts, meaning that it outputs any real number between min and max as its action. In the case of the puck, the action is the applied heat (in Watts). However, this agent has proven very slow to train on the LIGO-Caltech cluster (despite the networks being very small: 3 dense layers, 2->10->5->output for both critic and policy networks), so I only have a very rough training done, of 20 episodes Here is a plot:
  
  Here T_ref is represented by the red line and the blue line is the output bahvior of the policy. Keep in mind that this is only 20 episodes, and at the time of writing the training was killed (don't know why) but the measured total reward at 25 episodes was around 10% better than the above policy (sadly, no plot for that).
• DQN: this agent works on discrete environments, meaning that I had to modify the training environment to be dsicrete (which can be done in 1 line of code using a wrapper, see env.ipynb). There is an argument to be made that the actuator (in our case the PWM of the Raspberry Pi) has only a limited number of possible outputs: for example, the RPi PWM doesn't produce any measureable difference in duty cycles that differ by less than 0.05%. As Rana suggested, it would be best to discretize according to the number of bits that the plant can work with (I am not sure how many that is for the case of the RPi). Anyways, I am currently training the DQN on the env discretized to 100 actions, but it currently seems very unstable, even after I lowered the learning rate. The great thing about DQN is that it has been very fast to train, even on a cpu. Hence, I think the next step would be to just increase the size of the q-network, as the currrent 3 dense layers seems inadequate (2->10->5->100). I will come back with a better plot of this agent very soon.

Moving on, I want to:

1. Implement a reverb replay server for the PPO in hopes that it will speed up the training
2. Add T_env noise and sensor noise simulations to the environment.
3. Start hyperparameter tuning
4. Check results and compare with lthe inear controller

Update

1. I have finished setting up Neptune for easy access to the train/eval metrics of the RL models. Here is a link where you can track my progress. Neptune is an extremely useful tool that allows you to see in real time how the models are bahving: you can see nice plots of losses and returns, as well as my source code, the stdout and stderr, and even the resources plots for the node that my jobs got allocated to. Feel free to play around with it.
2. Currently, there are are only 2 runs (I just started them): one with DQN (RLCON-8), the other with PPO (RLCON-7).
3. As outlined before, I have tried improving the PPO agent by replacing the replay buffer with a reverb server (which is supposed to be the fast way of doing a replay buffer). However, no big speed ups have been achieved. The problem is that PPO is using a special type of layer: a distributional layer. I am unsure as to how this layer works, but certainly it is not a normal "dense" layer. After changing all other components, it seems that the PPO algorithm is very slow not because of the replay as I initially thought, but because of the actual training of the agent at line agent.train(experience).  However, I still have a lead: there is an error signal about tensorflow retracing (see run RLCON-8). I am now looking into that.
4. In the mean time, I will start increasing the size of the Q-net for the DQN and increasing the num_actions to 201 (so a resolution of 0.5% in duty cycle). That is because RLCON-7 seems to not learn too much: the loss is really unstable and the evaluation avg_return is terrible.

Aside on avg_return if you wonder what this number means: basically, at each step of the environment (the code that simulates the temperature of the puck) I am takig the error to be . Then, for an entire episode, the reward is given by the average of these errors (so it gives you average error for all steps in one episode). Then, the reported avg_return is calculated by taking the mean of average errors for 5 (as of now) episodes. Basically, sqrt(|avg_return|) gives you an average deviation away from T_ref: so if avg_return=-25, then you should expect the model to be consistently around 5 degrees away from T_ref.

Update

In the past 2 weeks I have finalized the pipeline for hyperparameter testing and speeding up the training dramatically. Here is a rundown of the most recent developments:

1. I have pushed a lot of code in the git repo with pipeline for training models using the TF Agent library. Now, all of the models' hyperparameters are specified via config files (config_ppo.yaml), allocated resources on HPC and starting a new run are managed via a bash script (start.sh), and a basic plotting functionality now comes built-in for each model run for easy plotting and evalution of the trained policy.
2. I have increased the speed of the code by making use of more advanced training methods from TF Agents: Compared to PPO_v5.py which uised to take around 25 seconds per iteration, PPO_v6_2.py takes 4 seconds per iteration (given the same training parameters). This allows not only for very fast training, but also much faster debug.
3. I have managed to train a model that effectively solved the environment with constant T_env and no measurement noise. Here is a plot of the policy applied for one episode:
   
   This particular plot comes from the policy of RLCON-60 run, which ran for 12.5 hours, but which seems to have reached "peak" performance in about 2-3 hours judging by the evaluation curve (from neptune.ai):
   
   I know this plot is not great, but basically the plateau has a return between -11064 and -11061 (so effectively constant). The evaluation runs are done 50 interations from each other, which in the case of this run corresponds to around 40 mins of training. Please feel free to check out more details about RLCON-60 at the provided link.
4. I have written an extensive manual in the same git repo with guidance as to how to use the code, and also a mini tutorial on Reinforcement Learning and TF Agents. The manual is not exhaustive and I am still adding to it regularly, but I am always happy to answer questions about the code (I am particularly active in the git issue tracker for the repo).

Moving on, I am ready to start making the environment more complicated: I now need to add things like varying T_env, measurement noise, heating delays, etc. Luckily, these changes can be done independently in PuckEnv.py, allowing us to just use the same code for training once the environment is up and bug-free. For testing purposes of the environments, I also have the env.ipynb, although it is not as streamlined as the rest of the code.

Dismantled the seismometer circuit

At Rana's request, Deven and I have pulled all the wires out of the seismometer at the X end that Adviat set up the heater and sensors for. Right now, the seismometer is uncovered (no can) and the can I had to move next to the server rack that is close to the X end. I am attaching pictures of both the seismometer and the can.

I now realize that I have not turned off the power supply for the heater. If someone can please turn off the power supply that is shown in Advait's elog, please turn it off and reply to this elog. If you are unsure which power supply to turn off, read the first line of the elog that I replied to. Thanks!

Advait's circuits have been removed from the seismometer and stored in a box underneath the desk at the end of the control room (see picture). There is even a label on the box which reads "ANDREI CIRCUITS". At the seismometer there are still a few BNC cables left there: these are the cables that were used to interact with Advait's circuits. They have labels on them so that I can put back the circuits when I come back/when the seismometer can is available again

1. Before doing anything, we centered the IOO QPDs.
2. With the WFS enabled, we offloaded the control signals onto the bias sliders. Then we saved the slider values. The MC LSC diode had a DC value of ~0.5
3. Turned down power with half wave plate before PMC.  Power injected to vacuum ~ 100mW.
4. We did a beam scan of MC REFL, it looks smaller than what Andres predicted based on the MC eigenmode by 10-20%.
5. We made many changes on the table, pictures to be added by Andres.
6. We didn't have the 80% reflector we wanted to increase the WFS power, so it's still a 98%.
6. Beams were aligned on MC REFL PL, camera, beam dumps, WFSs.
7. Clean up
8. PSL power increased to 1.2W, MC locked right away.
9 We didn't change the IOO WFS output matrix, but we changed some signs and gains to make everything stable. MC autolocker brings it back from cold just fine.
10. All time bombs that we've left will be E.Q.'s to clean up. Sorry.\
11. Yay

Kevin, Gautam and Arijit

We made a measurement of the MC_REFL photodiode transfer function using the network analyzer. We did it for two different power input (0dB and -10dB) to the test measurement point of the MC_REFL photodiode. This was important to ensure the measurements of the transfer function of the MC_REFL photodiode was in the linear regime. The measurements are shown in attachment 1. We corrected for phase noise for the length of cable (50cm) used for the measurement. With reference to ELOG 10406, in comparison to the transimpedance measurement performed by Riju and Koji, there is a much stronger peak around 290MHz as observed by our measurement.

We also did a noise measurement for the MC_REFL photodiode. We did it for three scenarios: 1. Without any light falling on the photodiode 2. With light falling on the photodiode, the MC misaligned and the NPRO noise eater was OFF 3. With light falling on the photodiode, the MC misaligned and the NPRO noise eater was ON. We observed that the noise eater does reduce the noise being observed from 80kHz to 20MHz. This is shown in attachment 2.

We did the noise modelling of the MC_REFL photodiode using LISO and tried matching the expected noise from the model with the noise measurements we made earlier. The modeled noise is in good agreement with the measured noise with 10Ohms in the output resistance. The schematic for the MC_REFL photodiode however reveals a 50Ohm resistance being used. The measured noise shows excess noise ~ 290MHz. This is not predicted from the simplied LISO model of the photodiode we took.

Discussion with Koji and Gautam revealed that we do not have the exact circuit diagram for the MC_REFL photodiode. Hence the simplified model that was assumed earlier is not able to predict the excess noise at high frequencies. One thing to note however, is that the excess noise is measured with the same amplitude even with no light falling on the MC_REFL photodiode. This means that there is a positive feedback and oscillation in the op-amp (MAX4107) at high frequencies. One way to refine the LISO model would be to physically examine the photodiode circuit.

We also recorded the POX and POY RF monitor photodiode outputs when the interferometer arms are independently stabilized to the laser. Given the noise outputs from the RF photodiodes were similar, we have only plotted the POY RF monitor output for the sake of clarity and convenience.

Kevin, Gautam and Arijit

We did a optical measurement of the MCREFL_PD transimpedance using the Jenny Laser set-up. We used 0.56mW @1064nm on the NewFocus 1611 Photodiode as reference and 0.475mW @1064nm on the MCREFL_PD. Transfer function was measured using the AG4395 network analyzer. We also fit the data using the refined LISO model. From the optical measurement, we can see that we do not have a prominent peak at about 300MHz like the one we had from the electrical transimpedence measurement. We also put in the electrical transimpedence measurement as reference. RMS contribution of 300MHz peak to follow.

As per Rana`s advice I have updated the entry with information on the LISO fit quality and parameters used. I have put all the relevant files concerning the above measurement as well as the LISO fit and output files as a zip file "LISO_fit" . I also added a note describing what each file represents. I have also updated the plot with fit parameters and errors as in elog 10406.

Kevin, Gautam and Arijit

Today we performed the in-loop noise measurements of the MCREFL-PD using the SR785 to ascertain the effect of the Noise Eater on the laser. We took the measurements at the demodulated output channel from the MCREFL-PD. We performed a series of 6 measurements with the Noise Eater ''ON'' and ''OFF''. The first data set is an outlier probably, due to some transient effects. The remaining data sets were recorded in succession with a time interval of 5 minutes each between the Noise Eater in the ''ON'' and ''OFF'' state. We used the calibration factor of 13kHz/Vrms from elog 13696 to convert the V_rms to Hz-scale.

The conclusion is that the NOISE EATER does not have any noticeable effect on the noise measurements.

ALS beat spectrum and also the arm control signal look as they did before. coherence between arm control signals (in POX/POY lock) is high between 10-100Hz, so looks like there is still excess frequency noise in the MC transmitted light. Looking at POX as an OOL sensor with the arm under ALS control shows ~10x the noise at 100 Hz compared to the "nominal" level, consistent with what Koji and I observed ~3weeks ago.

We tried swapping out Marconis. Problem persists. So Marconi is not to blame. I wanted to rule this out as in Jan, Steve and I had installed a 10MHz reference to the rear of the Marconi.

I made a very slighly updated version of Yinzi's script that pulls the channel names and actuator hardstop limits from an external .ini config file. The idea was to avoid having as many versions of the script as there are implimentations of it. Seems like slighly better practice, but maybe I'm wrong. The config files are also easier to read. Its posted on the elog (PSL:1758) with lastest on the 40mSVN .../trunk/CTNLab/current/computing/scripts . 

If you're working off her first implimentation 'RCAV_thermalPID.py' then there is an indent issue after the if statement on line 88: only line 89 should be indended. If you deactivate the debug flag then the script fails to read in PID factors and dies.

I brought a bunch of SR560s over for repair from Bridge labs. This unit, picture attached (SN 49698), appears to still not be retaining charge. I’ve brought it back. 

Another cool feature is client side pre-commit hooks. They can be used to run checks on the local version at the time of commit and refuses to push until the pass/fail exits 0.

Can be the same as the Gitlab CI or just basic code quality checks.  I use them to prevent jupyter notebooks being commited with uncleared cells. It needs to be set up on the user's computer manually and is not automatically cloned with the directory: a script can be included in the repo to do this and run manually on first time clone.

I've taken a PI Piezo Actuator (P-810.10) from the 40m collection. I forgot to note it on the equipment checklist by the door, will do so when I next drop by.

I've borrowed the Busby Box for a day or so.  Location: QIL lab at Bridge West.

Edit  Sat Apr 20 21:16:46 2019 (awade): returned.

Borrowed Zurich HF2LI Lock in Amplifer to QIL lab Wed Apr 24 11:25:11 2019.

I checked out the cable that I took from you, and all of the connections looked right.  The only thing I did notice was that some of the soldered wires on the 37-pin connector had gotten hot enough to melt their insulation, and potentially short together.  I cut off that connector, and left it on your desk to check out.  I put on a new connector, and checked the pinout.  If the Guralps still doesn't work, we'll have to check out other possibilities.

So, near 2 of the trashcans in the control room and underneath a desk there are hundrends of ants. Is this normal?

Comparing  PSL-FSS-RMTEMP and PEM-MC1-TEMPS

So, to compare temp channels, I made a plot of PSL-FSS_RMTEMP and PEM-MC1_TEMPS(the test temp sensor channel after converting from cts to degC). This plot begins about 2 months ago t_initial=911805130. The temperature channels look kinda similar but MC1-TEMPS (the temp sensor clamped to MC1,3 chamber) is consistently higher in temperature than FSS_RMTEMP. See compare_temperature_channels.png.

MC1-TEMPS isn't exactly consistent with FSS-RMTEMP. I attached a few plots where I've zoomed in on a few hours or a few days. See compare_temperature_channels_zoom1.pdf & compare_temperature_channels_zoom2.pdf

Change the room temperature, see what happens to the chamber temperature

A while ago, somebody was fiddling around with the room temperature.  See compare_temperature_channels_zoom4.pdf.  This is a plot of PEM-MC1_TEMPS and PSL-FSS_RMTEMP at t0=911805130. You can see the chamber heating up and cooling down in happy-capacitory-fashion. Although, the PSL-FSS_RMTEMP and the PEM-MC1_TEMPS don't really line up so well. Maybe, the air in the location of the MC1,3 chamber is just warmer than the air in the PSL or maybe there's an offset in my calibration equation.

Calibration equation for PEM-MC1-TEMPS

For the calibration (cts to degC) I used the following equation based on the data-sheet for the LM34 and some measurements of the circuit:

TEMPERATURE[degC]=5/9*(((-CTS/16384/451.9/1.04094)-(.0499*10^-3))/(20*10^-6)-35);

How does the chamber temperature compare with the air temperature?

It looks like the chamber may be warmer than the air around it sometimes.

I wanted to check the temperature of the air and compare it with the temperature the sensor had been measuring. So, at t=918855087 gps, I took the temp sensor off of the mc1-mc3 chamber and let it hang freely, close to the chamber but not touching anything. See compare_temperature_chamber_air.png. MC1_TEMPS increases in temperature when I am handling the temp-sensor and then cools down to below the chamber temperature, close to FSS_RMTEMP, indicating the air temperature was less than the chamber temperature.

 When, I reattached temp sensor to the chamber at t=919011131 gps, the the temperature of the chamber was again higher than the temperature of the air. See compare_temperature_air2chamber.pdf.

Also, as one might expect, when the temp-sensor is clamped to the chamber, the temperature varies less, & when it's detached from the chamber, the temperature varies more. See compare_temperature_air_1day.pdf & compare_temperature_chamber_1day.pdf.

New temp-sensor power supply vs old temp-sensor power supply

The new temp-sensor is less noisy and seems to work OK. It's not completely consistent with PSL-FSS_RMTEMP, but neither was the old temp-sensor. And even the air just outside the chamber isn't the same temperature as the chamber. So, the channels shouldn't line up perfectly anyways.

I unplugged the 'old' temp-sensor power supply for a few hours and plugged in the 'new' one, which doesn't have a box but has some capacitors and and 2 more voltage regulators. The MC1_TEMPS channel became less noisy. See noisetime.png & noisefreq.pdf. For that time, the minute trend shows that with the old temp-sensor power supply the temp sensor varies +/-30cts and with the new power supply, it is more like +/-5cts (and Volt/16,384cts * 1degF/10mV -->  apprx +/-0.03degF). So, it's less noisy. 

I kept the new temp-sensor power supply plugged in for about 8 hours, checking if new temp sensor power supply worked ok. Compared it with PSL-FSS_RMTEMP after applying an approximate calibration equation. See ver2_mc1_rmtemp_8hr_appxcal.png.

Just for kicks

Measuring time constant of temp sensor when detached from chamber. At 918858981, I heated up the temp sensor on of the mc1-mc3 chamber with my hand. Took hand off sensor at  918859253 and let it cool down to the room temperature. See temperature_sensor_tau.pdf. 

I plugged the Guralp cables back into the PEM ADCU

       Guralp NS1b ---> #11

       Guralp Vert1b --->#10

       Guralp EW1b --->#12

For several of the channels on the PEM ADCU, zeros are occuring at the same time. Does anyone know why that might happen or how to fix it?

I unplugged Guralp EW1b and Guralp Vert1b and plugged in temp sensors temporarily. Guralp NS1b is still plugged in.