40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 48 of 341  Not logged in ELOG logo
IDdown Date Author Type Category Subject
  14791   Sun Jul 21 17:17:03 2019 KruthiUpdateLoss MeasurementMC2 loss map

The camera server keeps throwing the error: failed to grab frames. Milind suggested that it might a problem with the ethernet cable, so I even unplugged it and connected it again; it still had the same issue. One more thing I noticed was, it does take snapshots sometimes with the terminal command caput C1:CAM-ETMX_SNAP 1, but produces a segmentation fault when ezca.Ezca().write(C1:CAM-ETMX_SNAP, 1) ezca.Ezca().write(CAM-ETMX_SNAP, 1) is used via ipython. When the terminal command also fails to take snapshots, I noticed that the SNAP button on the GigE medm screen remains on and doesn't switch back to OFF like it is supposed to.

Quote:

Can you please be more specific about what the error is? Is this the usual instability with the camera server code? Or was it something new?

Quote:

The camera server is throwing an error and is not grabbing snapshots :(

  14790   Sun Jul 21 12:55:38 2019 gautamUpdateCDSCM board Latch Enable test script

DATED, SEE ELOG14941 for the most up-to-date info on latch.py.

I wrote (/cvs/cds/caltech/target/c1iscaux3/latch.py) and tested the logic illustrated in Attachment #1. Results of a test are shown in Attachment #2, the various channels change as expected. Note that for negative values of the gain channel, the corresponding "BITS" channel will take on values like 65536 - this is because the mbboDirect data type is a 16 bit data type, and presumably the MSB is the sign bit. A bit mask is applied to this channel before the actual BIO unit bits are set - we should verify that the correct behavior happens, but I don't immediately see any problems.

To me, this is a robust logic, but it will benefit from more sets of eyes giving it a look over. The idea is to run this continuously on the Supermicro machine.

Apart from this, I also fixed some errors in the mbboDirect record syntax - so now I am able to start up the EPICS server without it throwing any error messages. It remains to verify that changing an EPICS gain slider results in the appropriate gain bits being flipped in the correct way (on the hardware side, I think the correct behavior is happening on the software end). For this testing, I turned off the old c1iscaux crate at ~10am, and started up the server on c1iscaux3. I am reverting to the nominal config now (~1pm).

Further testing will require the wiring inside the Acromag chassis to be completed. This should be the priority task for next week.

*Update 1130 22 July 2019: I've now installed the required dependencies on c1iscaux3 and setup the latch.py script to run as a systemctl process dependent on modbusIOC.service.

Attachment 1: LatchLogic.pdf
LatchLogic.pdf
Attachment 2: LatchLogicTest.png
LatchLogicTest.png
  14789   Sun Jul 21 12:54:18 2019 gautamUpdateLoss MeasurementMC2 loss map

Can you please be more specific about what the error is? Is this the usual instability with the camera server code? Or was it something new?

Quote:

The camera server is throwing an error and is not grabbing snapshots :(

  14788   Sun Jul 21 02:07:04 2019 KruthiUpdateLoss MeasurementMC2 loss map

I'm running the MC2 loss map scripts on pianosa now. The camera server is throwing an error and is not grabbing snapshots :(

Update: I finished taking the readings for MC2 loss map. I couldn't get the snapshots with the script, so I manually took some 4-5 pictures.

  14787   Sat Jul 20 14:43:45 2019 MilindUpdateCamerasCNNs for beam tracking || Analysis of results

<Adding details>

See Attachment #2.

Quote:

Make the MSE a subplot on the same axes as the time series for easier interpretation.

Training dataset:

  1. Peak to peak amplitue in physical units: ?
  2. Dither frequency: 0.2 Hz
  3. Video data: zoomed in video of the beam spot obtained from GigE camera 198.162.113.153 at 500us exposure time. Each frame has a resolution of 640 x 480 which I have cropped to 350 x 350. Attachment #1 is one such frame.
  4. Yes, therefore I am going to obtain video at lower amplitudes. I think that should help me avoid the problem of not-nominal-maximum value?
  5. Other details of the training dataset:
    1. Dataset created from four vides of duration ~ 30, 60, 60, 60 s at 25 FPS.
    2. 4032 training data points
      1. Input (one example/ data point): 10 successive frames stacked to form a 3D volume of shape 350 x 350 x 10
      2. Output (2 dimensional vector): QPD readings (C1:IOO-MC_TRANS_PIT_ERR, C1:IOO-MC_TRANS_YAW_ERR)
    3. Pre-processing: none
    4. Shuffling: Dataset was shuffled before every epoch
    5. No thresholding: Binary images are gonna be of little use if the expectation is that the network will learn to interpret intensity variations of pixels.

Do I need to provide any more details here?

Quote

Describe the training dataset - what is the pk-to-pk amplitude of the beam spot motion you are using for training in physical units? What was the frequency of the dither applied? Is this using a zoomed-in view of the spot or a zoomed out one with the OSEMs in it? If the excursion is large, and you are moving the spot by dithering MC2, the WFS servos may not have time to adjust the cavity alignment to the nominal maximum value.

?

Quote:

What is the minimum detectable motion given the CCD resolution?

see attachment #4.

Quote:
  1. Please upload a cartoon of the network architecture for easier visualization. What is the algorithm we are using? Is the approach the same as using the bright point scatterers to signal the beam spot motion that Gabriele demonstrated successfully

 

I wrote what I think is a handy script to observe if the frames are saturated. I thought this might be handy for if/when I collect data with higher exposure times. I assumed there was no saturation in the images because I'd set the exposure value to something low. I thought it'd be useful to just verify that. Attachment #3 has log scale on the x axis.

Quote:

What is the significance of Attachment #6? I think the x-axis of that plot should also be log-scaled.

 

Quote:
  1. Is the performance of the network still good if you feed it a time-shuffled test dataset? i.e. you have (pictures,Xcoord,Ycoord) tuples, which don't necessarily have to be given to the network in a time-ordered sequence in order to predict the beam spot position (unless the network is somehow using the past beam position to predict the new beam position).
  2. Is the time-sync problem Koji raised limiting this approach?

 

Attachment 1: frame0.pdf
frame0.pdf
Attachment 2: subplot_yaw_test.pdf
subplot_yaw_test.pdf
Attachment 3: intensity_histogram.mp4
Attachment 4: network2.pdf
network2.pdf
  14786   Sat Jul 20 12:16:39 2019 gautamUpdateCamerasCNNs for beam tracking || Analysis of results
  1. Make the MSE a subplot on the same axes as the time series for easier interpretation.
  2. Describe the training dataset - what is the pk-to-pk amplitude of the beam spot motion you are using for training in physical units? What was the frequency of the dither applied? Is this using a zoomed-in view of the spot or a zoomed out one with the OSEMs in it? If the excursion is large, and you are moving the spot by dithering MC2, the WFS servos may not have time to adjust the cavity alignment to the nominal maximum value.
  3. What is the minimum detectable motion given the CCD resolution?
  4. Please upload a cartoon of the network architecture for easier visualization. What is the algorithm we are using? Is the approach the same as using the bright point scatterers to signal the beam spot motion that Gabriele demonstrated successfully?
  5. What is the significance of Attachment #6? I think the x-axis of that plot should also be log-scaled.
  6. Is the performance of the network still good if you feed it a time-shuffled test dataset? i.e. you have (pictures,Xcoord,Ycoord) tuples, which don't necessarily have to be given to the network in a time-ordered sequence in order to predict the beam spot position (unless the network is somehow using the past beam position to predict the new beam position).
  7. Is the time-sync problem Koji raised limiting this approach?
  14785   Sat Jul 20 11:57:39 2019 gautamSummaryCDSP2 interface board

The boards arrived. I soldered on a DIN96 connector, and tested that the goemetry will work. It does yes. The only constraint is that the P2 interface board has to be installed before the P1 interface is installed. Next step is to confirm that the pin-mapping is correct. The pin mapping from the DIN96 connector to the DB15 was also verified.

*Maybe it isn't obvious from the picture, but there shouldn't be any space constraint even with the DB37/DB15 cables connected to the respective adapter boards.

Attachment 1: IMG_7773.JPG
IMG_7773.JPG
  14784   Sat Jul 20 11:24:04 2019 gautamUpdateGeneralrossa bricked

Summary:

SnapPy scripts made to work on Pianosa.

Details:

Of course rossa was the only machine in the lab that could run the python scripts to interface with the GigE camera. And it is totally bricked now. Lame.

So I installed several packages. The key was to install pypylon - if you go to the basler webpage, pypylon1.4.0 does not offer python2.7 support for x86_64 architecture, so I installed pypylon1.3.0. Here are the relevant lines from the changelog:

gstreamer-plugins-bad-0.10.23-5.el7.x86_64    Sat 20 Jul 2019 11:22:21 AM PDT
gstreamer-plugins-good-0.10.31-13.el7.x86_64  Sat 20 Jul 2019 11:22:11 AM PDT
gstreamer-plugins-ugly-0.10.19-31.el7.x86_64  Sat 20 Jul 2019 11:20:08 AM PDT
gstreamer-python-devel-0.10.22-6.el7.x86_64   Sat 20 Jul 2019 10:34:35 AM PDT
pygtk2-devel-2.24.0-9.el7.x86_64              Sat 20 Jul 2019 10:34:34 AM PDT
pygobject2-devel-2.28.6-11.el7.x86_64         Sat 20 Jul 2019 10:34:33 AM PDT
pygobject2-codegen-2.28.6-11.el7.x86_64       Sat 20 Jul 2019 10:34:33 AM PDT
gstreamer-devel-0.10.36-7.el7.x86_64          Sat 20 Jul 2019 10:34:32 AM PDT
gstreamer-python-0.10.22-6.el7.x86_64         Sat 20 Jul 2019 10:34:31 AM PDT
gtk2-devel-2.24.31-1.el7.x86_64               Sat 20 Jul 2019 10:34:30 AM PDT
libXrandr-devel-1.5.1-2.el7.x86_64            Sat 20 Jul 2019 10:34:28 AM PDT
pango-devel-1.42.4-1.el7.x86_64               Sat 20 Jul 2019 10:34:27 AM PDT
harfbuzz-devel-1.7.5-2.el7.x86_64             Sat 20 Jul 2019 10:34:26 AM PDT
graphite2-devel-1.3.10-1.el7_3.x86_64         Sat 20 Jul 2019 10:34:26 AM PDT
pycairo-devel-1.8.10-8.el7.x86_64             Sat 20 Jul 2019 10:34:25 AM PDT
cairo-devel-1.15.12-3.el7.x86_64              Sat 20 Jul 2019 10:34:25 AM PDT
mesa-libEGL-devel-18.0.5-3.el7.x86_64         Sat 20 Jul 2019 10:34:24 AM PDT
libXi-devel-1.7.9-1.el7.x86_64                Sat 20 Jul 2019 10:34:24 AM PDT
pygtk2-doc-2.24.0-9.el7.noarch                Sat 20 Jul 2019 10:34:23 AM PDT
atk-devel-2.28.1-1.el7.x86_64                 Sat 20 Jul 2019 10:34:21 AM PDT
libXcursor-devel-1.1.15-1.el7.x86_64          Sat 20 Jul 2019 10:34:20 AM PDT
fribidi-devel-1.0.2-1.el7.x86_64              Sat 20 Jul 2019 10:34:20 AM PDT
pixman-devel-0.34.0-1.el7.x86_64              Sat 20 Jul 2019 10:34:19 AM PDT
libXinerama-devel-1.1.3-2.1.el7.x86_64        Sat 20 Jul 2019 10:34:19 AM PDT
libXcomposite-devel-0.4.4-4.1.el7.x86_64      Sat 20 Jul 2019 10:34:19 AM PDT
libicu-devel-50.1.2-15.el7.x86_64             Sat 20 Jul 2019 10:34:18 AM PDT
gdk-pixbuf2-devel-2.36.12-3.el7.x86_64        Sat 20 Jul 2019 10:34:17 AM PDT
pygobject2-doc-2.28.6-11.el7.x86_64           Sat 20 Jul 2019 10:34:16 AM PDT
pygtk2-codegen-2.24.0-9.el7.x86_64            Sat 20 Jul 2019 10:34:15 AM PDT

Camera server is running on a tmux session on pianosa. But it keeps throwing up some gstreamer warnings/errors, and periodically (~every 20 mins) crashes. Kruthi tells me that this behavior was seen on Rossa as well, so whatever the problem is, doesn't seem to be because I missed out on installing some packages on pianosa. Moreover, if the server is in fact running, I am able to take a snapshot - but the camera client does not run.

  14783   Sat Jul 20 01:03:37 2019 gautamUpdate Dataviewer error

What channels are you trying to read?

Quote:

I'm not able to get trends of the TM adjustment test that Rana had asked us to perform, from the dataviewer. It's throwing the following error:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
Server error 7: connect() failed
datasrv: DataWrite failed: daq_send: Resource temporarily unavailable
T0=19-07-20-01-27-39; Length=600 (s)
No data output.

  14782   Fri Jul 19 22:48:08 2019 KruthiUpdate Dataviewer error

I'm not able to get trends of the TM adjustment test that Rana had asked us to perform, from the dataviewer. It's throwing the following error:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
Server error 7: connect() failed
datasrv: DataWrite failed: daq_send: Resource temporarily unavailable
T0=19-07-20-01-27-39; Length=600 (s)
No data output.

  14781   Fri Jul 19 19:44:03 2019 gautamUpdateCDSDatabase file test

Summary:

The database files for C1ISCAUX seem to work file - the exception being the mbbo channels for the CM board.

Details:

This was just a software test - the actual functionality of the channels will have to be tested once the Acromag crate has been installed in the rack. One change I had to make on the MEDM screen for the LSC PD whitening gains was to get rid of the "NMS" suffix on the EPICS channel names for whitening gain sliders/drop-down-menus. I suspect this has to do with the EPICS version we are using, 7.0.1. Furthermore, AS165 and POP55 no longer exist - I hold off removing them from the MEDM screen for the moment.

Next steps:

From the software point of view, the major steps are:

  1. Fix the mbbo channel notation in the database files
  2. Write and test the latch enabling code
  3. Figure out what scripted tests can be done to test the functionality of the new Acromag box.

I am stopping the EPICS server on the new machine and restarting the old VME crate over the weekend.

Attachment 1: Whitening.png
Whitening.png
  14780   Fri Jul 19 17:42:58 2019 gautamUpdateGeneralrossa Xdisp bricked

For some reason, rossa's Xdisplay won't start up anymore. This happened right after the UPS reset. Koji and I tried ~1.5 hours of debugging, got nowhere.

  14779   Fri Jul 19 16:47:06 2019 MilindUpdateCamerasCNNs for beam tracking || Analysis of results

I did a whole lot of hyperparameter tuning for convolutional networks (without 3d convolution). Of the results I obtained, I am attaching the best results below.

Define "best"?

The lower the power of the error signal (difference between the true and predicted X and Y positions), essentially mse, on the test data, the better the performance of the model. Of the trained models I had, I chose the one with the lowest mse.

Attached results:

  1. Attachment 1: Training configuration
  2. Attachment 2: Predicted motion along the Y direction for the test data
  3. Attachment 3: Predicted motion along the Y direction for the training data
  4. Attachment 4: Learning curves
  5. Attachment 5: Error in test predictions
  6. Attachment 6: Video of image histogram plots
  7. Attachment 7: Plot of percentage of pixels with intensity over 240 with time

(Note: Attachment 6 and 7 present information regarding a fraction of the data. However, the behaviour remains the same for the rest of the data.)

Observations and analysis:

  1. Data:
    1. From attachemtns 2, 3, 5: Maximum deviation from true labels at the peaks of applied dither/motion. Possible reasons:
      1. Stupid Cropping? I checked (by watching the video of cropped frames, i.e visually) to ensure that the entire motion of the beam spot is captured. Therefore, this is not the case.
      2. Intensity variation: The intensity (brightness?) of the beam spot varies (decreases) significantly at the maximum displacement. This, I think, is creating a skewed dataset with very few frames with low intensity pixels. Therefore, I think it makes sense to even this out and get more data points (frames) with similar (lower) pixel intensities. I can think of two ways of doing this:
        1. Collect more data with lower amplitude of sinusoidal dither. I used an amplitude of 80 cts to dither the optic. Perhaps something like 40 is more feasible. This will ensure the dataset isn't too skewed.
        2. Increase exposure time. I used an exposure time of 500us to capture data. Perhaps a higher exposure time will ensure that the image of the beam spot doesn't fade out at the peak of motion.
    2. From attachment 5, Saturated images?: We would like to gun for a maximum deviation of 10% (0.1 in this case) from the true values in the predicted labels (Tbh, I'm not sure why this is a good baseline, I ought to give that some thought. I think the maximum deviation of the OpenCV thing I did at the start might also be a good baseline?). Clearly, we're not meeting that. One possible reason is that the video might be saturated- (too many pixels at 255, bleeding into surrounding pixles) leading to loss of information. I set the exposure time to 500us precisely to avoid this. However, I also created videos of the image histograms of the frames to make sure the frames weren't saturated (Is there some better standard way of doing it?). From attachements 6 and 7, I think it's evident that saturation is not an issue. Consequently, I think increasing the exposure time and collecting data is a good idea.
  2. The network:
    1. From attachment 4: Training post 25 epochs seems to produce overfitting, though it doesn't seem too terrible (from attachments 2 and 3). The network is still learning after 75 epochs, so I'll tinker with the learning rate, dropout and maybe put in annealing.
    2. I don't think there is a need to change the architecture yet. The model seems to generalize okay (valdiation error is close to training error), therefore I think it'll be a good idea to increase dropout for the fully connected layers and train for longer/ with a higher learning rate.

 


 

P.S. I will also try the 2D convolution followed by the 1D convolution thing now. 

P.P.S. Gabriele suggested that I try average pooling instead of max pooling as this is a regression task. I'll give that a shot.

 

Attachment 1: readme.txt
Experiment file: train_both.py
batch_size: 32
dropout_probability: 0.5
eta: 0.0001
filter_size: 1
filter_type: median
initializer: Xavier
memory_size: 10
num_epochs: 75
activation_function: relu
... 22 more lines ...
Attachment 2: yaw_motion_test.pdf
yaw_motion_test.pdf
Attachment 3: yaw_motion_train.pdf
yaw_motion_train.pdf
Attachment 4: Learning_curves_replotted.pdf
Learning_curves_replotted.pdf
Attachment 5: yaw_error_test.pdf
yaw_error_test.pdf
Attachment 6: intensity_histogram.mp4
Attachment 7: saturation_percentage.pdf
saturation_percentage.pdf
  14778   Fri Jul 19 15:54:47 2019 gautamUpdateGeneralControl room UPS Batteries need replacement

The control room UPS started making a beeping noise saying batteries need replacement. I hit the "Test" button and the beeping went away. According to the label on it, the batteries were last repalced in March 2016, so maybe it is time for a replacement, @Chub, please look into this.

  14777   Fri Jul 19 15:51:55 2019 gautamUpdateGeneralProjector lightbulb blown out

[chub, gautam]

Bulb replaced. Projector is back on.

  14776   Fri Jul 19 12:50:10 2019 gautamUpdateSUSDC bias actuation options for SOS

Rana and I talked about some (genius) options for the large range DC bias actuation on the SOS, which do not require us to supply high-voltage to the OSEMs from outside the vacuum.

What we came up with (these are pretty vague ideas at the moment):

  1. Some kind of thermal actuation.
  2. Some kind of electrical actuation where we supply normal (+/- 10 V) from outside the vacuum, and some mechanism inside the chamber integrates (and hence also low-pass filters) the applied voltage to provide a large DC force without injecting a ton of sensor noise.
  3. Use the blue piers as a DC actuator to correct for the pitch imbalance --- Kruthi and Milind are going to do some experiments to investigate this possibility later today.

For the thermal option, I remembered that (exactly a year ago to the day!) when we were doing cavity mode scans, once the heaters were turned on, I needed to apply significant correction to the DC bias voltage to bring the cavity alignment back to normal. The mechanism of this wasn't exactly clear to me - furthermore, we don't have a FLIRcam picture of where the heater radiation patter was centered prior to my re-centering of it on the optic earlier this year, so we don't know what exactly we were heating. Nevertheless, I decided to look at the trend data from that night's work - see Attachment #1. This is a minute trend of some ETMY channels from 0000 UTC on 18 July 2018, for 24 hours. Some remarks:

  1. We did multiple trials that night, both with the elliptical reflector and the cylindrical setup that Annalisa and Terra implemented. I think the most relevant part of this data is starting at 1500 UTC (i.e. ~8am PDT, which is around when we closed shop and went home). So that's when the heaters were turned off, and the subsequent drift of PIT/YAW are, I claim, due to whatever thermal transients were at play.
  2. Just prior to that time, we were running the heater at close to its maximum rated current - so this relaxation is indicative of the range we can get out of this method of actuation.
  3. I had wrongly claimed in my discussion with Rana this morning that the change in alignment was mostly in pitch - in fact, the data suggests the change is almost equal in the two DoFs. Oplev and OSEMs report different changes though, by almost a factor of 2....
  4. The timescale of the relaxation is ~20 minutes - what part(s) of the suspension take this timescale to heat up/cool down? Unlikely to be the wire/any metal parts because the thermal conductivity is high? 
  5. In the optimistic scenario, let's say we get 100 urad of actuation range - over 40m, this corresponds to a beam spot motion of ~8mm, which isn't a whole lot. Since the mechanism of what is causing this misalignment is unclear, we may end up with significantly less actuation range as well.
  6. I will repeat the test (i.e. drive the heater and look for drift in the suspension alignment using OSEMs/Oplev) in the afternoon - now I claim the radation pattern is better centered on the optic so maybe we will have a better understanding of what mechanisms are at play.

Also see this elog by Terra.

Attachment #2 shows the results from today's heating. I did 4 steps, which are obvious in the data - I=0.6A, I=0.76A, I=0.9A, and I=1.05A.


In science, one usually tries to implement some kind of interpretation. so as to translate the natural world into meaning.

Attachment 1: heaterPitch_2018.pdf
heaterPitch_2018.pdf
Attachment 2: Screenshot_from_2019-07-19_16-39-21.png
Screenshot_from_2019-07-19_16-39-21.png
  14775   Thu Jul 18 22:34:40 2019 KojiSummaryCDSiscaux electronics modifications

The whitening filter modules have been restored to the crates. The SMA cables have been restored and fastened by a spanner. The ribbon cable to the antialiasing board was also connected. The backplane cables have not been moved from the upper DIN96 connector to the lower one.

Everything is expected to be good, but just keep eyes on the LSC signals as the boards were not quantitatvely tested yet. If you find something suspicious, report on the elog.

  14774   Thu Jul 18 22:03:00 2019 KruthiUpdateCamerasMC2 and cameras

[Kruthi, Yehonathan, Gautam]

Today evening, Yehonathan and I aligned the MC2 cameras. As of now there are 2 GigEs in the MC2 enclosure. For the temporary GigE (which is the analog camera's place), we are using an ethernet cable connection from the Netgear switch in 1x6. The MC2 was misaligned and the autolocker wasn't able to lock the mode cleaner. So, Gautam disabled the autolocker and manually changed the settings; the autolocker was able to take over eventually.

  14773   Thu Jul 18 19:58:56 2019 gautamUpdateCDSWork on Acromag chassis

Now that the .db files were prepared, I wanted to test for errors. So I did the following:

  1. Acromags were mounted on the DIN rails. Attachment #1 shows the grouping of ADC, DAC and BIO units. They are labelled with their IP addresses.
  2. Wiring of power:
    • Chub had already prepared the backplane with the power connectors, switches and indicator LEDs.
    • So I just had to daisy chain the +24 V (RED) and GND (BLACK) terminals for all the acromags together, which I did using 24 AWG wire (we may want to use heavier gauge given the current draw).
  3. Ethernet cables were used to daisy chain the network connectivity between the various units.  Attachment #1 shows the current state of the chassis box.
  4. Front panel pieces were attached and labelled, see Attachment #2
    • I found it was sufficient to use the front - we may use the rear panel slots when we want to add connections for controlling the c1aux machine channels.
    • The D15 P2 connector panel for the CM board will arrive tomorrow and will be installed then.
  5. Entire setup was connected to power and ethernet, see Attachment #3
    • As usual, the current draw is significant for the collection of Acromags, I got around this problem by using the bench supply to "Parallel" mode to enhance the current driving capacity.
    • For the ethernet connection, I used the office space port #6, which I connected at the network rack end to the eth1 port of the Supermicro.

All the Acromags are seen on the 192.168.114 subnet on c1iscaux3 yes- however, when I run the modbusIOC process, I see various errors in the logfile no, so more debugging is required. Nevertheless, progress.

Update 2245: Turns out the errors were indeed due to a copy/paste error - I had changed the IP addresses for the ADCs from the .115 subnet c1susaux was using, but forgot to do so for the DACs and BIOs. Now, if I turn off the existing c1iscaux so that there aren't any EPICS clashes, the EPICS server initializes correctly. There are still some errors in the log file - these pertain to (i) the mbbo notation, which I have to figure out, and (ii) the fact that this version of EPICS, 7.0.1, does not support channel descriptions longer than 28 characters (we have several that exceed this threshold). I think the latter isn't a serious problem.

Getting closer... Note that I turned off the c1iscaux VME crate to prevent any EPICS server clashes. I will turn it back on tomorrow.

Attachment 1: IMG_7771.JPG
IMG_7771.JPG
Attachment 2: IMG_7770.JPG
IMG_7770.JPG
Attachment 3: IMG_7772.JPG
IMG_7772.JPG
  14771   Thu Jul 18 10:46:04 2019 gautamUpdateCDSDatabase files made

I completed the translation of the .db files for the EPICS database records from the VME notation to the Acromag/Modbus/Asyn notation. The channels are now organized into 5 database files, located in /cvs/cds/caltech/target/c1iscaux3/,  for convenience:

  1. C1_ISC-AUX_LSCPDs.db -------- This handles whitening gain, AA enable/bypass, Demodulator FE, and PD Interface Board channels for REFL11, REFL55, REFL33, REFL165, POP22, POP110, POX11, POY11, AS55 and AS110 photodiodes.
  2. C1_ISC-AUX_CM.db -------------- This handles all channels for the CM board. The mbbo addressing notation needs to be checked.
  3. C1_ISC-AUX_QPDs.db ----------- This handles all channels for the IPPOS QPD.
  4. C1_ISC-AUX_ALS.db ------------- This handles all channels for the IR ALS DFD LO and RF power monitoring.
  5. C1_ISC-AUX_SPARE.db ---------- This handles the unused channels for the various whitening, AA and PD interface boards.

For reasons unknown to me, the database files in the other Acromag system target directories (e.g. c1susaux, c1auxex) all had 755 level access permission - maybe this is required for systemctl to handle the EPICS serving? Anyways, I upgraded the permission level of the above 5 files using chmod.

There are almost certainly typos / other errors, and I may have missed copying over some soft/calibrated channels, but I hope that this way of grouping by subsystem will make the debugging less painful. Once Chub connects up the power lines to the Acromags, I will run the soft tests. For this purpose, I've also made a C1_ISC-AUX.cmd file and a C1_ISC-AUX.env file in the above target directory, and also made the modbusIOC.service file in /etc/systemd/system on the supermicro.

  14770   Thu Jul 18 00:51:52 2019 KojiSummaryCDSiscaux electronics modifications

Along with the plan in ELOG 14744, the ISC PD interface and the whitening filter board have been modificed. The ISC PD I/Fs were restored to the crate and the cables were connected. The whitening filteres are still on the electronics bench for some more tests before being returned to the crate.

The updated schematics were uploaded as https://dcc.ligo.org/D1900318 and https://dcc.ligo.org/D1900319

- Modification of the ISC PD interface: Jumpers between DIN96 P1 and P2. Replace all AD797s with OP27. In fact only I/F #1 (the left most)  had total 12 AD797 but the other units already had OP27s.

- Modification of the whitening filter: Jumpers between DIN96 P1 and P2.

Attachment 1: LSC_whitening2.jpg
LSC_whitening2.jpg
Attachment 2: LSC_whitening.jpg
LSC_whitening.jpg
  14769   Wed Jul 17 21:22:41 2019 gautamUpdateCDSCM board Latch Enable subtlety

[koji, gautam]

Koji pointed out an important subtlety pertaining to the "LATCH ENABLE" signal line on the CM board. The purpose of this line is to smoothly facilitate the transition of a change in the "multi-bit-binary-outputs", a.k.a. "mbbo", that are controlled by MEDM gain sliders, to the analog electronics on the CM board. Why is this necessary? Imagine changing the gain from 7dB (=0111 in mbbo representation) to 8dB (=1000 in mbbo representation). In order to realize this change, all 4 bits have to change their state. But this almost certainly doesn't happen synchronously, because our EPICS interface isn't synchronous. So at some intermediate times, the mbbo representation could be 0100 (=4dB), or 1111 (=15dB), or many other possible values, which are all significantly different from either the initial value or the desired final state. This is clearly undesirable.

In order to protect against this kind of error, a Latched output part, 74ALS573, is used to buffer the physical digital logic levels from the switches in the analog gain stages. So in the default state, the "LATCH ENABLE" signal line is held "LOW". When a change happens in the EPICS value corresponding to a gain slider, the "LATCH ENABLE" state is quickly toggled to "HIGH", so as to enable the appropriate analog gain stages to be switched, and then again to "LOW", at which point the latch holds its output state. This logic is currently implemented by a piece of code called "latch.o", which is the compiled version of "latch.st", which may be found in /cvs/cds/caltech/target/c1iool0 where it presumably was written for the IMC servo board, but not in /cvs/cds/caltech/target/c1iool0  , which is where the CM board database files reside. The only elog reference I can find pertaining to this particular piece of code is from Alan, and doesn't say anything about the actual logic.

For the new c1iscaux, we need to implement this logic somehow. After discussion between Koji and me, we feel that a piece of python code is sufficient. This would continuously run in the background on the supermicro server machine. The channel hierarchy for each gain channes is as follows (I've taken the example of C1:LSC-CM_REFL1_GAIN):

  • C1:LSC-CM_REFL1_GAIN ------ this is the channel tied to an MEDM slider, and so is a "soft" channel
  • C1:LSC-CM_REFL1_SET ------- this is a "soft" channel that gets converted to an mbbo
  • C1:LSC-CM_REFL1_BITS ------ this is a channel that actually controls (multiple) physical binary outputs on the Acromag

So the logic will be that it continuously scans the EPICS channel C1:LSC-CM_REFL1_GAIN  for a change in set value. When a change is detected, it has to update the C1:LSC-CM_REFL1_SET channel. In the next EPICS refresh cycle, this would result in the mbbo bits, C1:LSC-CM_REFL1_BITS , all changing to the appropriate values. After these changes have happened, we need to toggle the LATCH ENABLE in order to allow the changes to propagate to the analog gain stage switches. Need to think about what's the best way to do this.

  14768   Wed Jul 17 20:12:26 2019 KruthiUpdateCamerasAnother GigE in place of analog camera

I've taken the MC2 analog camera down and put another GigE (unit 151) in its place. This is just temporary and I'll put the analog camera back once I finish the MC2 loss map calibration. I'm using a 25mm focal length camera lens with it and it gives a view of MC2 similar to the analog camera one. But I don't think it is completely focused yet (pictures attached).

...more to follow

gautam - Attachment #3 is my (sad) attempt at finding some point scatterers - Kruthi is going to play around with photUtils to figure out the average size of some point scatterers.

Attachment 1: zoomed_out_gige.png
zoomed_out_gige.png
Attachment 2: osems_mc2.png
osems_mc2.png
Attachment 3: MC2.pdf
MC2.pdf
  14767   Wed Jul 17 17:56:18 2019 KojiConfigurationComputersGave resolv.conf to giada

Kruthi noticed that she could not login to rossa from giada.

I checked /etc/resolv.conf and it was

nameserver 127.0.0.1

so obviously it is useless to refer localhost (i.e. giada) as a nameserver.

I copied our usual resolv.conf to giada as following:

nameserver 192.168.113.104
nameserver 131.215.125.1
nameserver 8.8.8.8

search martian

Giada's ssh known_host had unupdated entry for rossa, so I had to clean it up, but after that we can connect to rossa from giada just by "ssh rossa".

Case closed.

  14766   Wed Jul 17 03:05:01 2019 KruthiUpdateASSMC spot position measurement scripts

[Kruthi, Gautam, Rana]

Gautam installed Atom text editor on Pianosa yesterday.


MC spot position measurement scripts (these can be found in /scripts/ASS/MC directory)

  • Changed the power threshold for MC2 lock loss check from 15000 to 12000 (volts) in the MeasureSpotPositions.py script. This is because, the C1:I00-MC_TRANS_SUM reads a value, usually, greater than 14000 and with 15000 as the threshold, the script will always say the MC isn't locked even though it is!. Also, to account for additional variation we have a margin of 2000.
  • Issues with datetime: though MeasureSpotPositions.py was creating a .dat file, MC_spotMeasurement_history.py threw an error because the .dat file's name was not in the required format. I fixed this bug.
  • Just running the MeasureSpotPositions.py doesn't enter the results into the log file, instead ./mcassMCdecenter should be run
  • MC_spotMeasurement_history.py just plots the spot positions (in mm) vs days since 2013, using the log file. It still has some bugs
  14765   Tue Jul 16 16:00:01 2019 gautamUpdateCDSc1iscaux Supermicro setup

I worked on preparing for the c1iscaux upgrade a bit today.

  1. Attachment #1: This shows where the 120 GB solid-state hard-drive and the 2 RAM cards (2GB each) are installed.
    • I found that it required considerable application of force to get the RAM cards into their slots.
    • Note: the 4GB RAM is broken up into two separate physical cards, each 2GB. The labeling is a bit confusing, as each card suggests it is by itself 4GB.
  2. OS install for c1iscaux:
    • I followed Jon's instructions (and added some of mine to the wiki page to hopefully make this process even less thinking-intensive).
    • To be able to use the IP address 192.168.113.83, removed "bscteststand" from chiara martian.hosts and rev.113.168.192.in-addr.arpa as the last mention I could find of this machine was from 2009 (and I'm pretty sure it isn't an active unit anymore). I then restarted the bind9 process. 
    • The hostname for this machine is currently "c1iscaux3" for testing purposes, I will change it once we do the actual install.
    • There was an error in the installation instructions to allow incoming ssh connections - it is openssh-server that is required, not openssh-client. This has now been fixed on the wiki page instructions.
  3. Acromag static IP assignment:
    • Assigned 2 ADCs (XT1221), 5 DACs (XT1541) and 5 sinking BIO units (XT1111) static IP addresses (and labelled them for easy reference) using the windows laptop and the Acromag IP config utility.
    • I saw no reason not to use the 192.168.114.yyy scheme for the Acromag subnet on this machine, even though c1auxex and c1vac both have subnets with this addressing prefix. For reasons unknown to me, Jon opted to use 192.168.115.yyy for the c1susaux Acromag subnet.
  4. Followed the excellent step-by-step to install EPICS, Modbus and Asyn.
    • This took a while, ~1 hour, dominated by the building of EPICS. The other two took only a couple of minutes each.
    • The same combination suggested on Jon's wiki, of Modbus R2-11, EPICS base-7.0.1 and asyn4-33, are the most current at the time of installation.
    • Couple of typos that prevented straight up copy-pasting were fixed on the wiki.
  5. Playground for testing new database files:
    • made a directory /cvs/cds/caltech/target/c1iscaux3 and copied over the .db files from /cvs/cds/caltech/target/c1iscaux and /cvs/cds/caltech/target/c1iscaux2 over.
    • Johannes said he did not develop any code to automate the process of translating the old .db files into the new ones for the Acromag - I won't invest the time in developing any either as I think just manually editing the files will be faster. 
    • I think I will follow the c1susaux convention of grouping .db files by the physical electronics system where possible (e.g. REFL11 channels in one file, CM channels in one file etc), as I think this makes for easier debugging.
    • There is an old "PZT_AI.db" file which I think consists completely of obsolete channels.
  6. Next steps:
    • Wire up the crate [Chub]
    • Make the database files and modbus files for talking to the Acromags on the internal subnet [Gautam], check the .db files [Koji]
    • Wiring of whitening switching from P1 to P2 connector, Issue #1 in this elog (this will also requrie the installation of the DIN shrouds) [Koji]
    • Soldering of P2 interface boards [Gautam]
    • Bench testing [Gautam, Koji, Chub]
    • Installation and in-situ testing [Gautam, Koji, Chub]

All the required additional parts should be here by the end of the week - I'd like to aim for Wednesday 7/24 for the installation in 1Y3 and in-situ testing. While talking to Rana, I realized that we should also factor in the c1aux slow channels into this acromag crate - there is no need for a separate machine to handle the shutters and illuminators. But let's not worry about that for now, those channels can simply be added later.

Attachment 1: IMG_7769.JPG
IMG_7769.JPG
  14764   Tue Jul 16 15:17:57 2019 KojiHowToCDSFinal bit bug of the BIO CDS module

Yutaro talked about the BIO bug in KAGRA elog. http://klog.icrr.u-tokyo.ac.jp/osl/?r=9536

I think I made the similar change for the 40m model somewhere (don't remember), but be aware of the presense of this bug.

  14763   Tue Jul 16 15:00:03 2019 gautamUpdateSUSMultiple small EQs

There were several small/medium earthquakes in Ridgecrest and one medium one in Blackhawk CA at about 2000 UTC (i.e. ~ 2 hours ago), one of which caused BS, ITMY, and ETM watchdogs to trip. I restored the damping just now.

  14762   Mon Jul 15 18:55:05 2019 gautamUpdateIOOMegatron hard-rebooted

[koji, gautam]

In addition to c1psl needing a reboot, megatron was un-ssh-able (although it was responding to ping). Clue was that the NPRO PZT control voltage was drifting a lot on the StripTool trace. Koji hard-rebooted the machine. Now IMC is locked, and FSS slow servo is also running.

  14761   Mon Jul 15 14:53:40 2019 MilindUpdateIOOkeyed psl crate, unstick.py

Mode cleaner was not locked today. Koji came in and concluded that PSL had died. So we keyed it. Then we ran my unstick.py code. Mode cleaner is locked now.

Quote:

Today, Gautam keyed the C1PSL crate and we got to test my unstick.py code. It seems to be working fine. Remarks:

  1. Gautam moved the unstick.py code to /opt/rtcds/caltech/c1/scripts/cds. Therefore, the steps to run this code are now:
    1. cd /opt/rtcds/caltech/c1/scripts/cds
    2. python unstick.py c1psl (for the c1psl machine)
  2. There is now a sleepTime global variable in the code which defines the amount of delay between successive channel toggles. We set this to 1ms and it took the code around 3s to run.
  3. Gautam was curious to see if this would work even if we set the sleepTime parameter to 0 but decided that that could be tested the next time something was keyed.
  4. I still need to add the signal handling thing to this code.

Following this, we tested my PMC autolocker code. The code ran for about a minute before achieveing lock. Remarks:

  1. Gautam moved my code (pmc_autolocker.py and autolocker_config.yaml) to /cvs/cds/rtcds/caltech/c1/scripts/PSL/PMC/ . Therefore, the steps to run this code are now:
    1. cd /cvs/cds/rtcds/caltech/c1/scripts/PSL/PMC/
    2. python pmc_autolocker.py (check code or use --help to see what the command line arguments do which is only for when you wanna override the details in the .yaml file)
  2. Gautam suggested that I add some delay between succesive steps of DC output adjust so that it locks quickly. I'll do that ASAP. For now, it works.
  14760   Mon Jul 15 14:09:07 2019 MilindUpdateCamerasCNN LSTM for beam tracking

I've set up network with a CNN encoder (front end) feeding into a single LSTM cell followed by the output layer (see attachment #1). The network requires significantly more memory than the previous ones. It takes around 30s for one epoch of training. Attached are the predicted yaw motion and the fft of the same. The FFT looks rather curious. I still haven't done any tuning and these are only the preliminary results.

Quote:

 Rana also suggested I try LSTMs today. I'll maybe code it up tomorrow. What I have in mind- A conv layer encoder, flatten, followed by an LSTM layer (why not plain RNNs? well LSTMs handle vanishing gradients, so why the hassle).

Well, what about the previous conv nets?

What I did:

  1. Extensive tuning - of learning rate, batch size, dropout ratio, input size using a grid search
  2. Trained each network for 75 epochs and obtained weights, predicted motion and corresponding FFT, error etc.

What I observed:

  1. Loss curves look okay, validation loss isn't going up, so I don't think overfitting is the issue
  2. Training for over (even) 75 epochs seems to be pointless.

What I think is going wrong:

  1. Input size- relatively large input size: 350 x 350. Here, the input image size seems to be 128 x 128.
  2. Inadequate pre-processing.
    1. I have not applied any filters/blurs etc. to the frames.
    2. I have also not tried dimensionality reduction techniques such as PCA

What I will try now:

  1. Collect new data: with smaller amplitudes and different frequencies
  2. Tune the LSTM network for the data I have
  3. Try new CNN architectures with more aggressive max pooling and fewer parameters
  4. Ensembling the models (see this and this). Right now, I have multiple models trained either with same architecture and different hyperparameters or with different architectures. As a first pass, I intend to average the predictions of all the models and see if that improves performance.
Attachment 1: cnn-lstm.png
cnn-lstm.png
Attachment 2: fft_yaw.pdf
fft_yaw.pdf
Attachment 3: yaw_motion.pdf
yaw_motion.pdf
  14759   Mon Jul 15 03:30:47 2019 KruthiUpdateCalibration-RepairWhite paper as a Lambertian scatterer

I made some rough measurements, using the setup I had used for CCD calibration, to get an idea of how good of a Lambertian scatterer the white paper is. Following are the values I got:

Angle (degrees) Photodiode reading (V)  Ps (W) BRDF (per str) % error
12 0.864 2.54E-06 0.334 20.5
24 0.926 2.72E-06 0.439 19.0
30 1.581 4.65E-06 0.528 19.0
41 0.94 2.76E-06 0.473 19.8
49 0.545 1.60E-06 0.423 22.5
63 0.371 1.09E-06 0.475 28

Note: All the measurements are just rough ones and are prone to larger errors than estimated.

I also measured the transmittance of the white paper sample being used (it consists of 2 white papers wrapped together). It was around 0.002

Attachment 1: BRDF_paper.png
BRDF_paper.png
  14758   Mon Jul 15 03:15:24 2019 KruthiUpdateLoss MeasurementImaging scatterometer

On Friday, Koji helped me find various components required for the scatterometer setup. Like he suggested, I'll first set it up on the SP table and try it out with an usual mirror. Later on, once I know it's working, I'll move the setup to the flow bench near the south arm and measure the BRDF of a spare end test mass.

  14757   Sun Jul 14 00:24:29 2019 KruthiUpdateCamerasCCD Calibration

On Friday, I took images for different power outputs of LED. I calculated the calibration factor as explained in my previous elog (plots attached).

Vcc (V) Photodiode
reading(V)

Power incident on photodiode (W)

Power incident on GigE (W)
Slope (counts/​𝝁s)
Uncertainity in
 slope (counts/​𝝁s)
CF (W-sec/counts)
16 0.784 2.31E-06 3.89E-07 180.4029 1.02882 2.16E-15
18 0.854 2.51E-06 4.24E-07 207.7314 0.7656 2.04E-15
20 0.92 2.71E-06 4.57E-07 209.8902 1.358 2.18E-15
22 0.969 2.85E-06 4.81E-07 222.3862 1.456 2.16E-15
25 1.026 3.02E-06 5.09E-07 235.2349 1.53118 2.17E-15
  Average  2.14E-15

To estimate the uncertainity, I assumed an error of at most 20mV (due to stray lights or difference in orientation of GigE and photodiode) for the photodiode reading. Using the uncertainity in slope from the linear fit, I expect an uncertainity of maximum 4%. Note: I haven't accounted for the error in the responsivity value of the photodiode.

GigE area 10.36 sq.mm
PDA area 61.364 sq.mm
Responsivity 0.34 A/W
Transimpedance gain (at gain = 20dB) 10^6 V/W +/- 0.1%
Pixel format used Mono 8 bit

Johannes had reported CF as 0.0858E-15 W-sec/counts for 12 bit images, with measured a laser source. This value and the one I got are off by a factor of 25. Difference in the pixel formats and effect of coherence of the light used might be the possible reasons.

Attachment 1: CCD_calibration.png
CCD_calibration.png
  14756   Fri Jul 12 18:54:47 2019 KojiUpdateGeneralItem loan: optical chopper from Cryo Lab

Optical chopper borrowed from CryoLab to 40m

https://nodus.ligo.caltech.edu:8081/Cryo_Lab/2458

  14755   Fri Jul 12 07:37:48 2019 gautamUpdateSUSM4.9 EQ in Ridgecrest

All suspension watchdogs were tripped ~90mins ago. I restored the damping. IMC is locked.

ITMX was stuck. I set it free. But notice that the UL Sensor RMS is higher than the other 4? I thought ITMY UL was problematic, but maybe ITMX has also failed, or maybe it's coincidence? Something for IFOtest to figure out I guess. I don't think there is a cable switch between ITMX/ITMY as when I move the ITMX actuators, the ITMX sensors respond and I can also see the optic moving on the camera.

Took me a while to figure out what's going on because we don't have the seis BLRMS - i moved the usual projector striptool traces to the TV screen for better diagnostic ability.

Update 16 July 1515: Even though the RMS is computed from the slow readback channels, for diagnosis, I looked at the spectra of the fast PD monitoring channels (i.e. *_SENSOR_*) for ITMX - looks like the increased UL RMS is coming from enhanced BR-mode coupling and not of any issues with the whitening switching (which seems to work as advertised, see Attachment #3, where the LL traces are meant to be representative of LL, LR, SD and UR channels).

Attachment 1: 56.png
56.png
Attachment 2: ITMXunstick.png
ITMXunstick.png
Attachment 3: ITMX_UL.pdf
ITMX_UL.pdf
  14754   Thu Jul 11 18:15:22 2019 gautamSummaryElectronicsPSL/IOO rack checkout

I looked at the PSL/IOO racks to check for which boards, if any, require an additional P2 interface, so that we can try and design a generic one for the IMC/CM boards and whatever else may require it. While searching the elog, I saw that Koji and Johannes had already done this, see Koji's elog in this thread. Some remarks:

  1. D990155 seems to be unused in both PSL and IOO racks. The one in the PSL rack has some LEMO cables plugged in to the front panel, but they go nowhere. So I think that both of these are redundant (in the assessment below, only one was marked redundant).
  2. In the PSL rack, the "TTFSS Interface", "PSL PMC SERVO", and "DAQ INTERFACE" (which I think is obsolete) cards all have their P2 connectors daisy chained together, going to a cross-connect. Kruthi and I traced this to be going to a cross connect marked "J23-PSLRACK-CCP". In the PSL wiring diagram of which we have a hardcopy in the control room, it looks like these channels are related to the RefCav? So I think this is not required to be interfaced to our new Acromag DAQ system. 

Conclusion: Only the IMC Servo and CM boards need their P2 connectors connected to Acromag.It would be helpful to remove the TTFSS Interface board and figure out what exactly the pin-mapping for the backplane connectors are, but I didn't do this today because there is a "High Voltage" line going to the Interface Board and I'm not actually sure of the signal chain for the FSS servo.

  14753   Thu Jul 11 17:58:38 2019 gautamUpdateEquipment loanTT suspension --> Downs

Arnaud has taken 1 TT suspension from the 40m clean lab to Downs for modal testing. Estimated time of return is tomorrow evening.

  14752   Thu Jul 11 16:22:54 2019 KruthiUpdateGeneralProjector lightbulb blown out

I heard a popping sound in the control room; the projector lightbulb has blown out.sad

  14750   Thu Jul 11 13:09:22 2019 gautam SummaryCDSP2 interface board

it will connect to a 15 pin breakout board in the Acromag chassis

Quote:

It's nice and compact, and the cost of new 15-pin DSUB cables shouldn't be a factor here.  What does the 15p cable connect to?

  14749   Thu Jul 11 13:08:36 2019 ChubSummaryCDSP2 interface board

It's nice and compact, and the cost of new 15-pin DSUB cables shouldn't be a factor here.  What does the 15p cable connect to?

  14747   Thu Jul 11 12:42:35 2019 gautamSummaryCDSP2 interface board

I looked into the design of the P2 interface board. The main difficulty here is geometric - we have to somehow accommodate sufficient number of D-sub connectors in the tight space between the two P-type connectors. 

I think the least painful option is to stick with Johannes' design for the P1 connector. For the CM board, the P2 connector only uses 6 pairs of conductors for signals. So we can use a D-15 connector instead of 2 D-37 connectors. Then we can change the PCB shape such that the P1 connector can be accommodated (see Attachment #1). The other alternative would be to have 2 P-type connectors and 3 D-subs on the same PCB, but then we have to be extra careful about the relative positioning of the P-type connectors (otherwise they wont fit onto the Eurocrate). So I opted to still have two separate PCBs.

I took a first pass at the design, the files may be found here. I just auto-routed the connections, this is just an electrical feedthrough so I don't think we need to be too concerned about the PCB trace routing? If this looks okay, we should send out the piece for fab ASAP.

I will work on putting together the EPICS server machine (SuperMicro) this afternoon.

Quote:

2. D040180 / D1500308 Common Mode Board

CM servo board itself doesn't need any modification. The CM board uses P1 and P2. So we need to manufacture a special connector for CM Board P2. (cf The adapter board for P1 T1800260). See also D1700058.

Attachment 1: IMG_7728.JPG
IMG_7728.JPG
  14746   Wed Jul 10 22:32:38 2019 MilindUpdateCamerasConvolutional neural networks for beam tracking

I trained a bunch (around 25 or so - to tune hyperparameters) of networks today. They were all CNNs. They all produced garbage. I also looked at lstm networks with CNN encoders (see this very useful link) and gave some thought to what kind of architecture we want to use and how to go about programming it (in Keras, will use tensorflow if I feel like I need more control). I will code it up tomorrow after some thought and discussion. I am not sure if abandoning CNNs is the right thing to do or if I should continue probing this with more architectures and tuning attempts. Any thoughts?

Right now, after speaking to Stuart (ldas_admin) I've decided on coding up the LSTM thing and then running that on one machine while probing the CNN thing on another.

 


Update on 10 July, 2019: I'm attaching all the results of training here in case anyone is interested in the future.

Quote:

I received access today. After some incredible hassle, I was able to set up my repository and code on the remote system. Following this, Gautam wrote to Gabriele to ask him about which GPUs to use and if there was a previously set up environment I could directly use. Gabriele suggested that I use pcdev2 / pcdev3 / pcdev11 as they have good gpus. He also said that I could use source ~gabriele.vajente/virtualenv/bin/activate to use a virtualenv with tensorflow, numpy etc. preinstalled. However, I could not get that working, Therefore I created my own virtual environment with the necessary tensorflow, keras, scipy, numpy etc. libraries and suitable versions. On ssh-ing into the cluster, it can be activated using source /home/millind.vaddiraju/beamtrack/bin/activate. How do I know everything works? Well, I trained a network on it! With the new data. Attached (see attachment #1) is the prediction data for completely new test data. Yeah, its not great, but I got to observe the time it takes for the network to train for 50 epochs-

  1. On pcdev5 CPU: one epoch took ~1500s which is roughly 25 minutes (see Attachment #2). Gautam suggested that I try to train my networks on Optimus. I think this evidence should be sufficient to decide against that idea.
  2. On my GTX 1060: one epoch took ~30s. Which is 25 minutes (for 50 epochs) to train a network.
  3. On pcdev11 GPU (Titan X I think): each epoch took ~16s which is a far more reasonable time.

Therefore, I will carry out all training only on this machine from now.

 


Note to self:

Steps to repeat what you did are:

  1. ssh in to the cluster using ssh albert.einstein@ssh.ligo.org as described here.
  2. activate virtualenv as descirbed above
  3. navigate to code  and run it.
Quote:

 I attempted to train a bunch of networks on the new data to test if the code was alright but realised quickly that, training on my local machine is not feasilble at all as training for 10 epochs took roughly 6 minutes. Therefore, I have placed a request for access to the cluster and am waiting for a reply. I will now set up a bunch of experiments to tune hyperparameters for this data and see what the results are.

  14745   Wed Jul 10 16:53:22 2019 gautamUpdateSUSPRM watchdog condition modified

[koji, gautam]

We noticed that the PRM watchdog was tripping frequently. This is a period of enhanced seismic activity. The reason PRM in particular trips often is because the SIDE OSEM has 5x increased transimpedance. We implemented a workaround by modifying the watchdog tripping condition to scale the SD channel RMS by a factor of 0.2 (relative to the UL and LL channels). We restarted the modbus process on c1susaux and tested that the new logic works. Here is the relevant snippet of code:

# Disable fast DAC if variation tests too high
# PRM Side is special, see elog 14745
record(calc,"C1:SUS-PRM_LOGIC")
{
    field(DESC,"Tests whether RMS too high")
    field(SCAN,"1 second")
    field(PHAS,"1")
    field(PREC,"0")
    field(HOPR,"1")
        field(LOPR,"0")
        field(CALC,"(A<B)&(C<B)&(0.2*D<B)")
        field(INPA,"C1:SUS-PRM_ULPD_VAR  NPP  NMS")
        field(INPB,"C1:SUS-PRM_PD_MAX_VAR  NPP  NMS")
        field(INPC,"C1:SUS-PRM_LLPD_VAR  NPP  NMS")
        field(INPD,"C1:SUS-PRM_SDPD_VAR  NPP  NMS")
}

The db file has a note about this as well so that future debuggers aren't mystified by a factor of 0.2.

  14744   Wed Jul 10 14:57:01 2019 KojiSummaryCDSChannel recipe for iscaux upgrade

The list of the iscaux channels and pin assignments were posted to google drive.
The spreadsheet can be viewable by the link sent to the 40m ML. It was shared with foteee@gmail for full access.

Summary

  • We need
    4 ADC modules
    5 DAC modules
    5 Binary I/O modules
  • Be aware that there are bundled multiple digital I/O channels such as "mbboDirect" and "mbbi".
  • The full db record of the new channels need to be inferred from the existing channels.

Necessary electronics modification

1. D990694 whitening filter modification (4 modules)

This module shares the fast and slow channels on the top DIN96pin (P1) connector. Also, the whitening selector (done by an analog signal per channel) is assigned over 17pin of the P1 connector, resulting in the necessity of the second DSUB cable. By migrating the fast channels, we can swap the cable from the P1 to P2.  Also, the whitening selectors are concentrated on the first Dsub. (See Attachment1 P1)

2. D040180 / D1500308 Common Mode Board

CM servo board itself doesn't need any modification. The CM board uses P1 and P2. So we need to manufacture a special connector for CM Board P2. (cf The adapter board for P1 T1800260). See also D1700058.

3. D990543A1 LSC Photodiode Interface

PD I/F board has the DC mon channels spread over the 16pin limit. P1 21A can be connected to 6A so that we can accommdate it in the first Dsub.
Also the board uses AD797s. This is not necessary. We can replace them to OP27s. I actually don't know what is happening to those bias control, temp mon, enable, and status. These features should be disables at the I/F and the PDs. (See Attachment2 P1)

Attachment 1: D990694-B.pdf
D990694-B.pdf D990694-B.pdf D990694-B.pdf D990694-B.pdf D990694-B.pdf D990694-B.pdf D990694-B.pdf D990694-B.pdf
Attachment 2: D990543A1.PDF
D990543A1.PDF D990543A1.PDF D990543A1.PDF
  14743   Wed Jul 10 14:55:32 2019 KojiUpdateGeneralProjector lightbulb blown out

In fact the projector is still working. The lamp timer showed ~8200hrs. I just reset the timer, but not sure it was the cause of the shutdown. I also set the fan mode to be "High Altitude" to help cooling.

  14742   Wed Jul 10 10:04:09 2019 gautamUpdateSUSTip-Tilt moved from South clean cabinet to bake lab cleanroom

Arnaud and I moved one of the two spare TT suspensions from the south clean cabinet to the bake lab clean room. The main purpose was to inspect the contents of the packaging. According to the label, this suspension was cleaned to Class A standards, so we tried to be clean while handling it (frocks, gloves, masks etc). We found that the foil wrapping contained one suspension cage, with what looked like all the parts in a semi-assembled state. There were no OSEMs or electronics together with the suspension cage. Pictures were taken and uploaded to gPhoto. Arnaud is going to plan his tests, so in the meantime, this unit has been stored in Cabinet #6 in the bake lab cleanroom.

  14741   Tue Jul 9 22:13:26 2019 MilindUpdateCamerasConvolutional neural networks for beam tracking

I received access today. After some incredible hassle, I was able to set up my repository and code on the remote system. Following this, Gautam wrote to Gabriele to ask him about which GPUs to use and if there was a previously set up environment I could directly use. Gabriele suggested that I use pcdev2 / pcdev3 / pcdev11 as they have good gpus. He also said that I could use source ~gabriele.vajente/virtualenv/bin/activate to use a virtualenv with tensorflow, numpy etc. preinstalled. However, I could not get that working, Therefore I created my own virtual environment with the necessary tensorflow, keras, scipy, numpy etc. libraries and suitable versions. On ssh-ing into the cluster, it can be activated using source /home/millind.vaddiraju/beamtrack/bin/activate. How do I know everything works? Well, I trained a network on it! With the new data. Attached (see attachment #1) is the prediction data for completely new test data. Yeah, its not great, but I got to observe the time it takes for the network to train for 50 epochs-

  1. On pcdev5 CPU: one epoch took ~1500s which is roughly 25 minutes (see Attachment #2). Gautam suggested that I try to train my networks on Optimus. I think this evidence should be sufficient to decide against that idea.
  2. On my GTX 1060: one epoch took ~30s. Which is 25 minutes (for 50 epochs) to train a network.
  3. On pcdev11 GPU (Titan X I think): each epoch took ~16s which is a far more reasonable time.

Therefore, I will carry out all training only on this machine from now.

 


Note to self:

Steps to repeat what you did are:

  1. ssh in to the cluster using ssh albert.einstein@ssh.ligo.org as described here.
  2. activate virtualenv as descirbed above
  3. navigate to code  and run it.
Quote:

 I attempted to train a bunch of networks on the new data to test if the code was alright but realised quickly that, training on my local machine is not feasilble at all as training for 10 epochs took roughly 6 minutes. Therefore, I have placed a request for access to the cluster and am waiting for a reply. I will now set up a bunch of experiments to tune hyperparameters for this data and see what the results are.

Attachment 1: predicted_motion_first.pdf
predicted_motion_first.pdf
Attachment 2: pcdev5_time.png
pcdev5_time.png
  14740   Tue Jul 9 18:42:15 2019 gautamUpdateALSEX green doubling oven temperature controller power was disconnected

There was no green light even though the EX NPRO was on. I checked the doubling oven temperature controller and found that its power cable was loose on the rear. I reconnected it, and now there is green light again. 

  14739   Tue Jul 9 18:17:48 2019 gautamUpdateGeneralProjector lightbulb blown out

Last documented replacement in Nov 2018, so ~7 months, which I believe is par for the course. I am disconnecting its power supply cable.

ELOG V3.1.3-