I tried restarting the fb in two different ways.  Neither of them re-established the connection to dtt or epics.

1) I restarted the fb from the control room console with the 'shutdown' command.  No change.

2) I halted the machine with 'shutdown -h now' and restarted it with the hardware reset button on its front-panel. No change.

The console connected to the fb showed that the network file systems did not load.  Could this have resulted in failure to start several services since it could not find the files which are stored on the network file system?

The fb is otherwise healthy since I am able to ssh into it and browse the directory structure.

[Mike, Rana]

The fb is okay.  Rana found that it works on Pianosa, but not on Allegra or Rossa.  It also works on Rosalba, on which Jamie recently installed Ubuntu.

The white fields on the medm  'Status' screen for fb are an unrelated problem.

 [Koji, Suresh]

  We tried to locate the sixteen analog output channels we need to control the four tip-tilts (four coils on each).  We have only 8 available channels on the C1SUS machine. 

  So we will have to plug-in a new DAC output card on one of the machines and it would be logical to do that on the C1IOO machine as the active tip-tilts are conceptually part of the IOO sub-system.   We have to procure this card if we do not already have it.  We have to make an interface between this card output and a front panel on the 1X2 rack.  We may have to move some of the sub-racks on the 1X2 rack to accommodate this front panel.

  We checked out the availability of cards (De-whitening, Anti-imaging, SOS coil drivers)  yesterday.  In summary: we have all the cards we need (and some spares too).  As the De-whitening and Anti-imaging cards each have 8 channels, we need only two of each to address the sixteen channels.  And we need four of the SOS coil drivers, one for each tip-tilt.  There are 9 slots available on the C1IOO satellite expansion chassis (1X1 rack), where these eight cards could be accommodated.

  There are two 25 pin feed-thoughs, where the PZT drive signals currently enter the BS chamber.  We will have to route the SOS coil driver outputs to these two feed-throughs. 

   Inside the BS chamber, there are cables which carry the PZT signals from the chamber wall to the the table top, where they are anchored to a post (L- bracket).  We need a 25-pin-to-25-pin cable (~2m length) to go from the post to the tip-tilt (one for each tip-tilt).  And then, of course, we need quadrapus cables (requested from Rich) which fit inside each tip-tilt to go to the BOSEMs.

   I am summarising it all here to give an overview of the work involved.

    The only script that can currently take this action is the MC autolocker.  If that is disabled first and the PMC unlocks later, the WFS will not be turned off.  During the last round of discussions we had about the autolocker script, sometime last Nov, we decided that too much automation is not desirable and that the autolocker should be kept as simple as possible.

   Here is the suggested layout of the MC health check web page layout. I will update the Omnigraffle file as people comment and suggest changes.  If you want the file let me know.

[Yuta, Suresh]

We found that the MC was not locking and that the alignment between PSL and MC was too poor to obtain a TEM00 mode in the MC.   To correct the situation we went through the following steps:

1) We burt restored the MC alignment slider values to their values at 3:07 AM of today

2) We turned off the MC-autolocker and the ASC signal to the coils.   Then aligned the PSL beam into the MC (with the MC servo loop off) to obtain the TEM00 mode.  We had to adjust the zig-zag at the PSL output by quite a bit to maximise MC transmission.

3) We then centered the spot on the MC2 face and centered the transmitted beam on the MC2_TRANS_QPD

4) Next, we centered the beams on the MC_WFS sensors.

5) Turning on the WFS loops after this showed that everything works fine and WFS loops do not accumulate large offsets.

[Koji, Yuta, Suresh]

We measured the MC spot positions after re-aligning the MC.  The spot positions are listed below:

spot positions in mm (MC1,2,3 pit MC1,2,3 yaw):
    3.9073    6.6754    2.8591   -7.6985   -0.9492    7.0423

Procedure:

1) In the directory /opt/rtcds/caltech/c1/scripts/ASS/MC  we have the following scripts

      a) mcassUp:    This sets up the MCASS lockins to excite each of the MC mirrors at a different frequency

      b) mcassOn:    This sets the MCASS output matrix to actually send the excitation signals to the mirrors

      c) senseMCdecenter:  This sequentially introduces a 10% offset into the coil gains of each mirror degree of freedom.   It also sends the lockin output data to the screen. 

      d) sensemcass.m :  This is a matlab file which digests the data gathered by the senseMCdecenter script to print a couple of plots and compute the spot positions.

      e) MCASS_StripTool.stp:  This is a set-up file for the StripTool which allows us to see the MCASS-lockin_outputs.  It is nice to see the action of senseMCdecenter  script  at work.

2) So the series of commands to use are

      a) ./StripTool  <-- MCASS_StripTool.stp

      b) ./mcassUp

      c) ./mcassOn

      d) ./senseMCdecenter | tee Output_file

       e) ./mcassOff

      f) ./mcassDown

       g) matlab <-- sensemcass.m  <---- Output_file

Mike Pedraza came by today to install a new wireless network router configured for the 40m lab network.  It has a 'secret' SSID i.e. not meant for public use outside the lab.  You can look up the password and network name on the rack.  Pictures below show the location of the labels.

I have made a preliminary sketch of the cabling involved in connecting the Tip-tilt coil drivers.   This is a preliminary document. 

 The scheme currently is to run a 25pin Kapton strip cable from BS to IMC table inside the chamber.  However we cannot do the same for the OMC table since it will cross the bellows which we often remove.  So we need to supply the one tip-tilt on the OMC table from outside.  I could not spot a spare unused feedthough on the OMC chamber.  We will have to swap one of the blank flanges for one with a few feed throughs.

We do not have any of the cables.   So everything listed has to be arranged for.   The pics are from the existing coil driver system on the SUS machine.

Recent History

The lower blades which I had given to the Physics Workshop for making a vacuum relief hole (using a sinker-EDM process) came back about ten days ago.   Merih Eken <meken@caltech.edu>,  the supervisor at the Physics Dept workshop, handled this matter for us.  The blades were sent to a local EDM machineshop and returned in about three working days ( a weekend intervened). 

Bob cleaned and handed them over to me yesterday evening.  

Current status

Today I have reassembled the four tip-tilts.  I have repacked them in clean bags (double bagged) shifted them to Clean Optics Cabinet (near the ETMX chamber).  The four tip-tilts are in the bottom-most shelf in the cabinet.  There are also some tip-tilt spares in a separate envelope.

Note:  The mirror holder is now held tightly by the eddy current dampers.  This was done for safety of the wires during transportation from LHO.  The eddy current damper in the front of the mirror has to be retracted to allow the mirror holder to swing free.  It has be to about 1mm away from the suspended mirror holder

Work Remaining

1) We need to install the quadrapus cables.  The connector placement on the BOSEM side will have some issues.  It is best to loosen the BOSEM seating as well as the connector seating screws and then push the cable connector into place.  Caution:  when the connector seating screws on the BOSEM are loosened the flexible ckt could be damaged by the loose connector.

2) Insert the mirrors into the mirror holders and balance the suspension such that the mirror's hang vertical and do not have a large yaw offset.

3) Adjust the wire suspension point height so that the flags are in the center of the BOSEM aperture.  Else they will strike against the

4) We need to adjust the position of the BOSEMs such that the shadow sensor signals are at 50%.  This ensures that all the magnets hang at an appropriate distance from their respective coils.

5) To do (3) we need to set up a shadow sensor read-out set-up for one tip-tilt (four sensors)

The reason for using Alberto's laser is that some amount of work has already gone into characterising its phase noise.  Ref elog entry 2788

We have positioned the guide rod and the wire-stand-off on the optic in the axial direction. 

We have selected six magnets whose magnetic strength is +/-5% of their mean strength (180 Gauss).  The measurement was made as follows:

1) each magnet was placed on its  end, on the top of a beaker held upside down. 

2) The Hall probe was placed directly under the magnet touching the glass from the other side (the inside of the beaker). 

This ensures that the relative position of the magnet and the probe remains fixed during a measurement.  And ensures that their separation is the same for each of the magnets tested. 

With this procedure the variation in the measured B field is less than +/- 10% in the sample of magnets tested.

[Jenne, Suresh]

The ETM assembly has moved forward a couple of steps.  We have completed the following:

1) Positioning the guide rod and wire stand-off on both the ETMs (5 and 7)

2) The magnets had to be cleaned with an acetone wash as they had touched the plastic Petri-dish (not cleaned for vacuum).

3) The magnets and the Al dumb-bells have been glued together and left to cure in the gluing fixture.

4) The guide-rod and wire stand-offs have also been glued to the optic and left to cure for 24 hrs.

JD:  As you can see in my nifty status table, we are nearing the end of the suspension story.  

We are going to try (but can't guarantee) to get ETMX to Bob for baking by Friday at lunchtime, that way we can re-suspend it on ~Monday, and place it in the chamber.  Then we could potentially begin Green arm locking next week.  Steve has (hopefully!!) ordered the spring plungers for ETMY.  The receiving and baking of the spring plungers is the only current delay that I can foresee, and that only is relevant for one of the optics. 

We (who is going to be in charge of this?) still need to move the SRM OSEMs & cables & connectors to the ITMY chamber from the BS chamber. 

[Jenne, Suresh]

 Jenne found the X-end table enclosure had been left open.  She replaced the lid on it.

We did several things today+night.  The final goal was to lock the PRC so that we could obtain the POX, POY and POP beams.  However there were large number of steps to get there.

1) We moved the ITMY into its place and balanced the table

2) We then aligned the Y-arm cavity to the green beam which was set up as a reference before we moved the ETMY and ITMY to adjust the OSEMS.  We had the green flashing in Y-arm

3) We checked the beam position on PR2. It was okay. This confirmed that we were ready to send the beam onto the Y arm.

4) We then roughly aligned the IR beam on ETMY where Jamie had placed an Al foil with a hole.  We got the arm flashing in both IR and green. 

5) We used the PZTs to make the green and IR beams co-incident and flashing in the Y arm.  This completed the alignment of the IR beam into the Y-arm.

6) The IPPO (pick-off) window had to be repositioned to avoid clipping.  The IPANG beam was aligned such that it exits the ETMY chamber onto the ETMY table.  It can now be easily sent to the IPANG QPD.

7) Then BS was aligned to direct the IR beam into the X-arm and had the X-arm flashing.  It had already been aligned to its green.

8) It was now the turn of the SRC.  The beam spots on all the SRC related optics were off centered.  We aligned all the optics in the AS path to get the AS beam on to the AP table.

9) The AS beam was very faint so we repositioned the AS camera to the place intended for AS11 PD, since there was a brighter beam available there. 

10) We could then obtain reflections from ITMY, ITMX and PRM at the AS camera. 

11) Problems:

      a) ITMY osems need to be readjusted to make sure that they are in mid-range.  Several are out of range and so the damping is not effective.

      b) When we tried to align SRC the yaw OSEM had to be pushed to its full range.  We therefore have to turn the SRM tower to get it back into range.

 12)  We stopped here since moving the SRM is not something to be attempted at the end of a rather long day. Kiwamu is posting a plan for the rest of the day.

PR2's side magnet height was adjusted and its roll was balanced (attachment 1,2). I verified that the OpLev beam is still aligned. The pitch was balanced: First, using an iris for rough adjustment. Then, with the QPD. I locked the counterweight setscrew.

I turned off the HEPAs, damped PR2, and measured the QPD spectra (attachment 3). Major peaks are at 690mHz, 953mHz, and 1.05Hz. I screwed back the lower OSEM plate. The wires were clamped to the suspension block and were cut. Winch adapter plate removed. I wanted to push OSEMs into the OSEM plates but the wiki is down so I can't tell what was the plan. This will have to wait for tomorrow. Also here like with AS1 we need to apply glue to the counterweights.

 Yes, it should be. However, what I did was calculating thermal noise of MC. I'm not sure about the 40m IMC's actual noise level. The plot in the entry was taken from LLO's MC in 2003.

Also deleted the ~50GB error files from ldas to prevent rsync from copying them to nodus again. With the new update to GWsumm, there are new error messages that initially didn't seem to affect the summary pages functionality, but in the extreme case can populated the error files the repeated warnings on the form "Loading: FrSerData", "Loading: FrSerData::n4294967295", "Loading: FrSummary","Loading: FrSerDataLoading: FrSerData" and many more combinations until we get file sizes of the order of ~50GB. So I have updated the checkstatus script to parse the error files and strip out the majority of these error messages. Work is ongoing to get them all.

In light of these large files generation, I decided to look in the summary pages folder to see if there are other large files that we need to keep track of and it turns there are indeed a collection of files in the archive folder that bloats the summary pages on ldas to ~1TB. Luckily these are not synced to nodus so no problem here. However, since the beginning of the year, the archive folders that hold data used for each day's computation have not been cleared. We have a script for doing this but it has not been run for a while now and it only delete archive files for a specific month which is hardcoded to two months from the date the file is run. I have modified the code to allow archive deletion for a range of months so we can clear data from Jan to July. 

[paco]

This morning, I spent some time restoring the jupyter notebook server running in allegra. This server was first set up by Anchal to be able to use the latest nds python API tools which is handy for the calibration stuff. The process to restore the environment was to run "source ~/bashrc.d/*" to restore some of the aliases, variables, paths, etc... that made the nds server work. I then ran ssh -N -f -L localhost:8888:localhost:8888 controls@allegra from pianosa and carry on with the experiment.

[paco, hang, tega]

We started a notebook under /users/paco/20210906_XARM_Cal/XARM_Cal.ipynb on which the first part was doing the following;

• Set up list of excitations for C1:LSC-XARM_EXC (for example three sine waveforms) using awg.py
• Make sure the arm is locked
• Read a reference time trace of the C1:LSC-XARM_IN2 channel for some duration
• Start excitations (one by one at the moment, ramptime ~ 3 seconds, same duration as above)
• Get data for C1:LSC-XARM_IN2 for an equal duration (raw data in Attachment #1)
• Generate the excitation sine and cosine waveforms using numpy and demodulate the raw timeseries using a 4th order lowpass filter with fc ~ 10 Hz
• Estimate the correct demod phase by computing arctan(Q / I) and rerunning the demodulation to dump the information into the I quadrature (Attachment #2).
• Plot the estimated ASD of all the quadratures (Attachment #3)

[paco, hang, tega]

Estimation of open loop gain:

• Grab data from the C1:LSC-XARM_IN1 and C1:LSC-XARM_IN2 test points
• Infer excitation from their differnce, i.e. C1:LSC-XARM_EXC = C1:LSC-XARM_IN2 - C1:LSC-XARM_IN1
• Compute the open loop gain as follows : G(f) = csd(EXC,IN1)/csd(EXC,IN2), where csd computes the cross spectra density of the input arguments
• For the uncertainty in G, dG, we repeat steps (1) to (3) with & without signal injection in the C1:LSC-XARM_EXC channel. In the absence of signal injection, the signal in C1:LSC-XARM_IN2 is of the form: Y_ref = Noise/(1-G), whereas with nonzero signal injection, the signal in C1:LSC-XARM_IN2 has the form: Y_cal = EXC/(1-G) + Noise/(1-G), so their ratio, Y_cal/Y_ref = EXC/Noise, gives the SNR, which we can then invert to give the uncertainty in our estimation of G, i.e dG = Y_ref/Y_cal.
• For the excitation at 53 Hz, our measurtement for the open loop gain comes out to about 5 dB whiich is consistent with previous measurement.
• We seem to have an SNR in excess of 100 at measurement time of 35 seconds and 1 count of amplitude which gives a relative uncertainty of G of 0.1%
• The analysis details are ongoing. Feedback is welcome.

I finally got the modbus part working on chiara, so we can now view the temperature data on any machine on the martian network, see Attachment 1. 

I also updated the entries on /opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini, as suggested by Koji, to include the SensorGatway temperature channels, but I still don't see their EPICs channels on https://ldvw.ligo.caltech.edu/ldvw/view. This means the channels are not available via nds so I think the temperature data is not being to be written to frame files on framebuilder but I am not sure what this entails, since I assumed C0EDCU.ini is the framebuilder daq channel list.

When the EPICs channels are available via nds, we should be able to display the temperature data on the summary pages.

Anchal mentioned it would be good to put more details about how I arrived at the values needed to configure the modbus drive for the temperature sensor, since this information is not in the manual and is hard to find on the internet, so here is a breakdown.

So the generic format is:

drvAsynIPPortConfigure("<TCP_PORT_NAME>","<UNIT_IP_ADDRESS>:502",0,0,1)
modbusInterposeConfig("<TCP_PORT_NAME>",0,5000,0)
drvModbusAsynConfigure("<PORT_NAME>","<TCP_PORT_NAME>",<slaveAddress>,<modbusFunction>,<modbusStartAddress>,<modbusLength>,<dataType>,<pollMsec>,<plcType>)

which in our case become:

drvAsynIPPortConfigure("c1pemxendtemp","192.168.113.240:502",0,0,1)
modbusInterposeConfig("c1pemxendtemp",0,5000,0)
drvModbusAsynConfigure("C1PEMXENDTEMP","c1pemxendtemp",0,4,199,2,0,1000,"ServerCheck")

As can be seen, the parameters of the first two functions "drvAsynIPPortConfigure" and "modbusInterposeConfig" are straight forward, so we restrict our discussion to the case of third function "drvModbusAsynConfigure". Well, after hours of trolling the internet, I was able to piece together a coherent picture of what needs doing and I have summarised them in the table below.

drvModbusAsynConfigure

Once the asyn IP or serial port driver has been created, and the modbusInterpose driver has been configured, a modbus port driver is created with the following command:

drvModbusAsynConfigure(portName,                # used by channel definitions in .db file to reference this unit)
                       tcpPortName,             # reference to portName created with drvAsynIPPortConfigure command
                       slaveAddress,            # 
                       modbusFunction,          # 
                       modbusStartAddress,      # 
                       modbusLength,            # length in dataType units
                       dataType,                # 
                       pollMsec,                # how frequently to request a value in [ms]
                       plcType);                #

Useful links

https://nodus.ligo.caltech.edu:8081/40m/16214

https://nodus.ligo.caltech.edu:8081/40m/16269

https://nodus.ligo.caltech.edu:8081/40m/16270

https://nodus.ligo.caltech.edu:8081/40m/16274

http://manuals.serverscheck.com/InfraSensing_Sensors_Platform.pdf

http://manuals.serverscheck.com/InfraSensing_Modbus_manualv5.pdf

https://community.serverscheck.com/discussion/comment/7419#Comment_7419

https://wiki-40m.ligo.caltech.edu/CDS/SlowControls

https://www.slac.stanford.edu/grp/ssrl/spear/epics/site/modbus/modbusDoc.html#Creating_a_modbus_port_driver

https://github.com/riptideio/pymodbus

https://en.wikipedia.org/wiki/Modbus

https://deltamotion.com/support/webhelp/rmctools/Communications/Ethernet/Supported_Protocols/Ethernet_Modbus_TCP.htm

[Tega, Anchal, Paco]

After talking to Anchal, it was made clear that chiara is not the place to host the modbus service for the temperature sensors. The obvious machine is c1pem, but the startup cmd script loads c object files and it is not clear how easy it would integrate the modbus functionality since we can only login via telnet, so we decided to instead host the service on c1susaux. We also modified the /etc/motd file on c1susaucx which displays the welcome message during login to inform the user that this machine hosts the modbus service for the temperature sensor. Anchal plans to also document this information on the temperature sensor wiki at some point in the future when the page is updated to include what has been learnt so far.

We might also consider updating the database file to a more modern way of reading the temperature sensor data using FLOAT32_LE which is available on EPICs version 3.14 and above, instead of the current method which works but leaves the reader bemused by the bitwise operations that convert the two 16 bits words (A and B) to IEEE-754 32-bit float, via 

field(CALC, "(A&D?(A&C?-1:1):0)*((G|A&E)*J+B)*2^((A&D)/G-F)")

where 

   field(INPA, "$HiWord")
   field(INPB, "$LoWord")
   field(INPC, "0x8000")   # Hi word, sign bit
   field(INPD, "0x7F80")   # Hi word, exponent mask
   field(INPE, "0x00FF")   # Hi word, mantissa mask (incl hidden bit)
   field(INPF, "150")      # Exponent offset plus 23-bit mantissa shift
   field(INPG, "0x0080")   # Mantissa hidden bit
   field(INPJ, "65536")    # Hi/Lo mantissa ratio
   field(CALC, "(A&D?(A&C?-1:1):0)*((G|A&E)*J+B)*2^((A&D)/G-F)")
   field(PREC, "4")

as opposed to the more modern form

field(INP,"@asyn($(PORT) $(OFFSET))FLOAT32_LE")

We can now view the minute trend of the temperature sensors under the PEM tab of the summary pages. See attachment 1 for an example of today's temperature readings. 

Running update of Sat Amp modification work, which involves the following procedure (x8) per unit:

1. Replace R20 & R24 with 4.99K ohms, R23 with 499 ohms, and remove C16.
2. (Testing) Connect LEDDrive output to GND and check that
   • TP4 is ~ 5V
   •  TP5-8 ~ 0V. 
3. Install 40m Satellite to Flange Adapter (D2100148-v1)

[Koji, Tega]

Decided to do a quick check of the remaining Sat Amp units before component replacement to identify any unit with defective LED circuits. Managed to examine 5 out of 10 units, so still have 5 units remaining. Also installed the photodiode bias voltage jumper (JP1) on all the units processed so far.

Defective unit with updated resistors and capacitors in the previous elog

Debugging complete.

All units now have the correct TP4 voltage reading needed to drive a nominal current of 35 mA through to OSEM LED. The next step is to go ahead and replace the components and test afterward that everything is OK.

Trying to finish 2 more Sat Amp units so that we have the 7 units needed for the X-arm install. 

S2100736 - All good

S2100737 - This unit presented with an issue on the PD1 circuit of channel 1-4 PCB where the voltage reading on TP6, TP7 and TP8 are -15.1V,  -14.2V, and +14.7V respectively, instead of ~0V.  The unit also has an issue on the PD2 circuit of channel 1-4 PCB because the voltage reading on TP7 and TP8 are  -14.2V, and +14.25V respectively, instead of ~0V.

[Tega, Koji]

(S2100737) - Debugging showed that the opamp, AD822ARZ, for PD2 circuit was not working as expected so we replaced with a spare and this fixed the problem. Somehow, the PD1 circuit no longer presents any issues, so everything is now fine with the unit.

(S2100741) - All good.

[S2100738, S2100745, S2100751] Completed three more Sat Amp units modification with seven remaining.

Modifications and testing of SatAmp units COMPLETE. Attachments 1 & 2 show all 19 units, one installed unit and the remaining 18 units are stacked and ready for install. Detailed notes of the modification for each unit are presented in the summary document in the dcc.

[Ian,Tega]

Today we continued working on setting up the 6 degrees of freedom model for testing the suspension which we copied over from  "/cvs/cds/rtcds/userapps/release/sus/c1/models/c1sup.mdl" to c1sp2.mdl in the same folder. We then changed the host from c1lsc to c1sus2, changed cpu # from 7 to 3 bcos c1sus2 has 6 cores. Then ran the following commands to build and install the model on c1sus2:

$ ssh c1sus2

$ rtcds make c1sp2

$ rtcds install c1sp2

where we encountered the following installation error:

It seems that changing the model name and host did not change the node allocation, so will remove the previous entries in testpoint.par to see if that helps. After deleting the following lines

[C-node62]
hostname=c1lsc
system=c1sup

from the file "/opt/rtcds/caltech/c1/target/gds/param/testpoint.par", the installation went fine and the above entries were replaced by 

[C-node62]
hostname=c1sus2
system=c1sp2

BTW, I now believe the reason we had the node conflict earlier was bcos both models still had the same value of  dcuid=62, so I think changing this value in our model file would be a better solution. Work is ongoing.

[Tega, Ian]

TODO

1. Investigate cross-coupling btw the various degrees of freedom (dof) - turn on noise for each dof in the plant model and measure the transfer function of the other dofs.

2. Get a closed-loop transfer function using noise injection and give a detailed outline of the procedure in elog - IN1/IN2 for each TM_RESP filter while the others are turned off.

3. Derive analytic model of the closed-loop transfer functions for comparison.

4. Adapt control filters to fit optimized analytical solutions.

The new SUS screen can be reached via sitemap -> IFO SUS button -> NEW ETMX dropdown menu link. Please use and provide feedback. Not sure exactly if we need/want the display screens after the IOP model on the right of the medm screen. I have not been able to locate the corresponding channels but did not want to remove them until I was sure that we don't plan to add these features to our screens. When all bugs have been ironed out, we can use appropriate macro substitution for the other optics.

The next feature to add is the BLRMS to the coil and PD channels. I plan to combine the PEM BLRMS medm implementation with the sus_single_BLRMS model block (located in  /opt/rtcds/userapps/release/cds/c1/models). This way we use the latest BLRMS block in "/opt/rtcds/userapps/release/cds/common/models/BLRMS.mdl" whilst also leveraging the previous work done on the sus_single_BLRMS model, which neatly fits into our current SUS model.

Work on the medm screen for SUS RMS monitor is ongoing. The next step would be to incorporate this into the SUS medm screen, add the BLRMS model to the SUS controller model, recompile, check that the channels are being correctly addressed, then load the appropriate bandpass and lowpass filters.  

Turns out the BLRMS monitoring channels for MC1, MC2, MC3, ITMY and SRM already exist in c1pem. So I modified the new SUS screen to display the BLRMS info for the aforementioned optics. Next step is to add the BLRMS monitor for PRM, ITMX, ETMX and ETMY. This would require extending the number of inputs for the "SUS" block in c1pem to accomodate the additional inputs from the remaining optics.

[temperature sensor]

git repo: https://git.ligo.org/40m/tempsensor.git

todo

Update the temp sensor channels to fit with cds format, ie. "C1:PEM-TEMP_EX", "C1:PEM-TEMP_EY", "C1:PEM-TEMP_BS"

- Use FLOAT32_LE data format for the database file (/cvs/cds/caltech/target/c1pem1/tempsensor/C1PEMaux.db) to create the new channels.

- Keep the old datadase code and channels so we can compare with new temp channels afterwards. Also we need a 1-month overlap b4 deleting the old channels.

[sus medm screen]

git repo: https://git.ligo.org/40m/susmedmscreen.git

todo (from talk with Koji)

- Link stateword display to open "C1CDS_FE_STATUS.adl"

- Damp filter and Lock filter buttons should open a 3x1 filter screen so that the 6 filters are opened by 2 buttons compared to the old screen that has 3 buttons connected to 2X1 filter screen

- Make the LOCKIN signla modulation flow diagramlook more like the old 40m screen since that is a better layout

- Move load coefficient button to top of sus medm screen (beside stateword)

- The rectangular red outline around the oplev display is confusing and needs to be modified for clarity

- COMM tag block should not be 3D as this suggests it is a button. Make it flat and change tag name to indicate individual watchdog control as this better reflect its functionality. Rename current watchdog switch to watchdog master is it does what the 5 COMM switches do at once.

- Macro pass need to be better documented so that when we call the sus screens from locations other than sitemap, we should know what macro variables to pass in, like DCU_ID etc.

- Edit sitemap.adl to point only to the new screens. Then create a button on the new screen that points to the old screen. This way, we can still access the old screen without clogging sitemap.

- Move the new screen location to a subfolder of where the current sus screens reside, /opt/rtcds/userapps/trunk/sus/c1/medm/templates

- Rename the overview screen (SUS_CUST_HSSS_OVERVIEW.adl) to use the SUS_SINGLE nomenclature, i.e. SUS_SINGLE_OVERVIEW.adl

- Keep an eye of the cpu usage of c1pem as we add BLRMS block for other optics. 

[Temperature sensor]

Added new temp EPICs channels to database file (/cvs/cds/caltech/target/c1pem1/tempsensor/C1PEMaux.db)

Added new temp EPICs channels to slow channels ini file (/opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini)

[SUS medm screen]

Moved new SUS screen to location : /opt/rtcds/userapps/trunk/sus/c1/medm/templates/NEW_SUS_SCREENS

Place button on the new screen to link to the old screen and replace old screens link on sitemap.

Fixed Load Coefficient button location issue

Fixed LOCKIN flow diagram issue

Fixed watchdog labelling issue

Linked STATE WORD block to FrontEnd STATUS screen

Replaced the 2x1 pit/yaw filter screens for LOCK and DAMP fliters with 3x1 LPY filter screen

*Need some more time to figure out the OPTLEV red indicator

Seen. Thanks.

Red Arrow: The channel was labeled incorrectly as INMON instead of OUTPUT

Green Arrow: OK, I will create a custom medm screen for this.

Blue arrow: Hmm, OK I will look into this. Doing this work remotely is a pain as the medm response is quite slow for poking around.

Orange circle: OK, I'll move this to the left side of the line.

Note to self: I also noticed another error on the side (LPYS blue box just b4 the sum). The channel is pointing to YAW instead of the side, so needs to be fixed as well.

Started working on this. First created a git repo for tracking https://git.ligo.org/40m/susaux.git

I have looked through the folder to see what needs doing and I think I know what needs to be done for the final case by just following the same pattern for the other optics, which I am listing below

- Create database file for the BHD optics, say C1_SUS-AUX_LO1.db by copying another optic database file say C1_SUS-AUX_SRM. Then replace the optic name.

- Insert a new line "C1:SUS-LO1_LATCH_OFF" in the file autoBurt_watchdogs.req

- Populate the file autoBurt.req with the appropriate channels for LO1

- Populate the file C1SUSaux_post.sh with the corresponding commands for LO1

- Add the line dbLoadDatabase("/cvs/cds/caltech/target/c1susaux/C1_SUS-AUX_LO1.db") to the file C1SUSaux.cmd

For the temporary watchdog, we comment everything I have just talked about, and do only what come next.

My question is the following:

I understand that we need to use the OUT16 slow channel as a temporary watchdog since we don't currently have access to the slow channels bcos the Acromag units have not been installed. My guess from Koji's instructions is that we need to update the channels in the last two fields "INPA" and "INPB" below

record(calc,"C1:SUS-LO1_UL_CALC")
{
        field(DESC,"ANDs Enable with Watchdog")
        field(SCAN,"1 second")
        field(PHAS,"1")
        field(PREC,"2")
        field(HOPR,"40")
        field(LOPR,"-40")
        field(CALC,"A&B")
        field(INPA,"C1:SUS-LO1_UL_COMM  PP  NMS")
        field(INPB,"C1:SUS-LO1_LATCH_OFF  PP  MS")
}

Suppose we replace the channel for INPA with C1:SUS-LO1_ULCOIL_OUT16, what about INPB. Is this even the right thing to do as I am just guessing here?

After some work on the reference database file, we now have a template for temporary watchdog implementation for LO1 located here "/cvs/cds/caltech/target/c1susaux/C1_SUS-AUX_LO1.db".

Basically, what I have done is swap the EPICS asyn analog input readout for the COIL and OSEM to accessible medm channels, then write out watchdog enable/disable to coil filter SW2 switch. Everything else in the file remains the same. I am worried about some of the conversions but the only way to know more is to see the output on the medm screen.

To test, I restarted c1su2 but this did not make the LO1 database available, so I am guessing that we also need to restart the c1sus, which can be done tomorrow.

Temp software watchdog now operational for LO1 and the remaining optics!

Koji helped me understand how to write to switches and we tried for a while to only turnoff the output switch of the filters instead of the writing a zero that resets everything in the filter.

Eventually, I was able to move this effort foward by realising that I can pass the control trigger along multiple records using the forwarding option 'FLNK'. When I added this field to the trigger block, record(dfanout,"C1:SUS-LO1_PUSH_ALL"), and subsequent calculation blocks, record(calcout,"C1:SUS-LO1_COILSWa") to record(calcout,"C1:SUS-LO1_COILSWd"), everything started working right.

All done (almost)! I still have not sorted the issue of pitch and yaw gains growing together when modified using ramping time. Image of custom ADC and DAC panel is attached.

[Ian, Tega]

Connected the New SUS screens to the controller for the simplant model. Because of hard-coded links in the medm screen links, it was necessary to create the following path in the c1sim computer, where the new medm screen files are located:

We noticed a few problems:

1. Some of the medm files still had C1 hard coded, so we need to replace them with $IFO instead, in order for the custom damping filter screen to be useful.

2. The "Load coefficient" button was initially blank on the new sus screen, but we were able to figure out that the problem came from setting the top-level DCU_ID to 63.

[TODO]

Get the data showing the controller damping the pendulum. This will involve tweaking some gains and such to fine-tune the settings in the controller medm screen. Then we will be able to post some data of the working controller.

[Useful aside]

We should have a single place with all the instructions that are currently spread over multiple elogs so that we can better navigate the simplant computer.

[Ian, Paco, Tega]

Last night we set up the four main matrices that handle the conversion between the degrees of freedom bases and the sensor bases. We also wrote a bash script to automatically set up the system. The script sets the four change of bases matrices and activates the filters that control the plant. this script should fully set up the plant to its most basic form. The script also turns off all of the built-in noise generators. 

After this, we tried damping the optic. The easiest part of the system to damp is the side or y motion of the optic because it is separate from the other degrees of freedom in both of the bases. We were able to damp that easily. in attachment 1 you can see that the last graph in the ndscope screen the side motion of the optic is damped. Today we decided to revisit the problem.

Anyways, looking at the problem with fresh eyes today, I noticed the in pit2pit coupling has the largest swing of all the plant filters and thought this might be the reason why the inputs (UL,UR,LR,LL) to the controller was hitting the rails for pit DoF. I reduce the gain of the pit2pit filter then slowly increased it back to one. I also reduced the gain in the OSEM input filter from 1 to 1/100. The attached image (Attachment2) is the output from this trial. This did not solve the problem. The output when all OSEM input filter gain set to one is shown in Attachment2.

We will try to continue to tweak the coefficients. We are probably going to ask Anchal and Paco to sit down with us and really hone in on the right coefficients. They have more experience and should be able to really get the right values.

git repo: git@git.ligo.org:tega-edo/charmirrormap.git

The analysis code takes in a set of raw images, 10 in our case,  for each mirror and calculates the zernike aberration coefficients for each image, then takes their average. This average value is used to reconstruct the mirror height map.  Finally, the residual error between the reconstructed image and the raw data is calculated.

We repeat the analysis for different field of views (FoV) namely 10mm, 20mm, 30mm, 40mm and 46.5mm and save the results in the output folder of the repo.

The analysis output for a 10mm FoV aperture at the mirror center is shown in the attachement. These three images show the input data, the reconstructed mirror surface map and the residual error.

Finally got the SIMPLANT damping to work following Rana's suggestion to try damping one DoF at a time, woo-hoo!

At first, things didn't look good even when we only focus on the POS DoF. I then noticed that the input value (X1:SUS-OPT_PLANT_TM_RESP_1_1_IN1) to the plant was always zero. This was odd bcos it meant the control signal was not making its way to the plant. So I decided to look at the sensor data

(X1:SUS-OPT_PLANT_COIL_IN_UL_OUTPUT, X1:SUS-OPT_PLANT_COIL_IN_UR_OUTPUT, X1:SUS-OPT_PLANT_COIL_IN_LR_OUTPUT, X1:SUS-OPT_PLANT_COIL_IN_LL_OUTPUT)

that adds up via the C2DOF matrix to give the POS DoF and I noticed that these interior nodes can take on large values but always sum up to zero because the pair (UL, LL) was always the negative of (UR,LR). These things should have the same sign, at least in our case where only the POS DoF is excited, so I tracked the issue back to the alternating (-,+,-,+,-) convention for the gains

(X1:SUS-OPT_CTRL_ULCOIL_GAIN, X1:SUS-OPT_CTRL_URCOIL_GAIN, X1:SUS-OPT_CTRL_LRCOIL_GAIN, X1:SUS-OPT_CTRL_LLCOIL_GAIN, X1:SUS-OPT_CTRL_SDCOIL_GAIN)

of the Coil Output filters used in the real system, which we adopted in the hopes that all was well. Anyways, I changed them all back to +1. This also means that we need to change the sign of the gain for the SIDE filter, which I have done also (and check that it damps OK). I decided to reduce the magnitude of the SIDE damping from 1 to 0.1 so that we can see the residuals since the value of -1 quickly sends the error to zero.  I also increased the gain magnitude for the other DoF to 4.

When looking at the plot remember that the values actually represent counts with a scaling of 2^15 (or 32768) from the ADC. I switched back to the original filters on FM1 (e.g. pit_pit ) without damping coefficients present in the FM2 filter (e.g. pit_pit_damp).

FYI, Rana used the ETMY suspension MEDM screen to illustrate the working of the single suspension to me and changed maybe POS and PITCH gains while doing so.

Also, the Medify purifier 'replace filter' indicator issue occurred bcos the moonlight button should have been pressed for 3 seconds to reset the 'replace filter' indicator after filter replacement.

I was looking into plotting temperature sensor data trend and why we currently do not have frame data written to file (on /frames) since Friday, and noticed that the FE models were not running. So I spoke to Anchal about it and he mentioned that we are currently unable to ssh into the FE machines, therefore we have been unable to start the models. I recalled the last time we enountered this problem Koji resolved it on Chiara, so I search the elog for Koji's fix and found it here, https://nodus.ligo.caltech.edu:8081/40m/16310. I followed the procedure and restarted c1sus and c1lsc machine and we are now able to ssh into these machines. Also restarted the remaining FE machines and confirm that can ssh into them. Then to start models, I ssh into each FE machine (c1lsc, c1sus, c1ioo, c1iscex, c1iscey, c1sus2) and ran the command

rtcds start --all

to start all models on the FE machine. This procedure worked for all the FE machines but failed for c1lsc. For some reason after starting the first the IOP model - c1x04, c1lsc and c1ass, the ssh connection to the machine drops. When we try to ssh into c1lsc after this event, we get the following error :  "ssh: connect to host c1lsc port 22: No route to host".  I reset the c1lsc machine and deecided to to start the IOP model for now. I'll wait for Anchal or Paco to resolve this issue.

[Anchal, Tega]

I informed Anchal of the problem and ask if he could take a look. It turn out 9 FE models across 3 FE machines (c1lsc, c1sus, c1ioo) have a certain interdependece that requires careful consideration when starting the FE model. In a nutshell, we need to first start the IOP models in all three FE machines before we start the other models in these machines. So we turned off all the models and shutdown the FE machines mainly bcos of a daq issue, since the DC (data concentrator) indicator was not initialised. Anchal looked around in fb1 to figure out why this was happening and eventually discovered that it was the same as the ms_stream issue encountered earlier in fb1 clone (https://nodus.ligo.caltech.edu:8081/40m/16372). So we restarted fb1 to see if things clear up given that chiara dhcp sever is now working fine. Upon restart of fb1, we use the info in a previous elog that shows if the DAQ network is working or not, r.e. we ran the command

 The output shows that MX device was not initialiesd during the reboot as can also be seen below.

NOTE: We commented out the line

Restart=always

in the file "/etc/systemd/system/daqd_dc.service" in order to see the error, BUT MUST UNDO THIS AFTER THE PROBLEM IS FIXED!

I will make a detailed elog later today giving a detailed outline of the connection from the Agilent gauge controller to the vacuum subnet and the work I have been doing over the past two days to get data from the unit to EPICs channels. I just want to mention that I have plugged the XGS-600 gauge controller into the serial server on the vacuum subnet. I check the vacuum medm screen and I can confirm that the other sensors did not experience and issues are a result of this. I also currently have two of the FRG-700 connected to the controller but I have powered the unit down after the checks.

I have attached a flow diagram of my understanding of how the gauges are connected to the network.

Earlier today, I connected the XGS-600 gauge controller to the IOLAN Serial Device Server on port 192.168.114.22 .

The plan is a follows:

1. Update the serial device yaml file to include this new ip entry for the XGS-600 gauge controller

2. Create a serial gauge class "serial_gauge_xgs.py" for the XGS-600 gauge controller that inherits from the serial gauge parent class for EPICS communication with a serial device via TCP sockets.

• Might be better to use the current channels of the devices that are being replaced initially, i.e.

• C1:Vac-FRG1_pressure	C1:Vac-CC1_pressure
  C1:Vac-FRG2_pressure	C1:Vac-CCMC_pressure
  C1:Vac-FRG3_pressure	C1:Vac-PTP1_pressure
  C1:Vac-FRG4_pressure	C1:Vac-CC4_pressure
  C1:Vac-FRG5_pressure	C1:Vac-IG1_pressure

3. Modify the launcher file to include the XGS gauge controller. Following the same pattern used  to start the service for the other serial gauges, we can start the communication between the XGS-600 gauge controller and the IOLAN serial server and write data to EPICS channels using

If we are able to establish communication between the XGS-600 gauge controller and write it gause data to EPICS channels, go on to steps 4.

4. Create a serial service file "serial_XGS600.service" and place it in the service folder

5. Add the new EPICS channels to the database file

6. Add the "serial_XGS600.service" to line 10 and 11 of modbusIOC.service

7. Later on, when we are ready, we can restart the updated modbusIOC service

For vacuum signal flow and Acromag channel assignments see [1]  and [2] respectively. For the 16 port IOLAN SDS (Serial Device Server) ethernet connections, see [3]. 

[1] https://wiki-40m.ligo.caltech.edu/Vacuum-Upgrade-2018?action=AttachFile&do=view&target=40m_Vacuum_System_Signal_Flow.pdf

[2] https://wiki-40m.ligo.caltech.edu/Vacuum-Upgrade-2018?action=AttachFile&do=view&target=AcromagChannelAssignment.pdf

[3] https://git.ligo.org/40m/vac/-/blob/master/python/serial/serial_devices.yaml