40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 168 of 339  Not logged in ELOG logo
ID Date Authorup Type Category Subject
  5244   Tue Aug 16 04:25:34 2011 Suresh, KiwamuUpdateSUSalignment of MC output to Y-arm using PZTs

We did several things today+night.  The final goal was to lock the PRC so that we could obtain the POX, POY and POP beams.  However there were large number of steps to get there.

1) We moved the ITMY into its place and balanced the table

2) We then aligned the Y-arm cavity to the green beam which was set up as a reference before we moved the ETMY and ITMY to adjust the OSEMS.  We had the green flashing in Y-arm

3) We checked the beam position on PR2. It was okay. This confirmed that we were ready to send the beam onto the Y arm.

4) We then roughly aligned the IR beam on ETMY where Jamie had placed an Al foil with a hole.  We got the arm flashing in both IR and green. 

5) We used the PZTs to make the green and IR beams co-incident and flashing in the Y arm.  This completed the alignment of the IR beam into the Y-arm.

6) The IPPO (pick-off) window had to be repositioned to avoid clipping.  The IPANG beam was aligned such that it exits the ETMY chamber onto the ETMY table.  It can now be easily sent to the IPANG QPD.

7) Then BS was aligned to direct the IR beam into the X-arm and had the X-arm flashing.  It had already been aligned to its green.

8) It was now the turn of the SRC.  The beam spots on all the SRC related optics were off centered.  We aligned all the optics in the AS path to get the AS beam on to the AP table.

9) The AS beam was very faint so we repositioned the AS camera to the place intended for AS11 PD, since there was a brighter beam available there. 

10) We could then obtain reflections from ITMY, ITMX and PRM at the AS camera. 

11) Problems:

      a) ITMY osems need to be readjusted to make sure that they are in mid-range.  Several are out of range and so the damping is not effective.

      b) When we tried to align SRC the yaw OSEM had to be pushed to its full range.  We therefore have to turn the SRM tower to get it back into range.

 12)  We stopped here since moving the SRM is not something to be attempted at the end of a rather long day. Kiwamu is posting a plan for the rest of the day.

  16624   Tue Jan 25 18:37:12 2022 TYehonathanUpdateBHDPR2 Suspension

PR2's side magnet height was adjusted and its roll was balanced (attachment 1,2). I verified that the OpLev beam is still aligned. The pitch was balanced: First, using an iris for rough adjustment. Then, with the QPD. I locked the counterweight setscrew.

I turned off the HEPAs, damped PR2, and measured the QPD spectra (attachment 3). Major peaks are at 690mHz, 953mHz, and 1.05Hz. I screwed back the lower OSEM plate. The wires were clamped to the suspension block and were cut. Winch adapter plate removed. I wanted to push OSEMs into the OSEM plates but the wiki is down so I can't tell what was the plan. This will have to wait for tomorrow. Also here like with AS1 we need to apply glue to the counterweights.

Attachment 1: PR2_magnet_height.png
PR2_magnet_height.png
Attachment 2: PR2_roll_balance.png
PR2_roll_balance.png
Attachment 3: FreeSwingingSpectra_div_50mV.pdf
FreeSwingingSpectra_div_50mV.pdf
  7911   Thu Jan 17 01:27:54 2013 Tara(?)SummaryIOONoise budget for MC

Quote:

I missed the point.
Do you mean that we can measure the coating thermal noise of the ref cav at the 40m as the IMC is quiet enough?

 Yes, it should be. However, what I did was calculating thermal noise of MC. I'm not sure about the 40m IMC's actual noise level. The plot in the entry was taken from LLO's MC in 2003.

  16314   Fri Sep 3 02:03:15 2021 TegaSummaryComputersStrip down large error files

Also deleted the ~50GB error files from ldas to prevent rsync from copying them to nodus again. With the new update to GWsumm, there are new error messages that initially didn't seem to affect the summary pages functionality, but in the extreme case can populated the error files the repeated warnings on the form "Loading: FrSerData", "Loading: FrSerData::n4294967295", "Loading: FrSummary","Loading: FrSerDataLoading: FrSerData" and many more combinations until we get file sizes of the order of ~50GB. So I have updated the checkstatus script to parse the error files and strip out the majority of these error messages. Work is ongoing to get them all.

In light of these large files generation, I decided to look in the summary pages folder to see if there are other large files that we need to keep track of and it turns there are indeed a collection of files in the archive folder that bloats the summary pages on ldas to ~1TB. Luckily these are not synced to nodus so no problem here. However, since the beginning of the year, the archive folders that hold data used for each day's computation have not been cleared. We have a script for doing this but it has not been run for a while now and it only delete archive files for a specific month which is hardcoded to two months from the date the file is run. I have modified the code to allow archive deletion for a range of months so we can clear data from Jan to July. 

Quote:

[tega, paco]

We found the files that took excess space in the chiara filesystem (see Attachment 1). They were error files from the summary pages that were ~ 50 GB in size or so located under /home/cds/caltech/users/public_html/detcharsummary/logs/. We manually removed them and then copied the rest of the summary page contents into the main file system drive (this is to preserve the information backup before it gets deleted by the cron job at the end of today) and checked carefully to identify the actual issue for why these files were as large in the first place.

We then copied the /detcharsummary directory from /media/40mBackup into /home/cds to match the two disks.

 

  16315   Tue Sep 7 18:00:54 2021 TegaSummaryCalibrationSystem Identification via line injection

[paco]

This morning, I spent some time restoring the jupyter notebook server running in allegra. This server was first set up by Anchal to be able to use the latest nds python API tools which is handy for the calibration stuff. The process to restore the environment was to run "source ~/bashrc.d/*" to restore some of the aliases, variables, paths, etc... that made the nds server work. I then ran ssh -N -f -L localhost:8888:localhost:8888 controls@allegra from pianosa and carry on with the experiment.


[paco, hang, tega]

We started a notebook under /users/paco/20210906_XARM_Cal/XARM_Cal.ipynb on which the first part was doing the following;

  • Set up list of excitations for C1:LSC-XARM_EXC (for example three sine waveforms) using awg.py
  • Make sure the arm is locked
  • Read a reference time trace of the C1:LSC-XARM_IN2 channel for some duration
  • Start excitations (one by one at the moment, ramptime ~ 3 seconds, same duration as above)
  • Get data for C1:LSC-XARM_IN2 for an equal duration (raw data in Attachment #1)
  • Generate the excitation sine and cosine waveforms using numpy and demodulate the raw timeseries using a 4th order lowpass filter with fc ~ 10 Hz
  • Estimate the correct demod phase by computing arctan(Q / I) and rerunning the demodulation to dump the information into the I quadrature (Attachment #2).
  • Plot the estimated ASD of all the quadratures (Attachment #3)

[paco, hang, tega]

Estimation of open loop gain:

  • Grab data from the C1:LSC-XARM_IN1 and C1:LSC-XARM_IN2 test points
  • Infer excitation from their differnce, i.e. C1:LSC-XARM_EXC = C1:LSC-XARM_IN2 - C1:LSC-XARM_IN1
  • Compute the open loop gain as follows : G(f) = csd(EXC,IN1)/csd(EXC,IN2), where csd computes the cross spectra density of the input arguments
  • For the uncertainty in G, dG, we repeat steps (1) to (3) with & without signal injection in the C1:LSC-XARM_EXC channel. In the absence of signal injection, the signal in C1:LSC-XARM_IN2 is of the form: Y_ref = Noise/(1-G), whereas with nonzero signal injection, the signal in C1:LSC-XARM_IN2 has the form: Y_cal = EXC/(1-G) + Noise/(1-G), so their ratio, Y_cal/Y_ref = EXC/Noise, gives the SNR, which we can then invert to give the uncertainty in our estimation of G, i.e dG = Y_ref/Y_cal.
  • For the excitation at 53 Hz, our measurtement for the open loop gain comes out to about 5 dB whiich is consistent with previous measurement.
  • We seem to have an SNR in excess of 100 at measurement time of 35 seconds and 1 count of amplitude which gives a relative uncertainty of G of 0.1%
  • The analysis details are ongoing. Feedback is welcome.
Attachment 1: raw_timeseries.pdf
raw_timeseries.pdf
Attachment 2: demod_signals.pdf
demod_signals.pdf
Attachment 3: cal_noise_asd.pdf
cal_noise_asd.pdf
  16319   Mon Sep 13 04:12:01 2021 TegaUpdateGeneralAdded temperature sensors at Yend and Vertex too

I finally got the modbus part working on chiara, so we can now view the temperature data on any machine on the martian network, see Attachment 1. 

I also updated the entries on /opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini, as suggested by Koji, to include the SensorGatway temperature channels, but I still don't see their EPICs channels on https://ldvw.ligo.caltech.edu/ldvw/view. This means the channels are not available via nds so I think the temperature data is not being to be written to frame files on framebuilder but I am not sure what this entails, since I assumed C0EDCU.ini is the framebuilder daq channel list.

When the EPICs channels are available via nds, we should be able to display the temperature data on the summary pages.

Quote:

I've added the other two temperature sensor modules on Y end (on 1Y4, IP: 192.168.113.241) and in the vertex on (1X2, IP: 192.168.113.242). I've updated the martian host table accordingly. From inside martian network, one can go to the browser and go to the IP address to see the temperature sensor status . These sensors can be set to trigger alarm and send emails/sms etc if temperature goes out of a defined range.

I feel something is off though. The vertex sensor shows temperature of ~28 degrees C, Xend says 20 degrees C and Yend says 26 degrees C. I believe these sensors might need calibration.

Remaining tasks are following:

  • Modbus TCP solution:
    • If we get it right, this will be easiest solution.
    • We just need to add these sensors as streaming devices in some slow EPICS machine in there .cmd file and add the temperature sensing channels in a corresponding database file.
  • Python workaround:
    • Might be faster but dirty.
    • We run a python script on megatron which requests temperature values every second or so from the IP addresses and write them on a soft EPICs channel.
    • We still would need to create a soft EPICs channel fro this and add it to framebuilder data acquisition list.
    • Even shorted workaround for near future could be to just write temperature every 30 min to a log file in some location.

[anchal, paco]

We made a script under scripts/PEM/temp_logger.py and ran it on megatron. The script uses the requests package to query the latest sensor data from the three sensors every 10 minutes as a json file and outputs accordingly. This is not a permanent solution.

 

Attachment 1: Screen_Shot_2021-09-13_at_4.16.22_AM.png
Screen_Shot_2021-09-13_at_4.16.22_AM.png
  16323   Mon Sep 13 17:05:04 2021 TegaSummaryPEMInfrasensing temperature sensor modbus configuration

Anchal mentioned it would be good to put more details about how I arrived at the values needed to configure the modbus drive for the temperature sensor, since this information is not in the manual and is hard to find on the internet, so here is a breakdown.

So the generic format is:

drvAsynIPPortConfigure("<TCP_PORT_NAME>","<UNIT_IP_ADDRESS>:502",0,0,1)
modbusInterposeConfig("<TCP_PORT_NAME>",0,5000,0)
drvModbusAsynConfigure("<PORT_NAME>","<TCP_PORT_NAME>",<slaveAddress>,<modbusFunction>,<modbusStartAddress>,<modbusLength>,<dataType>,<pollMsec>,<plcType>)

which in our case become:

drvAsynIPPortConfigure("c1pemxendtemp","192.168.113.240:502",0,0,1)
modbusInterposeConfig("c1pemxendtemp",0,5000,0)
drvModbusAsynConfigure("C1PEMXENDTEMP","c1pemxendtemp",0,4,199,2,0,1000,"ServerCheck")

As can be seen, the parameters of the first two functions "drvAsynIPPortConfigure" and "modbusInterposeConfig" are straight forward, so we restrict our discussion to the case of third function "drvModbusAsynConfigure". Well, after hours of trolling the internet, I was able to piece together a coherent picture of what needs doing and I have summarised them in the table below.

 

drvModbusAsynConfigure

Once the asyn IP or serial port driver has been created, and the modbusInterpose driver has been configured, a modbus port driver is created with the following command:

drvModbusAsynConfigure(portName,                # used by channel definitions in .db file to reference this unit)
                       tcpPortName,             # reference to portName created with drvAsynIPPortConfigure command
                       slaveAddress,            # 
                       modbusFunction,          # 
                       modbusStartAddress,      # 
                       modbusLength,            # length in dataType units
                       dataType,                # 
                       pollMsec,                # how frequently to request a value in [ms]
                       plcType);                #

drvModbusAsynConfigure command
Parameter Data type Description
portName string Name of the modbus port to be created.
 
tcpPortName string Name of the asyn IP or serial port previously created.

tcpPortName = { 192.168.113.240:502192.168.113.241:502192.168.113.242:502 }
 
slaveAddress int The address of the Modbus slave. This must match the configuration of the Modbus slave (PLC) for RTU and ASCII. For TCP the slave address is used for the "unit identifier", the last field in the MBAP header. The "unit identifier" is ignored by most PLCs, but may be required by some.

ServersCheck API ignores this value, as confirmed with pymodbus query, so set to default value: 
slaveAddress = 0
 
modbusFunction int

modbus supports the following 8 Modbus function codes:

Modbus Function Codes
Access Function description Function code
Bit access Read Coils 1
Bit access Read Discrete Inputs 2
Bit access Write Single Coil 5
Bit access Write Multiple Coils 15
16-bit word access Read Input Registers 4
16-bit word access Read Holding Registers 3
16-bit word access Write Single Register 6
16-bit word access Write Multiple Registers 16
modbusStartAddress int Start address for the Modbus data segment to be accessed.
(0-65535 decimal, 0-0177777 octal).

Modbus addresses are specified by a 16-bit integer address. The location of inputs and outputs within the 16-bit address space is not defined by the Modbus protocol, it is vendor-specific. Note that 16-bit Modbus addresses are commonly specified with an offset of 400001 (or 300001). This offset is not used by the modbus driver, it uses only the 16-bit address, not the offset.

For ServersCheck, the offset is "30001", so that

modbusStartAddress = 30200 - 30001 = 199

modbusLength int The length of the Modbus data segment to be accessed.
This is specified in bits for Modbus functions 1, 2, 5 and 15.
It is specified in 16-bit words for Modbus functions 3, 4, 6 and 16.
Length limit is 2000 for functions 1 and 2, 1968 for functions 5 and 15,
125 for functions 3 and 4, and 123 for functions 6 and 16.

ServersCheck uses two's complement 32-bits word (with big-endian byte order & little-endian word order) format to store floating-point data, as confirmed with pymodbus query, so that:

modbusLength = 2
 
modbusDataType int The modbusDataType is used to tell the driver the format of the Modbus data. The driver uses this information to convert the number between EPICS and Modbus. Data is transferred to and from EPICS as epicsUInt32, epicsInt32, and epicsFloat64 numbers.

Modbus data type:
0 = binary, twos-complement format
1 = binary, sign and magnitude format
2 = BCD, unsigned
3 = BCD, signed

Some Modbus devices (including ServersCheck) use floating point numbers, typically by storing a 32-bit float in two consecutive 16-bit registers. This is not supported by the Modbus specification, which only supports 16-bit registers and single-bit data. The modbus driver does not directly support reading such values, because the word order and floating point format is not specified.

Note that if it is desired to transmit BCD numbers untranslated to EPICS over the asynInt32 interface, then data type 0 should be used, because no translation is done in this case. 

For ServersCheck, we wish to transmit the untranslated data, so:

modbusDataType = 0
 
pollMsec int Polling delay time in msec for the polling thread for read functions.
For write functions, a non-zero value means that the Modbus data should
be read once when the port driver is first created.

ServersCheck recommends setting sensor polling interval between 1-5 seconds, so we can try:

pollMsec = 1000
 
plcType string Type of PLC (e.g. Koyo, Modicon, etc.).
This parameter is currently used only to print information in asynReport.
In the future it could be used to modify the driver behavior for a specific PLC.

plcType = "ServersCheck"
 

 

Useful links

https://nodus.ligo.caltech.edu:8081/40m/16214

https://nodus.ligo.caltech.edu:8081/40m/16269

https://nodus.ligo.caltech.edu:8081/40m/16270

https://nodus.ligo.caltech.edu:8081/40m/16274

 

http://manuals.serverscheck.com/InfraSensing_Sensors_Platform.pdf

http://manuals.serverscheck.com/InfraSensing_Modbus_manualv5.pdf

https://community.serverscheck.com/discussion/comment/7419#Comment_7419

 

https://wiki-40m.ligo.caltech.edu/CDS/SlowControls

https://www.slac.stanford.edu/grp/ssrl/spear/epics/site/modbus/modbusDoc.html#Creating_a_modbus_port_driver

 

https://github.com/riptideio/pymodbus

 

https://en.wikipedia.org/wiki/Modbus

https://deltamotion.com/support/webhelp/rmctools/Communications/Ethernet/Supported_Protocols/Ethernet_Modbus_TCP.htm

  16324   Mon Sep 13 18:19:25 2021 TegaUpdateComputer Scripts / ProgramsMoved modbus service from chiara to c1susaux

[Tega, Anchal, Paco]

After talking to Anchal, it was made clear that chiara is not the place to host the modbus service for the temperature sensors. The obvious machine is c1pem, but the startup cmd script loads c object files and it is not clear how easy it would integrate the modbus functionality since we can only login via telnet, so we decided to instead host the service on c1susaux. We also modified the /etc/motd file on c1susaucx which displays the welcome message during login to inform the user that this machine hosts the modbus service for the temperature sensor. Anchal plans to also document this information on the temperature sensor wiki at some point in the future when the page is updated to include what has been learnt so far.

We might also consider updating the database file to a more modern way of reading the temperature sensor data using FLOAT32_LE which is available on EPICs version 3.14 and above, instead of the current method which works but leaves the reader bemused by the bitwise operations that convert the two 16 bits words (A and B) to IEEE-754 32-bit float, via 

field(CALC, "(A&D?(A&C?-1:1):0)*((G|A&E)*J+B)*2^((A&D)/G-F)")

where 

   field(INPA, "$HiWord")
   field(INPB, "$LoWord")
   field(INPC, "0x8000")   # Hi word, sign bit
   field(INPD, "0x7F80")   # Hi word, exponent mask
   field(INPE, "0x00FF")   # Hi word, mantissa mask (incl hidden bit)
   field(INPF, "150")      # Exponent offset plus 23-bit mantissa shift
   field(INPG, "0x0080")   # Mantissa hidden bit
   field(INPJ, "65536")    # Hi/Lo mantissa ratio
   field(CALC, "(A&D?(A&C?-1:1):0)*((G|A&E)*J+B)*2^((A&D)/G-F)")
   field(PREC, "4")

as opposed to the more modern form

field(INP,"@asyn($(PORT) $(OFFSET))FLOAT32_LE")
  16338   Thu Sep 16 12:06:17 2021 TegaUpdateComputer Scripts / ProgramsTemperature sensors added to the summary pages

We can now view the minute trend of the temperature sensors under the PEM tab of the summary pages. See attachment 1 for an example of today's temperature readings. 

Attachment 1: TempPlot_2021-09-16_12.04.19PM.png
TempPlot_2021-09-16_12.04.19PM.png
  16349   Mon Sep 20 20:43:38 2021 TegaUpdateElectronicsSat Amp modifications

Running update of Sat Amp modification work, which involves the following procedure (x8) per unit:

  1. Replace R20 & R24 with 4.99K ohms, R23 with 499 ohms, and remove C16.
  2. (Testing) Connect LEDDrive output to GND and check that
    • TP4 is ~ 5V
    •  TP5-8 ~ 0V. 
  3. Install 40m Satellite to Flange Adapter (D2100148-v1)

 

Unit Serial Number Issues Status
S1200740 NONE DONE
S1200742 NONE DONE
S1200743 NONE DONE
S1200744

TP4 @ LED1,2 on PCB S2100568 is 13V instead of 5V

TP4 @ LED4 on PCB S2100559 is 13V instead of 5V

DONE
S1200752 NONE DONE

 

 

 

Attachment 1: IMG_20210920_203456226.jpg
IMG_20210920_203456226.jpg
  16356   Wed Sep 22 17:22:59 2021 TegaUpdateElectronicsSat Amp modifications

[Koji, Tega]

 

Decided to do a quick check of the remaining Sat Amp units before component replacement to identify any unit with defective LED circuits. Managed to examine 5 out of 10 units, so still have 5 units remaining. Also installed the photodiode bias voltage jumper (JP1) on all the units processed so far.

Unit Serial Number Issues Debugging Status
S1200738

TP4 @ LED3 on chan 1-4 PCB was ~0.7 V instead of 5V

Koji checked the solder connections of the various components, then swapped out the IC OPAMP. Removed DB9 connections to the front panel to get access to the bottom of the board. Upon close inspection, it looked like an issue of a short connection between the Emitter & Base legs of the Q1 transistor.

Solution - Remove the short connection between the Emitter & Base legs of the Q1 transistor legs.

DONE
S1200748 TP4 @ LED2 on chan 1-4 PCB was ~0.7 V instead of 5V

This issue was caused by a short connection between the Emitter & Base legs of the Q1 transistor.

Solution - Remove the short connection between the Emitter & Base legs of the Q1 transistor legs.

DONE
S1200749 NONE N/A DONE
S1200750 NONE N/A DONE
S1200751 NONE N/A DONE

 

Defective unit with updated resistors and capacitors in the previous elog

Unit Serial Number Issues Debugging Status
S1200744

TP4 @ LED1,2 on PCB S2100568 is 13V instead of 5V

TP4 @ LED4 on PCB S2100559 is 13V instead of 5V

This issue was caused by a short between the Collector & Base legs of the Q1 transistor.

Solution - Remove the short connection between the Collector & Base legs of the Q1 transistor legs

 

Complications - During the process of flipping the board to get access to the bottom of the board, a connector holding the two middle black wires, on P1, came loose. I resecured the wires to the connector and checked all TP4s on the board afterwards to make sure things are as expected.

DONE

 

 

 

Quote:

Running update of Sat Amp modification work, which involves the following procedure (x8) per unit:

  1. Replace R20 & R24 with 4.99K ohms, R23 with 499 ohms, and remove C16.
  2. (Testing) Connect LEDDrive output to GND and check that
    • TP4 is ~ 5V
    •  TP5-8 ~ 0V. 
  3. Install 40m Satellite to Flange Adapter (D2100148-v1)

 

Unit Serial Number Issues Status
S1200740 NONE DONE
S1200742 NONE DONE
S1200743 NONE DONE
S1200744

TP4 @ LED1,2 on PCB S2100568 is 13V instead of 5V

TP4 @ LED4 on PCB S2100559 is 13V instead of 5V

DONE
S1200752 NONE DONE

 

 

 

 

  16357   Thu Sep 23 14:17:44 2021 TegaUpdateElectronicsSat Amp modifications debugging update

Debugging complete.

All units now have the correct TP4 voltage reading needed to drive a nominal current of 35 mA through to OSEM LED. The next step is to go ahead and replace the components and test afterward that everything is OK.

 

Unit Serial Number Issues Debugging Status
S1200736 TP4 @ LED4 on chan 1-4 PCB reads 13V instead of 5V

This issue was caused by a short between the Collector & Base legs of the Q1 transistor.

Solution - Remove the short connection between the Collector & Base legs of the Q1 transistor legs

DONE
S1200737 NONE N/A DONE
S1200739 NONE N/A DONE
S1200746 TP4 @ LED3 on chan 5-8 PCB reads 0.765 V instead of 5V

This issue was caused by a short between the Emitter & Base legs of the Q1 transistor.

Solution - Remove the short connection between the Emitter & Base legs of the Q1 transistor legs

 

Complications - I was extra careful this time because of the problem of loose cable from the last flip-over of the right PCB containing chan 5-8. Anyways, after I was done I noticed one of the pink wires (it carries the +14V to the left PCB) had come off on P1. At least this time I could also see that the corresponding front panel green LED turn off as a result. So I resecured the wire to the connector (using solder as my last attempt yesterday to reattach the via crimping didn't work after a long time trying. I hope this is not a problem.) and checked the front panel LED turns on when the unit is powered before closing the unit. These connectors are quite flimsy.

DONE
S1200747 TP4 @ LED2 on chan 1-4 PCB reads 13V instead of 5V

This issue was caused by a short between the Collector & Base legs of the Q1 transistor.

Solution - Remove the short connection between the Collector & Base legs of the Q1 transistor legs

DONE

 

 

 

  16379   Mon Oct 4 21:58:17 2021 TegaUpdateElectronicsSat Amp modifications

Trying to finish 2 more Sat Amp units so that we have the 7 units needed for the X-arm install. 

S2100736 - All good

S2100737 - This unit presented with an issue on the PD1 circuit of channel 1-4 PCB where the voltage reading on TP6, TP7 and TP8 are -15.1V,  -14.2V, and +14.7V respectively, instead of ~0V.  The unit also has an issue on the PD2 circuit of channel 1-4 PCB because the voltage reading on TP7 and TP8 are  -14.2V, and +14.25V respectively, instead of ~0V.

 

  16386   Wed Oct 6 16:31:02 2021 TegaUpdateElectronicsSat Amp modifications

[Tega, Koji]

(S2100737) - Debugging showed that the opamp, AD822ARZ, for PD2 circuit was not working as expected so we replaced with a spare and this fixed the problem. Somehow, the PD1 circuit no longer presents any issues, so everything is now fine with the unit.

(S2100741) - All good.

Quote:

Trying to finish 2 more Sat Amp units so that we have the 7 units needed for the X-arm install. 

S2100736 - All good

S2100737 - This unit presented with an issue on the PD1 circuit of channel 1-4 PCB where the voltage reading on TP6, TP7 and TP8 are -15.1V,  -14.2V, and +14.7V respectively, instead of ~0V.  The unit also has an issue on the PD2 circuit of channel 1-4 PCB because the voltage reading on TP7 and TP8 are  -14.2V, and +14.25V respectively, instead of ~0V.

 

 

  16411   Mon Oct 18 16:48:32 2021 TegaUpdateElectronicsSat Amp modifications

[S2100738, S2100745, S2100751] Completed three more Sat Amp units modification with seven remaining.

 

Attachment 1: IMG_20211018_162918574.jpg
IMG_20211018_162918574.jpg
  16427   Tue Oct 26 13:27:07 2021 TegaSummaryElectronicsSat Amp modification Summary

Modifications and testing of SatAmp units COMPLETE. Attachments 1 & 2 show all 19 units, one installed unit and the remaining 18 units are stacked and ready for install. Detailed notes of the modification for each unit are presented in the summary document in the dcc.

 

 

Attachment 1: SapAmpModStack.jpg
SapAmpModStack.jpg
Attachment 2: SatAmpInstalled.jpg
SatAmpInstalled.jpg
  16449   Thu Nov 4 18:29:51 2021 TegaUpdateSUSSetting up suspension test model

[Ian,Tega]

Today we continued working on setting up the 6 degrees of freedom model for testing the suspension which we copied over from  "/cvs/cds/rtcds/userapps/release/sus/c1/models/c1sup.mdl" to c1sp2.mdl in the same folder. We then changed the host from c1lsc to c1sus2, changed cpu # from 7 to 3 bcos c1sus2 has 6 cores. Then ran the following commands to build and install the model on c1sus2:

$ ssh c1sus2

$ rtcds make c1sp2

$ rtcds install c1sp2

where we encountered the following installation error:

ERROR: This node 62 is already installed as:

hostname=c1lsc

system=c1sup

The new entry you are trying to write is as follows:

hostname=c1sus2

system=c1sp2

This script will not overwrite existing entries in testpoint.par

If this is an attempt to move an existing system from one host to another,

please remove conflicting entry from testpoint.par file

It seems that changing the model name and host did not change the node allocation, so will remove the previous entries in testpoint.par to see if that helps. After deleting the following lines

[C-node62]
hostname=c1lsc
system=c1sup

from the file "/opt/rtcds/caltech/c1/target/gds/param/testpoint.par", the installation went fine and the above entries were replaced by 

[C-node62]
hostname=c1sus2
system=c1sp2

BTW, I now believe the reason we had the node conflict earlier was bcos both models still had the same value of  dcuid=62, so I think changing this value in our model file would be a better solution. Work is ongoing.

 

  16478   Mon Nov 22 16:38:26 2021 TegaSummaryComputer Scripts / ProgramsSUS Plant Plan for New Optics

[Tega, Ian]

TODO

1. Investigate cross-coupling btw the various degrees of freedom (dof) - turn on noise for each dof in the plant model and measure the transfer function of the other dofs.

2. Get a closed-loop transfer function using noise injection and give a detailed outline of the procedure in elog - IN1/IN2 for each TM_RESP filter while the others are turned off.

3. Derive analytic model of the closed-loop transfer functions for comparison.

4. Adapt control filters to fit optimized analytical solutions.

  16495   Thu Dec 9 00:32:56 2021 TegaUpdateCDSNew SUS medm screen update

The new SUS screen can be reached via sitemap -> IFO SUS button -> NEW ETMX dropdown menu link. Please use and provide feedback. Not sure exactly if we need/want the display screens after the IOP model on the right of the medm screen. I have not been able to locate the corresponding channels but did not want to remove them until I was sure that we don't plan to add these features to our screens. When all bugs have been ironed out, we can use appropriate macro substitution for the other optics.

The next feature to add is the BLRMS to the coil and PD channels. I plan to combine the PEM BLRMS medm implementation with the sus_single_BLRMS model block (located in  /opt/rtcds/userapps/release/cds/c1/models). This way we use the latest BLRMS block in "/opt/rtcds/userapps/release/cds/common/models/BLRMS.mdl" whilst also leveraging the previous work done on the sus_single_BLRMS model, which neatly fits into our current SUS model.

Attachment 1: Screen_Shot_2021-12-09_at_12.29.30_AM.png
Screen_Shot_2021-12-09_at_12.29.30_AM.png
Attachment 2: Screen_Shot_2021-12-09_at_12.42.35_AM.png
Screen_Shot_2021-12-09_at_12.42.35_AM.png
  16496   Thu Dec 9 18:22:36 2021 TegaUpdateCDSNew SUS medm screen update

Work on the medm screen for SUS RMS monitor is ongoing. The next step would be to incorporate this into the SUS medm screen, add the BLRMS model to the SUS controller model, recompile, check that the channels are being correctly addressed, then load the appropriate bandpass and lowpass filters.  

Attachment 1: Screen_Shot_2021-12-09_at_6.21.09_PM.png
Screen_Shot_2021-12-09_at_6.21.09_PM.png
  16500   Fri Dec 10 18:55:58 2021 TegaUpdateCDSNew SUS medm screen update

Turns out the BLRMS monitoring channels for MC1, MC2, MC3, ITMY and SRM already exist in c1pem. So I modified the new SUS screen to display the BLRMS info for the aforementioned optics. Next step is to add the BLRMS monitor for PRM, ITMX, ETMX and ETMY. This would require extending the number of inputs for the "SUS" block in c1pem to accomodate the additional inputs from the remaining optics.

Attachment 1: BLRMS_ITMY_screenshot.png
BLRMS_ITMY_screenshot.png
  16503   Mon Dec 13 15:05:47 2021 TegaUpdatePEMgit repo for temp sensor and sus medm

[temperature sensor]

git repo: https://git.ligo.org/40m/tempsensor.git

todo

Update the temp sensor channels to fit with cds format, ie. "C1:PEM-TEMP_EX", "C1:PEM-TEMP_EY", "C1:PEM-TEMP_BS"

- Use FLOAT32_LE data format for the database file (/cvs/cds/caltech/target/c1pem1/tempsensor/C1PEMaux.db) to create the new channels.

- Keep the old datadase code and channels so we can compare with new temp channels afterwards. Also we need a 1-month overlap b4 deleting the old channels.

 

[sus medm screen]

git repo: https://git.ligo.org/40m/susmedmscreen.git

todo (from talk with Koji)

- Link stateword display to open "C1CDS_FE_STATUS.adl"

- Damp filter and Lock filter buttons should open a 3x1 filter screen so that the 6 filters are opened by 2 buttons compared to the old screen that has 3 buttons connected to 2X1 filter screen

- Make the LOCKIN signla modulation flow diagramlook more like the old 40m screen since that is a better layout

- Move load coefficient button to top of sus medm screen (beside stateword)

- The rectangular red outline around the oplev display is confusing and needs to be modified for clarity

- COMM tag block should not be 3D as this suggests it is a button. Make it flat and change tag name to indicate individual watchdog control as this better reflect its functionality. Rename current watchdog switch to watchdog master is it does what the 5 COMM switches do at once.

- Macro pass need to be better documented so that when we call the sus screens from locations other than sitemap, we should know what macro variables to pass in, like DCU_ID etc.

- Edit sitemap.adl to point only to the new screens. Then create a button on the new screen that points to the old screen. This way, we can still access the old screen without clogging sitemap.

- Move the new screen location to a subfolder of where the current sus screens reside, /opt/rtcds/userapps/trunk/sus/c1/medm/templates

- Rename the overview screen (SUS_CUST_HSSS_OVERVIEW.adl) to use the SUS_SINGLE nomenclature, i.e. SUS_SINGLE_OVERVIEW.adl

- Keep an eye of the cpu usage of c1pem as we add BLRMS block for other optics. 

 

 

  16504   Tue Dec 14 11:33:29 2021 TegaUpdatePEMgit repo for temp sensor and sus medm

[Temperature sensor]

Added new temp EPICs channels to database file (/cvs/cds/caltech/target/c1pem1/tempsensor/C1PEMaux.db)

Added new temp EPICs channels to slow channels ini file (/opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini)

 

[SUS medm screen]

Moved new SUS screen to location : /opt/rtcds/userapps/trunk/sus/c1/medm/templates/NEW_SUS_SCREENS

Place button on the new screen to link to the old screen and replace old screens link on sitemap.

Fixed Load Coefficient button location issue

Fixed LOCKIN flow diagram issue

Fixed watchdog labelling issue

Linked STATE WORD block to FrontEnd STATUS screen

Replaced the 2x1 pit/yaw filter screens for LOCK and DAMP fliters with 3x1 LPY filter screen

*Need some more time to figure out the OPTLEV red indicator

  16570   Tue Jan 11 10:46:07 2022 TegaUpdateCDSSUS screen debugging

Seen. Thanks.

Red Arrow: The channel was labeled incorrectly as INMON instead of OUTPUT

Green Arrow: OK, I will create a custom medm screen for this.

Blue arrow: Hmm, OK I will look into this. Doing this work remotely is a pain as the medm response is quite slow for poking around.

Orange circle: OK, I'll move this to the left side of the line.

Note to self: I also noticed another error on the side (LPYS blue box just b4 the sum). The channel is pointing to YAW instead of the side, so needs to be fixed as well.

Quote:

Indicated by the red arrow:
Even when the side damping servo is off, the number appears at the input of the output matrix

Indicated by the green arrows:
The face magnets and the side magnets use different ADCs. How about opening a custom ADC panel that accommodates all ADCs at once? Same for the DAC.

Indicated by the blue arrows:
This button opens a custom FM window. When the pitch gain was modified with a ramping time, the pitch and yaw gain grows at the same time even though only the pitch gain was modified.

Indicated by the orange circle:
The numbers are not indicated here, but they are input-related numbers (for watchdogging) rather than output-related numbers. It is confusing to place them here.

 

  16571   Tue Jan 11 10:58:58 2022 TegaUpdateSUSTemporary watchdog

Started working on this. First created a git repo for tracking https://git.ligo.org/40m/susaux.git

I have looked through the folder to see what needs doing and I think I know what needs to be done for the final case by just following the same pattern for the other optics, which I am listing below

- Create database file for the BHD optics, say C1_SUS-AUX_LO1.db by copying another optic database file say C1_SUS-AUX_SRM. Then replace the optic name.

- Insert a new line "C1:SUS-LO1_LATCH_OFF" in the file autoBurt_watchdogs.req

- Populate the file autoBurt.req with the appropriate channels for LO1

- Populate the file C1SUSaux_post.sh with the corresponding commands for LO1

- Add the line dbLoadDatabase("/cvs/cds/caltech/target/c1susaux/C1_SUS-AUX_LO1.db") to the file C1SUSaux.cmd

 

For the temporary watchdog, we comment everything I have just talked about, and do only what come next.

My question is the following:

I understand that we need to use the OUT16 slow channel as a temporary watchdog since we don't currently have access to the slow channels bcos the Acromag units have not been installed. My guess from Koji's instructions is that we need to update the channels in the last two fields "INPA" and "INPB" below

record(calc,"C1:SUS-LO1_UL_CALC")
{
        field(DESC,"ANDs Enable with Watchdog")
        field(SCAN,"1 second")
        field(PHAS,"1")
        field(PREC,"2")
        field(HOPR,"40")
        field(LOPR,"-40")
        field(CALC,"A&B")
        field(INPA,"C1:SUS-LO1_UL_COMM  PP  NMS")
        field(INPB,"C1:SUS-LO1_LATCH_OFF  PP  MS")
}

Suppose we replace the channel for INPA with C1:SUS-LO1_ULCOIL_OUT16, what about INPB. Is this even the right thing to do as I am just guessing here?

 

  16600   Wed Jan 19 21:39:22 2022 TegaUpdateSUSTemporary watchdog

After some work on the reference database file, we now have a template for temporary watchdog implementation for LO1 located here "/cvs/cds/caltech/target/c1susaux/C1_SUS-AUX_LO1.db".

Basically, what I have done is swap the EPICS asyn analog input readout for the COIL and OSEM to accessible medm channels, then write out watchdog enable/disable to coil filter SW2 switch. Everything else in the file remains the same. I am worried about some of the conversions but the only way to know more is to see the output on the medm screen.

To test, I restarted c1su2 but this did not make the LO1 database available, so I am guessing that we also need to restart the c1sus, which can be done tomorrow.

  16606   Thu Jan 20 17:21:21 2022 TegaUpdateSUSTemporary watchdog

Temp software watchdog now operational for LO1 and the remaining optics!

Koji helped me understand how to write to switches and we tried for a while to only turnoff the output switch of the filters instead of the writing a zero that resets everything in the filter.

Eventually, I was able to move this effort foward by realising that I can pass the control trigger along multiple records using the forwarding option 'FLNK'. When I added this field to the trigger block, record(dfanout,"C1:SUS-LO1_PUSH_ALL"), and subsequent calculation blocks, record(calcout,"C1:SUS-LO1_COILSWa") to record(calcout,"C1:SUS-LO1_COILSWd"), everything started working right.

Quote:

After some work on the reference database file, we now have a template for temporary watchdog implementation for LO1 located here "/cvs/cds/caltech/target/c1susaux/C1_SUS-AUX_LO1.db".

Basically, what I have done is swap the EPICS asyn analog input readout for the COIL and OSEM to accessible medm channels, then write out watchdog enable/disable to coil filter SW2 switch. Everything else in the file remains the same. I am worried about some of the conversions but the only way to know more is to see the output on the medm screen.

To test, I restarted c1su2 but this did not make the LO1 database available, so I am guessing that we also need to restart the c1sus, which can be done tomorrow.

 

  16611   Fri Jan 21 12:46:31 2022 TegaUpdateCDSSUS screen debugging

All done (almost)! I still have not sorted the issue of pitch and yaw gains growing together when modified using ramping time. Image of custom ADC and DAC panel is attached.

 

Quote:

Seen. Thanks.

 
Quote:

Indicated by the red arrow:
Even when the side damping servo is off, the number appears at the input of the output matrix

Indicated by the green arrows:
The face magnets and the side magnets use different ADCs. How about opening a custom ADC panel that accommodates all ADCs at once? Same for the DAC.

Indicated by the blue arrows:
This button opens a custom FM window. When the pitch gain was modified with a ramping time, the pitch and yaw gain grows at the same time even though only the pitch gain was modified.

Indicated by the orange circle:
The numbers are not indicated here, but they are input-related numbers (for watchdogging) rather than output-related numbers. It is confusing to place them here.

 

 

Attachment 1: Custom_ADC_DAC_monitors.png
Custom_ADC_DAC_monitors.png
  16615   Mon Jan 24 17:10:25 2022 TegaSummaryComputer Scripts / ProgramsSUS Plant Plan for New Optics

[Ian, Tega]

Connected the New SUS screens to the controller for the simplant model. Because of hard-coded links in the medm screen links, it was necessary to create the following path in the c1sim computer, where the new medm screen files are located:

/opt/rtcds/userapps/trunk/sus/c1/medm/templates/NEW_SUS_SCREENS

 

We noticed a few problems:

1. Some of the medm files still had C1 hard coded, so we need to replace them with $IFO instead, in order for the custom damping filter screen to be useful.

2. The "Load coefficient" button was initially blank on the new sus screen, but we were able to figure out that the problem came from setting the top-level DCU_ID to 63.

medm -x -macro "IFO=X1,OPTIC=OPT_CTRL,DCU_ID=63" SUS_SINGLE_OVERVIEW.adl

 

[TODO]

Get the data showing the controller damping the pendulum. This will involve tweaking some gains and such to fine-tune the settings in the controller medm screen. Then we will be able to post some data of the working controller.

 

[Useful aside]

We should have a single place with all the instructions that are currently spread over multiple elogs so that we can better navigate the simplant computer.

Attachment 1: Screen_Shot_2022-01-24_at_5.33.15_PM.png
Screen_Shot_2022-01-24_at_5.33.15_PM.png
  16626   Thu Jan 27 16:40:57 2022 TegaSummaryComputer Scripts / ProgramsSUS Plant Plan for New Optics

[Ian, Paco, Tega]

Last night we set up the four main matrices that handle the conversion between the degrees of freedom bases and the sensor bases. We also wrote a bash script to automatically set up the system. The script sets the four change of bases matrices and activates the filters that control the plant. this script should fully set up the plant to its most basic form. The script also turns off all of the built-in noise generators. 

After this, we tried damping the optic. The easiest part of the system to damp is the side or y motion of the optic because it is separate from the other degrees of freedom in both of the bases. We were able to damp that easily. in attachment 1 you can see that the last graph in the ndscope screen the side motion of the optic is damped. Today we decided to revisit the problem.

Anyways, looking at the problem with fresh eyes today, I noticed the in pit2pit coupling has the largest swing of all the plant filters and thought this might be the reason why the inputs (UL,UR,LR,LL) to the controller was hitting the rails for pit DoF. I reduce the gain of the pit2pit filter then slowly increased it back to one. I also reduced the gain in the OSEM input filter from 1 to 1/100. The attached image (Attachment2) is the output from this trial. This did not solve the problem. The output when all OSEM input filter gain set to one is shown in Attachment2.

We will try to continue to tweak the coefficients. We are probably going to ask Anchal and Paco to sit down with us and really hone in on the right coefficients. They have more experience and should be able to really get the right values.

Attachment 1: simplant_control_1.png
simplant_control_1.png
Attachment 2: simplant_control_0.png
simplant_control_0.png
  16636   Tue Feb 1 20:16:09 2022 TegaUpdateBHDPR2 candidate mirror analysis

git repo: git@git.ligo.org:tega-edo/charmirrormap.git

The analysis code takes in a set of raw images, 10 in our case,  for each mirror and calculates the zernike aberration coefficients for each image, then takes their average. This average value is used to reconstruct the mirror height map.  Finally, the residual error between the reconstructed image and the raw data is calculated.

We repeat the analysis for different field of views (FoV) namely 10mm, 20mm, 30mm, 40mm and 46.5mm and save the results in the output folder of the repo.

The analysis output for a 10mm FoV aperture at the mirror center is shown in the attachement. These three images show the input data, the reconstructed mirror surface map and the residual error.

Attachment 1: PR2_M2_data.png
PR2_M2_data.png
Attachment 2: PR2_M2_recon_FoV_10mm.png
PR2_M2_recon_FoV_10mm.png
Attachment 3: PR2_M2_residual_FoV_10mm.png
PR2_M2_residual_FoV_10mm.png
  16645   Thu Feb 3 17:15:23 2022 TegaSummaryComputer Scripts / ProgramsSUS Plant Plan for New Optics

Finally got the SIMPLANT damping to work following Rana's suggestion to try damping one DoF at a time, woo-hoo!

At first, things didn't look good even when we only focus on the POS DoF. I then noticed that the input value (X1:SUS-OPT_PLANT_TM_RESP_1_1_IN1) to the plant was always zero. This was odd bcos it meant the control signal was not making its way to the plant. So I decided to look at the sensor data

(X1:SUS-OPT_PLANT_COIL_IN_UL_OUTPUT, X1:SUS-OPT_PLANT_COIL_IN_UR_OUTPUT, X1:SUS-OPT_PLANT_COIL_IN_LR_OUTPUT, X1:SUS-OPT_PLANT_COIL_IN_LL_OUTPUT)

that adds up via the C2DOF matrix to give the POS DoF and I noticed that these interior nodes can take on large values but always sum up to zero because the pair (UL, LL) was always the negative of (UR,LR). These things should have the same sign, at least in our case where only the POS DoF is excited, so I tracked the issue back to the alternating (-,+,-,+,-) convention for the gains

(X1:SUS-OPT_CTRL_ULCOIL_GAIN, X1:SUS-OPT_CTRL_URCOIL_GAIN, X1:SUS-OPT_CTRL_LRCOIL_GAIN, X1:SUS-OPT_CTRL_LLCOIL_GAIN, X1:SUS-OPT_CTRL_SDCOIL_GAIN)

of the Coil Output filters used in the real system, which we adopted in the hopes that all was well. Anyways, I changed them all back to +1. This also means that we need to change the sign of the gain for the SIDE filter, which I have done also (and check that it damps OK). I decided to reduce the magnitude of the SIDE damping from 1 to 0.1 so that we can see the residuals since the value of -1 quickly sends the error to zero.  I also increased the gain magnitude for the other DoF to 4.

When looking at the plot remember that the values actually represent counts with a scaling of 2^15 (or 32768) from the ADC. I switched back to the original filters on FM1 (e.g. pit_pit ) without damping coefficients present in the FM2 filter (e.g. pit_pit_damp).


FYI, Rana used the ETMY suspension MEDM screen to illustrate the working of the single suspension to me and changed maybe POS and PITCH gains while doing so.


Also, the Medify purifier 'replace filter' indicator issue occurred bcos the moonlight button should have been pressed for 3 seconds to reset the 'replace filter' indicator after filter replacement.

Attachment 1: Screen_Shot_2022-02-03_at_8.23.07_PM.png
Screen_Shot_2022-02-03_at_8.23.07_PM.png
  16650   Mon Feb 7 16:14:37 2022 TegaUpdateComputersrealtime system reboot problem

I was looking into plotting temperature sensor data trend and why we currently do not have frame data written to file (on /frames) since Friday, and noticed that the FE models were not running. So I spoke to Anchal about it and he mentioned that we are currently unable to ssh into the FE machines, therefore we have been unable to start the models. I recalled the last time we enountered this problem Koji resolved it on Chiara, so I search the elog for Koji's fix and found it here, https://nodus.ligo.caltech.edu:8081/40m/16310. I followed the procedure and restarted c1sus and c1lsc machine and we are now able to ssh into these machines. Also restarted the remaining FE machines and confirm that can ssh into them. Then to start models, I ssh into each FE machine (c1lsc, c1sus, c1ioo, c1iscex, c1iscey, c1sus2) and ran the command

rtcds start --all

to start all models on the FE machine. This procedure worked for all the FE machines but failed for c1lsc. For some reason after starting the first the IOP model - c1x04, c1lsc and c1ass, the ssh connection to the machine drops. When we try to ssh into c1lsc after this event, we get the following error :  "ssh: connect to host c1lsc port 22: No route to host".  I reset the c1lsc machine and deecided to to start the IOP model for now. I'll wait for Anchal or Paco to resolve this issue.


[Anchal, Tega]

I informed Anchal of the problem and ask if he could take a look. It turn out 9 FE models across 3 FE machines (c1lsc, c1sus, c1ioo) have a certain interdependece that requires careful consideration when starting the FE model. In a nutshell, we need to first start the IOP models in all three FE machines before we start the other models in these machines. So we turned off all the models and shutdown the FE machines mainly bcos of a daq issue, since the DC (data concentrator) indicator was not initialised. Anchal looked around in fb1 to figure out why this was happening and eventually discovered that it was the same as the ms_stream issue encountered earlier in fb1 clone (https://nodus.ligo.caltech.edu:8081/40m/16372). So we restarted fb1 to see if things clear up given that chiara dhcp sever is now working fine. Upon restart of fb1, we use the info in a previous elog that shows if the DAQ network is working or not, r.e. we ran the command

$ /opt/mx/bin/mx_info
MX:fb1:mx_init:querying driver:error 5(errno=2):No MX device entry in /dev.

 The output shows that MX device was not initialiesd during the reboot as can also be seen below.

$ sudo systemctl status daqd_dc -l

● daqd_dc.service - Advanced LIGO RTS daqd data concentrator
   Loaded: loaded (/etc/systemd/system/daqd_dc.service; enabled)
   Active: failed (Result: exit-code) since Mon 2022-02-07 18:02:02 PST; 12min ago
  Process: 606 ExecStart=/usr/bin/daqd_dc_mx -c /opt/rtcds/caltech/c1/target/daqd/daqdrc.dc (code=exited, status=1/FAILURE)
 Main PID: 606 (code=exited, status=1/FAILURE)

Feb 07 18:01:56 fb1 systemd[1]: Starting Advanced LIGO RTS daqd data concentrator...
Feb 07 18:01:56 fb1 systemd[1]: Started Advanced LIGO RTS daqd data concentrator.
Feb 07 18:02:00 fb1 daqd_dc_mx[606]: [Mon Feb  7 18:01:57 2022] Unable to set to nice = -20 -error Unknown error -1
Feb 07 18:02:00 fb1 daqd_dc_mx[606]: Failed to do mx_get_info: MX not initialized.
Feb 07 18:02:00 fb1 daqd_dc_mx[606]: 263596
Feb 07 18:02:02 fb1 systemd[1]: daqd_dc.service: main process exited, code=exited, status=1/FAILURE
Feb 07 18:02:02 fb1 systemd[1]: Unit daqd_dc.service entered failed state.


NOTE: We commented out the line

Restart=always

in the file "/etc/systemd/system/daqd_dc.service" in order to see the error, BUT MUST UNDO THIS AFTER THE PROBLEM IS FIXED!

  16682   Sat Feb 26 01:01:40 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

I will make a detailed elog later today giving a detailed outline of the connection from the Agilent gauge controller to the vacuum subnet and the work I have been doing over the past two days to get data from the unit to EPICs channels. I just want to mention that I have plugged the XGS-600 gauge controller into the serial server on the vacuum subnet. I check the vacuum medm screen and I can confirm that the other sensors did not experience and issues are a result of this. I also currently have two of the FRG-700 connected to the controller but I have powered the unit down after the checks.

  16683   Sat Feb 26 15:45:14 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

I have attached a flow diagram of my understanding of how the gauges are connected to the network.

Earlier today, I connected the XGS-600 gauge controller to the IOLAN Serial Device Server on port 192.168.114.22 .

The plan is a follows:

1. Update the serial device yaml file to include this new ip entry for the XGS-600 gauge controller

2. Create a serial gauge class "serial_gauge_xgs.py" for the XGS-600 gauge controller that inherits from the serial gauge parent class for EPICS communication with a serial device via TCP sockets.

  • Might be better to use the current channels of the devices that are being replaced initially, i.e.
  • C1:Vac-FRG1_pressure C1:Vac-CC1_pressure
    C1:Vac-FRG2_pressure C1:Vac-CCMC_pressure
    C1:Vac-FRG3_pressure C1:Vac-PTP1_pressure
    C1:Vac-FRG4_pressure C1:Vac-CC4_pressure
    C1:Vac-FRG5_pressure C1:Vac-IG1_pressure

3. Modify the launcher file to include the XGS gauge controller. Following the same pattern used  to start the service for the other serial gauges, we can start the communication between the XGS-600 gauge controller and the IOLAN serial server and write data to EPICS channels using

controls@c1vac> python launcher.py XGS600

If we are able to establish communication between the XGS-600 gauge controller and write it gause data to EPICS channels, go on to steps 4.

4. Create a serial service file "serial_XGS600.service" and place it in the service folder

5. Add the new EPICS channels to the database file

6. Add the "serial_XGS600.service" to line 10 and 11 of modbusIOC.service

7. Later on, when we are ready, we can restart the updated modbusIOC service

 

For vacuum signal flow and Acromag channel assignments see [1]  and [2] respectively. For the 16 port IOLAN SDS (Serial Device Server) ethernet connections, see [3]. 

[1] https://wiki-40m.ligo.caltech.edu/Vacuum-Upgrade-2018?action=AttachFile&do=view&target=40m_Vacuum_System_Signal_Flow.pdf

[2] https://wiki-40m.ligo.caltech.edu/Vacuum-Upgrade-2018?action=AttachFile&do=view&target=AcromagChannelAssignment.pdf

[3] https://git.ligo.org/40m/vac/-/blob/master/python/serial/serial_devices.yaml

Attachment 1: Vac-gauges-flow-diagram.png
Vac-gauges-flow-diagram.png
  16688   Mon Feb 28 19:15:10 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

I decided to create an independent service for the XGS data readout so we can get this to work first before trying to integrate into current system. After starting the service, I noticed that the EPICS channel were not updating as expected. So I started to debug the problem and managed to track it down to an ip socket connect() error, i.e. we get a connection error for the ip address assigned to the LAN port to which the XGS box was connected. After trying a few things and searching the internet, I think the error indicates that this particular LAN port is not yet configured. I reached this conclusion after noting that only a select number of LAN ports connected without issues and these are the ports that already had devices connected. So it must be the case that the LAN ports were somehow configured. The next step is to look at the IOLAN manual to figure out how to configure the ip port for the XGS controller. Fingers crossed.

  16691   Tue Mar 1 20:38:49 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

During my investigation, I inadvertently overwrote the serial port configuration for the connected devices. So I am now working to get it all back. I have attached screenshots of the config settings that brought back communication that is not garbled. There is no physical connection to port 6, which I guess was initially used for the UPS serial communication but not anymore. Also, ports 9 and 10 are connected to Hornet and SuperBee, both of which have not been communicating for a while and are to be replaced, so there is no way to confirm communication with them. Otherwise, the remaining devices seem to be communicating as before.

I still could not establish communication with the XGS-600 controller using the serial port settings given in the manual, which also happen to work via Serial to USB adapter, so I will revisit the problem later. My immediate plan is to do a Serial Ethernet, then Ethernet to Serial, and then Serial to USB connection to see if the USB code still works. If it does then at least I know the problem is not coming from the Serial to Ethernet adapters. Then I guess I will replace the controller with my laptop and see what signal comes through when I send a message to the controller via the IOLAN serial device server. Hopefully, I can discover what's wrong by this point.

 

Note to self: Before doing anything, do a sanity check by comparing the settings on the IOLAN SDS and the config settings that worked for the Serial to USB communication and post an elog for this for reference.

Attachment 1: Working_Serial_Port_List_1.png
Working_Serial_Port_List_1.png
Attachment 2: Working_Serial_Port_List_2.png
Working_Serial_Port_List_2.png
Attachment 3: Working_Config_Ports#1-5.png
Working_Config_Ports#1-5.png
Attachment 4: Working_Config_Ports#7-8.png
Working_Config_Ports#7-8.png
  16692   Wed Mar 2 11:50:39 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

Here is the IOLAN SDS TCP socket setting and the USBserial setting for comparison.

I have also included the python script and output from the USBserial test from earlier.

Attachment 1: XGS600_IOLAN_settings_1.png
XGS600_IOLAN_settings_1.png
Attachment 2: XGS600_IOLAN_settings_2.png
XGS600_IOLAN_settings_2.png
Attachment 3: XGS600_USBserial_settings.png
XGS600_USBserial_settings.png
Attachment 4: XGS600_comm_test.py
#!/usr/bin/env python

#Created 2/24/22 by Tega Edo
'''Script to read/write to the XGS-600 Gauge Controller'''

import serial
import sys,os,math,time

ser = serial.Serial('/dev/cu.usbserial-1410') # open serial port 

... 74 more lines ...
Attachment 5: XGS600_comm_test_result.txt
----- Multiple Sensor Read Commands -----

Sent to XGS-600 -> #0001\r : Read XGS contents
response : >FE4CFE4CFE4C

Sent to XGS-600 -> #0003\r : Read Setpoint States
response : >0000

Sent to XGS-600 -> #0005\r : Read software revision
response : >0206,0200,0200,0200
... 69 more lines ...
  16693   Wed Mar 2 12:40:08 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

Connector Test:

A quick test to rule out any issue with the Ethernet to Serial adapter was done using the setup shown in Attachment 1. The results rule out any connector problem.

 

IOLAN COMM test (as per Koji's suggestion):

The next step is to swap the controller with a laptop set up to receive serial commands using the same settings as the XGS600 controller. Basically, run a slightly modified version of python script where we go into listening mode. Then send commands to the TCP socket on the IOLAN SDS unit using c1vac and check what data makes its way to the laptop USBserial terminal. After working on this for a bit, I realized that we do not need to do anything on the c1vac machine. We only need to start the service as it would work normally. So I wrote a small python code for a basic XGS-600 controller emulator, see Attachment 4. The outputs from the laptop and c1vac terminals are Attachments 5 and 6 respectively. 

These results show that we can communicate via the assigned IP address "192.168.114.22" and the commands that are sent from c1vac reaches the laptop in the correct format. Furthermore, the serial_XSG service, a part modbusIOC_XGS service, which usually exits with an error seems fine now after successfully communicating with the laptop. I don't know why it did not die after the tests. I also found a bug in my code as a result of the test, where the status field for the fourth gauge didn't get written to. 

 

Pressure reading issue:

I noticed that the pressure reading was not giving the atmospheric value of ~760 Torrs as expected. Looking through my previous readouts, it seems the unit showed this atm value of ~761 Torrs when the first gauge was attached. However, a closer look at the issue revealed a transient behavior, i.e. when the unit is turned on the reading dips to atm value but eventually rises up to 1000 Torrs. I don't think this is a calibration problem bcos the value of 1000 Torrs is the maximum value for the gauge range. I also found out that when the XGS-controller has been running for a while, a power cycle does not have this transient behavior. So maybe a faulty capacitor somewhere? I have attached a short video clip that shows what happens when the XGS-controller unit is turned on.

Attachment 1: IMG_20220302_123529382.jpg
IMG_20220302_123529382.jpg
Attachment 2: XGS600_Serial2Ethernet2Serial2USB_comm_test_result.txt
$ python3 XGS600_comm_test.py  

----- Multiple Sensor Read Commands -----

Sent to XGS-600 -> #0001\r : Read XGS contents
response : >FE4CFE4CFE4C

Sent to XGS-600 -> #0003\r : Read Setpoint States
response : >0000

... 73 more lines ...
Attachment 3: VID-20220302-WA0001.mp4
Attachment 4: comm_test_c1vac_to_laptop_via_iolansds.py
#!/usr/bin/env python

#Created 3/2/22 by Tega Edo
'''Script to emulate XGS-600 controller using laptop USBserial port'''

import serial
import sys,os,math,time

ser = serial.Serial('/dev/cu.usbserial-1410') # open serial port 

... 19 more lines ...
Attachment 5: laptop_terminal.txt
(base) tega.edo@Tegas-MBP serial % python3 comm_test_c1vac_to_laptop_via_iolansds.py

----- Listen for USBserial command and asynchronously send data in XGS600 format -----

Command received from c1vac [1] : 

Data sent to c1vac [1] : >1.000E+00,NOCBL    ,NOCBL    ,NOCBL    ,2.00E+00,NOCBL\r
Command received from c1vac [2] : 

Data sent to c1vac [2] : >2.000E+00,NOCBL    ,NOCBL    ,NOCBL    ,3.00E+00,NOCBL\r
... 54 more lines ...
Attachment 6: c1vac_terminal.txt
controls@c1vac:/opt/target/python/serial$ caget C1:Vac-FRG1_status && caget C1:Vac-FRG2_status && caget C1:Vac-FRG3_status && caget C1:Vac-FRG4_status && caget C1:Vac-FRG5_status
C1:Vac-FRG1_status             1.530E+02
C1:Vac-FRG2_status             OFF
C1:Vac-FRG3_status             OFF
C1:Vac-FRG4_status             NO COMM
C1:Vac-FRG5_status             1.55E+02
controls@c1vac:/opt/target/python/serial$ caget C1:Vac-FRG1_status && caget C1:Vac-FRG2_status && caget C1:Vac-FRG3_status && caget C1:Vac-FRG4_status && caget C1:Vac-FRG5_status
C1:Vac-FRG1_status             1.630E+02
C1:Vac-FRG2_status             OFF
C1:Vac-FRG3_status             OFF
... 70 more lines ...
  16704   Sun Mar 6 18:14:45 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

Following repeated failure to establish communication between c1vac and the XGS600 controller via Perle IOLAN serial device server, I decided to monitor the signal voltage of the communication channels (pin#2, pin#3 and pin#5) using an oscilloscope. The result of this investigation is presented in the attached pdf document. In summary, it seems I have used a crossed wired RS232 serial cable instead of a normal RS232 serial cable, so the c1vac read request command is being relayed on the wrong comm channel (pin#2 instead of pin#3). I will swap out the cable to see if this resolves the problem.  

Attachment 1: iolan_xgs_comm_investigation.pdf
iolan_xgs_comm_investigation.pdf iolan_xgs_comm_investigation.pdf iolan_xgs_comm_investigation.pdf iolan_xgs_comm_investigation.pdf
  16706   Mon Mar 7 13:53:40 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

So it appears that my deduction from the pictures of needing a cable swap was correct, however, it turns out that the installed cable was actually the normal RS232 and what we need instead is the RS232 null cable. After the swap was done, the communication between c1vac and the XGS600 controller became active. Although, the data makes it all to the to c1vac without any issues, the scope view of it shows that it is mainly utilizing the upper half of the voltage range which is just over 50% of the available range. I don't know what to make of this.

 

I guess, the only remaining issue now is the incorrect atmospheric pressure reading of 1000 Torrs. 

 

Quote:

Following repeated failure to establish communication between c1vac and the XGS600 controller via Perle IOLAN serial device server, I decided to monitor the signal voltage of the communication channels (pin#2, pin#3 and pin#5) using an oscilloscope. The result of this investigation is presented in the attached pdf document. In summary, it seems I have used a crossed wired RS232 serial cable instead of a normal RS232 serial cable, so the c1vac read request command is being relayed on the wrong comm channel (pin#2 instead of pin#3). I will swap out the cable to see if this resolves the problem.  

 

Attachment 1: iolan_xgs_comm_live.pdf
iolan_xgs_comm_live.pdf
  16713   Tue Mar 8 12:08:47 2022 TegaUpdateVACOngoing work to get the FRG gauges readouts to EPICs channels

OMG, it worked! It was indeed a calibration issue and all I had to do was press the "OK" button after selecting the "CAL" tab beside the pressure reading. Wow.

Quote:

Great trouble shoot!

> I guess, the only remaining issue now is the incorrect atmospheric pressure reading of 1000 Torrs. 

This is just a calibration issue. The controller should have the calibration function.
(The other Pirani showing 850Torr was also a calibration issue although I didn't bother to correct it. I think the pirani's typically has large distribution of the calibration values and requires individual calibration)

 

Attachment 1: XGS600_calibration.pdf
XGS600_calibration.pdf
  16747   Tue Mar 29 15:07:11 2022 TegaSummaryVACOpening of MC and ETM chambers

[Anchal, Chub, Ian, Paco, Tega]

Today, we opened the MC chamber, ETMX chamber and ETMY chamber.

  16751   Fri Apr 1 14:26:50 2022 TegaUpdateOptical LeversSimplified sketch on MC table

Here is an early sketch of the MC table. 

 

Quote:

I have made an editable draw.io diagram of the planned simplified BHD setup on the ITMY table (see attached).  10 pts = 1 inch.

This is very sketchy but easily adjustable since we are removing everything but the ITMY Oplev from that table.

 

Attachment 1: MC_Table.drawio.pdf
MC_Table.drawio.pdf
  16755   Mon Apr 4 15:49:06 2022 TegaUpdateOptical LeversSimplified sketch on BS table

I have updated the BS table using feedback from Koji and Paco and the attached pdf document is the latest iteration.

Attachment 1: BS_Table.drawio.pdf
BS_Table.drawio.pdf
  16773   Wed Apr 13 12:50:07 2022 TegaUpdateVACNew Pressure Gauge Install/Pump Spool Vent

[Jordan, JC, Tega]

We have installed all the FRGs and updated the VAC medm screens to display their sensor readings. The replacement map is CC# -> FRG#, where # in [1..4] and PRP1 -> FRG5. We now need to clean up the C1VAC python code so that it is not overloaded with non-function gauges (CC1,CC2,CC3,CC4,PRP1). Also, need to remove the connection cables for the old replaced gauges.

Attachment 1: VAC_FRG_UPGRADE.png
VAC_FRG_UPGRADE.png
  16796   Thu Apr 21 16:36:56 2022 TegaUpdateVACcleanup work for vacuum git repo

git repo - https://git.ligo.org/40m/vac

Finally incorporated the FRGs into the main modbusIOC service and everything seems to be working fine. I have also removed the old sensors (CC1,CC2,CC3,CC4,PTP1,IG1) from the serial client list and their corresponding EPICS channels. Furthermore, the interlock service python script has been updated so that all occurrence of old sensors (turns out to be only CC1) were replaced by their corresponding new FRG sensor (FRG1) and a redundnacy was also enacted for P1a where the interlock condition is replicated with P1a being replaced with FRG1 because they both sense the main volume pressure.

  16800   Thu Apr 21 18:42:47 2022 TegaUpdateBHDPart V of BHR upgrade - Align LO and AS beams to BHD BS

[Tega, Yuta, Paco]

We tried aligning the LO and AS beams on to the BHD beamsplitter. During the alignment process, we noticed that the damping loop for AS1 was not working. Paco drew our attention to the fact that the UR OSEM signal was alway close to zero, so we checked to ensure that the magnet was still within the OSEM recess and it looks OK. Next we checked the electrical connection at the interface between the copper OSEM cables to the blue in-vacuum flat cable and this too looks alright also. Since the AS1 coil driver was recently modified, it is possible we might find the problem there, so I'll ask Koji about this.


So Koji clarified that the coil driver board and SATAMP boards are different so we should connect this issue to the coil driver board.

  16803   Fri Apr 22 12:03:09 2022 TegaUpdateBHDAS1 UR OSEM problem localized in the chamber

[JC, Tega, Ian, Paco]

We found that the UR cable was clamped to the table by one of the ITMY OPLEV steering mirror mounts that was recently installed. After freeing the cable, the UR signal is now active again.

Quote:

This indicates that the AS1 UR OSEM problem is localized in the chamber. Please check if the DSUB pins are touching the table or something else.

 

  16830   Wed May 4 15:08:59 2022 TegaUpdateBHDXARM alignment to get flashing

[Yuta, Tega]

We aligned the BS, ITMY, and ETMY PIT and YAW to get the flashing on X-arm whilst also keeping the flashing of Y-arm. From attachment 1, it is clear that POXDC photodiode is not receiveing any light, so our next task is to work on POX alignment.

 

Quote:
  • Align X-arm cavity and regain flashing.
  • Fix the Oplev path for ITMX.
  • Tune POX11 phase angle to get an error signal with which we can lock the cavity.
  • Finish AS beam path setup.

 

Attachment 1: IFO_XY_ARMS_Flashing.png
IFO_XY_ARMS_Flashing.png
ELOG V3.1.3-