40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 301 of 330  Not logged in ELOG logo
ID Date Author Type Categorydown Subject
  16061   Wed Apr 21 11:01:37 2021 RanaUpdateCDS40m LSC simPlant model

The controller would be in the c1sus model, and connects to the c1sup plant model. So the controller doesn't go in the plant model.

Both the controller and the plant can be modeled using a single filter module in each separate model as you've drawn, but they go in separate models.


  16081   Fri Apr 23 15:52:19 2021 Ian MacMillanUpdateCDS40m LSC simPlant model

I have attached the framework that I am using for the full system. Plantframework.pdf has the important aspects that I will be changed. Right now I am trying to keep it mostly as is, but I have disconnected the Optic Force Noise and hope to disconnect the Suspension Position Noise. The Optic Force Noise Block is additive to the signal so eliminating it from the system should make it less realistic but simpler. It can be added back easily by reconnecting it.

The next step is adding my plant response, which is simply the transfer function and measurement from the last post. These should be inserted in the place of the red TM_RESP in the model.

The TM_RESP block takes in a vector of dimension 12 and returns a vector of dimension 6. The confusing part is that the block does not seem to do anything. it simply passes the vector through with no changes. I'm not sure why this is the case and I am looking for documentation to explain it but haven't found anything. As to how a 12 vector turns into a 6 vector I am also lost. I will probably just disconnect everything but the x position.

I tried to just throw in my model (see Simple_Plant.pdf) and see what happened but the model would not let me add built-in blocks to the model. This is weird because all the blocks that I am adding are part of the basic library. My guess is that this mode will only accept blocks from the CDL library. I will either need to change my blocks to be made from blocks in the CDL library or maybe I can pass the signal out of the plant framework model then into my model then back to the plant framework model. I think this is just a Matlab thing that I don't know about yet. (Jon probably knows)

I have also attached an image of the controls model for reference. It looks like a mess but I'm sure there is a method. I won't get lost in going through it just assume it works... for now.

The next question I have been asking is how do I show that the system works. When anchal and I made a python simulation of the system, we tested it by seeing the evolution of the degrees of freedom over time given some initial conditions. We could see the pendulum propagating and some of the coupling between the DOFs. This is a fast and dirty way to check if everything is working and should be easy to add. I simply recorded the POS signal and graph it over time. Once we get to a state-space model we can test it by taking the transfer function, but since our plant is literally already just a transfer function there isn't much of a point yet.

Also, I need to add color to my Simple_Plant.pdf model because it looks really boring :(

Attachment 1: Plant_framework.pdf
Attachment 2: Simple_Plant.pdf
Attachment 3: Controls.pdf
  16084   Sun Apr 25 21:21:02 2021 ranaUpdateCDSSUS simPlant model
  1. I suggest not naming this the LSC model, since it has no LSC stuff.
  2. Also remove all the diagnostic stuff in the plant model. We need nothing except a generic filter Module, like in the SUS controller.
  3. Also, the TF looks kind of weird to me. I would like to see how you derive that eq.
  4. Connect the models and show us some plots of the behavior in physical units using FOTON to make the filters and diaggui/DTT (at first) to make the plots.
  16088   Tue Apr 27 15:15:17 2021 Ian MacMillanUpdateCDSSUS simPlant model

The first version of the single filter plant is below. Jon and I went through compiling a model and running it on the docker (see this post)

We activated an early version of the plant model (from about 10 years ago) but this model was not designed to run on its own so we had to ground lots of unconnected pieces. the model compiled and ran so we moved on to the x1sus_single_plant model that I prepared.

This model is shown in the first attachment wasn't made to be run alone because it is technically a locked library (see the lock in the bottom left). It is supposed to be referenced by another file: x1sup.mdl (see the second attachment). This works great in the Simulink framework. I add the x1sus_single_plant model to the path and Matlab automatically attaches the two. but the docker does not seem to be able to combine the two. Starting the cymac it gives these errors:

cymac    | Can't find sus_single_plant.mdl; RCG_LIB_PATH=/opt/rtcds/userapps:/opt/rtcds/userapps/lib:/usr/share/advligorts/src/src/epics/simLink/:/usr/share/advligorts/src/src/epics/simLink/lib:/usr/share/advligorts/src/src/epics/simLink
cymac    | make[1]: *** [Makefile:30: x1sup] Error 2
cymac    | make: *** [Makefile:35: x1sup] Error 1

I have tried putting the x1sus_single_plant.mdl file everywhere as well as physically dragging the blocks that I need into the x1sup.mdl file but it always seems to throw an error. Basically, I want to combine them into one file that is not referencing anything other than the CDS library but I cant figure out how to combine them.

Okay but the next problem is the medm screen generation. When we had the original 2010 model running the sitemap did not include it. It included models that weren't even running before but not the model Jon and I had added. I think this is because the other models that were not running had medm screens made for them. I need to figure out how to generate those screens. I need to figure out how to use the tool Chris made to auto-generate medm screens from Simulink but I can't seem to figure it out. And honestly, it won't be much use to me until I can actually connect the plant block to its framework. One option is to just copy each piece over one by one. this will take forever but at this point, I am frustrated enough to try it. I'll try to give another update later tonight.

Attachment 1: x1sus_single_plant.pdf
Attachment 2: x1sup.pdf
  16092   Wed Apr 28 18:56:57 2021 Yehonathan, JonUpdateCDSUpdated c1auxey wiring plan

We took a Supermicro from the lab (along with a keyboard, a mouse, and a screen taken from a table on the Y arm) and placed it near the Acromag chassis.

We installed Debian 10 on the machine. I followed the steps on the slow machine wiki for setting up the host machine. Some steps had to be updated. Most importantly, in the new Debian, the network interfaces are given random names like enp3s0 and enp4s0 instead of eth0 and eth1. I updated the wiki accordingly.

To operate the chassis using one 15V source I disconnected the +24V cable from the Acromag units and jumpered the +15V wire into the power input instead. I started up the Acromags. They draw 0.7A. I connected an Ethernet cable to the front interface. I checked that all the Acromags are connected to the local network of the host machine by pinging them one by one.


  16093   Thu Apr 29 10:51:35 2021 JonUpdateCDSI/O Chassis Assembly


Yesterday I unpacked and installed the three 18-bit DAC cards received from Hanford. I then repeated the low-level PCIe testing outlined in T1900700, which is expanded upon below. I did not make it to DAC-ADC loopback testing because these tests in fact revealed a problem with the new hardware. After a combinatorial investigation that involved swapping cards around between known-to-be-working PCIe slots, I determined that one of the three 18-bit DAC cards is bad. Although its "voltage present" LED illuminates, the card is not detected by the host in either I/O chassis.

I installed one of the two working DACs in the c1bhd chassis. This now 100% completes this system. I installed the other DAC in the c1sus2 chassis, which still requires four more 18-bit DACs. Lastly, I reran the PCIe tests for the final configurations of both chassis.

PCIe Card Detection Tests

For future reference, below is the set of command line tests to verify proper detection and initialization of ADC/DAC/BIO cards in I/O chassis. This summarizes the procedure described in T1900700 and also adds the tests for 18-bit DAC and 32-channel BO cards, which are not included in the original document.

Each command should be executed on the host machine with the I/O chassis powered on:

$ sudo lspci -v | grep -B 1 xxxx

where xxxx is a four-digit device code given in the following table.

Device Device Code
General Standards 16-bit ADC 3101
General Standards 16-bit DAC 3120
General Standards 18-bit DAC 3357
Contec 16-channel BIO 8632
Contec 32-channel BO 86e2
Dolphin IPC host adapter 0101

The command will return a two-line entry for each PCIe device of the specified type that is detected. For example, on a system with a single ADC this command should return:

10:04.0 Bridge: PLX Technology, Inc. PCI9056 32-bit 66MHz PCI IOBus Bridge (rev ac)
             Subsystem: PLX Technology, Inc. Device 3101
  16096   Thu Apr 29 13:41:40 2021 Ian MacMillanUpdateCDSSUS simPlant model

To add the required library: put the .mdl file that contains the library into the userapps/lib folder. That will allow it to compile correctly

I got these errors:

Module ‘mbuf’ symvers file could not be found.
Module ‘gpstime’ symvers file could not be found.
***ERROR: IPCx parameter file /opt/rtcds/zzz/c1/chans/ipc/c1.ipc not found
make[1]: *** [Makefile:30: c1sup] Error 2
make: *** [Makefile:35: c1sup] Error 1

I removed all IPC parts (as seen in Attachment 1) and that did the trick. IPC parts (Inter-Process Communication) were how this model was linked to the controller so I don't know how exactly how I can link them now. 

I also went through the model and grounded all un-attached inputs and outputs. Now the model compiles

Also, The computer seems to be running very slowly in the past 24 hours. I know Jon was working on it so I'm wondering if that had any impact. I think it has to do with the connection speed because I am connected through X2goclient. And one thing that has probably been said before but I want to note again is that you don't need a campus VPN to access the docker.

Attachment 1: Non-IPC_Plant.pdf
  16097   Thu Apr 29 15:11:33 2021 gautamUpdateCDSRFM

The problem here was that the RFM errors cropped up again - seems like it started ~4am today morning judging by TRX trends. Of course without the triggering signal the arm cavity couldn't lock. I rebooted everything (since just restarting the rfm senders/receivers did not do the trick), now arm locking works fine again. It's a bit disappointing that the Rogue Master setting did not eliminate this problem completely, but oh well...

It's kind of cool that in this trend view of the TRX signal, you can see the drift of the ETMX suspension. The days are getting hot again and the temp at EX can fluctuate by >12C between day and night (so the "air-conditioning" doesn't condition that much I guess 😂 ), and I think that's what drives the drift (idk what the transfer function to the inside of the vacuum chamber is but such a large swing isn't great in any case). Not plotted here but i hypothesize TRY levels will be more constant over the day (modulo TT drift which affects both arms).

The IMC suspension team should double check their filters are on again. I am not familiar with the settings and I don't think they've been added to the SDF.

Attachment 1: RFM_errs.png
Attachment 2: Screenshot_2021-04-29_15-12-56.png
  16098   Thu Apr 29 16:35:51 2021 YehonathanUpdateCDSUpdated c1auxey wiring plan

I installed the EPICs base, asyn and modbus modules according to Jon's instructions.

Since the modbus configurations files were already writtten for c1auxey1 (see elog 15292) the only thing I did was to change the IP addresses in ETMYaux.cmd to match the actual assigned IPs.

I followed the rest of the instructions as written.

The modbus service was activated succesfully.

The only thing left to do is to change ETMYaux.db to reflect to new channels that were added. I believe these are BI channels named C1:SUS-ETMY_xx_ENABLEMon.


  16099   Thu Apr 29 17:43:16 2021 KojiUpdateCDSRFM

The other day I felt hot at the X end. I wondered if the Xend A/C was off, but the switch right next to the SP table was ON (green light).
I could not confirm if the A/C was actually blowing or not.

  16100   Thu Apr 29 17:43:48 2021 AnchalUpdateCDSF2A Filters double check

I double checked today and the F2A filters in the output matrices of MC1, MC2 and MC3 in the POS column are ON. I do not get what SDF means? Did we need to add these filters elsewhere?


The IMC suspension team should double check their filters are on again. I am not familiar with the settings and I don't think they've been added to the SDF.

Attachment 1: F2AFiltersON.png
  16103   Thu Apr 29 19:55:45 2021 YehonathanUpdateCDSUpdated c1auxey wiring plan

We received a stock of DB9 male feed-through connectors. That allowed me to complete the remaining wiring on the c1auxey Acromag chassis. The only thing left to be done is the splicing to the RTS.


  16105   Fri Apr 30 00:20:30 2021 gautamUpdateCDSF2A Filters double check

The SDF system is supposed to help with restoring the correct settings, complementary to burt. My personal opinion is that there is no need to commit these filters to SDF until we're convinced that they help with the locking / noise performance.


I double checked today and the F2A filters in the output matrices of MC1, MC2 and MC3 in the POS column are ON. I do not get what SDF means? Did we need to add these filters elsewhere

  16106   Fri Apr 30 12:52:14 2021 Ian MacMillanUpdateCDSSUS simPlant model

Now that the model is finally compiled I need to make an medm screen for it and put it in the c1sim:/home/controls/docker-cymac/userapps/medm/ directory. 

But before doing that I really want to test it using the autogenerated medm screens which are in the virtual cymac in the folder /opt/rtcds/tst/x1/medm/x1sup. In Jon's post he said that I can use the virtual path for sitemap after running $ eval $(./env_cymac)

  16107   Fri Apr 30 19:18:51 2021 Yehonathan, JonUpdateCDSUpdated c1auxey wiring plan

We finished the installation procedure on the c1auxey1 host machine. There were some adjustments that had to be made for Debian 10. The slow machine wiki page has been updated.

A test database file was made were all the channel names were changed from C1 to C2 in order to not interfere with the existing channels.

We starting testing the channels one by one to check the wiring and the EPICs software. We found some misswirings and fixed them.

Channel Name Type EPICs Test Acromag windows software test
C2:SUS-ETMY_SideVMon AI Pass Pass

Its getting late. I'll continue with the rest of the channels on Monday.

Notice that for all the AI channels the RTN was disconnected while testing.







  16109   Mon May 3 13:35:12 2021 Ian MacMillanUpdateCDSSUS simPlant model

When the cymac is started it gives me a list of channels shown below. 

 $  Initialized TP interface node=8, host=98e93ecffcca
 $  Creating X1:DAQ-DC0_X1IOP_STATUS
 $  Creating X1:DAQ-DC0_X1IOP_CRC_CPS
 $  Creating X1:DAQ-DC0_X1IOP_CRC_SUM
 $  Creating X1:DAQ-DC0_X1SUP_STATUS
 $  Creating X1:DAQ-DC0_X1SUP_CRC_CPS
 $  Creating X1:DAQ-DC0_X1SUP_CRC_SUM

But when I enter it into the Diaggui I get an error:

The following channel could not be found:

My guess is that need to connect to the Diaggui to something that can access those channels. I also need to figure out what those channels are.

  16114   Mon May 3 20:36:46 2021 Yehonathan, JonUpdateCDSUpdated c1auxey wiring plan

It seemed like the BIO channels were not working, both the inputs and the outputs. The inputs were working on the windows machine though. That is, when we shorted the BIO channel to the return, or put 0V on it, we could see the LED turn on on the I/O testing screen and when we ramped up the voltage above 3 the LED turned off. This is the expected behavior from a sinking digital input. However, the EPICs caget didn't show any change. All the channels were stuck on Disabled.

We checked the digital outputs by connecting the channels to a fluke. Initially, the fluke showed 13V. We tried to toggle the digital output channels with caput and that didn't work. We checked the outputs with the windows software. For that, we needed to stop the Modbus. To our surprise, the windows software was not able to flip the channels either. We realized that this BIO Acromag unit is probably defective. We replaced it with a different unit and put a warning sticker on the defective unit. Now, the digital outputs were working as expected. When we turned them on the voltage output dropped to 0V. We checked the channels with the EPICs software. We realized that these channels were locked with the closed loop definition. We turned on the channels tied to these output channels (watchdog and toggles) and it worked. The output channels can be flipped with the EPICs software. We checked all the digital output channels and fixed some wiring issues along the way.

The digital input channels were still not working. This is a software issue that we will have to deal with later.

(Yehonathan) Rana noticed that the BNC leads on the chassis front panel didn't have isolation on them so I redid them with shrinking tubes.

  16116   Tue May 4 07:38:36 2021 JonUpdateCDSI/O Chassis Assembly

IOP models created

With all the PCIe issues now resolved, yesterday I proceeded to build an IOP model for each of new FEs. I assigned them names and DCUIDs consist with the 40m convention, listed below. These models currently exist on only the cloned copy of /opt/rtcds running on the test stand. They will be copied to the main network disk later, once the new systems are fully tested.

Model Host CPU DCUID
c1x06 c1bhd 1 23
c1x07 c1sus2 1 24

The models compile and install successfully. The RCG runtime diagnostics indicate that all is working except for the timing synchronization and DAQD data transmission. This is as expected because neither of these have been set up yet.

Timing system set-up

The next step is to provide the 65 kHz clock signals from the timing fanout via LC optical fiber. I overlooked the fact that an SPX optical transceiver is required to interface the fiber to the timing slave board. These were not provided with the timing slaves we received. The timing slaves require a particular type of transceiver, 100base-FX/OC-3, which we did not have on hand. (For future reference, there is a handy list of compatible transceivers in E080541, p. 14.) I placed a Digikey order for two Finisar FTLF1217P2BTL, which should arrive within two days.

Attachment 1: Screen_Shot_2021-05-03_at_4.16.06_PM.png
  16118   Tue May 4 14:55:38 2021 Ian MacMillanUpdateCDSSUS simPlant model

After a helpful meeting with Jon, we realized that I have somehow corrupted the sitemap file. So I am going to use the code Chris wrote to regenerate it. 

Also, I am going to connect the controller using the IPC parts. The error that I was having before had to do with the IPC parts not being connected properly. 

  16122   Wed May 5 15:11:54 2021 Ian MacMillanUpdateCDSSUS simPlant model

I added the IPC parts back to the plant model so that should be done now. It looks like this again here.

I can't seem to find the control model which should look like this. When I open sus_single_control.mdl, it just shows the C1_SUS_SINGLE_PLANT.mdl model. Which should not be the case.

  16124   Thu May 6 16:13:24 2021 Ian MacMillanUpdateCDSSUS simPlant model

When using mdl2adl I was getting the error:

$  cd /home/controls/mdl2adl
$  ./mdl2adl x1sup.mdl
error: set $site and $ifo environment variables

to set these in the terminal use the following commands:

$  export site=tst
$  export ifo=x1

On most of the systems, there is a script that automatically runs when a terminal is opened that sets these but that hasn't been added here so you must run these commands every time you open the terminal when you are using mdl2adl.

  16126   Fri May 7 11:19:29 2021 Ian MacMillanUpdateCDSSUS simPlant model

I copied c1scx.mdl to the docker to attach to the plant using the commands:

$  ssh nodus.ligo.caltech.edu
[Enter Password]
$  cd opt/rtcds/userapps/release/isc/c1/models/simPlant
$  scp c1scx.mdl controls@c1sim:/home/controls/docker-cymac/userapps
  16130   Tue May 11 16:29:55 2021 JonUpdateCDSI/O Chassis Assembly

Timing system set-up

The next step is to provide the 65 kHz clock signals from the timing fanout via LC optical fiber. I overlooked the fact that an SPX optical transceiver is required to interface the fiber to the timing slave board. These were not provided with the timing slaves we received. The timing slaves require a particular type of transceiver, 100base-FX/OC-3, which we did not have on hand. (For future reference, there is a handy list of compatible transceivers in E080541, p. 14.) I placed a Digikey order for two Finisar FTLF1217P2BTL, which should arrive within two days.

Today I brought and installed the new optical transceivers (Finisar FTLF1217P2BTL) for the two timing slaves. The timing slaves appear to phase-lock to the clocking signal from the master fanout. A few seconds after each timing slave is powered on, its status LED begins steadily blinking at 1 Hz, just as in the existing 40m systems.

However, some other timing issue remains unresolved. When the IOP model is started (on either FE), the DACKILL watchdog appears to start in a tripped state. Then after a few minutes of running, the TIM and ADC indicators go down as well. This makes me suspect the sample clocks are not really phase-locked. However, the models do start up with no error messages. Will continue to debug...

Attachment 1: Screen_Shot_2021-05-11_at_3.03.42_PM.png
  16131   Tue May 11 17:43:09 2021 KojiUpdateCDSI/O Chassis Assembly

Did you match the local PC time with the GPS time?

  16134   Wed May 12 13:06:15 2021 Ian MacMillanUpdateCDSSUS simPlant model

Working with Chris, we decided that it is probably better to use a simple filter module as a controller before we make the model more complicated. I will use the plant model that I have already made (see attachment 1 of this). then attach a single control filter module to that: as seen in attachment 1. because I only want to work with one degree of freedom (position) I will average the four outputs which should give me the position. Then by feeding the same signal to all four inputs I should isolate one degree of freedom while still using the premade plant model.

The model I made that is shown in attachment 2 is the model I made from the plan. And it complies! yay! I think there is a better way to do the average than the way I showed. And since the model is feeding back on itself I think I need to add a delay which Rana noted a while ago. I think it was a UnitDelay (see page 41 of RTS Developer’s Guide). So I will add that if we run into problems but I think there is enough going on that it might already be delayed.

Since our model (x1sup_isolated.mdl) has compiled we can open the medm screens for it. I provide a procedure below which is based on Jon's post

[First start the cymac and have the model running]
$  cd docker-cymac
$  eval $(./env_cymac)

$  medm -x /opt/rtcds/tst/x1/medm/x1sup_isolated/X1SUP_ISOLATED_GDS_TP.adl

To see a list of all medm screens use:

$  cd docker-cymac
$  ./login_cymac
 #  cd /opt/rtcds/tst/x1/medm/x1sup_isolated
 #  ls

Some of the other useful ones are:

adl screen Description
X1SUP_ISOLATED_Control_Module.adl This is the control filter module shown in attachment 2 at the top in the center. This module will represent the control system.

See attachment 4. This screen shows the POS plant filter module that will be filled by the filter representing the transfer function of a damped harmonic oscillator:        \frac{x}{F}=\frac{\omega_0^2}{\omega_0^2+i\frac{\omega_0 \omega}{Q}-\omega^2}


The first one of these screens that are of interest to us (shown in attachment 3) is the X1SUP_ISOLATED_GDS_TP.adl screen, which is the CDS runtime diagnostics screen. This screen tells us "the success/fail state of the model and all its dependencies." I am still figuring out these screens and the best guide is T1100625.

The next step is taking some data and seeing if I can see the position damp over time. To do this I need to:

  1. Edit the plant filter for the model and add the correct filter.
  2. Figure out a filter for the control system and add it to that. (I can leave it as is to see what the plant is doing) 
  3. Take some position data to show that the plant is a harmonic oscillator and is damping away.
Attachment 1: SimplePlant_SingleContr.pdf
Attachment 2: x1sup_isolated.pdf
Attachment 3: X1SUP_ISOLATED_GDS_TP.png
Attachment 4: X1SUP_ISOLATED_C1_SUS_SINGLE_PLANT_Plant_POS_Mod.png
  16151   Fri May 21 09:44:52 2021 Ian MacMillanUpdateCDSSUS simPlant model

The transfer function given in the previous post was slightly incorrect the units did not make sense the new function is:

\frac{x}{F}=\frac{1}{m\omega_0^2-m\omega^2+im\frac{\omega_0 \omega }{Q}}

I have attached a quick derivation below in attachment 1

Attachment 1: Transfer_Function_of_Damped_Harmonic_Oscillator.pdf
  16153   Fri May 21 14:36:20 2021 Ian MacMillanUpdateCDSSUS simPlant model

The plant transfer function of the pendulum in the s domain is:


Using Foton to make a plot of the TF needed and using m=40kg, w0=3Hz, and Q=50 (See attachment 1). It is easiest to enter the above filter using RPoly and saved it as Plant_V1

Attachment 1: Plant_Mod_TF.pdf
  16154   Sun May 23 18:28:54 2021 JonUpdateCDSOpto-isolator for c1auxey

The new HAM-A coil drivers have a single DB9 connector for all the binary inputs. This requires that the dewhitening switching signals from the fast system be spliced with the coil enable signals from c1auxey. There is a common return for all the binary inputs. To avoid directly connecting the grounds of the two systems, I have looked for a suitable opto-isolator for the c1auxey signals.

I best option I found is the Ocean Controls KTD-258, a 4-channel, DIN-rail-mounted opto-isolator supporting input/output voltages of up to 30 V DC. It is an active device and can be powered using the same 15 V supply as is currently powering both the Acromags and excitation. I ordered one unit to be trialed in c1auxey. If this is found to be good solution, we will order more for the upgrades of c1auxex and c1susaux, as required for compatibility with the new suspension electronics.

  16166   Fri May 28 10:54:59 2021 JonUpdateCDSOpto-isolator for c1auxey

I have received the opto-isolator needed to complete the new c1auxey system. I left it sitting on the electronics bench next to the Acromag chassis.

Here is the manufacturer's wiring manual. It should be wired to the +15V chassis power and to the common return from the coil driver, following the instructions herein for NPN-style signals. Note that there are two sets of DIP switches (one on the input side and one on the output side) for selecting the mode of operation. These should all be set to "NPN" mode.

Attachment 1: optoisolator.jpeg
  16167   Fri May 28 11:16:21 2021 JonUpdateCDSFront-End Assembly and Testing

An update on recent progress in the lab towards building and testing the new FEs.

1. Timing problems resolved / FE BIOS changes

The previously reported problem with the IOPs losing sync after a few minutes (16130) was resolved through a change in BIOS settings. However, there are many required settings and it is not trivial to get these right, so I document the procedure here for future reference.

The CDS group has a document (T1300430) listing the correct settings for each type of motherboard used in aLIGO. All of the machines received from LLO contain the oldest motherboards: the Supermicro X8DTU. Quoting from the document, the BIOS must be configured to enforce the following:

• Remove hyper-threading so the CPU doesn’t try to run stuff on the idle core, as hyperthreading simulate two cores for every physical core.
• Minimize any system interrupts from hardware, such as USB and Serial Ports, that might get through to the ‘idled’ core. This is needed on the older machines.
• Prevent the computer from reducing the clock speed on any cores to ‘save power’, etc. We need to have a constant clock speed on every ‘idled’ CPU core.

I generally followed the T1300430 instructions but found a few adjustments were necessary for diskless and deterministic operation, as noted below. The procedure for configuring the FE BIOS is as follows:

  1. At boot-up, hit the delete key to enter the BIOS setup screen.
  2. Before changing anything, I recommend photographing or otherwise documenting the current working settings on all the subscreens, in case for some reason it is necessary to revert.
  3. T1300430 assumes the process is started from a known state and lists only the non-default settings that must be changed. To put the BIOS into this known state, first navigate to Exit > Load Failsafe Defaults > Enter.
  4. Configure the non-default settings following T1300430 (Sec. 5 for the X8DTU motherboard). On the IPMI screen, set the static IP address and netmask to their specific assigned values, but do set the gateway address to all zeros as the document indicates. This is to prevent the IPMI from trying to initiate outgoing connections.
  5. For diskless booting to continue to work, it is also necessary to set Advanced > PCI/PnP Configuration > Load Onboard LAN 1 Option Rom > Enabled.
  6. I also found it was necessary to re-enable IDE direct memory access and WHEA (Windows Hardware Error Architecture) support. Since these machines have neither hard disks nor Windows, I have no idea why these are needed, but I found that without them, one of the FEs would hang during boot about 50% of the time.
    • Advanced > PCI/PnP configuration > PCI IDE BusMaster  > Enabled.
    • Advanced > ACPI Configuration > WHEA Support > Enabled.

After completing the BIOS setup, I rebooted the new FEs about six times each to make sure the configuration was stable (i.e., would never hang during boot).

2. User models created for FE testing

With the timing issue resolved, I proceeded to build basic user models for c1bhd and c1sus2 for testing purposes. Each one has a simple structure where M ADC inputs are routed through IIR filters to an output matrix, which forms linear signal combinations that are routed to N DAC outputs. This is shown in Attachment 1 for the c1bhd case, where the signals from a single ADC are conditioned and routed to a single 18-bit DAC. The c1sus2 case is similar; however the Contec BO modules still needed to be added to this model.

The FEs are now running two models each: the IOP model and one user model. The assigned parameters of each model are documented below.

Model Host CPU DCUID Path
c1x06 c1bhd 1 23 /opt/rtcds/userapps/release/cds/c1/models/c1x06.mdl
c1x07 c1sus2 1 24 /opt/rtcds/userapps/release/cds/c1/models/c1x07.mdl
c1bhd c1bhd 2 25 /opt/rtcds/userapps/release/isc/c1/models/c1bhd.mdl
c1sus2 c1sus2 2 26 /opt/rtcds/userapps/release/sus/c1/models/c1sus2.mdl

The user models were compiled and installed following the previously documented procedure (15979). As shown in Attachment 2, all the RTS processes are now working, with the exception of the DAQ server (for which we're still awaiting hardware). Note that these models currently exist only on the cloned copy of the /opt/rtcds disk running on the test stand. The plan is to copy these models to the main 40m disk later, once the new FEs are ready to be installed.

3. AA and AI chassis installed

I installed several new AA and AI chassis in the test stand to interface with the ADC and DAC cards. This includes three 16-bit AA chassis, one 16-bit AI chassis, and one 18-bit AI chassis, as pictured in Attachment 3. All of the AA/AI chassis are powered by one of the new 15V DC power strips connected to a bench supply, which is housed underneath the computers as pictured in Attachment 4.

These chassis have not yet been tested, beyond verifying that the LEDs all illuminate to indicate that power is present.

Attachment 1: c1bhd.png
Attachment 2: gds_tp.png
Attachment 3: teststand.jpeg
Attachment 4: bench_supply.jpeg
  16177   Thu Jun 3 13:06:47 2021 Ian MacMillanUpdateCDSSUS simPlant model

I was able to measure the transfer function of the plant filter module from the channel X1:SUP-C1_SUS_SINGLE_PLANT_Plant_POS_Mod_EXC to X1:SUP-C1_SUS_SINGLE_PLANT_Plant_POS_Mod_OUT. The resulting transfer function is shown below. I have also attached the raw data for making the graph.

Next, I will make a script that will make the photon filters for all the degrees of freedom and start working on the matrix version of the filter module so that there can be multiple degrees of freedom.

Attachment 1: SingleSusPlantTF.pdf
Attachment 2: SUS_PLANT_TF.zip
  16178   Thu Jun 3 17:15:17 2021 YehonathanUpdateCDSOpto-isolator for c1auxey

As Jon wrote we need to use the NPN configuration (see attachments). I tested the isolator channels in the following way:

1. I connected +15V from the power supply to the input(+) contact.

2. Signal wire from one of the digital outputs was connected to I1-4

3. When I set the digital output to HIGH, the LED on the isolator turns on.

4. I measure the resistance between O1-4 to output(-) and find it to be ~ 100ohm in the HIGH state and an open circuit in the LOW state, as expected from an open collector output.

Unlike the Acromag output, the isolator output is not pulled up in the LOW state. To do so we need to connect +15V to the output channel through a pull-up resistor. For now, I leave it with no pull-up. According to the schematics of the HAM-A Coil Driver, the digital output channels drive an electromagnetic relay (I think) so it might not need to be pulled up to switch back. I'm not sure. We will need to check the operation of these outputs at the installation.

During the testing of the isolator outputs pull-up, I accidentally ran a high current through O2, frying it dead. It is now permanently shorted to the + and - outputs rendering it unusable. In any case, we need another isolator since we have 5 channels we need to isolate.

I mounted the isolator on the DIN rail and started wiring the digital outputs into it. I connected the GND from the RTS to output(-) such that when the digital outputs are HIGH the channels in the coil driver will be sunk into the RTS GND and not the slow one avoiding GND contamination.

Attachment 1: Optical_Isolator_NPN_Input.png
Attachment 2: Optical_Isolator_NPN_Output.png
  16181   Thu Jun 3 22:08:00 2021 KojiUpdateCDSOpto-isolator for c1auxey

- Could you explain what is the blue thing in Attachment 1?

- To check the validity of the signal chain, can you make a diagram summarizing the path from the fast BO - BO I/F - Acromag - This opto-isolator - the coil driver relay? (Cut-and-paste of the existing schematics is fine)


  16182   Fri Jun 4 14:49:23 2021 YehonathanUpdateCDSOpto-isolator for c1auxey

I made a diagram (Attached). I think it explains the blue thing in the previous post.

I don't know what is the grounding situation in the RTS so I put a ground in both the coil driver and the RTS. Hopefully, only one of them is connected in reality.


- Could you explain what is the blue thing in Attachment 1?

- To check the validity of the signal chain, can you make a diagram summarizing the path from the fast BO - BO I/F - Acromag - This opto-isolator - the coil driver relay? (Cut-and-paste of the existing schematics is fine)



Attachment 1: Optical_isolator_Wiring.pdf
  16183   Fri Jun 4 17:46:25 2021 unYehonathanUpdateCDSOpto-isolator for c1auxey

I mounted the optoisolator on the DIN rail and connected the 3 first channels



to the optoisolator inputs 1,3,4 respectively. I connected the +15V input voltage into the input(+) of the optoisolator.

The outputs were connected to DB9F-2 where those channels were connected before.

I added DB9F-1 to the front panel to accept channels from the RTS. I connected the fast channels to connectors 1,2,3 from DB9F-1 to DB9F-2 according to the wiring diagram. The GND from DB9F-1 was connected to both connector 5 of DB9F-2 and the output (-).

I tested the channels: I connected a DB9 breakout board to DB9F-2. I measured the resistance between the RTS GND and the isolated channels while switching them on and off. In the beginning, when I turned on the binary channels the resistance was behaving weird - oscillating between low resistance and open circuit. I pulled up the channels through a 100Kohm resistor to observe whether the voltage behavior is reasonable or not. Indeed I observed that in the LOW state the voltage between the isolated channel and slow GND is 15V and 0.03V in the HIGH state. Then I disconnected the pull up from the channels and measured the resistance again. It showed ~ stable 170ohm in the HIGH state and an open circuit in the LOW state. I was not able to reproduce the weird initial behavior. Maybe the optoisolator needs some warmup of some sort.


We still need to wire the rest of the fast channels to DBF9-3 and isolate the channels in DBF9-4. For that, we need another optoisolator.


There is still an open issue with the BI channels not read by EPICS. They can still be read by the Windows machine though.

Attachment 1: 20210604_173420.jpg
  16184   Sun Jun 6 03:02:14 2021 KojiUpdateCDSOpto-isolator for c1auxey

This RTS also use the BO interface with an opto isolator. https://dcc.ligo.org/LIGO-D1002593

Could you also include the pull up/pull down situations?

  16185   Sun Jun 6 08:42:05 2021 JonUpdateCDSFront-End Assembly and Testing

Here is an update and status report on the new BHD front-ends (FEs).


The changes to the FE BIOS settings documented in [16167] do seem to have solved the timing issues. The RTS models ran for one week with no more timing failures. The IOP model on c1sus2 did die due to an unrelated "Channel hopping detected" error. This was traced back to a bug in the Simulink model, where two identical CDS parts were both mapped to ADC_0 instead of ADC_0/1. I made this correction and recompiled the model following the procedure in [15979].

Model naming standardization

For lack of a better name, I had originally set up the user model on c1sus2 as "c1sus2.mdl" This week I standardized the name to follow the three-letter subsystem convention, as four letters lead to some inconsistency in the naming of the auto-generated MEDM screens. I renamed the model c1sus2.mdl -> c1su2.mdl. The updated table of models is below.

Model Host CPU DCUID Path
c1x06 c1bhd 1 23 /opt/rtcds/userapps/release/cds/c1/models/c1x06.mdl
c1x07 c1sus2 1 24 /opt/rtcds/userapps/release/cds/c1/models/c1x07.mdl
c1bhd c1bhd 2 25 /opt/rtcds/userapps/release/isc/c1/models/c1bhd.mdl
c1su2 c1su2 2 26 /opt/rtcds/userapps/release/sus/c1/models/c1su2.mdl

Renaming an RTS model requires several steps to fully propagate the change, so I've documented the procedure below for future reference.

On the target FE, first stop the model to be renamed:

controls@c1sus2$ rtcds stop c1sus2

Then, navigate to the build directory and run the uninstall and cleanup scripts:

controls@c1sus2$ cd /opt/rtcds/caltech/c1/rtbuild/release
controls@c1sus2$ make uninstall-c1sus2
controls@c1sus2$ make clean-c1sus2

Unfortunately, the uninstall script does not remove every vestige of the old model, so some manual cleanup is required. First, open the file /opt/rtcds/caltech/c1/target/gds/param/testpoint.par and manually delete the three-line entry corresponding to the old model:


If this is not removed, reinstallation of the renamed model will fail because its assigned DCUID will appear to already be in use. Next, find all relics of the old model using:

controls@c1sus2$ find /opt/rtcds/caltech/c1 -iname "*sus2*"

and manually delete each file and subdirectory containing the "sus2" name. Finally, rename, recompile, reinstall, and relaunch the model:

controls@c1sus2$ cd /opt/rtcds/userapps/release/sus/c1/models
controls@c1sus2$ mv c1sus2.mdl c1su2.mdl
controls@c1sus2$ cd /opt/rtcds/caltech/c1/rtbuild/release
controls@c1sus2$ make c1su2
controls@c1sus2$ make install-c1su2
controls@c1sus2$ rtcds start c1su2

Sitemap screens

I used a tool developed by Chris, mdl2adl, to auto-generate a set of temporary sitemap/model MEDM screens. This package parses each Simulink file and generates an MEDM screen whose background is an .svg image of the Simulink model. Each object in the image is overlaid with a clickable button linked to the auto-generated RTS screens. An example of the screen for the C1BHD model is shown in Attachment 1. Having these screens will make the testing much faster and less user-error prone.

I generated these screens following the instructions in Chris' README. However, I ran this script on the c1sim machine, where all the dependencies including Matlab 2021 are already set up. I simply copied the target .mdl files to the root level of the mdl2adl repo, ran the script (./mdl2adl.sh c1x06 c1x07 c1bhd c1su2), and then copied the output to /opt/rtcds/caltech/c1/medm/medm_teststand. Then I redefined the "sitemap" environment variable on the chiara clone to point to this new location, so that they can be launched in the teststand via the usual "sitemap" command.

Current status and plans

Is it possible to convert 18-bit AO channels to 16-bit?

Currently, we are missing five 18-bit DACs needed to complete the c1sus2 system (the c1bhd system is complete). Since the first shipment, we have had no luck getting additional 18-bit DACs from the sites, and I don't know when more will become available. So, this week I took an inventory of all the 16-bit DACs available at the 40m. I located four 16-bit DACs, pictured in Attachment 2. Their operational states are unknown, but none were labeled as known not to work.

The original CDS design would call for 40 more 18-bit DAC channels. Between the four 16-bit DACs there are 64 channels, so if only 3/4 of these DACs work we would have enough AO channels. However, my search turned up zero additional 16-bit DAC adapter boards. We could check if first Rolf or Todd have any spares. If not, I think it would be relatively cheap and fast to have four new adapters fabricated.

DAQ network limitations and plan

To get deeper into the signal-integrity aspect of the testing, it is going to be critical to get the secondary DAQ network running in the teststand. Of all the CDS tools (Ndscope, Diaggui, DataViewer, StripTool), only StripTool can be used without a functioning NDS server (which, in turn, requires a functioning DAQ server). StripTool connects directly to the EPICS server run by the RTS process. As such, StripTool is useful for basic DC tests of the fast channels, but it can only access the downsampled monitor channels. Ian and Anchal are going to carry out some simple DAC-to-ADC loopback tests to the furthest extent possible using StripTool (using DC signals) and will document their findings separately.

We don't yet have a working DAQ network because we are still missing one piece of critical hardware: a 10G switch compatible with the older Myricom network cards. In the older RCG version 3.x used by the 40m, the DAQ code is hardwired to interface with a Myricom 10G PCIe card. I was able to locate a spare Myricom card, pictured in Attachment 3, in the old fb machine. Since it looks like it is going to take some time to get an old 10G switch from the sites, I went ahead and ordered one this week. I have not been able to find documentation on our particular Myricom card, so it might be compatible with the latest 10G switches but I just don't know. So instead I bought exactly the same older (discontinued) model as is used in the 40m DAQ network, the Netgear GSM7352S. This way we'll also have a spare. The unit I bought is in "like-new" condition and will unfortunately take about a week to arrive.

Attachment 1: c1bhd.png
Attachment 2: 16bit_dacs.png
Attachment 3: myricom.png
  16186   Sun Jun 6 12:15:16 2021 JonUpdateCDSOpto-isolator for c1auxey

Since this Ocean Controls optoisolator has been shown to be compatible, I've gone ahead and ordered 10 more:

  • (1) to complete c1auxey
  • (2) for the upgrade of c1auxex
  • (7) for the upgrade of c1susaux

They are expected to arrive by Wednesday.

  16187   Sun Jun 6 15:59:51 2021 YehonathanUpdateCDSOpto-isolator for c1auxey

According to the BO interface circuit board https://dcc.ligo.org/D1001266, PCIN wires are connected to the coil driver and they are not pulled either way.

That means that they're either grounded or floating. I updated the drawing.


This RTS also use the BO interface with an opto isolator. https://dcc.ligo.org/LIGO-D1002593

Could you also include the pull up/pull down situations?


Attachment 1: Optical_isolator_Wiring.pdf
  16188   Sun Jun 6 16:33:47 2021 JonUpdateCDSBI channels on c1auxey

There is still an open issue with the BI channels not read by EPICS. They can still be read by the Windows machine though.

I looked into the issue that Yehonathan reported with the BI channels. I found the problem was with the .cmd file which sets up the Modbus interfacing of the Acromags to EPICS (/cvs/cds/caltech/target/c1auxey1/ETMYaux.cmd).

The problem is that all the channels on the XT1111 unit are being configured in Modbus as output channels. While it is possible to break up the address space of a single unit, so that some subset of channels are configured as inputs and another as outputs, I think this is likely to lead to mass confusion if the setup ever has to be modified. A simpler solution (and the convention we adopted for previous systems) is just to use separate Acromag units for BI and BO signals.

Accordingly, I updated the wiring plan to include the following changes:

  • The five EnableMon BI channels are moved to a new Acromag XT1111 unit (BIO01), whose channels are configured in Modbus as inputs.
  • One new DB37M connector is added for the 11 spare BI channels on BIO01.
  • The five channels freed up on the existing XT1111 (BIO00) are wired to the existing connector for spare BO channels.

So, one more Acromag XT1111 needs to be added to the c1auxey chassis, with the wiring changes as noted above. I have already updated the .cmd and EPICS database files in /cvs/cds/caltech/target/c1auxey1 to reflect these changes.

  16189   Mon Jun 7 13:14:20 2021 YehonathanUpdateCDSBI channels on c1auxey

I added a new XT1111 Acromag module to the c1auxey chassis. I sanitized and configured it according to the slow machines wiki instructions.

Since all the spare BIOs fit one DB37 connector I didn't add another feedthrough and combined them all on one and the same DB37 connector. This was possible because all the RTNs of the BIOs are tied to the chassis ground and therefore need only one connection. I changed the wiring spreadsheet accordingly.

I did a lot of rewirings and also cut short several long wires that were protruding from the chassis. I tested all the wires from the feedthroughs to the Acromag channels and fixed some wiring mistakes.

Tomorrow I will test the BIs using EPICs.

  16191   Mon Jun 7 17:49:19 2021 Ian MacMillanUpdateCDSSUS simPlant model

Added difference to the graph. I included the code so that others could see what it looks like and use it for easy use.

Attachment 1: SingleSusPlantTF.pdf
Attachment 2: TF_Graph_Code.zip
  16193   Tue Jun 8 11:54:39 2021 YehonathanUpdateCDSBI channels on c1auxey

I tested the digital inputs the following way: I connected a DB9 breakout to DB9M-5 and DB9M-6 where digital inputs are hosted. I shorted the channel under test to GND to turn it on.

I observed the channels turn from Disabled to Enabled using caget when I shorted the channel to GND and from Enabled to Disabled when I disconnected them.

I did this for all the digital inputs and they all passed the test.

I am still waiting for the other isolator to wire the rest of the digital outputs.

Next, I believe we should take some noise spectra of the Y end before we do the installation.


Tomorrow I will test the BIs using EPICs.


  16195   Wed Jun 9 13:50:48 2021 Ian MacMillanUpdateCDSSUS simPlant model

I have attached an updated transfer function graph with the residual easier to see. I thought here I would include a better explanation of what this transfer function was measuring.

This transfer function was mainly about learning how to use DTT and Foton to make and measure transfer functions. Therefore it is just measuring across a single CDS filter block. X1SUP_ISOLATED_C1_SUS_SINGLE_PLANT_Plant_POS_Mod block to be specific. This measurement shows that the block is doing what I programmed it to do with Foton. The residual is probably just because the measured TF had fewer points than the calculated one.

The next step is to take a closed-loop TF of the system and the control module.

After that, I want to add more degrees of freedom to the model. both in the plant and in the controls.

Attachment 1: SingleSusPlantTF.pdf
  16199   Mon Jun 14 15:31:30 2021 YehonathanUpdateCDSOpto-isolator for c1auxey

I checked the BI situation on the HAM-A coil driver. It seems like these are sinking BIs and indeed need to be isolated from the Acromag unit GND to avoid contamination.

The BIs will have to be isolated on a different isolator. Now, the wires coming from the field (red) are connected to the second isolator's input and the outputs are connected to the Acromag BI module and the Acromag's RTN.

I updated the wiring diagram (attached) and the wiring spreadsheet.

In the diagram, you can notice that the BI isolator (the right one) is powered by the Acromag's +15V and switched when the coil driver's GND is supplied. I am not sure if it makes sense or not. In this configuration, there is a path between the coil driver's GND and the Acromag's GND but its resistance is at least 10KOhm. The extra careful option is to power the isolator by the coil driver's +V but there is no +V on any of the connectors going out of the coil driver.

I installed an additional isolator on the DIN rail and wired the remaining BOs (C1:SUS-ETMY_SD_ENABLE, C1:SUS-ETMY_LR_ENABLE) through it to the DB9F-4 feedthrough. I also added DB9F-3 for incoming wires from the RTS and made the required connection from it to DB9F-4.

I tested the new isolated BOs using the Windows machine (after stopping Modbus). As before, I measure the resistance between pin 5 (coil driver GND) and the channel under test. When I turn on the BO I see the resistance drops from inf to 166ohm and back to inf when I turn it off. Both channels passed the test.


Attachment 1: Optical_isolator_Wiring.pdf
  16201   Tue Jun 15 11:46:40 2021 Ian MacMillanUpdateCDSSUS simPlant model

I have added more degrees of freedom. The model includes x, y, z, pitch, yaw, roll and is controlled by a matrix of transfer functions (See Attachment 2). I have added 5 control filters to individually control UL, UR, LL, LR, and side. Eventually, this should become a matrix too but for the moment this is fine.

Note the Unit delay blocks in the control in Attachment 1. The model will not compile without these blocks.

Attachment 1: x1sup_isolated-6-15-v1.pdf
Attachment 2: C1_SUS_SINGLE_PLANT-6-15-v1.pdf
  16203   Tue Jun 15 21:48:55 2021 KojiUpdateCDSOpto-isolator for c1auxey

If my understanding is correct, the (photo receiving) NPN transistor of the optocoupler is energized through the acromag. The LED side should be driven by the coil driver circuit. It is properly done for the "enable mon" through 750Ohm and +V. However, "Run/Acquire" is a relay switch and there is no one to drive the line. I propose to add the pull-up network to the run/acquire outputs. This way all 8 outputs become identical and symmetric.

We should test the configuration if this works properly. This can be done with just a manual switch, R=750Ohm, and a +V supply  (+18V I guess).

Attachment 1: Acromag_RTS_BI_config.jpg
  16205   Wed Jun 16 17:24:29 2021 YehonathanUpdateCDSOpto-isolator for c1auxey

I updated the wiring diagram according to Koji's suggestion. According to the isolator manual, this configuration requires that the isolator input be configured as PNP.

Additionally, when the switch in the coil driver is open the LED in the isolator is signaling an on-state. Therefore, we might need to configure the Acromag to invert the input.

There are the Run/Aquire channels that we might need to add to the wiring diagram. If we do need to read them using slow channels, we will have to pull them up like the EnableMon channels to use them like in the wiring diagram.

Attachment 1: Optical_isolator_Wiring.pdf
  16207   Wed Jun 16 20:32:39 2021 YehonathanUpdateCDSOpto-isolator for c1auxey

I installed 2 additional isolators in the Acromag chassis. I set all the input channels to PNP. I ran the digital inputs (EnableMon channels) through these isolators according to the previous post.

I tested the digital inputs in the following way:

I connected an 18V voltage source to the signal wire under test through a 1Kohm resistor. I connected the GND of the voltage source to the RTN wire of the feedthrough. When the voltage source was connected, the LED on the isolator turned on and the EPICs channel under test was Enabled. When I disconnected the voltage source or shorted the signal wire to GND the LED on the isolator turned off and the EPICs channel showed a Disabled state.

  16208   Thu Jun 17 11:19:37 2021 Ian MacMillanUpdateCDSCDS Upgrade

Jon and I tested the ADC and DAC cards in both of the systems on the test stand. We had to swap out an 18-bit DAC for a 16-bit one that worked but now both machines have at least one working ADC and DAC.

[Still working on this post. I need to look at what is in the machines to say everything ]

ELOG V3.1.3-