After fixing multiple issues, the model webviews are updating, should be done by tomorrow. It should be obvious from the timestamps on the index page which are the new ones. These new screens are better than the old ones and offer more info/details. Please look at them, and let me know if there are broken links etc. Once we are happy with this new webview, we can archive the old files and clean up the top directory a bit. I don't think this adds anything to the channel accounting effort but it's a nice thing to have up-to-date webviews, I found the LLO ones really useful in setting up the AS WFS model.

BTW, the crontab on megatron is set to run every day at 0844. The process of updating the models is pretty heavy because of whatever MATLAB overhead. Do we really need to have this run every day? I modified the crontab to run every other Saturday, and we can manually run the update when we modify a model. Considering this hasn't been working for ~3 years, I think this is fine, but if anyone has strong preference you can edit the crontab.

If someboy can volunteer to fix the MEDM screenshot that would be useful.

Taking inspiration from SR785 on how it reads differential signal, I figured that acromag too always need a way to return current through RTN ports always. That must be the reason why everything goes haywire when RTN is not connected to IN-. Now for single ended signals, we can always short RTN to IN- and keep same GND but then we need to be careful in avoiding ground loops. I'm gonna post a wiring diagram in next post to show how if two signal sources connect to each other separately, a GND loop can be formed if we tie each IN- port to RTN on an acromag.
Coming to the issue of reading a differential signal, what SR785 does is that it connects 50 Ohm resistance between Earth GND and differential signal shields (which are supposed to signal GND). In a floating GND setting, SR785 connects a 1 MOhm resistor between input shield and Earth GND. This can be used to read a differential signal through a single BNC cable since the shiled can take arbitrary voltages thanks ti the 1 MOhm resistor.

We can do the same in acromag. Instead of shorting RTN to IN- ports, we can connect them through a large resistor which would let IN- float but will give a path for current to return through RTN ports. Attached here are few scenarios where I connected IN- to RTN throguh wire, 820 Ohms, 10kOhms and 1MOhms in two sub cases where RTN was left open or was shorted to Earth GND. In all cases, the signal was produced by a 9V battery outputing roughly 8.16V. It seems that 10kOhm resistor between RTN and IN- with RTN connected to Earth GND is the best scenario noise wise. I'll post more results and a wiring diagram soon.

Here I present few wiring diagrams when using Acromag to avoid noisy behavior and ground loops.

Case 1: Only single-ended sources

• Attachment 1 gives a functioning wiring diagram when all sources are single ended.
• One should always short the RTN to IN- pin if the particular GND carried by that signal has not been shorted before to RTN for some other signal.
• So care is required to mark different GNDs of different powersupply separately and follow where they inadvertently get shorted, for example when a photodiode output is connected to FSS Box.
• Acromag should serve as the node of all GNDs concerned and all these grounds must not be connected to Earth GND at power supply ends or in any of the signal sources.
• I think this is a bit complicated thing to do.

Case 2: Some single and some differential sources

• Connect all single ended sources same as above keeping care of not building any ground loops.
• The differential source can be connected to IN+ and IN- pins, but there should be a high resistance path between IN- and RTN. See Attachment 2.
• Why this is the case, I'm not sure since I could not get access to internal wiring of Acromag (no response from them). But I have empirically found this.
• This helps IN- to float with respect to RTN according to the negative signal value. I've found that 10kOhm resistance works good. See 40m/15778.
• If RTN is shorted to nearby Earth GND (assuming none of the other power supply GNDs have been shorted to Earth GND) shows a reduction in noise for differential input. So this is recommended.
• This wiring diagram carries all complexity of previous case along with the fact that RTN and anything connected to it is at Earth GND now.

Case 3: Signal agnostic wiring

• Attachment 3 gives a wiring diagram which mimics the high resistance shorting of RTN to IN- in all ports regardless of the kind of signal it is used for reading.
• In this case, instead of being the node of the star configuration for GND, acromag gets detached from any ground loops.
• All differences in various GNDs would be kept by the power supplies driving small amounts of current through the 10 kOhm resistors.
• This is a much simpler wiring diagram as it avoids shorting various signal sources or their GNDs with each other, avoiding some of the ground loops.

Edit Wed Jan 27 13:38:19 2021 :

This solution is not acceptable as well. Even if it is successfull in reading the value, connecting resistor between IN- and RTN will not break the ground loops and the issue of ground loops will persist. Further, IN- connection to RTN breaks the symmetry between IN-  and IN+, and hence reduces the common mode rejection which is the intended purpose of differential signal anyways. I'll work more on this to find a way to read differential signals without connecitng IN- and RTN. My first guess is that it would need the GND on the source end to be connected to EarthGND and RTN on acromag end to be connected to EarthGND as well.

I found a white paper  from Acromag which discusses how to read differential signal using Acromag units. The document categorically says that differential signals are always supposed to be transmitted in three wires. I provides the two options of either using the RTN to connect to the signal ground (as done in Attachment 3) or locally place 10k-100k resistors between return and IN+ and IN- both (Attachment 2).

I have provided possible scenarios for these.

Using two wires to carry differential signal (Attachment 1):

• I assume this is our preferential way to connect.
• We can also assume all single-ended inputs as differential and do a signal condition agnostic wiring.
• Attachment 3 show what were the results for different values of resistors when a 2Hz 0.5V amplitude signal from SR785 which as converted to differential signal using D1900068 was measured by acromag.
• The connection to RTN is symmetrical for both inputs.

Using three wires to carry differential signal (Attachment 2):

• This is recommended method by the document in which it asks to carry the GND from signal source and connect it to RTN.
• If we use this, we'll have to be very cautious on what GND has been shorted through the acromag RTN terminals.
• This would probably create a lot of opportunities for ground loops to form.

Using an acromag card without making any connection with RTN is basically not allowed as per this document.

I worked around the racks and the feedthru flanges this afternoon and evening. This inevitably crashed c1lsc real-time process.
Rebooting c1lsc caused multiple crashes (as usual) and I had to hard reboot c1lsc/c1sus/c1ioo
This made the "DC" indicator of the IOPs for these hosts **RED**.

This looked like the usual timing issue. It looked like "ntpdate" is not available in the new system. (When was it updated?)

The hardware clock (RTC) of these hosts are set to be PST while the functional end host showed UTC. So I copied the time of the UTC time from the end to the vertex machines.
For the time adjustment, the standard "date" command was used

> sudo date -s "2021-02-03 07:11:30"

This made the trick. Once IOP was restarted, the "DC" indicators returned to **Green**, restarting the other processes were straight forward and now the CDS indicators are all green.

controls@c1iscex:~ 0$ timedatectl
      Local time: Wed 2021-02-03 07:35:12 UTC
  Universal time: Wed 2021-02-03 07:35:12 UTC
        RTC time: Wed 2021-02-03 07:35:26
       Time zone: Etc/UTC (UTC, +0000)
     NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
      DST active: n/a

NTP synchronization is not active. Is this OK?

With the recovered CDS, the IMC was immediately locked and the autolocker started to function after a few pokes (like manually running of the "mcup" script). However, I didn't see any light on the AS/REF cameras as well as the test mass faces. I'm sure the IMC alignment is OK. This means the TTs are not well aligned.

So, burtrestored c1assepics with 12:19 snapshot. This immediately brought the spots on the REFL/AS.

Then the arm were aligned, locked, and ASSed. I tried to lock the FP arms. The transmissions were at the level of 0.1~0.3. So some manual alignment of ITMY and BS were necessary. After having the TRs of ~0.8, I still could not lock the arms. The signs of the servo gains were flipped to -0.143 for X arm and -0.012 for Y arm, and the arms were locked. ASS worked well and the ASS offsets were offloaded to the SUSs.

Didn't get a chance to comment during the meeting - This was almost certainly a coincidence. I have never had to do this - I assert, based on the ~10 labwide reboots I have had to do in the last two years, that whether the timing errors persist on reboot or not is not deterministic. But this is beyond my level of CDS knowledge and so I'm happy for Rolf / Jamie to comment. I use the reboot script - if that doesn't work, I use it again until the systems come back without any errors.

I don't think this is a problem, the NTP synchronization is handled by timesyncd now.

I defer restoring the LSC settings etc since I guess there is not expected to be any interferometer activity for a while.

Really!? I didn't reboot the machines between "sudo date" and "rtcds start c1x0*". I tried rtcds. If it didn't work, it used date. Then tried rtcds. (repeat) The time was not synched at all wrt the time zones and also the time. There were 1~3 sec offset besides the TZ problem.

I am just reporting my experience - this may be a new failure mode but I don't think so. In the new RTCDS, the ntp server for the FEs are the FB, to which they are synced by timesyncd. The FB machine itself has the ntpd service installed, and so is capable of synching to an NTP server over the internet, but also serving as an NTP server for the FEs. The timesyncd daemon may not have started correctly, or the NTP serving from FB got interrupted (for example), but that's all just speculation.

The defaults cds-crtools didn't come with some of the older ezcautils (like ezcaread, ezcawrite etc). This is now packaged for debian, so I installled them with sudo apt update && sudo apt install dtt-ezca-tools on rossa. Now, we don't have to needlessly substitute the commands in our old shell scripts with the more modern z read, z write etc.

I am wondering if there is a relative implicit minus sign between the z servo and ezcaservo commands...

I've started writing up a rough testing sequence for getting the three new front-ends operational (c1bhd, c1sus2, c1ioo). Since I anticipate this plan undergoing many updates, I've set it up as a Google doc which everyone can edit (log in with LIGO.ORG credentials).

Link to planning document

Please have a look and add any more tests, details, or concerns. I will continue adding to it as I read up on CDS documentation.

I've set up one 2U server unit received from KT in the control room area (the fans in it are pretty loud but other than that, no major issues with it being tested here). The IPMI interface is enabled and the machine is also hooked up to the martian network for diskless boot (usual login creds). I also installed a Dolphin interface card and the one-stop-systems host side card, and both seem to be recognized (the OSSI card has the "PWR" LED on, the Dolphin card shows up in the list of PCIe devices, but has no LEDs on at the moment). I actually can't find the OSSI card in the list of PCI devices, but maybe I'm not looking for the right device ID, or it needs the cable to be connected to the I/O chassis side to be recognized. Anyways, let the testing begin.

The machine previously christened c1bhd has been turned off and completely disconnected from the martian network (though I didn't bother removing it from the rack for now).

BTW - I think most of our 19" racks are deformed from years of loading - I spent 5 mins trying to install the rails (at 1Y1 and 1X7) to mount the supermicro on, and couldn't manage it. I could be missing some technique/trick, but i don't think so.

Today I moved the c1bhd machine from the control room to a new test area set up behind (west of) the 1X6 rack. The test stand is pictured in Attachment 1. I assembled one of the new IO chassis and connected it to the host.

I/O Chassis Assembly

• LIGO-style 24V feedthrough replaced with an ATX 650W switching power supply
• Timing slave installed
• Contec DO-1616L-PE card installed for timing control
• One 16-bit ADC and one 32-channel DO module were installed for testing

The chassis was then powered on and LED lights illuminated indicating that all the components have power. The assembled chassis is pictured in Attachment 2.

Chassis-Host Communications Testing

Following the procedure outlined T1900700, the system failed the very first test of the communications link between chassis and host, which is to check that all PCIe cards installed in both the host and the expansion chassis are detected. The Dolpin host adapter card is detected:

07:06.0 PCI bridge: Stargen Inc. Device 0102 (rev 02) (prog-if 00 [Normal decode])
    Flags: bus master, fast devsel, latency 0
    Bus: primary=07, secondary=0e, subordinate=0e, sec-latency=0
    I/O behind bridge: 00002000-00002fff
    Prefetchable memory behind bridge: 00000000c0200000-00000000c03fffff
    Capabilities: [40] Power Management version 2
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [60] Express Downstream Port (Slot+), MSI 00
    Capabilities: [80] Subsystem: Device 0000:0000
    Kernel driver in use: pcieport

However the OSS PCIe adapter card linking the host to the IO chassis was not detected, nor were any of the cards in the expansion chassis. Gautam previously reported that the OSS card was not detected by the host (though it was not connected to the chassis then). Even now connected to the IO chassis, the card is still not detected. On the chassis-side OSS card, there is a red LED illuminated indicating "HOST CARD RESET" as pictured in Attachment 3. This may indicate a problem with the card on the host side. Still more debugging to be done.

Today I continued with assembly and testing of the new front-ends. The main progress is that the IO chassis is now communicating with the host, resolving the previously reported issue.

Hardware Issues to be Resolved

Unfortunately, though, it turns out one of the two (host-side) One Stop Systems PCIe cards sent from Hanford is bad. After some investigation, I ultimately resolved the problem by swapping in the second card, with no other changes. I'll try to procure another from Keith Thorne, along with some spares.

Also, two of the three switching power supplies sent from Livingston (250W Channel Well PSG400P-89) appear to be incompatible with the Trenton BPX6806 PCIe backplanes in these chassis. The power supply cable has 20 conductors and the connector on the board has 24. The third supply, a 650W Antec EA-650, does have the correct cable and is currently powering one of the IO chassis. I'll confirm this situation with Keith and see whether they have any more Antecs. If not, I think these supplies can still be bought (not obsolete).

I've gone through all the hardware we've received, checked against the procurement spreadsheet. There are still some missing items:

• 18-bit DACs (Qty 14; but 7 are spares)
• ADC adapter boards (Qty 5)
• DAC adapter boards (Qty 9)
• 32-channel DO modules (Qty 2/10 in hand)

Testing Progress

Once the PCIe communications link between host and IO chassis was working, I carried out the testing procedure outlined in T1900700. This performs a series checks to confirm basic operation/compatibility of the hardware and PCIe drivers. All of the cards installed in both the host and the expansion chassis are detected and appear correctly configured, according to T1900700. In the below tree, there is one ADC, one 16-ch DIO, one 32-ch DO, and one DolphinDX card:

+-05.0-[05-20]----00.0-[06-20]--+-00.0-[07-08]----00.0-[08]----00.0  Contec Co., Ltd Device 86e2
|                               +-01.0-[09]--
|                               +-03.0-[0a]--
|                               +-08.0-[0b-15]----00.0-[0c-15]--+-02.0-[0d]--
|                               |                               +-03.0-[0e]--
|                               |                               +-04.0-[0f]--
|                               |                               +-06.0-[10-11]----00.0-[11]----04.0  PLX Technology, Inc. PCI9056 32-bit 66MHz PCI <-> IOBus Bridge
|                               |                               +-07.0-[12]--
|                               |                               +-08.0-[13]--
|                               |                               +-0a.0-[14]--
|                               |                               \-0b.0-[15]--
|                               \-09.0-[16-20]----00.0-[17-20]--+-02.0-[18]--
|                                                               +-03.0-[19]--
|                                                               +-04.0-[1a]--
|                                                               +-06.0-[1b]--
|                                                               +-07.0-[1c]--
|                                                               +-08.0-[1d]--
|                                                               +-0a.0-[1e-1f]----00.0-[1f]----00.0  Contec Co., Ltd Device 8632
|                                                               \-0b.0-[20]--
\-08.0-[21-2a]--+-00.0  Stargen Inc. Device 0101
                \-00.1-[22-2a]--+-00.0-[23]--
                                +-01.0-[24]--
                                +-02.0-[25]--
                                +-03.0-[26]--
                                +-04.0-[27]--
                                +-05.0-[28]--
                                +-06.0-[29]--
                                \-07.0-[2a]--

Standalone Subnet

Before I start building/testing RTCDS models, I'd like to move the new front ends to an isolated subnet. This is guaranteed to prevent any contention with the current system, or inadvertent changes to it.

Today I set up another of the Supermicro servers sent by Livingston in the 1X6 test stand area. The intention is for this machine to run a cloned, bootable image of the current fb1 system, allowing it to function as a bootserver and DAQ server for the FEs on the subnet.

However, this hard disk containing the fb1 image appears to be corrupted and will not boot. It seems to have been sitting disconnected in a box since ~2018, which is not a stable way to store data long term. I wasn't immediately able to recover the disk using fsck. I could spend some more time trying, but it might be most time-effective to just make a new clone of the fb1 system as it is now.

I have recently been running into hitting the 4MB/s data rate limit on testpoints - basically, I can't run DTT TF and spectrum measurements that I was able to while locking the interferometer, which I certainly was able to this time last year. AFAIK, the major modification made was the addition of 4 DQ channels for the in-air BHD experiment - assuming the data is transmitted as double precision numbers, i estimate the additional load due to this change was ~500KB/s. Probably there is some compression so it is a bit more efficient (as this naive calc would suggest we can only record 32 channels and I counted 41 full rate channels in the model), but still, can't think of anything else that has changed. Anyway, I removed the unused parts and recompiled/re-installed the models (c1lsc and c1omc). Holding off on a restart until I decide I have the energy to recover the CDS system from the inevitable crash. For documentation I'm also attaching screenshot of the schematic of the changes made.

Anyway, the main point of this elog is that at the compilation stage, I got a warning I've never seen before:

This prompted me to check the system time on c1lsc and FB - you can see there is a 1 minute offset (it is not a delay in me issuing the command to the two machines)! I am suspecting this NTP action is the reason. So maybe a model reboot is in order. Sigh

Since Koji was in the lab I decided to bite the bullet and do the reboot. I've modified the reboot script - now, it prompts the user to confirm that the time recognized by the FEs are the same (use the IOP model's status screen, the GPSTIME is updated live on the upper right hand corner). So you would do sudo date --set="Thu 11 Mar 2021 06:48:30 PM UTC" for example, and then restart the IOP model. Why is this necessary? Who knows. It seems to be a deterministic way of getting things back up and running for now so we have to live with it. I will note that this was not a problem between 2017 and 2020 Oct, in which time I've run the reboot script >10 times without needing to take this step. But things change (for an as of yet unknown reason) and we must adapt. Once the IOPs all report a green "DC" status light on the CDS overview screen, you can let the script take you the rest of the way again.

The main point of this work was to relax the data rate on the c1lsc model, and this worked. It now registers ~3.2 MB/s, down from the ~3.8 MB/s earlier today. I can now measure 2 loop TFs simultaneously. This means that we should avoid adding any more DQ channels to the c1lsc model (without some adjustment/downsampling of others).

I want to emphasize the followings:

• FB's RTC (real-time clock/motherboard clock) is synchronized to NTP time.
• The other RT machines are not synchronized to NTP time.
• My speculation is that the RT machine timestamp is produced from the local RT machine time because there is no IRIG-B signal provided. -> If the RTC is more than 1 sec off from the FB time, the timestamp inconsistency occurs. Then it induces "DC" error.
• For now, we have to use "date" command to match the local RTC time to the FB's time.
   
• So: If NTP synchronization is properly implemented for the RT machines, we will be free from this silly manual time adjustment.

I looked into this a bit today morning. I forgot exactly what time we restarted the machines, but looking at the timesyncd logs, it appears that the NTP synchronization is in fact working (log below is for c1sus, similar on other FEs):

So, the service is doing what it is supposed to (using the FB, 192.168.113.201, as the ntpserver). You can see that the timesync was done a couple of seconds after the machine booted up (validated against "uptime"). Then, the service is periodically correcting drifts. idk what it means that the time wasn't in sync when we check the time using timedatectl or similar. Anyway, like I said, I have successfully rebooted all the FEs without having to do this weird time adjustment >10 times. 

I guess what I am saying is, I don't know what action is necessary for "implementing NTP synchronization properly", since the diagnostic logfile seems to indicate that it is doing what it is supposed to.

More worryingly, the time has already drifted in < 24 hours.

Some progress today towards setting up an isolated subnet for testing the new front-ends. I was able to recover the fb1 backup disk using the Rescatux disk-rescue utility and successfully booted an fb1 clone on the subnet. This machine will function as the boot server and DAQ server for the front-ends under test. (None of these machines are connected to the Martian network or, currently, even the outside Internet.)

Despite the success with the framebuilder, front-ends cannot yet be booted locally because we are still missing the DHCP and FTP servers required for network booting. On the Martian net, these processes are running not on fb1 but on chiara. And to be able to compile and run models later in the testing, we will need the contents of the /opt/rtcds directory also hosted on chiara.

For these reasons, I think it will be easiest to create another clone of chiara to run on the subnet. There is a backup disk of chiara and I attempted to boot it on one of the LLO front-ends, but without success. The repair tool I used to recover the fb1 disk does not find a problem with the chiara disk. However, the chiara disk is an external USB drive, so I suspect there could be a compatibility problem with these old (~2010) machines. Some of them don't even recognize USB keyboards pre-boot-up. I may try booting the USB drive from a newer computer.

Edit: I removed one of the new, unused Supermicros from the 1Y2 rack and set it up in the test stand. This newer machine is able to boot the chiara USB disk without issue. Next time I'll continue with the networking setup.

Now that I think about it, I may only have backed up the root file system of chiara, and not/home/cds/ (symlinked to /opt/ over NFS). I think we never revived the rsync backup to LDAS after the FB fiasco of 2017, else that'd have been the most convenient way to get files. So you may have to resort to some other technique (e.g. configure the second network interface of the chiara clone to be on the martian network and copy over files to the local disk, and then disconnect the chiara clone from the martian network (if we really want to keep this test stand completely isolated from the existing CDS network) - the /home/cds/ directory is rather large IIRC, but with 2TB on the FB clone, you may be able to get everything needed to get the rtcds system working). It may then be necessary to hook up a separate disk to write frames to if you want to test that part of the system out.

Good to hear the backup disk was able to boot though!

Summary

Today I finished setting up the subnet for new FE testing. There are clones of both fb1 and chiara running on this subnet (pictured in Attachment 2), which are able to boot FEs completely independently of the Martian network. I then assembled a second FE system (Supermicro host and IO chassis) to serve as c1sus2, using a new OSS host adapter card received yesterday from LLO. I ran the same set of PCIe hardware/driver tests as was done on the c1bhd system in 15890. All the PCIe tests pass.

Subnet setup

For future reference, below is the procedure used to configure the bootserver subnet.

• Select "Network" as highest boot priority in FE BIOS settings
• Connect all machines to subnet switch. Verify fb1 and chiara eth0 interfaces are enabled and assigned correct IP address.
• Add c1bhd and c1sus2 entries to chiara:/etc/dhcp/dhcpd.conf:

• Restart DHCP server to pick up changes:

• Add c1bhd and c1sus2 entries to fb1:/etc/hosts:

• Power on the FEs. If all was configured correctly, the machines will boot.

C1SUS2 I/O chassis assembly

• Installed in host:
  • DolphinDX host adapter
  • One Stop Systems PCIe x4 host adapter (new card sent from LLO)
• Installed in chassis:
  • Channel Well 250 W power supply (replaces aLIGO-style 24 V feedthrough)
  • Timing slave
  • Contec DIO-1616L-PE module for timing control

Next time, on to RTCDS model compilation and testing. This will require first obtaining a clone of the /opt/rtcds disk hosted on chiara.

Today I helped Yehonathan get started with assembly of the c1auxey (slow controls) Acromag chassis. This will replace the final remaining VME crate. We cleared the far left end of the electronics bench in the office area, as discussed on Wed. The high-voltage supplies and test equipment was moved together to the desk across the aisle.

Yehonathan has begun assembling the chassis frame (it required some light machining to mount the DIN rails that hold the Acromag units). Next, he will wire up the switches, LED indicator lights, and Acromag power connectors following the the documented procedure.

This evening I prepared a new 2 TB 3.5" disk to hold a copy of /opt/rtcds and /opt/rtapps from chiara. This is the final piece of setup before model compilation can be tested on the new front-ends. However chiara does not appear to support hot-swapping of disks, as the disk is not recognized when connected to the live machine. I will await confirmation before rebooting it. The new disk is not currently connected.

I finished prewiring the new c1auxey Acromag chassis (see attached pictures). I connected all grounds to the DIN rail to save some wiring. The power switches and LEDs work as expected.

I configured the DAQ modules using the old windows machine. I configured the gateway to be 192.168.114.1. The host machine still needs to be setup.

Next, the feedthroughs need to be wired and the channels need to be bench tested.

It might be a good idea to configure this box for the new suspension config - modern Satellite Amp, HV coil driver etc. It's a good opportunity to test the wiring scheme, "cross-connect" type adapters etc.

Cloning of chiara:/home/cvs underway

I returned today with a beefier USB-SATA adapter, which has an integrated 12 V supply for powering 3.5" disks. I used this to interface a new 6 TB 3.5" disk found in the FE supplies cabinet.

I decided to go with a larger disk and copy the full contents of chiara:/home/cds. Strictly, the FEs only strictly need the RTS executables in /home/cvs/rtcds and /home/cvs/rtapps. However, to independently develop models, the shared matlab binaries in /home/cvs/caltech/... also need to be exposed. And there may be others I've missed.

I began the clone around 12:30 pm today. To preserve bandwidth to the main disk, I am copying not the /home/cds disk directly, but rather its backup image at /media/40mBackup.

Set up of dedicated SimPlant host

Although not directly related to the FE testing, today I added a new machine to the test stand which will be dedicated to running sim models. Chris has developed a virtual cymac which we plan to run on this machine. It will provide a dedicated testbed for SimPlant and other development, and can host up to 10 user models.

I used one of the spare 12-core Supermicro servers from LLO, which I have named c1sim. I assigned it the IP address 192.168.113.93 on the Martian network. This machine will run in a self-contained way that will not depend on any 40m CDS services and also should not interfere with them. However, if there are concerns about having it present on the network, it can be moved to the outside-facing switch in the office area. It is not currently running any RTCDS processes.

Set-up was carried out via the following procedure:

• Installed Debian 10.9 on an internal 480 GB SSD.
• Installed cdssoft repos following Jamie's instructions.
• Installed RTS and Docker dependencies:
  $ sudo apt install cpuset advligorts-mbuf-dkms advligorts-gpstime-dkms docker.io docker-compose
• Configured scheduler for real-time operation:
  $ sudo /sbin/sysctl kernel.sched_rt_runtime_us = -1
• Reserved 10 cores for RTS user models (plus one for IOP model) by adding the following line to /etc/default/grub:
  GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=nohz,domain,1-11 nohz_full=1-11 tsc=reliable mce=off"
  followed by the commands:
  $ sudo update-grub
$ sudo reboot now
• Downloaded virtual cymac repo to /home/controls/docker-cymac.

I need to talk to Chris before I can take the setup further.

{Yehonathan, Jon}

We poked (looked in situ with a flashlight, not disturbing any connections) around c1auxex chassis to understand better what is the wiring scheme.

To our surprise, we found that nothing was connected to the RTNs of the analog input Acromag modules. From previous experience and the Acromag manual, there can't be any meaningful voltage measurement without it.

I also did some rewiring in the Acromag chassis to improve its reliability. In particular, I removed the ground wires from the DIN rail and connected them using crimp-on butt splices.

Progress today:

Outside Internet access for FE test stand

This morning Jordan and I ran an 85-foot Cat 6 Ethernet cable from the campus network switch in the office area (on the ligo.caltech.edu domain) to the FE test stand near 1X6. This is to allow the test-stand subnet to be accessed for remote testing, while keeping it invisible to the parallel Martian subnet.

Successful RTCDS model compilation on new FEs

The clone of the chiara:/home/cds disk completed overnight. Today I installed the disk in the chiara clone. The NFS mounts (/opt/rtcds, /opt/rtapps) shared with the other test-stand machines mounted without issue.

Next, I attempted to open the shared Matlab executable (/cvs/cds/caltech/apps/linux64/matlab/bin/matlab) and launch Simulink. The existing Matlab license (/cvs/cds/caltech/apps/linux64/matlab/licenses/license_chiara_865865_R2015b.lic) did not work on this new machine, as they are machine-specific, so I updated the license file. I linked this license to my personal license, so that the machine license for the real chiara would not get replaced. The original license file is saved in the same directory with a *.bak postfix. If this disk is ever used in the real chiara machine, this file should be restored. After the machine license was updated, Matlab and Simulink loaded and allowed model editing.

Finally, I tested RTCDS model compilation on the new FEs using the c1lsc model as a trial case. It encountered one path issue due to the model being located at /opt/rtcds/userapps/release/isc/c1/models/isc/ instead of /opt/rtcds/userapps/release/isc/c1/models/. This seems to be a relic of the migration of the 40m models from the SVN to a standalone git repo. This was resolved by simply symlinking to the expected location:

The model compilation then succeeded:

As did the installation:

We are ready to start building and testing models.

Yesterday Chris and I completed setup of the Supermicro machine that will serve as a dedicated host for developing and testing RTCDS sim models. It is currently sitting in the stack of machines in the FE test stand, though it should eventually be moved into a permanent rack.

It turns out the machine cannot run 10 user models, only 4. Hyperthreading was enabled in the BIOS settings, which created the illusion of there being 12 rather than 6 physical cores. Between Chris and Ian's sim models, we already have a fully-loaded machine. There are several more of these spare 6-core machines that could be set up to run additional models. But in the long term, and especially in Ian's case where the IFO sim models will all need to communicate with one another (this is a self-contained cymac, not a distributed FE system), we may need to buy a larger machine with 16 or 32 cores.

IPMI was set up for the c1sim cymac. I assigned the IPMI interface a static IP address on the Martian network (192.168.113.45) and registered it in the usual way with the domain name server on chiara. After updating the BIOS settings and rebooting, I was able to remotely power off and back on the machine following these instructions.

I/O chassis assembly

Yesterday I installed all the available ADC/DAC/BIO modules and adapter boards into the new I/O chassis (c1bhd, c1sus2). We are still missing three ADC adapter boards and six 18-bit DACs. A thorough search of the FE cabinet turned up several 16-bit DACs, but only one adapter board. Since one 16-bit DAC is required anyway for c1sus2, I installed the one complete set in that chassis.

Below is the current state of each chassis. Missing components are highlighted in yellow. We cannot proceed to loopback testing until at least some of the missing hardware is in hand.

C1BHD

C1SUS2

Gateway for remote access

To enable remote access to the machines on the test stand subnet, one machine must function as a gateway server. Initially, I tried to set this up using the second network interface of the chiara clone. However, having two active interfaces caused problems for the DHCP and FTS servers and broke the diskless FE booting. Debugging this would have required making changes to the network configuration that would have to be remembered and reverted, were the chiara disk to ever to be used in the original machine.

So instead, I simply grabbed another of the (unused) 1U Supermicro servers from the 1Y1 rack and set it up on the subnet as a standalone gateway server. The machine is named c1teststand. Its first network interface is connected to the general computing network (ligo.caltech.edu) and the second to the test-stand subnet. It has no connection to the Martian subnet. I installed Debian 10.9 anticipating that, when the machine is no longer needed in the test stand, it can be converted into another docker-cymac for to run additional sim models.

Currently, the outside-facing IP address is assigned via DHCP and so periodically changes. I've asked Larry to assign it a static IP on the ligo.caltech.edu domain, so that it can be accessed analogously to nodus.

5x 16bit ADC adapter boards (D0902006) assembled.

I installed three of the 16-bit ADC adapter boards assembled by Koji. Now, the only missing hardware is the 18-bit DACs (quantities below). As I mentioned this week, there are 2-3 16-bit DACs available in the FE cabinet. They could be used if more 16-bit adapter boards could be procured.

Summary

Yesterday I resurrected the 40m's LSC simPlant model, c1lsp. It is running on c1sim, a virtual, self-contained cymac that Chris and I set up for developing sim models (see 15997). I think the next step towards an integrated IFO model is incorporating the suspension plants. I am going to hand development largely over to Ian at this point, with continued support from me and Chris.

LSC Plant

This model dates back to around 2012 and appears to have last been used in ~2015. According to the old CDS documentation:

Here XEP, YEP, and VSP are respectively the x-end, y-end, and vertex suspension plant models. I haven't found any evidence that these were ever fully implemented for the entire IFO. However, it looks like SUS plants were later implemented for a single arm cavity, at least, using two models named c1sup and c1spx (appear in more recent CDS documentation). These suspension plants could likely be updated and then copied for the other suspended optics.

To represent the optical transfer functions, the model loads a set of SOS filter coefficients generated by an Optickle model of the interferometer. The filter-generating code and instructions on how to use it are located here. In particular, it contains a Matlab script named opt40m.m which defines the interferferometer. It should be updated to match the parameters in the latest 40m Finesse model, C1_w_BHD.kat. The calibrations from Watts to sensor voltages will also need to be checked and likely updated.

Model-Porting Procedure

For future reference, below are the steps followed to port this model to the virtual cymac.

1. Copy over model files.
    
   • The c1lsp model, chiara:/opt/rtcds/userapps/release/isc/c1/models/c1lsp.mdl, was copied to the userapps directory on the virtual cymac, c1sim:/home/controls/docker-cymac/userapps/x1lsp.mdl. In the filename, note the change in IFO prefix "c1" --> "x1," since this cymac is not part of the C1 CDS network.
      
   • This model also depends on a custom library file, chiara:/opt/rtcds/userapps/release/isc/c1/models/SIMPLANT.mdl, which was copied to c1sim:/home/controls/docker-cymac/userapps/lib/SIMPLANT.mdl.
      
2. Update model parameters in Simulink. To edit models in Simulink, see the instructions here and also here.
    
   • The main changes are to the cdsParameters block, which was updated as shown below. Note the values of dcuid and specific_cpu are specifically assigned to x1lsp and will vary for other models. The other parameters will be the same.
      
   • 
      
   • I also had to change the name of one of the user-defined objects from "ADC0" --> "ADC" and then re-copy the cdsAdc object (shown above) from the CDS_PARTS.mdl library. At least in newer RCG code, the cdsAdc object must also be named "ADC0." This namespace collision was causing the compiler to fail.
      
   • Note: Since Matlab is not yet set up on c1sim, I actually made these edits on one of the 40m machines (chiara) prior to copying the model.
      
3. Compile and launch the models. Execute the following commands on c1sim:
   • $ cd ~/docker-cymac
$ ./kill_cymac
$ ./start_cymac debug

   • The optional debug flag will print the full set of compilation messages to the terminal. If compilation fails, search the traceback for lines containing "ERROR" to determine what is causing the failure.
      

4. Accessing MEDM screens. Once the model is running, a button should be added to the sitemap screen (located at c1sim:/home/controls/docker-cymac/userapps/medm/sitemap.adl) to access one or more screens specific to the newly added model.

   • Custom-made screens should be added to c1sim:/home/controls/docker-cymac/userapps/medm/x1lsp (where the final subdirectory is the name of the particular model).

   • The set of available auto-generated screens for the model can be viewed by entering the virtual environment:

   • $ cd ~/docker-cymac
$ ./login_cymac                    #drops into virtual shell
# cd /opt/rtcds/tst/x1/medm/x1lsp  #last subdirectory is model name
# ls -l *.adl
# exit                             #return to host shell

   • The sitemap screen and any subscreens can link to the auto-generated screens in the usual way (by pointing to their virtual /opt/rtcds path). Currently, for the virtual path resolution to work, an environment script has to be run prior to launching sitemap, which sets the location of a virtual MEDM server (this will be auto-scripted in the future):

   • $ cd ~/docker-cymac
$ eval $(./env_cymac)
$ sitemap

   • One important auto-generated screen that should be linked for every model is the CDS runtime diagnostics screen, which indicates the success/fail state of the model and all its dependencies. T1100625 details the meaning of all the various indicator lights.

   • 

Added Matlab to the Docker machine. This should help immensely with workflow as well as keeping installed libraries consistent. Next step is outlining the project so coding is easier

Command to launch is:     $ matlab &

From Jon just for bookkeeping: 

Then in the Matlab command window, open the CDS parts library via:

addpath /home/controls/simLink/lib/

open /home/controls/simLink/CDS_PARTS.mdl

Then open an RTCDS model (for example, here the LSC plant) via:

open /home/controls/docker-cymac/userapps/x1lsp.mdl

I've updated the c1auxey wiring plan for compatibility with the new suspension electronics. Specifically it is based on wiring schematics for the new HAM-A coil driver (D1100117), satellite amplifier (D1002818), and HV bias driver (D1900163).

Changes:

• The PDMon, VMon, CoilEnable, and BiasAdj channels all move from DB37 to various DB9 breakout boards.
• The DB9 cables (x2) connecting the CoilEnable channels to the coil drivers must be spliced with the dewhitening switching signals from the RTS.
• As suggested, I added five new BI channels to monitor the state of the CoilEnable switches. For lack of a better name, they follow the naming convention C1:SUS-ETMY_xx_ENABLEMon.

@Yehonathan can proceed with wiring the chassis.

We received 10x 16bit ADC adapter boards from Todd. S2100687~S2100696

The number of soldered resistors seems to be less than that on the schematics. They are related to duotone, so check if it's OK upon use.

The x1SUSsim model on the docker was made in a more recent version of Simulink so I updated Matlab (see this)

I updated Matlab to 2021a so now the docker has 2020b and 2021a installed. This should also install Simulink 10.3 for the sus model to open. I used my account to activate it but I can change it over if I get a dedicated license for this. I am not sure what Jon did for the 2020b that he installed.

it is giving me "License error: -9,57" so I guess it didn't work... I will try to just make the model on the 2020b just so I have something.

-----------------------------------------------------------------------------------------------------------------------------

I was able to fix the problem with the activation (using this).

I can now open the x1sussim.mdl file. It is trying to access a BSFM_MASTER Library that it doesn't seem to have. the other files don't seem to have any warnings though. 

the simple suspension model can be found in home/controls/docker-cymac/userapps/c1simpsus.mdl on the docker system. This is where I will put my model. (right now it is just a copied file)

Also using Simulink on the docker is very slow. I think this is either a limit of the x2goclient software or the hardware that the docker is running on.

Except for the feed-throughs that require a DB9-M connector I finished wiring and labeling the Acromag, following Jon's updated wiring plan.

We can start testing the differential inputs until the missing connectors arrive.

So I am stuck on how to add the control block to my model. I am trying to make it as simple as possible with just a simple transfer function for a damped harmonic oscillator and then the control block (see overview.png). 

The transfer function I am using is:

For future generations: To measure the transfer function (to verify that it is doing what it says it is) I am using the discrete transfer function estimator block. To get a quick transfer function estimator Simulink program run ex_discrete_transfer_function_estimator in the Matlab command line. This works well for filters but it was hit or miss using the discrete transfer function. 

The roadblock I am running into right now is that I can't figure out how to add the controller to the model. Not on an interpretation level but in a very straightforward "can I drag it into my model and will it just work" kind of way.

I am also a little confused as to exactly which block would do the controlling. Because I want to just use the x of the pendulum (its position) I think I want to use the suspension controls which come are connected to in the suspension plant model. But where exactly is it and how can I get the model? I can't seem to find it. 

The controller would be in the c1sus model, and connects to the c1sup plant model. So the controller doesn't go in the plant model.

Both the controller and the plant can be modeled using a single filter module in each separate model as you've drawn, but they go in separate models.

I have attached the framework that I am using for the full system. Plantframework.pdf has the important aspects that I will be changed. Right now I am trying to keep it mostly as is, but I have disconnected the Optic Force Noise and hope to disconnect the Suspension Position Noise. The Optic Force Noise Block is additive to the signal so eliminating it from the system should make it less realistic but simpler. It can be added back easily by reconnecting it.

The next step is adding my plant response, which is simply the transfer function and measurement from the last post. These should be inserted in the place of the red TM_RESP in the model.

The TM_RESP block takes in a vector of dimension 12 and returns a vector of dimension 6. The confusing part is that the block does not seem to do anything. it simply passes the vector through with no changes. I'm not sure why this is the case and I am looking for documentation to explain it but haven't found anything. As to how a 12 vector turns into a 6 vector I am also lost. I will probably just disconnect everything but the x position.

I tried to just throw in my model (see Simple_Plant.pdf) and see what happened but the model would not let me add built-in blocks to the model. This is weird because all the blocks that I am adding are part of the basic library. My guess is that this mode will only accept blocks from the CDL library. I will either need to change my blocks to be made from blocks in the CDL library or maybe I can pass the signal out of the plant framework model then into my model then back to the plant framework model. I think this is just a Matlab thing that I don't know about yet. (Jon probably knows)

I have also attached an image of the controls model for reference. It looks like a mess but I'm sure there is a method. I won't get lost in going through it just assume it works... for now.

The next question I have been asking is how do I show that the system works. When anchal and I made a python simulation of the system, we tested it by seeing the evolution of the degrees of freedom over time given some initial conditions. We could see the pendulum propagating and some of the coupling between the DOFs. This is a fast and dirty way to check if everything is working and should be easy to add. I simply recorded the POS signal and graph it over time. Once we get to a state-space model we can test it by taking the transfer function, but since our plant is literally already just a transfer function there isn't much of a point yet.

Also, I need to add color to my Simple_Plant.pdf model because it looks really boring :(

1. I suggest not naming this the LSC model, since it has no LSC stuff.
2. Also remove all the diagnostic stuff in the plant model. We need nothing except a generic filter Module, like in the SUS controller.
3. Also, the TF looks kind of weird to me. I would like to see how you derive that eq.
4. Connect the models and show us some plots of the behavior in physical units using FOTON to make the filters and diaggui/DTT (at first) to make the plots.

The first version of the single filter plant is below. Jon and I went through compiling a model and running it on the docker (see this post)

We activated an early version of the plant model (from about 10 years ago) but this model was not designed to run on its own so we had to ground lots of unconnected pieces. the model compiled and ran so we moved on to the x1sus_single_plant model that I prepared.

This model is shown in the first attachment wasn't made to be run alone because it is technically a locked library (see the lock in the bottom left). It is supposed to be referenced by another file: x1sup.mdl (see the second attachment). This works great in the Simulink framework. I add the x1sus_single_plant model to the path and Matlab automatically attaches the two. but the docker does not seem to be able to combine the two. Starting the cymac it gives these errors:

I have tried putting the x1sus_single_plant.mdl file everywhere as well as physically dragging the blocks that I need into the x1sup.mdl file but it always seems to throw an error. Basically, I want to combine them into one file that is not referencing anything other than the CDS library but I cant figure out how to combine them.

Okay but the next problem is the medm screen generation. When we had the original 2010 model running the sitemap did not include it. It included models that weren't even running before but not the model Jon and I had added. I think this is because the other models that were not running had medm screens made for them. I need to figure out how to generate those screens. I need to figure out how to use the tool Chris made to auto-generate medm screens from Simulink but I can't seem to figure it out. And honestly, it won't be much use to me until I can actually connect the plant block to its framework. One option is to just copy each piece over one by one. this will take forever but at this point, I am frustrated enough to try it. I'll try to give another update later tonight.

We took a Supermicro from the lab (along with a keyboard, a mouse, and a screen taken from a table on the Y arm) and placed it near the Acromag chassis.

We installed Debian 10 on the machine. I followed the steps on the slow machine wiki for setting up the host machine. Some steps had to be updated. Most importantly, in the new Debian, the network interfaces are given random names like enp3s0 and enp4s0 instead of eth0 and eth1. I updated the wiki accordingly.

To operate the chassis using one 15V source I disconnected the +24V cable from the Acromag units and jumpered the +15V wire into the power input instead. I started up the Acromags. They draw 0.7A. I connected an Ethernet cable to the front interface. I checked that all the Acromags are connected to the local network of the host machine by pinging them one by one.

Summary

Yesterday I unpacked and installed the three 18-bit DAC cards received from Hanford. I then repeated the low-level PCIe testing outlined in T1900700, which is expanded upon below. I did not make it to DAC-ADC loopback testing because these tests in fact revealed a problem with the new hardware. After a combinatorial investigation that involved swapping cards around between known-to-be-working PCIe slots, I determined that one of the three 18-bit DAC cards is bad. Although its "voltage present" LED illuminates, the card is not detected by the host in either I/O chassis.

I installed one of the two working DACs in the c1bhd chassis. This now 100% completes this system. I installed the other DAC in the c1sus2 chassis, which still requires four more 18-bit DACs. Lastly, I reran the PCIe tests for the final configurations of both chassis.

PCIe Card Detection Tests

For future reference, below is the set of command line tests to verify proper detection and initialization of ADC/DAC/BIO cards in I/O chassis. This summarizes the procedure described in T1900700 and also adds the tests for 18-bit DAC and 32-channel BO cards, which are not included in the original document.

Each command should be executed on the host machine with the I/O chassis powered on:

where xxxx is a four-digit device code given in the following table.

The command will return a two-line entry for each PCIe device of the specified type that is detected. For example, on a system with a single ADC this command should return:

To add the required library: put the .mdl file that contains the library into the userapps/lib folder. That will allow it to compile correctly

I got these errors:

I removed all IPC parts (as seen in Attachment 1) and that did the trick. IPC parts (Inter-Process Communication) were how this model was linked to the controller so I don't know how exactly how I can link them now. 

I also went through the model and grounded all un-attached inputs and outputs. Now the model compiles

Also, The computer seems to be running very slowly in the past 24 hours. I know Jon was working on it so I'm wondering if that had any impact. I think it has to do with the connection speed because I am connected through X2goclient. And one thing that has probably been said before but I want to note again is that you don't need a campus VPN to access the docker.

The problem here was that the RFM errors cropped up again - seems like it started ~4am today morning judging by TRX trends. Of course without the triggering signal the arm cavity couldn't lock. I rebooted everything (since just restarting the rfm senders/receivers did not do the trick), now arm locking works fine again. It's a bit disappointing that the Rogue Master setting did not eliminate this problem completely, but oh well...

It's kind of cool that in this trend view of the TRX signal, you can see the drift of the ETMX suspension. The days are getting hot again and the temp at EX can fluctuate by >12C between day and night (so the "air-conditioning" doesn't condition that much I guess 😂 ), and I think that's what drives the drift (idk what the transfer function to the inside of the vacuum chamber is but such a large swing isn't great in any case). Not plotted here but i hypothesize TRY levels will be more constant over the day (modulo TT drift which affects both arms).

The IMC suspension team should double check their filters are on again. I am not familiar with the settings and I don't think they've been added to the SDF.

I installed the EPICs base, asyn and modbus modules according to Jon's instructions.

Since the modbus configurations files were already writtten for c1auxey1 (see elog 15292) the only thing I did was to change the IP addresses in ETMYaux.cmd to match the actual assigned IPs.

I followed the rest of the instructions as written.

The modbus service was activated succesfully.

The only thing left to do is to change ETMYaux.db to reflect to new channels that were added. I believe these are BI channels named C1:SUS-ETMY_xx_ENABLEMon.

The other day I felt hot at the X end. I wondered if the Xend A/C was off, but the switch right next to the SP table was ON (green light).
I could not confirm if the A/C was actually blowing or not.

I double checked today and the F2A filters in the output matrices of MC1, MC2 and MC3 in the POS column are ON. I do not get what SDF means? Did we need to add these filters elsewhere?

We received a stock of DB9 male feed-through connectors. That allowed me to complete the remaining wiring on the c1auxey Acromag chassis. The only thing left to be done is the splicing to the RTS.