To debug a problem with the new c1psl (later elog), we needed a Supermicro EPICS server that was using the shared EPICS/modbus/asyn binaries rather than a local install. Of those available in the lab (c1iscaux, c1vac, c1susaux being the others), this was the only one which uses the shared install. So I 

• turned the slow bias voltages to 0
• shutdown the watchdog
• disconnected the Acromag crate in 1X9 from the 192.168.114.xxx subnet at the supermicro end
• connected a test ADC to the local subnet using a different ethernet cable (leaving the original one dangling)
• ran some software tests to see if we could open up a communication line to the test ADC using modbus without any errors being thrown
• removed the test ADC and restored the ethernet connection.

At which point Jon reset the software end, I restored the slow bias voltage and re-enabled the local damping. The optic seems to have damped okay. The Oplev spot is back in ~center of the QPD and the green beam can be locked to a TEM00 mode (so the alignment is okay - the IR beam is unavailable while c1psl issues are being sorted but I judge that things are back to the nominal state now).

I have made a wiring + channel list that need to be included in the new C!AUXEY Acromag.

It was mostly copied from C1AUXEX. 

I ignored the IPANG channels since it is going to be removed from the table.

I'd like to re-measure the transfer function from driving MC2 position to the MC_L_DQ channel (for feedforward purposes). Swept sine would be one option, but I can't get the "Envelope" feature of DTT to work, the excitation amplitude isn't getting scaled as specified in the envelope, and so I'm unable to make the measurement near 1 Hz (which is where the FF is effective). I see some scattered mentions of such an issue in past elogs but no mention of a fix (I also feel like I have gotten the envelope function to work for some other loop measurement templates). So then I thought I'd try broadband noise injection, since that seems to have been the approach followed in the past. Again, the noise injection needs to be shaped around ~1 Hz to avoid knocking the IMC out of lock, but I can't get Foton to do shaped noise injections because it doesn't inherit the sample rate when launched from inside DTT/awggui - this is not a new issue, does anyone know the fix?

Note that we are using the gds2.15 install of foton, but the pre-packaged foton that comes with the SL7 installation doesn't work either.

Update:

The envelope feature for swept-sine wasn't working because i specified the frequency grid in the wrong order apparently. Eric von Reis has been notified to include a sorting algorithm in future DTT so that this can be in arbitrary order. fixing that allows me to run a swept sine with enveloped excitation amplitude and hence get the TF I want, but still no shaped noise injections via foton 😢 

do you really mean awggui cannot make shaped noise injections via its foton text box ? That has always worked for me in the past.

If this is broken I'm suspicious there's been some package installs to the shared dirs by someone.

The problem is that foton does not inherit the model sample rate when launched from DTT/awggui. This is likely some shared/linked/dynamic library issue, the binaries we are running are precompiled presumably for some other OS. I've never gotten this to work since we changed to SL7 (but I did use it successfully in 2017 with the Ubuntu12 install).

I used Yehonathan's wiring assignments to lay the rest of groundwork for the final slow controls machine upgrade, c1auxey. Actions completed:

• Created an internal wiring diagram for assembling the Acromag chassis (log in with LIGO.ORG credentials to view/edit)
• Created a new target directory on the network drive:

The "1" will be dropped after the new system is permanently installed.

• Populated the target directory with files:
  • modbusIOC.service - wraps the EPICS IOC as a systemd service
  • ETMYaux.env - defines the EPICS environment variables
  • ETMYaux.cmd - command file to set up the EPICS IOC
  • ETMYaux.sh - enables DAC outputs to the suspension (executed lastly)
• Created the EPICS channel databases:
  • ETMYaux.db - migration of the existing database
  • c1auxey_state.db - contains logic for loopback monitoring of the IOC "alive" state (visible from Sitemap > CDS > Slow Controls Status)

Hardware-wise, this system will require:

• 2 Acromag XT-1221 units (ADC)
• 1 Acromag XT-1541 unit (DAC)
• 1 Acromag XT-1111 unit (sinking BIO)

I know that we do have these quantities left on hand. The next steps are to set up the Supermicro host and begin assembling the Acromag chassis. Both of these activities require an in-person presence, so I think this is as far as we can advance this project for now.

We want to migrate the end shutter controls from c1aux to the end acromags. Could you include them to the list if not yet?

This will let us remove c1aux from the rack, I believe.

Yehonathan's list does include C1:AUX-GREEN_Y_Shutter and I copied its definition from /cvs/cds/caltech/target/c1aux/ShutterInterlock.db into the new ETMYaux.db file.

I noticed ShutterInterlock.db still contains about a dozen channels. Some of them appear to be ghosts (like the C1:AUX-PSL_Shutter[...] set, which has since become C1:PSL-PSL_Shutter[...] hosted on c1psl) but others like C1:AUX-GREEN_X_Shutter appear to still be in active use.

Summary:

Around 5pm local time, the three vertex FEs crashed. AFAIK, no one was in the lab or working on anything CDS related, so this is worrying.

Details:

• Reboot script was used to bring all FEs back - only soft reboots were required.
• The IMC and arms can now be locked.
• I think combination of burt + SDF would have reverted all the settings as they should be, but if something appears off, it could be that some EPICS value didn't get reset correctly.

The machine needed a hard reboot as it was un-ssh-able. 

The exact time that the machine went down is unknown because the blinkys were not DQ-ed. I've now added these to the EDCU to make these channels actually useful, and we may look back on the reliability (or otherwise) of the Acromag system. To my memory, this is the ~5th time one of the new Acromag servers has needed a hard reboot. While this may be less frequent (?) than the VME machines, perhaps there is some other reason for these dropouts. Maybe something to do with the martian network?

Anyway the machine is back up and running now.

Here is the procedure for setting up the three new BHD front-ends (c1bhd, c1sus2, c1ioo - replacement). This plan is based on technical advice from Rolf Bork and Keith Thorne.

The overall topology for each machine is shown here. As all our existing front-ends use (obsolete) Dolphin PCIe Gen1 cards for IPC, we have elected to re-use Dolphin Gen1 cards removed from the sites. Different PCIe generations of Dolphin cards cannot be mixed, so the only alternative would be to upgrade every 40m machine. However the drivers for these Gen1 Dolphin cards were last updated in 2016. Consequently, they do not support the latest Linux kernel (4.x) which forces us to install a near-obsolete OS for compatibility (Debian 8).

Hardware

• Servers: SuperMicro 5018R-M [RECEIVED]
• IPC cards: Dolphin DXH510-A0 (PCIe x4 Gen1) [LLO will provide; I've asked Keith Thorne to ship them]
• I/O cards: One Stop Systems OSS-PCIe-HIB25-x4-H-H (PCIe x4 Gen2) [Rich Abbott will provide; part of the I/O chassis hardware we are receiving from aLIGO]
• I/O cables: One Stop Systems OSS-PCIe-CBL-x4-3M (PCIe x4 3m) [RECEIVED]
• IPC cables: Data Storage Cables C5252-40ME-O (fiber optic 40m) [RECEIVED]

Software

• OS: Debian 8.11 (Linux kernel 3.16)
• IPC card driver: Dolphin DX 4.4.5 [works only with Linux kernel 2.6 to 3.x]
• I/O card driver: None required, per the manual

Install Procedure

1. Follow Keith Thorne's procedure for setting up Debian 8 front-ends
2. Apply the real-time kernel patches developed for Debian 9, but modified for kernel 3.16 [these are UNTESTED against Debian 8; Keith thinks they may work, but they weren't discovered until after the Debian 9 upgrade]
3. Install the PCIe expansion cards and Dolphin DX driver (driver installation procedure)

See Attachment #1. J8 was connected to a "LASTI timing slave" sitting in the rack that Chiara lives in - we don't use this for anything and I confirmed that there was no effect on the RTCDS when I pulled that fiber out. The LASTI timing slave also had a blinky that was blinking when the fiber was plugged in - which I take to believe that the slot works. 

Can we get away with just using these two available slots, J8 and J13? Do we really need three new expansion chassis?

I believe we will use two new chassis at most. We'll replace c1ioo from Sun to Supermicro, but we recycle the existing timing system.

That's great. I wonder if we can also get away with not adding new Dolphin infrastructure. I'd really like to avoid changing any IPC drivers.

The new dolphin eventually helps us. But the installation is an invasive change to the existing system and should be done at the installation stage of the 40m BHD.

I added the EPCIS channels for the c1omc model (gains, matrix elements etc) to the autoburt such that we have a record of these, since we expect these models to be running somewhat regularly now, and I also expect many CDS crashes.

That's great news we won't have to worry about a new timing fanout for the two new machines, c1bhd and c1sus2. And there's no plan to change Dolphin IPC drivers. The plan is only to install the same (older) version of the driver on the two new machines and plug into free slots in the existing switch.

I edited /diskless/root.jessie/home/controls/.bashrc so that I don't have to keep doing this every time I do a model recompile.

I had to make a CDS change to the c1lsc model in an effort to get a few more signals into the models. Rather than risk requiring hard reboots (typcially my experience if I try to restart a model), I opted for the more deterministic scripted reboot, at the expense of spending ~20mins to get everything back up and running.

Update 2230: this was more complicated than expected - a nuclear reboot was necessary but now everything is back online and functioning as expected. While all the CDS indicators were green when I wrote this up at ~1800, the c1sus model was having frequent CPU overflows (execution time > 60 us). Not sure why this happened, or why a hard power reboot of everything fixed it, but I'm not delving into this.

The point of all this was that I can now simultaneously digitize 4 channels - 2 DCPDs, and 2 demodulated quadratures of an RF signal.

Attachment #1 shows that the c1rfm model isn't able to receive any signals from the front end machines at EX and EY. Attachment #2 shows that the problem appears to have started at ~430am today morning - I certainly wasn't doing anything with the IFO at that time.

I don't know what kind of error this is - what does it mean that the receiving model shows errors but the sender shows no errors? It is not a new kind of error, and the solution in the past has been a series of model reboots, but it'd be nice if we could fix such issues because it eats up a lot of time to reboot all the vertex machines. There is no diagnostic information available in all the places I looked. I'll ask the CDS group for help, but I'm not sure if they'll have anything useful since this RFM technology has been retired at the sites (?).

In the meantime, arm cavity locking in the usual way isn't possible since we don't have the trigger signals from the arm cavity transmission. 

Update 1500 4 Oct: soft reboots of models didn't do the trick so I had to resort to hard reboots of all FEs/expansion chassis. Now the signals seem to be okay.

I'm starting the model restarts from remote. Then later I'll show up in the lab to do more hard resets.
==> It seems that the RFM errors are gone. Here are the steps.

1. Shutdown all the watchdogs
2. login to c1iscex. Shutdown all the realtime models: rtcds kill --all
3. login to c1iscey. Shutdown all the realtime models: rtcds kill --all
4. run scripts/cds/rebootC1LSC.sh on pianosa
5. reboot c1iscex
6. reboot c1isxey
7. Wait until all the machines/models are up by the script
8. restart c1iscex models
9. restart c1iscey models
10. some IPC errors are still visible on the CDS status screen. Lauch c1daf and c1oaf

I am working on the setup of a CDS FE, so please do not attempt any remote login to the IPMI interface of c1bhd until I'm done.

I was able to boot one of the 3 new Supermicro machines, which I christened c1bhd, in a diskless way (with the boot image hosted on fb, as is the case for all the other realtime FEs in the lab). This is just a first test, but it is reassuring that we can get this custom linux kernel to boot on the new hardware. Some errors about dolphin drivers are thrown at startup but this is to be expected since the server isn't connected to the dolphin network yet. We have the Dolphin adaptor card in hand, but since we have to get another PCIe card (supposedly from LLO according to the BHD spreadsheet), I defer installing this in the server chassis until we have all the necessary hardware on hand.

I also have to figure out the correct BIOS settings for this to really run effectively as a FE (we have to disable all the "un-necessary" system level services) - these machines have BIOS v3.2 as opposed to the older vintages for which there are instructions from K.T. et al.

There may yet be issues with drivers, but this is all the testing that can be done without getting an expansion chassis. After the vent and recovering the IFO, I may try experimenting with the c1ioo chassis, but I'd much prefer if we can do the testing offline on a subnet that doesn't mess with the regular IFO operation (until we need to test the IPC).

As discussed at the meeting, I commenced the recovery of the CDS status at 1750 local time.

• Started by attempting to just soft-restart the c1rfm model and see if that fixes the issue. It didn't and what's more, took down the c1sus machine.
• So hard reboots of the vertex machines was required. c1iscey also crashed. I was able to keep the EX machine up, but I soft-stopped all the RT models on it.
• All systems were recovered by 1815. For anyone checking, the DC light on the c1oaf model is red - this is a "known" issue and requires a model restart, but i don't want to get into that now and it doesn't disrupt normal operation.

Single arm POX/POY locking was checked, but not much more. Our IMC WFS are still out of service so I hand aligned the IMC a bit, IMC REFL DC went from ~0.3 to ~0.12, which is the usual nominal level.

I suspect what happened here is that the IP didn't get updated when we went from the 131.215.113.xxx system to 192.168.113.xxx system. I fixed it now and can access the web interface. This system is now ready for remote debugging (from inside the martian network obviously). The IP is 192.168.113.90.

Managed to pull this operation off without crashing the RFM network, phew.

BTW, a windows laptop that used to be in the VEA (I last remember it being on the table near MC2 which was cleared sometime to hold the spare suspensions) is missing. Anyone know where this is ?

As I was working on the IFO re-alignment just now, the rfm errors popped up again. I don't see any useful diagnostics on the web interface.

Do we want to take this opportunity to configure jumpers and set up the rogue master as Rolf suggested? Of course there's no guarantee that will fix anything, and may possibly make it impossible to recover the current state...

Attached is the layout for the "intermediate" CDS upgrade option, as was discussed on Wednesday. Under this plan:

• Existing FEs stay where they are (they are not moved to a single rack)

• Dolphin IPC remains PCIe Gen 1

• RFM network is entirely replaced with Dolphin IPC

Please send me any omissions or corrections to the layout.

I just want to point out that if you move all the FEs to the same rack they can all be connected to the Dolphin switch via copper, and you would only have to string a single fiber to every IO rack, rather than the multiple now (for network, dolphin, timing, etc.).

The CDS model change required to get the AS WFS signals into the RTCDS system are rather invasive.

• We use VCS for these models. Linus Torvald may question my taste but I also made local backups of the models, just in case...
• Particularly, the ADC1 card on c1ioo is completely reconfigured.
• I also think it's more natural to do all the ASC computations on this machine rather than the c1lsc machine (it is currently done in the c1ass model). So there are many IPC changes as well.
• I have documented everything carefully, and the compile/install went smoothly.
• Taking down all the FE servers at 1830 local time. 
  1. To propagagte the model changes
  2. To make a hardware change in the c1rfm card in the c1ioo machine to configure it as "ROGUE MASTER 0"
  3. To clear the RFM errors we are currently suffering from will require a model reboot anyways.
• Recovery was completed by 1930 - the RFM errors are also cleared, and now we have a "ROGUE MASTER 👾" on the network. Pretty smooth, no major issues with the CDS part of the procedure to report.
• The main issue is that in the AA chassis I built, Ch14 (with the first channel as Ch1) has the output saturated to 28V (differential). I'm not sure what kind of overvoltage protection the ADC has - we frequently have the inputs exceed the spec'd +/-20 V (e.g. when the whitening filters are engaged and the cavity is fringing), but pending further investigation, I am removing the SCSI connection from the rear of the AA chassis.

In terms of computational load, the c1ioo model seems to be able to handle the extra load no issues - ~35us/60us per cycle. The RFM model shows no extra computational time. 

After this work, the IMC locking and POX/POY locking, and dither alignment servos are working okay. So I have some confidence that my invasive work hasn't completely destroyed everything. There is some hardware around the rear of 1X2 that I will clear tomorrow.

Koji fixed the problematic channel - the issue was a bad solder joint on the input resistors to the THS4131. The board was re-installed. I also made a custom 2x4-pin LEMO-->DB9 cable, so we are now recording the PMC and FSS ERR/CTRL channel diagnostics again (spectra tomorrow). Note that Ch32 is recording some sort of DuoTone signal and so is not usable. This is due to a misconfiguration - ADC0 CH31 is the one which is supposed to be reserved for this timing signal, and not ADC1 as we currently have. When we swap the c1ioo hosts, we should fix this issue.

I also did most of the work to make the MEDM screens for the revised ASC topology, tried to mirror the site screens where possible. The overview screen remains to be done. I also loaded the anti-whitening filters (z:p 150:15) at the demodulated WFS input signal entry points. We don't have remote whitening switching capability at this time, so I'll test the switching manually at some point.

Summary:

1. There appears to be insufficient number of PCIe slots in the new Supermicro servers that were bought for the BHD upgrade.
2. Modulo a "riser card", we have all the parts in hand to put one of the end machines on the Dolphin network. If the Rogue Master doesn't improve the situation, we should consider installing a Dolphin card in the c1iscex server and connecting it to the Dolphin network at the next opportunity. 

Details:

Last night, I briefly spoke with Koji about the CDS upgrade plan. This is a follow up.

Each server needs a minimum of two peripheral devices added to the PCIe bus:

• A PCIe interface card that connects the server to the Expansion Chassis (copper or optical fiber depending on distance between the two).
• A Dolphin or RFM card that makes the IPC interconnects. 
• I'm pretty certain the expansion chasiss cannot support the Dolphin / RFM card (it's only meant to be for ADCs/DACs/BIO cards). At least, all the existing servers in the 40m have at least 2 PCIe cards installed, and I think we have enough to worry about without trying to engineer a hack.
• I attach some photos of new and old Supermicro servers to illustrate my point, see Attachment #1. 

As for the second issue, the main question is if we had an open PCIe slot on the c1iscex machine to install a Dolphin card. Looks like the 2 standard slots are taken (see Attachment #1), but a "low profile" slot is available. I can't find what the exact models of the Supermicro servers installed back in 2010 are, but maybe it's this one? It's a good match visually anyways. The manual says a "riser card" is required. I don't know if such a riser is already installed. 

Questions I have, Rolf is probably the best person to answer:

1. Can we even use the specified host adaptor, HIB25-X4, which is "PCIe Gen2", with the "PCIe Gen3" slots on the new Supermicro servers we bought? Anyway, the fact that the new servers have only 1 PCIe slot probably makes this a moot point.
2. Can we overcome this slot limitation by installing the Dolphin / RFM card in the expansion chassis?
3. In the short run (i.e. if it is much faster than the full CDS shipment we are going to receive), can we get (from the CDS test stand in Downs or the site) 
   • A riser card so that we may install the Gen 1 Dolphin card (which we have in hand) in the c1iscex server?
   • A compatible (not sure what PCIe generation we need) PCIe host to ECA kit so we can test out the replacement for the Sun Microsystems c1ioo server?
   • A spare RFM card (VMIC 5565, also for the above purpose). 
4. What sort of a test should we run to test the new Dolphin connection? Make a "null channel" differencing the same signal (e.g. TRX) sent via RFM and Dolphin? Or is there some better checksum diagnostic?

1. The diagram should clearly show the host machines and the expansion chassis and the interconnects between them.
2. We no longer have any Gentoo bootserver or diskless FEs.
3. The "c1lsc" host is in 1X4 not 1Y3.
4. The connection between c1lsc and Dolphin switch is copper not fiber. I don't know how many Gbps it is. But if the switch is 10 Gbps, are they really selling interface cables that have lower speed? The datasheet says 10 Gbps.
5. The control room workstations - Debian10 (rossa) is the way forward I believe. it is true pianosa remains SL7 (and we should continue to keep it so until all other machines have been upgraded and tested on Debian 10).
6. There is no "IOO/OAF". The host is called "c1ioo".
7. The interconnect between Dolphin switch and c1ioo host is via fiber not copper.
8. It'd be good to have an accurate diagram of the current situation as well (with the RFM network).
9. I'm not sure if the 1Y1 rack can accommodate 2 FEs and 2 expansion chassis. Maybe if we clear everything else there out...
10. There are 2 "2GB/s" Copper traces. I think the legend should make clear what's going on - i.e. which cables are ethernet (Cat 6? Cat 5? What's the speed limitation? The cable? Or the switch?), which are PCIe cables etc etc. 

I don't have omnigraffle - what about uploading the source doc in a format that the excellent (and free) draw.io can handle? I think we can do a much better job of making this diagram reflect reality. There should also be a corresponding diagram for the Acromag system (but that doesn't have to be tied to this task). Megatron (scripts machine) and nodus should be added to that diagram as well.

I used an Acromag XT1221 in CTN to play around with different wiring and see what works.  Following are my findings:

Referenced Single Ended Source (Attachment 1):

• If the source signal is referenced single ended, i.e. the signal is only on the positive output and the negative side is tied to GND on the source side AND this GND is also shared by the power supply powering the Acromag, then no additional wiring is required.
• The GND common to the power supply and the source is not required to be Earth GND but if done so, it should be at one point only.
• RTN terminal on Acromag can be left floating or tied to IN- terminal.

Floating Single Ended Source (Attachment 2):

• If the source signal is floating single-ended i.e. the signal is only on the positive output and the negative output is a floating GND on the source, the the IN- should be connected to RTN.
• This is the case for handheld calibrators or battery powered devices.
• Note that there is no need to connect GND of power supply to the floating GND on the source.

Differential Source (Attachment 3):

• If the source is differential output i.e. the signal is on both the positive output and the negative output, then connect one of the RTN terminals on Acromag to Earth GND. It has to be Earth GND for this to work.
• Note that you can no longer tie the IN- of different signals to RTN as they are all carrying different negative output from the source.
• Earth GND at RTN gives acromag a stable voltage reference to measure against the signals coming in IN+ and IN-. And the most stable voltage reference is of course Earth GND.

Conclusion:

• We might have a mix of these three types of signals coming to a single Acromag box. In that case, we have to make sure we are not connecting the different IN- to each other (maybe through RTN) as the differential negative inputs carry signal, not a constant voltage value.
• In this case, I think it would be fine to always use differential signal wiring diagram with the RTN  connected to Earth GND.
• There's no difference in software configuration for the two types of inputs, differential or single-ended.
• For cases in which we install the acromag box inside a rack mount chasis with an associated board (example: CTN/2248), it is ok and maybe the best to use the Attachment 1 wiring diagram.

Comments and suggestions are welcome.

Related elog posts:

40m/14841    40m/15134

Edit Tue Jan 26 12:44:19 2021 :

Note that the third wiring diagram mentioned actually does not work. It is an error in judgement. See 40m/15762 for seeing what happens during this.

Thanks for the systematic effort.

1. Can you please post some time domain plots (ndscope perferably or StripTool) to clearly show the different failure modes?
2. The majority of our AI channels are "Referenced Single Ended Source" in your terminology. At least on the c1psl Acromag crate, there are no channels that are truly differential drive (case #3 in your terminology). I think the point is that we saw noisy inputs when the IN- wasn't connected to RTN. e.g. the thorlabs photodiode has a BNC output with the shield connected to GND and the central conductor carrying a signal, and that channel was noisy when the RTN was unconnected. Is that consistent with your findings?
3. What is the prescription when we have multiple power supplies (mixture of Sorensens in multiple racks, Thorlabs photodiodes and other devices powered by an AC/DC converter) involved?
4. I'm still not entirely convinced of what the solution is, or that this is the whole picture. On 8 Jan, I disconnected (and then re-connected) the FSS RMTEMP sensor from the Acromag box, to check if the sensor output was noisy or if it was the Acromag. The problem surfaced on Dec 15, when I installed some new electronics in the rack (though none of them were connected to the Acromag directly, the only common point was the Sorensen DCPS. And between 8 Jan and today, the noise RMS has decreased back to the nominal level, without me doing anything to the grounding. How to reconcile this?

I'm working on a better wiring diagram that takes into account multiple power supplies, how their GND is passed forward to the circuits or sensors using those power supplies and what possible wiring configurations on Acromag would give low noise. I think I have two configurations in mind which I will test and update here with data and better diagrams.

I took some striptool images earlier yesterday. So I'm dumping them here for further comments or inferences.

I picked the boxes up this morning. The inventory per Fil's email looks accurate. Some comments:

1. They shipped the chassis and mounting parts (we should still get rails to mount these on, they're pretty heavy to just be supported on 4 rack nuts on the front). idk if we still need the two empty chassis that were requested from Rich.
2. Regarding the fibers - one of the fibers is pre-2012. These are known to fail (according to Rolf). One of the two that LHO shipped is from 2012 (judging by S/N, I can't find an online lookup for the serial number), the other is 2011. IIRC, Rolf offered us some fibers so we may want to take him up on that. We may also be able to use copper cables if the distances b/w server and expansion chassis are short.
3. The units are fitted with a +24V DC input power connector and not the AC power supplies that we have on all the rest of the chassis. This is probably just gonna be a matter of convenience, whether we want to stick to this scheme or revert to the AC input adaptor we have on all the other units. idk what the current draw will be from the Sorensen - I tested that the boards get power, and with noi ADCs/DACs/BIOs, the chassis draws ~1A (read off from DCPS display, not measured with a DMM). ~Half of this is for the cooling fans It seems like KT @ LLO has offered to ship AC power supplies so maybe we want to take them up on that offer.
4. Without the host side OSSI PCIe card, timing interface board, and supermicro servers that actually have enough PCIe slots, we still can't actually run any meaningful test. I ran just a basic diagnostic that the chassis can be powered on, and the indicator LEDs and cooling fans run.
5. Some photos of the contents are here. The units are stored along the east arm pending installation.

    >     Koji,
    >
    >     Barebones on this order.
    >
    >        1. Main PCIe board
    >        2. Backplane (Interface board)
    >        3. Power Board
    >        4. Fiber (One Stop) Interface Card, chassis side only
    >        5. Two One Stop Fibers
    >        6. No Timing Interface
    >        7. No Binary Cards.
    >        8. No ADC or DAC cards
    >
    >     Fil Clara
    >       

That's fine, we didn't actually request those. We bought and already have in hand new PCIe x4 cables for the chassis-host connection. They're 3 m copper cables, which was based on the assumption of the time that host and chassis would be installed in the same rack.

I think the "Rogue Master" setting on the RFM network may be doing some good. 5 mins, ago, all the CDS indicators were green, but I noticed an amber light on the c1rfm screen just now (amber = warning). Seems like at GPS time 1294691182, there was some kind of error on the RFM network. But the network hasn't gone down. I can clear the amber flag by running the global diag reset. Nevertheless, the upgrade of all RT systems to Dolphin should not be de-prioritized I think.

Koji asked me assemble a detailed breakdown of the parts received from LHO, which I do based on the high-res photos that Gautam posted of the shipment.

Parts in hand:

Parts still needed:

Issue with PCIe slots in new FEs

Also, I looked into the mix-up regarding the number of PCIe slots in the new Supermicro servers. The motherboard actually has six PCIe slots and is on the CDS list of boards known to be compatible. The mistake (mine) was in selecting a low-profile (1U) chassis that only exposes one of these slots. But at least it's not a fundamental limitation.

One option is to install an external PCIe expansion chassis that would be rack-mounted right above the FE. It is automatically configured by the system BIOS, so doesn't require any special drivers. It also supports hot-swapping of PCIe cards. There are also cheap ribbon-cable riser cards that would allow more cards to be connected for testing, although this is not as great for permanent mounting.

It may still be better to use the machines offered by Keith Thorne from LLO, as they're more powerful anyway. But if there is going to be an extended delay before those can be received, we should be able to use the machines we already have in conjunction with one of these PCIe expansion options.

Can you please provide a link to this "list of boards"? The only document I can find is T1800302. In that, under "Basic Requirements" (before considering specific motherboards), it is specified that the processor should be clocked @ >3GHz. The 3 new supermicros we have are clocked at 1.7 GHz. X10SRi-F boards are used according to that doc, but the processor is clocked at 3.6 or 3.2 GHz.

Please also confirm that there are no conflicts w.r.t. the generation of PCIe slots, and the interfaces (Dolphin, OSSI) we are planning to use - the new machines we have are "PCIe 2.0" (though i have no idea if this is the same as Gen 2). 

As for the CX4 cable - I still think it's good to have these on hand. Not good to be in a situation later where FE and expansion chassis have to be in different racks, and the copper cable can't be used.

Indeed T1800302 is the document I was alluding to, but I completely missed the statement about >3 GHz speed. There is an option for 3.4 GHz processors on the X10SRi-F board, but back in 2019 I chose against it because it would double the cost of the systems. At the time I thought I had saved us $5k. Hopefully we can get the LLO machines in the near term---but if not, I wonder if it's worth testing one of these to see whether the performance is tolerable.

I confirm that PCIe 2.0 motherboards are backwards compatible with PCIe 1.x cards, so there's no hardware issue. My main concern is whether the obsolete Dolphin drivers (requiring linux kernel <=3.x) will work on a new system, albeit one running Debian 8. The OSS PCIe card is automatically configured by the BIOS, so no external drivers are required for that one.

I've produced updated diagrams of the CDS layout, taking the comments in 15476 into account. I've also converted the 40m's diagrams from Omnigraffle ($150/license) to the free, cloud-based platform draw.io. I had never heard of draw.io, but I found that it has most all the same functionality. It also integrates nicely with Google Drive.

Attachment 1: The planned CDS upgrade (2 new FEs, fully replace RFM network with Gen 1 Dolphin IPC)
Attachment 2: The current 40m CDS topology

The most up-to-date diagrams are hosted at the following links:

• Planned 40m CDS topology
• Current 40m CDS topology

Please send me any further corrections or omissions. Anyone logged in with LIGO.ORG credentials can also directly edit the diagrams.

Not sure if 1Y1 can accommodate both c1sus2 and c1bhd as well as the various electronics chassis that will have to be installed. There may need to be some distribution between 1Y1 and 1Y3. Does Koji's new wiring also specify which racks hold which chassis?

Some minor improvements to the diagram:

1. The GPS receiver in 1X7 should be added. All the timing in the lab is synced to the 1pps from this.
2. We should add hyperlinks to the various parts datasheets (e.g. Dolphin switch, RFM switch, etc etc) so that the diagram will be truly informative and self-contained.
3. Megatron and nodus, but especially chiara (NFS server), should be added to the diagram. 

After fixing multiple issues, the model webviews are updating, should be done by tomorrow. It should be obvious from the timestamps on the index page which are the new ones. These new screens are better than the old ones and offer more info/details. Please look at them, and let me know if there are broken links etc. Once we are happy with this new webview, we can archive the old files and clean up the top directory a bit. I don't think this adds anything to the channel accounting effort but it's a nice thing to have up-to-date webviews, I found the LLO ones really useful in setting up the AS WFS model.

BTW, the crontab on megatron is set to run every day at 0844. The process of updating the models is pretty heavy because of whatever MATLAB overhead. Do we really need to have this run every day? I modified the crontab to run every other Saturday, and we can manually run the update when we modify a model. Considering this hasn't been working for ~3 years, I think this is fine, but if anyone has strong preference you can edit the crontab.

If someboy can volunteer to fix the MEDM screenshot that would be useful.

Taking inspiration from SR785 on how it reads differential signal, I figured that acromag too always need a way to return current through RTN ports always. That must be the reason why everything goes haywire when RTN is not connected to IN-. Now for single ended signals, we can always short RTN to IN- and keep same GND but then we need to be careful in avoiding ground loops. I'm gonna post a wiring diagram in next post to show how if two signal sources connect to each other separately, a GND loop can be formed if we tie each IN- port to RTN on an acromag.
Coming to the issue of reading a differential signal, what SR785 does is that it connects 50 Ohm resistance between Earth GND and differential signal shields (which are supposed to signal GND). In a floating GND setting, SR785 connects a 1 MOhm resistor between input shield and Earth GND. This can be used to read a differential signal through a single BNC cable since the shiled can take arbitrary voltages thanks ti the 1 MOhm resistor.

We can do the same in acromag. Instead of shorting RTN to IN- ports, we can connect them through a large resistor which would let IN- float but will give a path for current to return through RTN ports. Attached here are few scenarios where I connected IN- to RTN throguh wire, 820 Ohms, 10kOhms and 1MOhms in two sub cases where RTN was left open or was shorted to Earth GND. In all cases, the signal was produced by a 9V battery outputing roughly 8.16V. It seems that 10kOhm resistor between RTN and IN- with RTN connected to Earth GND is the best scenario noise wise. I'll post more results and a wiring diagram soon.

Here I present few wiring diagrams when using Acromag to avoid noisy behavior and ground loops.

Case 1: Only single-ended sources

• Attachment 1 gives a functioning wiring diagram when all sources are single ended.
• One should always short the RTN to IN- pin if the particular GND carried by that signal has not been shorted before to RTN for some other signal.
• So care is required to mark different GNDs of different powersupply separately and follow where they inadvertently get shorted, for example when a photodiode output is connected to FSS Box.
• Acromag should serve as the node of all GNDs concerned and all these grounds must not be connected to Earth GND at power supply ends or in any of the signal sources.
• I think this is a bit complicated thing to do.

Case 2: Some single and some differential sources

• Connect all single ended sources same as above keeping care of not building any ground loops.
• The differential source can be connected to IN+ and IN- pins, but there should be a high resistance path between IN- and RTN. See Attachment 2.
• Why this is the case, I'm not sure since I could not get access to internal wiring of Acromag (no response from them). But I have empirically found this.
• This helps IN- to float with respect to RTN according to the negative signal value. I've found that 10kOhm resistance works good. See 40m/15778.
• If RTN is shorted to nearby Earth GND (assuming none of the other power supply GNDs have been shorted to Earth GND) shows a reduction in noise for differential input. So this is recommended.
• This wiring diagram carries all complexity of previous case along with the fact that RTN and anything connected to it is at Earth GND now.

Case 3: Signal agnostic wiring

• Attachment 3 gives a wiring diagram which mimics the high resistance shorting of RTN to IN- in all ports regardless of the kind of signal it is used for reading.
• In this case, instead of being the node of the star configuration for GND, acromag gets detached from any ground loops.
• All differences in various GNDs would be kept by the power supplies driving small amounts of current through the 10 kOhm resistors.
• This is a much simpler wiring diagram as it avoids shorting various signal sources or their GNDs with each other, avoiding some of the ground loops.

Edit Wed Jan 27 13:38:19 2021 :

This solution is not acceptable as well. Even if it is successfull in reading the value, connecting resistor between IN- and RTN will not break the ground loops and the issue of ground loops will persist. Further, IN- connection to RTN breaks the symmetry between IN-  and IN+, and hence reduces the common mode rejection which is the intended purpose of differential signal anyways. I'll work more on this to find a way to read differential signals without connecitng IN- and RTN. My first guess is that it would need the GND on the source end to be connected to EarthGND and RTN on acromag end to be connected to EarthGND as well.

I found a white paper  from Acromag which discusses how to read differential signal using Acromag units. The document categorically says that differential signals are always supposed to be transmitted in three wires. I provides the two options of either using the RTN to connect to the signal ground (as done in Attachment 3) or locally place 10k-100k resistors between return and IN+ and IN- both (Attachment 2).

I have provided possible scenarios for these.

Using two wires to carry differential signal (Attachment 1):

• I assume this is our preferential way to connect.
• We can also assume all single-ended inputs as differential and do a signal condition agnostic wiring.
• Attachment 3 show what were the results for different values of resistors when a 2Hz 0.5V amplitude signal from SR785 which as converted to differential signal using D1900068 was measured by acromag.
• The connection to RTN is symmetrical for both inputs.

Using three wires to carry differential signal (Attachment 2):

• This is recommended method by the document in which it asks to carry the GND from signal source and connect it to RTN.
• If we use this, we'll have to be very cautious on what GND has been shorted through the acromag RTN terminals.
• This would probably create a lot of opportunities for ground loops to form.

Using an acromag card without making any connection with RTN is basically not allowed as per this document.

I worked around the racks and the feedthru flanges this afternoon and evening. This inevitably crashed c1lsc real-time process.
Rebooting c1lsc caused multiple crashes (as usual) and I had to hard reboot c1lsc/c1sus/c1ioo
This made the "DC" indicator of the IOPs for these hosts **RED**.

This looked like the usual timing issue. It looked like "ntpdate" is not available in the new system. (When was it updated?)

The hardware clock (RTC) of these hosts are set to be PST while the functional end host showed UTC. So I copied the time of the UTC time from the end to the vertex machines.
For the time adjustment, the standard "date" command was used

> sudo date -s "2021-02-03 07:11:30"

This made the trick. Once IOP was restarted, the "DC" indicators returned to **Green**, restarting the other processes were straight forward and now the CDS indicators are all green.

controls@c1iscex:~ 0$ timedatectl
      Local time: Wed 2021-02-03 07:35:12 UTC
  Universal time: Wed 2021-02-03 07:35:12 UTC
        RTC time: Wed 2021-02-03 07:35:26
       Time zone: Etc/UTC (UTC, +0000)
     NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
      DST active: n/a

NTP synchronization is not active. Is this OK?

With the recovered CDS, the IMC was immediately locked and the autolocker started to function after a few pokes (like manually running of the "mcup" script). However, I didn't see any light on the AS/REF cameras as well as the test mass faces. I'm sure the IMC alignment is OK. This means the TTs are not well aligned.

So, burtrestored c1assepics with 12:19 snapshot. This immediately brought the spots on the REFL/AS.

Then the arm were aligned, locked, and ASSed. I tried to lock the FP arms. The transmissions were at the level of 0.1~0.3. So some manual alignment of ITMY and BS were necessary. After having the TRs of ~0.8, I still could not lock the arms. The signs of the servo gains were flipped to -0.143 for X arm and -0.012 for Y arm, and the arms were locked. ASS worked well and the ASS offsets were offloaded to the SUSs.

Didn't get a chance to comment during the meeting - This was almost certainly a coincidence. I have never had to do this - I assert, based on the ~10 labwide reboots I have had to do in the last two years, that whether the timing errors persist on reboot or not is not deterministic. But this is beyond my level of CDS knowledge and so I'm happy for Rolf / Jamie to comment. I use the reboot script - if that doesn't work, I use it again until the systems come back without any errors.

I don't think this is a problem, the NTP synchronization is handled by timesyncd now.

I defer restoring the LSC settings etc since I guess there is not expected to be any interferometer activity for a while.

Really!? I didn't reboot the machines between "sudo date" and "rtcds start c1x0*". I tried rtcds. If it didn't work, it used date. Then tried rtcds. (repeat) The time was not synched at all wrt the time zones and also the time. There were 1~3 sec offset besides the TZ problem.