40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 59 of 344  Not logged in ELOG logo
ID Date Author Type Categoryup Subject
  15746   Wed Dec 23 23:06:45 2020 gautamConfigurationCDSUpdated CDS upgrade plan
  1. The diagram should clearly show the host machines and the expansion chassis and the interconnects between them.
  2. We no longer have any Gentoo bootserver or diskless FEs.
  3. The "c1lsc" host is in 1X4 not 1Y3.
  4. The connection between c1lsc and Dolphin switch is copper not fiber. I don't know how many Gbps it is. But if the switch is 10 Gbps, are they really selling interface cables that have lower speed? The datasheet says 10 Gbps.
  5. The control room workstations - Debian10 (rossa) is the way forward I believe. it is true pianosa remains SL7 (and we should continue to keep it so until all other machines have been upgraded and tested on Debian 10).
  6. There is no "IOO/OAF". The host is called "c1ioo".
  7. The interconnect between Dolphin switch and c1ioo host is via fiber not copper.
  8. It'd be good to have an accurate diagram of the current situation as well (with the RFM network).
  9. I'm not sure if the 1Y1 rack can accommodate 2 FEs and 2 expansion chassis. Maybe if we clear everything else there out...
  10. There are 2 "2GB/s" Copper traces. I think the legend should make clear what's going on - i.e. which cables are ethernet (Cat 6? Cat 5? What's the speed limitation? The cable? Or the switch?), which are PCIe cables etc etc. 

I don't have omnigraffle - what about uploading the source doc in a format that the excellent (and free) draw.io can handle? I think we can do a much better job of making this diagram reflect reality. There should also be a corresponding diagram for the Acromag system (but that doesn't have to be tied to this task). Megatron (scripts machine) and nodus should be added to that diagram as well.

Please send me any omissions or corrections to the layout.

  15760   Tue Jan 12 08:21:47 2021 anchalHowToCDSAcromag wiring investigation

I used an Acromag XT1221 in CTN to play around with different wiring and see what works.  Following are my findings:


Referenced Single Ended Source (Attachment 1):

  • If the source signal is referenced single ended, i.e. the signal is only on the positive output and the negative side is tied to GND on the source side AND this GND is also shared by the power supply powering the Acromag, then no additional wiring is required.
  • The GND common to the power supply and the source is not required to be Earth GND but if done so, it should be at one point only.
  • RTN terminal on Acromag can be left floating or tied to IN- terminal.

Floating Single Ended Source (Attachment 2):

  • If the source signal is floating single-ended i.e. the signal is only on the positive output and the negative output is a floating GND on the source, the the IN- should be connected to RTN.
  • This is the case for handheld calibrators or battery powered devices.
  • Note that there is no need to connect GND of power supply to the floating GND on the source.

Differential Source (Attachment 3):

  • If the source is differential output i.e. the signal is on both the positive output and the negative output, then connect one of the RTN terminals on Acromag to Earth GND. It has to be Earth GND for this to work.
  • Note that you can no longer tie the IN- of different signals to RTN as they are all carrying different negative output from the source.
  • Earth GND at RTN gives acromag a stable voltage reference to measure against the signals coming in IN+ and IN-. And the most stable voltage reference is of course Earth GND.

Conclusion:

  • We might have a mix of these three types of signals coming to a single Acromag box. In that case, we have to make sure we are not connecting the different IN- to each other (maybe through RTN) as the differential negative inputs carry signal, not a constant voltage value.
  • In this case, I think it would be fine to always use differential signal wiring diagram with the RTN  connected to Earth GND.
  • There's no difference in software configuration for the two types of inputs, differential or single-ended.
  • For cases in which we install the acromag box inside a rack mount chasis with an associated board (example: CTN/2248), it is ok and maybe the best to use the Attachment 1 wiring diagram.

Comments and suggestions are welcome.


Related elog posts:

40m/14841    40m/15134


Edit Tue Jan 26 12:44:19 2021 :

Note that the third wiring diagram mentioned actually does not work. It is an error in judgement. See 40m/15762 for seeing what happens during this.

  15761   Tue Jan 12 11:42:38 2021 gautamHowToCDSAcromag wiring investigation

Thanks for the systematic effort.

  1. Can you please post some time domain plots (ndscope perferably or StripTool) to clearly show the different failure modes?
  2. The majority of our AI channels are "Referenced Single Ended Source" in your terminology. At least on the c1psl Acromag crate, there are no channels that are truly differential drive (case #3 in your terminology). I think the point is that we saw noisy inputs when the IN- wasn't connected to RTN. e.g. the thorlabs photodiode has a BNC output with the shield connected to GND and the central conductor carrying a signal, and that channel was noisy when the RTN was unconnected. Is that consistent with your findings?
  3. What is the prescription when we have multiple power supplies (mixture of Sorensens in multiple racks, Thorlabs photodiodes and other devices powered by an AC/DC converter) involved?
  4. I'm still not entirely convinced of what the solution is, or that this is the whole picture. On 8 Jan, I disconnected (and then re-connected) the FSS RMTEMP sensor from the Acromag box, to check if the sensor output was noisy or if it was the Acromag. The problem surfaced on Dec 15, when I installed some new electronics in the rack (though none of them were connected to the Acromag directly, the only common point was the Sorensen DCPS. And between 8 Jan and today, the noise RMS has decreased back to the nominal level, without me doing anything to the grounding. How to reconcile this?
  15762   Wed Jan 13 16:09:29 2021 AnchalHowToCDSAcromag wiring investigation

I'm working on a better wiring diagram that takes into account multiple power supplies, how their GND is passed forward to the circuits or sensors using those power supplies and what possible wiring configurations on Acromag would give low noise. I think I have two configurations in mind which I will test and update here with data and better diagrams.


I took some striptool images earlier yesterday. So I'm dumping them here for further comments or inferences.

  15763   Thu Jan 14 11:46:20 2021 gautamUpdateCDSExpansion chassis from LHO

I picked the boxes up this morning. The inventory per Fil's email looks accurate. Some comments:

  1. They shipped the chassis and mounting parts (we should still get rails to mount these on, they're pretty heavy to just be supported on 4 rack nuts on the front). idk if we still need the two empty chassis that were requested from Rich.
  2. Regarding the fibers - one of the fibers is pre-2012. These are known to fail (according to Rolf). One of the two that LHO shipped is from 2012 (judging by S/N, I can't find an online lookup for the serial number), the other is 2011. IIRC, Rolf offered us some fibers so we may want to take him up on that. We may also be able to use copper cables if the distances b/w server and expansion chassis are short.
  3. The units are fitted with a +24V DC input power connector and not the AC power supplies that we have on all the rest of the chassis. This is probably just gonna be a matter of convenience, whether we want to stick to this scheme or revert to the AC input adaptor we have on all the other units. idk what the current draw will be from the Sorensen - I tested that the boards get power, and with noi ADCs/DACs/BIOs, the chassis draws ~1A (read off from DCPS display, not measured with a DMM). ~Half of this is for the cooling fans It seems like KT @ LLO has offered to ship AC power supplies so maybe we want to take them up on that offer.
  4. Without the host side OSSI PCIe card, timing interface board, and supermicro servers that actually have enough PCIe slots, we still can't actually run any meaningful test. I ran just a basic diagnostic that the chassis can be powered on, and the indicator LEDs and cooling fans run.
  5. Some photos of the contents are here. The units are stored along the east arm pending installation.

    >     Koji,
    >
    >     Barebones on this order.
    >
    >        1. Main PCIe board
    >        2. Backplane (Interface board)
    >        3. Power Board
    >        4. Fiber (One Stop) Interface Card, chassis side only
    >        5. Two One Stop Fibers
    >        6. No Timing Interface
    >        7. No Binary Cards.
    >        8. No ADC or DAC cards
    >
    >     Fil Clara
    >       
  15764   Thu Jan 14 12:19:43 2021 JonUpdateCDSExpansion chassis from LHO

That's fine, we didn't actually request those. We bought and already have in hand new PCIe x4 cables for the chassis-host connection. They're 3 m copper cables, which was based on the assumption of the time that host and chassis would be installed in the same rack.

Quote:
  1. Regarding the fibers - one of the fibers is pre-2012. These are known to fail (according to Rolf). One of the two that LHO shipped is from 2012 (judging by S/N, I can't find an online lookup for the serial number), the other is 2011. IIRC, Rolf offered us some fibers so we may want to take him up on that. We may also be able to use copper cables if the distances b/w server and expansion chassis are short.
  15765   Thu Jan 14 12:32:28 2021 gautamUpdateCDSRogue master may be doing something good?

I think the "Rogue Master" setting on the RFM network may be doing some good. 5 mins, ago, all the CDS indicators were green, but I noticed an amber light on the c1rfm screen just now (amber = warning). Seems like at GPS time 1294691182, there was some kind of error on the RFM network. But the network hasn't gone down. I can clear the amber flag by running the global diag reset. Nevertheless, the upgrade of all RT systems to Dolphin should not be de-prioritized I think.

  15766   Fri Jan 15 15:06:42 2021 JonUpdateCDSExpansion chassis from LHO

Koji asked me assemble a detailed breakdown of the parts received from LHO, which I do based on the high-res photos that Gautam posted of the shipment.

Parts in hand:

Qty Part Note(s)
2 Chassis body  
2 Power board and cooling fans As noted in 15763, these have the standard LIGO +24V input connector which we may want to change
2 IO interface backplane  
2 PCIe backplane  
2 Chassis-side OSS PCIe x4 card  
2 CX4 fiber cables These were not requested and are not needed

Parts still needed:

Qty Part Note(s)
2 Host-side OSS PCIe x4 card These were requested but missing from the LHO shipment 
2 Timing slave These were not originally requested, but we have recently learned they will be replaced at LHO soon

Issue with PCIe slots in new FEs

Also, I looked into the mix-up regarding the number of PCIe slots in the new Supermicro servers. The motherboard actually has six PCIe slots and is on the CDS list of boards known to be compatible. The mistake (mine) was in selecting a low-profile (1U) chassis that only exposes one of these slots. But at least it's not a fundamental limitation.

One option is to install an external PCIe expansion chassis that would be rack-mounted right above the FE. It is automatically configured by the system BIOS, so doesn't require any special drivers. It also supports hot-swapping of PCIe cards. There are also cheap ribbon-cable riser cards that would allow more cards to be connected for testing, although this is not as great for permanent mounting.

It may still be better to use the machines offered by Keith Thorne from LLO, as they're more powerful anyway. But if there is going to be an extended delay before those can be received, we should be able to use the machines we already have in conjunction with one of these PCIe expansion options.

  15767   Fri Jan 15 16:54:57 2021 gautamUpdateCDSExpansion chassis from LHO

Can you please provide a link to this "list of boards"? The only document I can find is T1800302. In that, under "Basic Requirements" (before considering specific motherboards), it is specified that the processor should be clocked @ >3GHz. The 3 new supermicros we have are clocked at 1.7 GHz. X10SRi-F boards are used according to that doc, but the processor is clocked at 3.6 or 3.2 GHz.

Please also confirm that there are no conflicts w.r.t. the generation of PCIe slots, and the interfaces (Dolphin, OSSI) we are planning to use - the new machines we have are "PCIe 2.0" (though i have no idea if this is the same as Gen 2). 

Quote:

The motherboard actually has six PCIe slots and is on the CDS list of boards known to be compatible.

As for the CX4 cable - I still think it's good to have these on hand. Not good to be in a situation later where FE and expansion chassis have to be in different racks, and the copper cable can't be used.

  15770   Tue Jan 19 13:19:24 2021 JonUpdateCDSExpansion chassis from LHO

Indeed T1800302 is the document I was alluding to, but I completely missed the statement about >3 GHz speed. There is an option for 3.4 GHz processors on the X10SRi-F board, but back in 2019 I chose against it because it would double the cost of the systems. At the time I thought I had saved us $5k. Hopefully we can get the LLO machines in the near term---but if not, I wonder if it's worth testing one of these to see whether the performance is tolerable.

Can you please provide a link to this "list of boards"? The only document I can find is T1800302....

I confirm that PCIe 2.0 motherboards are backwards compatible with PCIe 1.x cards, so there's no hardware issue. My main concern is whether the obsolete Dolphin drivers (requiring linux kernel <=3.x) will work on a new system, albeit one running Debian 8. The OSS PCIe card is automatically configured by the BIOS, so no external drivers are required for that one.

Please also confirm that there are no conflicts w.r.t. the generation of PCIe slots, and the interfaces (Dolphin, OSSI) we are planning to use - the new machines we have are "PCIe 2.0" (though i have no idea if this is the same as Gen 2). 

  15771   Tue Jan 19 14:05:25 2021 JonConfigurationCDSUpdated CDS upgrade plan

I've produced updated diagrams of the CDS layout, taking the comments in 15476 into account. I've also converted the 40m's diagrams from Omnigraffle ($150/license) to the free, cloud-based platform draw.io. I had never heard of draw.io, but I found that it has most all the same functionality. It also integrates nicely with Google Drive.

Attachment 1: The planned CDS upgrade (2 new FEs, fully replace RFM network with Gen 1 Dolphin IPC)
Attachment 2: The current 40m CDS topology

The most up-to-date diagrams are hosted at the following links:

Please send me any further corrections or omissions. Anyone logged in with LIGO.ORG credentials can also directly edit the diagrams.

  15772   Tue Jan 19 15:43:24 2021 gautamConfigurationCDSUpdated CDS upgrade plan

Not sure if 1Y1 can accommodate both c1sus2 and c1bhd as well as the various electronics chassis that will have to be installed. There may need to be some distribution between 1Y1 and 1Y3. Does Koji's new wiring also specify which racks hold which chassis?

Some minor improvements to the diagram:

  1. The GPS receiver in 1X7 should be added. All the timing in the lab is synced to the 1pps from this.
  2. We should add hyperlinks to the various parts datasheets (e.g. Dolphin switch, RFM switch, etc etc) so that the diagram will be truly informative and self-contained.
  3. Megatron and nodus, but especially chiara (NFS server), should be added to the diagram. 
  15775   Wed Jan 20 19:12:16 2021 gautamUpdateCDSSLwebview fixed

After fixing multiple issues, the model webviews are updating, should be done by tomorrow. It should be obvious from the timestamps on the index page which are the new ones. These new screens are better than the old ones and offer more info/details. Please look at them, and let me know if there are broken links etc. Once we are happy with this new webview, we can archive the old files and clean up the top directory a bit. I don't think this adds anything to the channel accounting effort but it's a nice thing to have up-to-date webviews, I found the LLO ones really useful in setting up the AS WFS model.

BTW, the crontab on megatron is set to run every day at 0844. The process of updating the models is pretty heavy because of whatever MATLAB overhead. Do we really need to have this run every day? I modified the crontab to run every other Saturday, and we can manually run the update when we modify a model. Considering this hasn't been working for ~3 years, I think this is fine, but if anyone has strong preference you can edit the crontab.

If someboy can volunteer to fix the MEDM screenshot that would be useful.

  15778   Tue Jan 26 12:59:51 2021 AnchalHowToCDSAcromag wiring investigation

Taking inspiration from SR785 on how it reads differential signal, I figured that acromag too always need a way to return current through RTN ports always. That must be the reason why everything goes haywire when RTN is not connected to IN-. Now for single ended signals, we can always short RTN to IN- and keep same GND but then we need to be careful in avoiding ground loops. I'm gonna post a wiring diagram in next post to show how if two signal sources connect to each other separately, a GND loop can be formed if we tie each IN- port to RTN on an acromag.
Coming to the issue of reading a differential signal, what SR785 does is that it connects 50 Ohm resistance between Earth GND and differential signal shields (which are supposed to signal GND). In a floating GND setting, SR785 connects a 1 MOhm resistor between input shield and Earth GND. This can be used to read a differential signal through a single BNC cable since the shiled can take arbitrary voltages thanks ti the 1 MOhm resistor.

We can do the same in acromag. Instead of shorting RTN to IN- ports, we can connect them through a large resistor which would let IN- float but will give a path for current to return through RTN ports. Attached here are few scenarios where I connected IN- to RTN throguh wire, 820 Ohms, 10kOhms and 1MOhms in two sub cases where RTN was left open or was shorted to Earth GND. In all cases, the signal was produced by a 9V battery outputing roughly 8.16V. It seems that 10kOhm resistor between RTN and IN- with RTN connected to Earth GND is the best scenario noise wise. I'll post more results and a wiring diagram soon.

  15779   Tue Jan 26 15:37:25 2021 AnchalHowToCDSAcromag wiring investigation

Here I present few wiring diagrams when using Acromag to avoid noisy behavior and ground loops.


Case 1: Only single-ended sources

  • Attachment 1 gives a functioning wiring diagram when all sources are single ended.
  • One should always short the RTN to IN- pin if the particular GND carried by that signal has not been shorted before to RTN for some other signal.
  • So care is required to mark different GNDs of different powersupply separately and follow where they inadvertently get shorted, for example when a photodiode output is connected to FSS Box.
  • Acromag should serve as the node of all GNDs concerned and all these grounds must not be connected to Earth GND at power supply ends or in any of the signal sources.
  • I think this is a bit complicated thing to do.

Case 2: Some single and some differential sources

  • Connect all single ended sources same as above keeping care of not building any ground loops.
  • The differential source can be connected to IN+ and IN- pins, but there should be a high resistance path between IN- and RTN. See Attachment 2.
  • Why this is the case, I'm not sure since I could not get access to internal wiring of Acromag (no response from them). But I have empirically found this.
  • This helps IN- to float with respect to RTN according to the negative signal value. I've found that 10kOhm resistance works good. See 40m/15778.
  • If RTN is shorted to nearby Earth GND (assuming none of the other power supply GNDs have been shorted to Earth GND) shows a reduction in noise for differential input. So this is recommended.
  • This wiring diagram carries all complexity of previous case along with the fact that RTN and anything connected to it is at Earth GND now.

Case 3: Signal agnostic wiring

  • Attachment 3 gives a wiring diagram which mimics the high resistance shorting of RTN to IN- in all ports regardless of the kind of signal it is used for reading.
  • In this case, instead of being the node of the star configuration for GND, acromag gets detached from any ground loops.
  • All differences in various GNDs would be kept by the power supplies driving small amounts of current through the 10 kOhm resistors.
  • This is a much simpler wiring diagram as it avoids shorting various signal sources or their GNDs with each other, avoiding some of the ground loops.

Edit Wed Jan 27 13:38:19 2021 :

This solution is not acceptable as well. Even if it is successfull in reading the value, connecting resistor between IN- and RTN will not break the ground loops and the issue of ground loops will persist. Further, IN- connection to RTN breaks the symmetry between IN-  and IN+, and hence reduces the common mode rejection which is the intended purpose of differential signal anyways. I'll work more on this to find a way to read differential signals without connecitng IN- and RTN. My first guess is that it would need the GND on the source end to be connected to EarthGND and RTN on acromag end to be connected to EarthGND as well.

 

  15785   Fri Jan 29 17:57:17 2021 AnchalHowToCDSAcromag wiring investigation

I found a white paper  from Acromag which discusses how to read differential signal using Acromag units. The document categorically says that differential signals are always supposed to be transmitted in three wires. I provides the two options of either using the RTN to connect to the signal ground (as done in Attachment 3) or locally place 10k-100k resistors between return and IN+ and IN- both (Attachment 2).

I have provided possible scenarios for these.

Using two wires to carry differential signal (Attachment 1):

  • I assume this is our preferential way to connect.
  • We can also assume all single-ended inputs as differential and do a signal condition agnostic wiring.
  • Attachment 3 show what were the results for different values of resistors when a 2Hz 0.5V amplitude signal from SR785 which as converted to differential signal using D1900068 was measured by acromag.
  • The connection to RTN is symmetrical for both inputs.

Using three wires to carry differential signal (Attachment 2):

  • This is recommended method by the document in which it asks to carry the GND from signal source and connect it to RTN.
  • If we use this, we'll have to be very cautious on what GND has been shorted through the acromag RTN terminals.
  • This would probably create a lot of opportunities for ground loops to form.

Using an acromag card without making any connection with RTN is basically not allowed as per this document.

  15791   Tue Feb 2 23:29:35 2021 KojiUpdateCDSCDS crash and CDS/IFO recovery

I worked around the racks and the feedthru flanges this afternoon and evening. This inevitably crashed c1lsc real-time process.
Rebooting c1lsc caused multiple crashes (as usual) and I had to hard reboot c1lsc/c1sus/c1ioo
This made the "DC" indicator of the IOPs for these hosts **RED**.

This looked like the usual timing issue. It looked like "ntpdate" is not available in the new system. (When was it updated?)

The hardware clock (RTC) of these hosts are set to be PST while the functional end host showed UTC. So I copied the time of the UTC time from the end to the vertex machines.
For the time adjustment, the standard "date" command was used

> sudo date -s "2021-02-03 07:11:30"

This made the trick. Once IOP was restarted, the "DC" indicators returned to **Green**, restarting the other processes were straight forward and now the CDS indicators are all green.

controls@c1iscex:~ 0$ timedatectl
      Local time: Wed 2021-02-03 07:35:12 UTC
  Universal time: Wed 2021-02-03 07:35:12 UTC
        RTC time: Wed 2021-02-03 07:35:26
       Time zone: Etc/UTC (UTC, +0000)
     NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
      DST active: n/a

NTP synchronization is not active. Is this OK?


With the recovered CDS, the IMC was immediately locked and the autolocker started to function after a few pokes (like manually running of the "mcup" script). However, I didn't see any light on the AS/REF cameras as well as the test mass faces. I'm sure the IMC alignment is OK. This means the TTs are not well aligned.

So, burtrestored c1assepics with 12:19 snapshot. This immediately brought the spots on the REFL/AS.

Then the arm were aligned, locked, and ASSed. I tried to lock the FP arms. The transmissions were at the level of 0.1~0.3. So some manual alignment of ITMY and BS were necessary. After having the TRs of ~0.8, I still could not lock the arms. The signs of the servo gains were flipped to -0.143 for X arm and -0.012 for Y arm, and the arms were locked. ASS worked well and the ASS offsets were offloaded to the SUSs.

 

  15792   Wed Feb 3 15:24:52 2021 gautamUpdateCDSCDS crash and CDS/IFO recovery

Didn't get a chance to comment during the meeting - This was almost certainly a coincidence. I have never had to do this - I assert, based on the ~10 labwide reboots I have had to do in the last two years, that whether the timing errors persist on reboot or not is not deterministic. But this is beyond my level of CDS knowledge and so I'm happy for Rolf / Jamie to comment. I use the reboot script - if that doesn't work, I use it again until the systems come back without any errors.

Quote:

This looked like the usual timing issue. It looked like "ntpdate" is not available in the new system. (When was it updated?)

The hardware clock (RTC) of these hosts are set to be PST while the functional end host showed UTC. So I copied the time of the UTC time from the end to the vertex machines.
For the time adjustment, the standard "date" command was used

> sudo date -s "2021-02-03 07:11:30"

This made the trick. Once IOP was restarted, the "DC" indicators returned to **Green**, restarting the other processes were straight forward and now the CDS indicators are all green.

I don't think this is a problem, the NTP synchronization is handled by timesyncd now.

Quote:

NTP synchronization is not active. Is this OK?

I defer restoring the LSC settings etc since I guess there is not expected to be any interferometer activity for a while.

  15794   Wed Feb 3 18:53:31 2021 KojiUpdateCDSCDS crash and CDS/IFO recovery

Really!? I didn't reboot the machines between "sudo date" and "rtcds start c1x0*". I tried rtcds. If it didn't work, it used date. Then tried rtcds. (repeat) The time was not synched at all wrt the time zones and also the time. There were 1~3 sec offset besides the TZ problem.

 

  15795   Wed Feb 3 21:28:02 2021 gautamUpdateCDSCDS crash and CDS/IFO recovery

I am just reporting my experience - this may be a new failure mode but I don't think so. In the new RTCDS, the ntp server for the FEs are the FB, to which they are synced by timesyncd. The FB machine itself has the ntpd service installed, and so is capable of synching to an NTP server over the internet, but also serving as an NTP server for the FEs. The timesyncd daemon may not have started correctly, or the NTP serving from FB got interrupted (for example), but that's all just speculation.

  15833   Mon Feb 22 14:53:47 2021 gautamUpdateCDSdtt-ezca-tools installed on rossa

The defaults cds-crtools didn't come with some of the older ezcautils (like ezcaread, ezcawrite etc). This is now packaged for debian, so I installled them with sudo apt update && sudo apt install dtt-ezca-tools on rossa. Now, we don't have to needlessly substitute the commands in our old shell scripts with the more modern z read, z write etc.

I am wondering if there is a relative implicit minus sign between the z servo and ezcaservo commands...

  15842   Wed Feb 24 22:13:47 2021 JonUpdateCDSPlanning document for front-end testing

I've started writing up a rough testing sequence for getting the three new front-ends operational (c1bhd, c1sus2, c1ioo). Since I anticipate this plan undergoing many updates, I've set it up as a Google doc which everyone can edit (log in with LIGO.ORG credentials).

Link to planning document

Please have a look and add any more tests, details, or concerns. I will continue adding to it as I read up on CDS documentation.

  15843   Thu Feb 25 14:30:05 2021 gautamUpdateCDSnew c1bhd setup - diskless boot

I've set up one 2U server unit received from KT in the control room area (the fans in it are pretty loud but other than that, no major issues with it being tested here). The IPMI interface is enabled and the machine is also hooked up to the martian network for diskless boot (usual login creds). I also installed a Dolphin interface card and the one-stop-systems host side card, and both seem to be recognized (the OSSI card has the "PWR" LED on, the Dolphin card shows up in the list of PCIe devices, but has no LEDs on at the moment). I actually can't find the OSSI card in the list of PCI devices, but maybe I'm not looking for the right device ID, or it needs the cable to be connected to the I/O chassis side to be recognized. Anyways, let the testing begin.

The machine previously christened c1bhd has been turned off and completely disconnected from the martian network (though I didn't bother removing it from the rack for now).

BTW - I think most of our 19" racks are deformed from years of loading - I spent 5 mins trying to install the rails (at 1Y1 and 1X7) to mount the supermicro on, and couldn't manage it. I could be missing some technique/trick, but i don't think so.

  15872   Fri Mar 5 17:48:25 2021 JonUpdateCDSFront-end testing

Today I moved the c1bhd machine from the control room to a new test area set up behind (west of) the 1X6 rack. The test stand is pictured in Attachment 1. I assembled one of the new IO chassis and connected it to the host.

I/O Chassis Assembly

  • LIGO-style 24V feedthrough replaced with an ATX 650W switching power supply
  • Timing slave installed
  • Contec DO-1616L-PE card installed for timing control
  • One 16-bit ADC and one 32-channel DO module were installed for testing

The chassis was then powered on and LED lights illuminated indicating that all the components have power. The assembled chassis is pictured in Attachment 2.

Chassis-Host Communications Testing

Following the procedure outlined T1900700, the system failed the very first test of the communications link between chassis and host, which is to check that all PCIe cards installed in both the host and the expansion chassis are detected. The Dolpin host adapter card is detected:

07:06.0 PCI bridge: Stargen Inc. Device 0102 (rev 02) (prog-if 00 [Normal decode])
    Flags: bus master, fast devsel, latency 0
    Bus: primary=07, secondary=0e, subordinate=0e, sec-latency=0
    I/O behind bridge: 00002000-00002fff
    Prefetchable memory behind bridge: 00000000c0200000-00000000c03fffff
    Capabilities: [40] Power Management version 2
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [60] Express Downstream Port (Slot+), MSI 00
    Capabilities: [80] Subsystem: Device 0000:0000
    Kernel driver in use: pcieport

However the OSS PCIe adapter card linking the host to the IO chassis was not detected, nor were any of the cards in the expansion chassis. Gautam previously reported that the OSS card was not detected by the host (though it was not connected to the chassis then). Even now connected to the IO chassis, the card is still not detected. On the chassis-side OSS card, there is a red LED illuminated indicating "HOST CARD RESET" as pictured in Attachment 3. This may indicate a problem with the card on the host side. Still more debugging to be done.

  15890   Tue Mar 9 16:52:47 2021 JonUpdateCDSFront-end testing

Today I continued with assembly and testing of the new front-ends. The main progress is that the IO chassis is now communicating with the host, resolving the previously reported issue.

Hardware Issues to be Resolved

Unfortunately, though, it turns out one of the two (host-side) One Stop Systems PCIe cards sent from Hanford is bad. After some investigation, I ultimately resolved the problem by swapping in the second card, with no other changes. I'll try to procure another from Keith Thorne, along with some spares.

Also, two of the three switching power supplies sent from Livingston (250W Channel Well PSG400P-89) appear to be incompatible with the Trenton BPX6806 PCIe backplanes in these chassis. The power supply cable has 20 conductors and the connector on the board has 24. The third supply, a 650W Antec EA-650, does have the correct cable and is currently powering one of the IO chassis. I'll confirm this situation with Keith and see whether they have any more Antecs. If not, I think these supplies can still be bought (not obsolete).

I've gone through all the hardware we've received, checked against the procurement spreadsheet. There are still some missing items:

  • 18-bit DACs (Qty 14; but 7 are spares)
  • ADC adapter boards (Qty 5)
  • DAC adapter boards (Qty 9)
  • 32-channel DO modules (Qty 2/10 in hand)

Testing Progress

Once the PCIe communications link between host and IO chassis was working, I carried out the testing procedure outlined in T1900700. This performs a series checks to confirm basic operation/compatibility of the hardware and PCIe drivers. All of the cards installed in both the host and the expansion chassis are detected and appear correctly configured, according to T1900700. In the below tree, there is one ADC, one 16-ch DIO, one 32-ch DO, and one DolphinDX card:

+-05.0-[05-20]----00.0-[06-20]--+-00.0-[07-08]----00.0-[08]----00.0  Contec Co., Ltd Device 86e2
|                               +-01.0-[09]--
|                               +-03.0-[0a]--
|                               +-08.0-[0b-15]----00.0-[0c-15]--+-02.0-[0d]--
|                               |                               +-03.0-[0e]--
|                               |                               +-04.0-[0f]--
|                               |                               +-06.0-[10-11]----00.0-[11]----04.0  PLX Technology, Inc. PCI9056 32-bit 66MHz PCI <-> IOBus Bridge
|                               |                               +-07.0-[12]--
|                               |                               +-08.0-[13]--
|                               |                               +-0a.0-[14]--
|                               |                               \-0b.0-[15]--
|                               \-09.0-[16-20]----00.0-[17-20]--+-02.0-[18]--
|                                                               +-03.0-[19]--
|                                                               +-04.0-[1a]--
|                                                               +-06.0-[1b]--
|                                                               +-07.0-[1c]--
|                                                               +-08.0-[1d]--
|                                                               +-0a.0-[1e-1f]----00.0-[1f]----00.0  Contec Co., Ltd Device 8632
|                                                               \-0b.0-[20]--
\-08.0-[21-2a]--+-00.0  Stargen Inc. Device 0101
                \-00.1-[22-2a]--+-00.0-[23]--
                                +-01.0-[24]--
                                +-02.0-[25]--
                                +-03.0-[26]--
                                +-04.0-[27]--
                                +-05.0-[28]--
                                +-06.0-[29]--
                                \-07.0-[2a]--

Standalone Subnet

Before I start building/testing RTCDS models, I'd like to move the new front ends to an isolated subnet. This is guaranteed to prevent any contention with the current system, or inadvertent changes to it.

Today I set up another of the Supermicro servers sent by Livingston in the 1X6 test stand area. The intention is for this machine to run a cloned, bootable image of the current fb1 system, allowing it to function as a bootserver and DAQ server for the FEs on the subnet.

However, this hard disk containing the fb1 image appears to be corrupted and will not boot. It seems to have been sitting disconnected in a box since ~2018, which is not a stable way to store data long term. I wasn't immediately able to recover the disk using fsck. I could spend some more time trying, but it might be most time-effective to just make a new clone of the fb1 system as it is now.

  15904   Thu Mar 11 14:27:56 2021 gautamUpdateCDStimesync issue?

I have recently been running into hitting the 4MB/s data rate limit on testpoints - basically, I can't run DTT TF and spectrum measurements that I was able to while locking the interferometer, which I certainly was able to this time last year. AFAIK, the major modification made was the addition of 4 DQ channels for the in-air BHD experiment - assuming the data is transmitted as double precision numbers, i estimate the additional load due to this change was ~500KB/s. Probably there is some compression so it is a bit more efficient (as this naive calc would suggest we can only record 32 channels and I counted 41 full rate channels in the model), but still, can't think of anything else that has changed. Anyway, I removed the unused parts and recompiled/re-installed the models (c1lsc and c1omc). Holding off on a restart until I decide I have the energy to recover the CDS system from the inevitable crash. For documentation I'm also attaching screenshot of the schematic of the changes made.

Anyway, the main point of this elog is that at the compilation stage, I got a warning I've never seen before:

Building front-end Linux kernel module c1lsc...
make[1]: Warning: File 'GNUmakefile' has modification time 13 s in the future
make[1]: warning:  Clock skew detected.  Your build may be incomplete.

This prompted me to check the system time on c1lsc and FB - you can see there is a 1 minute offset (it is not a delay in me issuing the command to the two machines)! I am suspecting this NTP action is the reason. So maybe a model reboot is in order. Sigh

  15905   Thu Mar 11 18:46:06 2021 gautamUpdateCDScds reboot

Since Koji was in the lab I decided to bite the bullet and do the reboot. I've modified the reboot script - now, it prompts the user to confirm that the time recognized by the FEs are the same (use the IOP model's status screen, the GPSTIME is updated live on the upper right hand corner). So you would do sudo date --set="Thu 11 Mar 2021 06:48:30 PM UTC" for example, and then restart the IOP model. Why is this necessary? Who knows. It seems to be a deterministic way of getting things back up and running for now so we have to live with it. I will note that this was not a problem between 2017 and 2020 Oct, in which time I've run the reboot script >10 times without needing to take this step. But things change (for an as of yet unknown reason) and we must adapt. Once the IOPs all report a green "DC" status light on the CDS overview screen, you can let the script take you the rest of the way again.

The main point of this work was to relax the data rate on the c1lsc model, and this worked. It now registers ~3.2 MB/s, down from the ~3.8 MB/s earlier today. I can now measure 2 loop TFs simultaneously. This means that we should avoid adding any more DQ channels to the c1lsc model (without some adjustment/downsampling of others).

Quote:

 Holding off on a restart until I decide I have the energy to recover the CDS system from the inevitable crash.

  15910   Fri Mar 12 03:28:51 2021 KojiUpdateCDScds reboot

I want to emphasize the followings:

  • FB's RTC (real-time clock/motherboard clock) is synchronized to NTP time.
  • The other RT machines are not synchronized to NTP time.
  • My speculation is that the RT machine timestamp is produced from the local RT machine time because there is no IRIG-B signal provided. -> If the RTC is more than 1 sec off from the FB time, the timestamp inconsistency occurs. Then it induces "DC" error.
  • For now, we have to use "date" command to match the local RTC time to the FB's time.
     
  • So: If NTP synchronization is properly implemented for the RT machines, we will be free from this silly manual time adjustment.

 

  15911   Fri Mar 12 11:02:38 2021 gautamUpdateCDScds reboot

I looked into this a bit today morning. I forgot exactly what time we restarted the machines, but looking at the timesyncd logs, it appears that the NTP synchronization is in fact working (log below is for c1sus, similar on other FEs):

-- Logs begin at Fri 2021-03-12 02:01:34 UTC, end at Fri 2021-03-12 19:01:55 UTC. --
Mar 12 02:01:36 c1sus systemd[1]: Starting Network Time Synchronization...
Mar 12 02:01:37 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).
Mar 12 02:01:37 c1sus systemd[1]: Started Network Time Synchronization.
Mar 12 02:02:09 c1sus systemd-timesyncd[243]: Using NTP server 192.168.113.201:123 (ntpserver).

So, the service is doing what it is supposed to (using the FB, 192.168.113.201, as the ntpserver). You can see that the timesync was done a couple of seconds after the machine booted up (validated against "uptime"). Then, the service is periodically correcting drifts. idk what it means that the time wasn't in sync when we check the time using timedatectl or similar. Anyway, like I said, I have successfully rebooted all the FEs without having to do this weird time adjustment >10 times

I guess what I am saying is, I don't know what action is necessary for "implementing NTP synchronization properly", since the diagnostic logfile seems to indicate that it is doing what it is supposed to.

More worryingly, the time has already drifted in < 24 hours.

Quote:

I want to emphasize the followings:

  • FB's RTC (real-time clock/motherboard clock) is synchronized to NTP time.
  • The other RT machines are not synchronized to NTP time.
  • My speculation is that the RT machine timestamp is produced from the local RT machine time because there is no IRIG-B signal provided. -> If the RTC is more than 1 sec off from the FB time, the timestamp inconsistency occurs. Then it induces "DC" error.
  • For now, we have to use "date" command to match the local RTC time to the FB's time.
     
  • So: If NTP synchronization is properly implemented for the RT machines, we will be free from this silly manual time adjustment.
  15924   Tue Mar 16 16:27:22 2021 JonUpdateCDSFront-end testing

Some progress today towards setting up an isolated subnet for testing the new front-ends. I was able to recover the fb1 backup disk using the Rescatux disk-rescue utility and successfully booted an fb1 clone on the subnet. This machine will function as the boot server and DAQ server for the front-ends under test. (None of these machines are connected to the Martian network or, currently, even the outside Internet.)

Despite the success with the framebuilder, front-ends cannot yet be booted locally because we are still missing the DHCP and FTP servers required for network booting. On the Martian net, these processes are running not on fb1 but on chiara. And to be able to compile and run models later in the testing, we will need the contents of the /opt/rtcds directory also hosted on chiara.

For these reasons, I think it will be easiest to create another clone of chiara to run on the subnet. There is a backup disk of chiara and I attempted to boot it on one of the LLO front-ends, but without success. The repair tool I used to recover the fb1 disk does not find a problem with the chiara disk. However, the chiara disk is an external USB drive, so I suspect there could be a compatibility problem with these old (~2010) machines. Some of them don't even recognize USB keyboards pre-boot-up. I may try booting the USB drive from a newer computer.

Edit: I removed one of the new, unused Supermicros from the 1Y2 rack and set it up in the test stand. This newer machine is able to boot the chiara USB disk without issue. Next time I'll continue with the networking setup.

  15925   Tue Mar 16 19:04:20 2021 gautamUpdateCDSFront-end testing

Now that I think about it, I may only have backed up the root file system of chiara, and not/home/cds/ (symlinked to /opt/ over NFS). I think we never revived the rsync backup to LDAS after the FB fiasco of 2017, else that'd have been the most convenient way to get files. So you may have to resort to some other technique (e.g. configure the second network interface of the chiara clone to be on the martian network and copy over files to the local disk, and then disconnect the chiara clone from the martian network (if we really want to keep this test stand completely isolated from the existing CDS network) - the /home/cds/ directory is rather large IIRC, but with 2TB on the FB clone, you may be able to get everything needed to get the rtcds system working). It may then be necessary to hook up a separate disk to write frames to if you want to test that part of the system out.

Good to hear the backup disk was able to boot though!

Quote:

And to be able to compile and run models later in the testing, we will need the contents of the /opt/rtcds directory also hosted on chiara.

For these reasons, I think it will be easiest to create another clone of chiara to run on the subnet. There is a backup disk of chiara and I attempted to boot it on one of the LLO front-ends, but without success.

  15947   Fri Mar 19 18:14:56 2021 JonUpdateCDSFront-end testing

Summary

Today I finished setting up the subnet for new FE testing. There are clones of both fb1 and chiara running on this subnet (pictured in Attachment 2), which are able to boot FEs completely independently of the Martian network. I then assembled a second FE system (Supermicro host and IO chassis) to serve as c1sus2, using a new OSS host adapter card received yesterday from LLO. I ran the same set of PCIe hardware/driver tests as was done on the c1bhd system in 15890. All the PCIe tests pass.

Subnet setup

For future reference, below is the procedure used to configure the bootserver subnet.

  • Select "Network" as highest boot priority in FE BIOS settings
  • Connect all machines to subnet switch. Verify fb1 and chiara eth0 interfaces are enabled and assigned correct IP address.
  • Add c1bhd and c1sus2 entries to chiara:/etc/dhcp/dhcpd.conf:
host c1bhd {
  hardware ethernet 00:25:90:05:AB:46;
  fixed-address 192.168.113.91;
}
host c1bhd {
  hardware ethernet 00:25:90:06:69:C2;
  fixed-address 192.168.113.92;
}
  • Restart DHCP server to pick up changes:
$ sudo service isc-dhcp-server restart
  • Add c1bhd and c1sus2 entries to fb1:/etc/hosts:
192.168.113.91    c1bhd
192.168.113.92    c1sus2
  • Power on the FEs. If all was configured correctly, the machines will boot.

C1SUS2 I/O chassis assembly

  • Installed in host:
    • DolphinDX host adapter
    • One Stop Systems PCIe x4 host adapter (new card sent from LLO)
  • Installed in chassis:
    • Channel Well 250 W power supply (replaces aLIGO-style 24 V feedthrough)
    • Timing slave
    • Contec DIO-1616L-PE module for timing control

Next time, on to RTCDS model compilation and testing. This will require first obtaining a clone of the /opt/rtcds disk hosted on chiara.

  15948   Fri Mar 19 19:15:13 2021 JonUpdateCDSc1auxey assembly

Today I helped Yehonathan get started with assembly of the c1auxey (slow controls) Acromag chassis. This will replace the final remaining VME crate. We cleared the far left end of the electronics bench in the office area, as discussed on Wed. The high-voltage supplies and test equipment was moved together to the desk across the aisle.

Yehonathan has begun assembling the chassis frame (it required some light machining to mount the DIN rails that hold the Acromag units). Next, he will wire up the switches, LED indicator lights, and Acromag power connectors following the the documented procedure.

  15959   Wed Mar 24 19:02:21 2021 JonUpdateCDSFront-end testing

This evening I prepared a new 2 TB 3.5" disk to hold a copy of /opt/rtcds and /opt/rtapps from chiara. This is the final piece of setup before model compilation can be tested on the new front-ends. However chiara does not appear to support hot-swapping of disks, as the disk is not recognized when connected to the live machine. I will await confirmation before rebooting it. The new disk is not currently connected.

  15962   Thu Mar 25 12:11:53 2021 YehonathanUpdateCDSc1auxey assembly

I finished prewiring the new c1auxey Acromag chassis (see attached pictures). I connected all grounds to the DIN rail to save some wiring. The power switches and LEDs work as expected.

I configured the DAQ modules using the old windows machine. I configured the gateway to be 192.168.114.1. The host machine still needs to be setup.

Next, the feedthroughs need to be wired and the channels need to be bench tested.

  15963   Thu Mar 25 14:16:33 2021 gautamUpdateCDSc1auxey assembly

It might be a good idea to configure this box for the new suspension config - modern Satellite Amp, HV coil driver etc. It's a good opportunity to test the wiring scheme, "cross-connect" type adapters etc.

Quote:

Next, the feedthroughs need to be wired and the channels need to be bench tested.

  15976   Mon Mar 29 17:55:50 2021 JonUpdateCDSFront-end testing

Cloning of chiara:/home/cvs underway

I returned today with a beefier USB-SATA adapter, which has an integrated 12 V supply for powering 3.5" disks. I used this to interface a new 6 TB 3.5" disk found in the FE supplies cabinet.

I decided to go with a larger disk and copy the full contents of chiara:/home/cds. Strictly, the FEs only strictly need the RTS executables in /home/cvs/rtcds and /home/cvs/rtapps. However, to independently develop models, the shared matlab binaries in /home/cvs/caltech/... also need to be exposed. And there may be others I've missed.

I began the clone around 12:30 pm today. To preserve bandwidth to the main disk, I am copying not the /home/cds disk directly, but rather its backup image at /media/40mBackup.

Set up of dedicated SimPlant host

Although not directly related to the FE testing, today I added a new machine to the test stand which will be dedicated to running sim models. Chris has developed a virtual cymac which we plan to run on this machine. It will provide a dedicated testbed for SimPlant and other development, and can host up to 10 user models.

I used one of the spare 12-core Supermicro servers from LLO, which I have named c1sim. I assigned it the IP address 192.168.113.93 on the Martian network. This machine will run in a self-contained way that will not depend on any 40m CDS services and also should not interfere with them. However, if there are concerns about having it present on the network, it can be moved to the outside-facing switch in the office area. It is not currently running any RTCDS processes.

Set-up was carried out via the following procedure:

  • Installed Debian 10.9 on an internal 480 GB SSD.
  • Installed cdssoft repos following Jamie's instructions.
  • Installed RTS and Docker dependencies:
    $ sudo apt install cpuset advligorts-mbuf-dkms advligorts-gpstime-dkms docker.io docker-compose
  • Configured scheduler for real-time operation:
    $ sudo /sbin/sysctl kernel.sched_rt_runtime_us = -1
  • Reserved 10 cores for RTS user models (plus one for IOP model) by adding the following line to /etc/default/grub:
    GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=nohz,domain,1-11 nohz_full=1-11 tsc=reliable mce=off"
    followed by the commands:
    $ sudo update-grub
    $ sudo reboot now
  • Downloaded virtual cymac repo to /home/controls/docker-cymac.

I need to talk to Chris before I can take the setup further.

  15978   Tue Mar 30 17:27:04 2021 YehonathanUpdateCDSc1auxey assembly

{Yehonathan, Jon}

We poked (looked in situ with a flashlight, not disturbing any connections) around c1auxex chassis to understand better what is the wiring scheme.

To our surprise, we found that nothing was connected to the RTNs of the analog input Acromag modules. From previous experience and the Acromag manual, there can't be any meaningful voltage measurement without it.

I also did some rewiring in the Acromag chassis to improve its reliability. In particular, I removed the ground wires from the DIN rail and connected them using crimp-on butt splices.

 

  15979   Tue Mar 30 18:21:34 2021 JonUpdateCDSFront-end testing

Progress today:

Outside Internet access for FE test stand

This morning Jordan and I ran an 85-foot Cat 6 Ethernet cable from the campus network switch in the office area (on the ligo.caltech.edu domain) to the FE test stand near 1X6. This is to allow the test-stand subnet to be accessed for remote testing, while keeping it invisible to the parallel Martian subnet.

Successful RTCDS model compilation on new FEs

The clone of the chiara:/home/cds disk completed overnight. Today I installed the disk in the chiara clone. The NFS mounts (/opt/rtcds, /opt/rtapps) shared with the other test-stand machines mounted without issue.

Next, I attempted to open the shared Matlab executable (/cvs/cds/caltech/apps/linux64/matlab/bin/matlab) and launch Simulink. The existing Matlab license (/cvs/cds/caltech/apps/linux64/matlab/licenses/license_chiara_865865_R2015b.lic) did not work on this new machine, as they are machine-specific, so I updated the license file. I linked this license to my personal license, so that the machine license for the real chiara would not get replaced. The original license file is saved in the same directory with a *.bak postfix. If this disk is ever used in the real chiara machine, this file should be restored. After the machine license was updated, Matlab and Simulink loaded and allowed model editing.

Finally, I tested RTCDS model compilation on the new FEs using the c1lsc model as a trial case. It encountered one path issue due to the model being located at /opt/rtcds/userapps/release/isc/c1/models/isc/ instead of /opt/rtcds/userapps/release/isc/c1/models/. This seems to be a relic of the migration of the 40m models from the SVN to a standalone git repo. This was resolved by simply symlinking to the expected location:

$ sudo ln -s /opt/rtcds/userapps/release/isc/c1/models/isc/c1lsc.mdl /opt/rtcds/userapps/release/isc/c1/models/c1lsc.mdl

The model compilation then succeeded:

controls@c1bhd$ cd /opt/rtcds/caltech/c1/rtbuild/release

controls@c1bhd$ make clean-c1lsc
Cleaning c1lsc...
Done

controls@c1bhd$ make c1lsc
Cleaning c1lsc...
Done
Parsing the model c1lsc...
Done
Building EPICS sequencers...
Done
Building front-end Linux kernel module c1lsc...
make[1]: Warning: File 'GNUmakefile' has modification time 28830 s in the
future
make[1]: warning:  Clock skew detected.  Your build may be incomplete.
Done
RCG source code directory:
/opt/rtcds/rtscore/branches/branch-3.4
The following files were used for this build:
/opt/rtcds/caltech/c1/userapps/release/cds/common/src/cdsToggle.c
/opt/rtcds/userapps/release/cds/c1/src/inmtrxparse.c
/opt/rtcds/userapps/release/cds/common/models/FILTBANK_MASK.mdl
/opt/rtcds/userapps/release/cds/common/models/rtbitget.mdl
/opt/rtcds/userapps/release/cds/common/models/SCHMITTTRIGGER.mdl
/opt/rtcds/userapps/release/cds/common/models/SQRT_SWITCH.mdl
/opt/rtcds/userapps/release/cds/common/src/DB2MAG.c
/opt/rtcds/userapps/release/cds/common/src/OSC_WITH_CONTROL.c
/opt/rtcds/userapps/release/cds/common/src/wait.c
/opt/rtcds/userapps/release/isc/c1/models/c1lsc.mdl
/opt/rtcds/userapps/release/isc/c1/models/IQLOCK_WHITENING_TRIGGERING.mdl
/opt/rtcds/userapps/release/isc/c1/models/PHASEROT.mdl
/opt/rtcds/userapps/release/isc/c1/models/RF_PD_WITH_WHITENING_TRIGGERING.mdl
/opt/rtcds/userapps/release/isc/c1/models/UGF_SERVO_40m.mdl
/opt/rtcds/userapps/release/isc/common/models/FILTBANK_TRIGGER.mdl
/opt/rtcds/userapps/release/isc/common/models/LSC_TRIGGER.mdl

Successfully compiled c1lsc
***********************************************
Compile Warnings, found in c1lsc_warnings.log:
***********************************************
[warnings suppressed]

As did the installation:

controls@c1bhd$ make install-c1lsc
Installing system=c1lsc site=caltech ifo=C1,c1
Installing /opt/rtcds/caltech/c1/chans/C1LSC.txt
Installing /opt/rtcds/caltech/c1/target/c1lsc/c1lscepics
Installing /opt/rtcds/caltech/c1/target/c1lsc
Installing start and stop scripts
/opt/rtcds/caltech/c1/scripts/killc1lsc
/opt/rtcds/caltech/c1/scripts/startc1lsc
Performing install-daq
Updating testpoint.par config file
/opt/rtcds/caltech/c1/target/gds/param/testpoint.par
/opt/rtcds/rtscore/branches/branch-3.4/src/epics/util/updateTestpointPar.pl
-par_file=/opt/rtcds/caltech/c1/target/gds/param/archive/testpoint_210330_170634.par
-gds_node=42 -site_letter=C -system=c1lsc -host=c1lsc
Installing GDS node 42 configuration file
/opt/rtcds/caltech/c1/target/gds/param/tpchn_c1lsc.par
Installing auto-generated DAQ configuration file
/opt/rtcds/caltech/c1/chans/daq/C1LSC.ini
Installing Epics MEDM screens
Running post-build script

safe.snap exists

We are ready to start building and testing models.

  15997   Tue Apr 6 07:19:11 2021 JonUpdateCDSNew SimPlant cymac

Yesterday Chris and I completed setup of the Supermicro machine that will serve as a dedicated host for developing and testing RTCDS sim models. It is currently sitting in the stack of machines in the FE test stand, though it should eventually be moved into a permanent rack.

It turns out the machine cannot run 10 user models, only 4. Hyperthreading was enabled in the BIOS settings, which created the illusion of there being 12 rather than 6 physical cores. Between Chris and Ian's sim models, we already have a fully-loaded machine. There are several more of these spare 6-core machines that could be set up to run additional models. But in the long term, and especially in Ian's case where the IFO sim models will all need to communicate with one another (this is a self-contained cymac, not a distributed FE system), we may need to buy a larger machine with 16 or 32 cores.

IPMI was set up for the c1sim cymac. I assigned the IPMI interface a static IP address on the Martian network (192.168.113.45) and registered it in the usual way with the domain name server on chiara. After updating the BIOS settings and rebooting, I was able to remotely power off and back on the machine following these instructions.

Set up of dedicated SimPlant host

Although not directly related to the FE testing, today I added a new machine to the test stand which will be dedicated to running sim models. Chris has developed a virtual cymac which we plan to run on this machine. It will provide a dedicated testbed for SimPlant and other development, and can host up to 10 user models.

I used one of the spare 12-core Supermicro servers from LLO, which I have named c1sim. I assigned it the IP address 192.168.113.93 on the Martian network. This machine will run in a self-contained way that will not depend on any 40m CDS services and also should not interfere with them.

  15998   Tue Apr 6 11:13:01 2021 JonUpdateCDSFE testing

I/O chassis assembly

Yesterday I installed all the available ADC/DAC/BIO modules and adapter boards into the new I/O chassis (c1bhd, c1sus2). We are still missing three ADC adapter boards and six 18-bit DACs. A thorough search of the FE cabinet turned up several 16-bit DACs, but only one adapter board. Since one 16-bit DAC is required anyway for c1sus2, I installed the one complete set in that chassis.

Below is the current state of each chassis. Missing components are highlighted in yellow. We cannot proceed to loopback testing until at least some of the missing hardware is in hand.

C1BHD

Component Qty Required Qty Installed
16-bit ADC 1 1
16-bit ADC adapter 1 0
18-bit DAC 1 0
18-bit DAC adapter 1 1
16-ch DIO 1 1

C1SUS2

Component Qty required Qty Installed
16-bit ADC 2 2
16-bit ADC adapter 2 0
16-bit DAC 1 1
16-bit DAC adapter 1 1
18-bit DAC 5 0
18-bit DAC adapter 5 5
32-ch DO 6 6
16-ch DIO 1 1

Gateway for remote access

To enable remote access to the machines on the test stand subnet, one machine must function as a gateway server. Initially, I tried to set this up using the second network interface of the chiara clone. However, having two active interfaces caused problems for the DHCP and FTS servers and broke the diskless FE booting. Debugging this would have required making changes to the network configuration that would have to be remembered and reverted, were the chiara disk to ever to be used in the original machine.

So instead, I simply grabbed another of the (unused) 1U Supermicro servers from the 1Y1 rack and set it up on the subnet as a standalone gateway server. The machine is named c1teststand. Its first network interface is connected to the general computing network (ligo.caltech.edu) and the second to the test-stand subnet. It has no connection to the Martian subnet. I installed Debian 10.9 anticipating that, when the machine is no longer needed in the test stand, it can be converted into another docker-cymac for to run additional sim models.

Currently, the outside-facing IP address is assigned via DHCP and so periodically changes. I've asked Larry to assign it a static IP on the ligo.caltech.edu domain, so that it can be accessed analogously to nodus.

  16008   Thu Apr 8 20:58:17 2021 KojiUpdateCDSADC adapter boards assembly

5x 16bit ADC adapter boards (D0902006) assembled.

  16012   Sat Apr 10 08:51:32 2021 JonUpdateCDSI/O Chassis Assembly

I installed three of the 16-bit ADC adapter boards assembled by Koji. Now, the only missing hardware is the 18-bit DACs (quantities below). As I mentioned this week, there are 2-3 16-bit DACs available in the FE cabinet. They could be used if more 16-bit adapter boards could be procured.

C1BHD    
Component Qty Required Qty Installed
16-bit ADC 1 1
16-bit ADC adapter 1 1
18-bit DAC 1 0
18-bit DAC adapter 1 1
16-ch DIO 1 1
C1SUS2    
Component Qty required Qty Installed
16-bit ADC 2 2
16-bit ADC adapter 2 2
16-bit DAC 1 1
16-bit DAC adapter 1 1
18-bit DAC 5 0
18-bit DAC adapter 5 5
32-ch DO 6 6
16-ch DIO 1 1
  16015   Sat Apr 10 11:56:14 2021 JonUpdateCDS40m LSC simPlant model

Summary

Yesterday I resurrected the 40m's LSC simPlant model, c1lsp. It is running on c1sim, a virtual, self-contained cymac that Chris and I set up for developing sim models (see 15997). I think the next step towards an integrated IFO model is incorporating the suspension plants. I am going to hand development largely over to Ian at this point, with continued support from me and Chris.

x1lsp_main.png

 

LSC Plant

This model dates back to around 2012 and appears to have last been used in ~2015. According to the old CDS documentation:

Name Description Communicates directly with
LSP Simulated length sensing model of the physical plant, handles light propagation between mirrors, also handles alignment modeling and would have to communicate ground motion to all the suspensions for ASS to work LSC, XEP, YEP, VSP

Here XEP, YEP, and VSP are respectively the x-end, y-end, and vertex suspension plant models. I haven't found any evidence that these were ever fully implemented for the entire IFO. However, it looks like SUS plants were later implemented for a single arm cavity, at least, using two models named c1sup and c1spx (appear in more recent CDS documentation). These suspension plants could likely be updated and then copied for the other suspended optics.

To represent the optical transfer functions, the model loads a set of SOS filter coefficients generated by an Optickle model of the interferometer. The filter-generating code and instructions on how to use it are located here. In particular, it contains a Matlab script named opt40m.m which defines the interferferometer. It should be updated to match the parameters in the latest 40m Finesse model, C1_w_BHD.kat. The calibrations from Watts to sensor voltages will also need to be checked and likely updated.

Model-Porting Procedure

For future reference, below are the steps followed to port this model to the virtual cymac.

  1. Copy over model files.
     
    • The c1lsp model, chiara:/opt/rtcds/userapps/release/isc/c1/models/c1lsp.mdl, was copied to the userapps directory on the virtual cymac, c1sim:/home/controls/docker-cymac/userapps/x1lsp.mdl. In the filename, note the change in IFO prefix "c1" --> "x1," since this cymac is not part of the C1 CDS network.
       
    • This model also depends on a custom library file, chiara:/opt/rtcds/userapps/release/isc/c1/models/SIMPLANT.mdl, which was copied to c1sim:/home/controls/docker-cymac/userapps/lib/SIMPLANT.mdl.
       
  2. Update model parameters in Simulink. To edit models in Simulink, see the instructions here and also here.
     
    • The main changes are to the cdsParameters block, which was updated as shown below. Note the values of dcuid and specific_cpu are specifically assigned to x1lsp and will vary for other models. The other parameters will be the same.
       

    •  
    • I also had to change the name of one of the user-defined objects from "ADC0" --> "ADC" and then re-copy the cdsAdc object (shown above) from the CDS_PARTS.mdl library. At least in newer RCG code, the cdsAdc object must also be named "ADC0." This namespace collision was causing the compiler to fail.
       
    • Note: Since Matlab is not yet set up on c1sim, I actually made these edits on one of the 40m machines (chiara) prior to copying the model.
       
  3. Compile and launch the models. Execute the following commands on c1sim:
    • $ cd ~/docker-cymac
      $ ./kill_cymac
      $ ./start_cymac debug
    • The optional debug flag will print the full set of compilation messages to the terminal. If compilation fails, search the traceback for lines containing "ERROR" to determine what is causing the failure.
       

  4. Accessing MEDM screens. Once the model is running, a button should be added to the sitemap screen (located at c1sim:/home/controls/docker-cymac/userapps/medm/sitemap.adl) to access one or more screens specific to the newly added model.

    • Custom-made screens should be added to c1sim:/home/controls/docker-cymac/userapps/medm/x1lsp (where the final subdirectory is the name of the particular model).

    • The set of available auto-generated screens for the model can be viewed by entering the virtual environment:

    • $ cd ~/docker-cymac
      $ ./login_cymac                    #drops into virtual shell
      # cd /opt/rtcds/tst/x1/medm/x1lsp  #last subdirectory is model name
      # ls -l *.adl
      # exit                             #return to host shell
    • The sitemap screen and any subscreens can link to the auto-generated screens in the usual way (by pointing to their virtual /opt/rtcds path). Currently, for the virtual path resolution to work, an environment script has to be run prior to launching sitemap, which sets the location of a virtual MEDM server (this will be auto-scripted in the future):

    • $ cd ~/docker-cymac
      $ eval $(./env_cymac)
      $ sitemap
    • One important auto-generated screen that should be linked for every model is the CDS runtime diagnostics screen, which indicates the success/fail state of the model and all its dependencies. T1100625 details the meaning of all the various indicator lights.

 

 

 

 

  16021   Tue Apr 13 16:24:38 2021 Ian MacMillanUpdateCDS40m LSC simPlant model

Added Matlab to the Docker machine. This should help immensely with workflow as well as keeping installed libraries consistent. Next step is outlining the project so coding is easier

Command to launch is:     $ matlab &

From Jon just for bookkeeping: 

Then in the Matlab command window, open the CDS parts library via:

addpath /home/controls/simLink/lib/

open /home/controls/simLink/CDS_PARTS.mdl

Then open an RTCDS model (for example, here the LSC plant) via:

open /home/controls/docker-cymac/userapps/x1lsp.mdl

 

 

  16037   Thu Apr 15 17:24:08 2021 JonUpdateCDSUpdated c1auxey wiring plan

I've updated the c1auxey wiring plan for compatibility with the new suspension electronics. Specifically it is based on wiring schematics for the new HAM-A coil driver (D1100117), satellite amplifier (D1002818), and HV bias driver (D1900163).

Changes:

  • The PDMon, VMon, CoilEnable, and BiasAdj channels all move from DB37 to various DB9 breakout boards.
  • The DB9 cables (x2) connecting the CoilEnable channels to the coil drivers must be spliced with the dewhitening switching signals from the RTS.
  • As suggested, I added five new BI channels to monitor the state of the CoilEnable switches. For lack of a better name, they follow the naming convention C1:SUS-ETMY_xx_ENABLEMon.

@Yehonathan can proceed with wiring the chassis.

Quote:

I finished prewiring the new c1auxey Acromag chassis (see attached pictures). I connected all grounds to the DIN rail to save some wiring. The power switches and LEDs work as expected.

I configured the DAQ modules using the old windows machine. I configured the gateway to be 192.168.114.1. The host machine still needs to be setup.

Next, the feedthroughs need to be wired and the channels need to be bench tested.

  16038   Thu Apr 15 18:33:39 2021 KojiUpdateCDSMore 16bit ADC adapter boards

We received 10x 16bit ADC adapter boards from Todd. S2100687~S2100696

The number of soldered resistors seems to be less than that on the schematics. They are related to duotone, so check if it's OK upon use.

  16050   Mon Apr 19 13:15:20 2021 Ian MacMillanUpdateCDS40m LSC simPlant model

The x1SUSsim model on the docker was made in a more recent version of Simulink so I updated Matlab (see this)

I updated Matlab to 2021a so now the docker has 2020b and 2021a installed. This should also install Simulink 10.3 for the sus model to open. I used my account to activate it but I can change it over if I get a dedicated license for this. I am not sure what Jon did for the 2020b that he installed.

it is giving me "License error: -9,57" so I guess it didn't work... I will try to just make the model on the 2020b just so I have something.

-----------------------------------------------------------------------------------------------------------------------------

I was able to fix the problem with the activation (using this).

I can now open the x1sussim.mdl file. It is trying to access a BSFM_MASTER Library that it doesn't seem to have. the other files don't seem to have any warnings though. 

the simple suspension model can be found in home/controls/docker-cymac/userapps/c1simpsus.mdl on the docker system. This is where I will put my model. (right now it is just a copied file)

Also using Simulink on the docker is very slow. I think this is either a limit of the x2goclient software or the hardware that the docker is running on.

  16052   Mon Apr 19 21:54:55 2021 YehonathanUpdateCDSUpdated c1auxey wiring plan

Except for the feed-throughs that require a DB9-M connector I finished wiring and labeling the Acromag, following Jon's updated wiring plan.

We can start testing the differential inputs until the missing connectors arrive.

 

  16059   Wed Apr 21 10:03:01 2021 Ian MacMillanUpdateCDS40m LSC simPlant model

So I am stuck on how to add the control block to my model. I am trying to make it as simple as possible with just a simple transfer function for a damped harmonic oscillator and then the control block (see overview.png). 

The transfer function I am using is:

 H(s)=\frac{Q^{-2} f^{-2}s^2+1}{f^{-4}s^4+(Q^{-2} f^{-2}-2f^{-2})s^2+1}

For future generations: To measure the transfer function (to verify that it is doing what it says it is) I am using the discrete transfer function estimator block. To get a quick transfer function estimator Simulink program run ex_discrete_transfer_function_estimator in the Matlab command line. This works well for filters but it was hit or miss using the discrete transfer function. 

The roadblock I am running into right now is that I can't figure out how to add the controller to the model. Not on an interpretation level but in a very straightforward "can I drag it into my model and will it just work" kind of way.

I am also a little confused as to exactly which block would do the controlling. Because I want to just use the x of the pendulum (its position) I think I want to use the suspension controls which come are connected to in the suspension plant model. But where exactly is it and how can I get the model? I can't seem to find it. 

ELOG V3.1.3-