40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Wed Feb 24 22:13:47 2021, Jon, Update, CDS, Planning document for front-end testing 
    Reply  Fri Mar 5 17:48:25 2021, Jon, Update, CDS, Front-end testing image_67203585.JPGimage_67216641.JPGimage_17185537.JPG
       Reply  Tue Mar 9 16:52:47 2021, Jon, Update, CDS, Front-end testing image_72192707.JPG
          Reply  Tue Mar 16 16:27:22 2021, Jon, Update, CDS, Front-end testing 
             Reply  Tue Mar 16 19:04:20 2021, gautam, Update, CDS, Front-end testing 
             Reply  Fri Mar 19 18:14:56 2021, Jon, Update, CDS, Front-end testing image_72192707_(1).JPGimage_50412545.JPG
                Reply  Wed Mar 24 19:02:21 2021, Jon, Update, CDS, Front-end testing 
                   Reply  Mon Mar 29 17:55:50 2021, Jon, Update, CDS, Front-end testing 
                      Reply  Tue Mar 30 18:21:34 2021, Jon, Update, CDS, Front-end testing 
                      Reply  Tue Apr 6 07:19:11 2021, Jon, Update, CDS, New SimPlant cymac 
                         Reply  Tue Apr 6 11:13:01 2021, Jon, Update, CDS, FE testing 
                            Reply  Sat Apr 10 08:51:32 2021, Jon, Update, CDS, I/O Chassis Assembly 
                               Reply  Thu Apr 29 10:51:35 2021, Jon, Update, CDS, I/O Chassis Assembly 
                                  Reply  Tue May 4 07:38:36 2021, Jon, Update, CDS, I/O Chassis Assembly Screen_Shot_2021-05-03_at_4.16.06_PM.png
                                     Reply  Tue May 11 16:29:55 2021, Jon, Update, CDS, I/O Chassis Assembly Screen_Shot_2021-05-11_at_3.03.42_PM.png
                                        Reply  Tue May 11 17:43:09 2021, Koji, Update, CDS, I/O Chassis Assembly 
                                           Reply  Fri May 28 11:16:21 2021, Jon, Update, CDS, Front-End Assembly and Testing c1bhd.pnggds_tp.pngteststand.jpegbench_supply.jpeg
                                              Reply  Sun Jun 6 08:42:05 2021, Jon, Update, CDS, Front-End Assembly and Testing c1bhd.png16bit_dacs.pngmyricom.png
                                                 Reply  Tue Jun 22 16:53:01 2021, Ian MacMillan, Update, CDS, Front-End Assembly and Testing 
                                                    Reply  Thu Jun 24 17:32:52 2021, Ian MacMillan, Update, CDS, Front-End Assembly and Testing C1-SU2_Channel_Responses.pdfC1-BHD_Channel_Responses.pdfCDS_Channel_Test.zip
                                                       Reply  Fri Jun 25 14:06:10 2021, Jon, Update, CDS, Front-End Assembly and Testing test_stand.JPG
Message ID: 16185     Entry time: Sun Jun 6 08:42:05 2021     In reply to: 16167     Reply to this: 16220
Author: Jon 
Type: Update 
Category: CDS 
Subject: Front-End Assembly and Testing 

Here is an update and status report on the new BHD front-ends (FEs).

Timing

The changes to the FE BIOS settings documented in [16167] do seem to have solved the timing issues. The RTS models ran for one week with no more timing failures. The IOP model on c1sus2 did die due to an unrelated "Channel hopping detected" error. This was traced back to a bug in the Simulink model, where two identical CDS parts were both mapped to ADC_0 instead of ADC_0/1. I made this correction and recompiled the model following the procedure in [15979].

Model naming standardization

For lack of a better name, I had originally set up the user model on c1sus2 as "c1sus2.mdl" This week I standardized the name to follow the three-letter subsystem convention, as four letters lead to some inconsistency in the naming of the auto-generated MEDM screens. I renamed the model c1sus2.mdl -> c1su2.mdl. The updated table of models is below.

Model Host CPU DCUID Path
c1x06 c1bhd 1 23 /opt/rtcds/userapps/release/cds/c1/models/c1x06.mdl
c1x07 c1sus2 1 24 /opt/rtcds/userapps/release/cds/c1/models/c1x07.mdl
c1bhd c1bhd 2 25 /opt/rtcds/userapps/release/isc/c1/models/c1bhd.mdl
c1su2 c1su2 2 26 /opt/rtcds/userapps/release/sus/c1/models/c1su2.mdl

Renaming an RTS model requires several steps to fully propagate the change, so I've documented the procedure below for future reference.

On the target FE, first stop the model to be renamed:

controls@c1sus2$ rtcds stop c1sus2

Then, navigate to the build directory and run the uninstall and cleanup scripts:

controls@c1sus2$ cd /opt/rtcds/caltech/c1/rtbuild/release
controls@c1sus2$ make uninstall-c1sus2
controls@c1sus2$ make clean-c1sus2

Unfortunately, the uninstall script does not remove every vestige of the old model, so some manual cleanup is required. First, open the file /opt/rtcds/caltech/c1/target/gds/param/testpoint.par and manually delete the three-line entry corresponding to the old model:

hostname=c1sus2
system=c1sus2
[C-node26]

If this is not removed, reinstallation of the renamed model will fail because its assigned DCUID will appear to already be in use. Next, find all relics of the old model using:

controls@c1sus2$ find /opt/rtcds/caltech/c1 -iname "*sus2*"

and manually delete each file and subdirectory containing the "sus2" name. Finally, rename, recompile, reinstall, and relaunch the model:

controls@c1sus2$ cd /opt/rtcds/userapps/release/sus/c1/models
controls@c1sus2$ mv c1sus2.mdl c1su2.mdl
controls@c1sus2$ cd /opt/rtcds/caltech/c1/rtbuild/release
controls@c1sus2$ make c1su2
controls@c1sus2$ make install-c1su2
controls@c1sus2$ rtcds start c1su2

Sitemap screens

I used a tool developed by Chris, mdl2adl, to auto-generate a set of temporary sitemap/model MEDM screens. This package parses each Simulink file and generates an MEDM screen whose background is an .svg image of the Simulink model. Each object in the image is overlaid with a clickable button linked to the auto-generated RTS screens. An example of the screen for the C1BHD model is shown in Attachment 1. Having these screens will make the testing much faster and less user-error prone.

I generated these screens following the instructions in Chris' README. However, I ran this script on the c1sim machine, where all the dependencies including Matlab 2021 are already set up. I simply copied the target .mdl files to the root level of the mdl2adl repo, ran the script (./mdl2adl.sh c1x06 c1x07 c1bhd c1su2), and then copied the output to /opt/rtcds/caltech/c1/medm/medm_teststand. Then I redefined the "sitemap" environment variable on the chiara clone to point to this new location, so that they can be launched in the teststand via the usual "sitemap" command.

Current status and plans

Is it possible to convert 18-bit AO channels to 16-bit?

Currently, we are missing five 18-bit DACs needed to complete the c1sus2 system (the c1bhd system is complete). Since the first shipment, we have had no luck getting additional 18-bit DACs from the sites, and I don't know when more will become available. So, this week I took an inventory of all the 16-bit DACs available at the 40m. I located four 16-bit DACs, pictured in Attachment 2. Their operational states are unknown, but none were labeled as known not to work.

The original CDS design would call for 40 more 18-bit DAC channels. Between the four 16-bit DACs there are 64 channels, so if only 3/4 of these DACs work we would have enough AO channels. However, my search turned up zero additional 16-bit DAC adapter boards. We could check if first Rolf or Todd have any spares. If not, I think it would be relatively cheap and fast to have four new adapters fabricated.

DAQ network limitations and plan

To get deeper into the signal-integrity aspect of the testing, it is going to be critical to get the secondary DAQ network running in the teststand. Of all the CDS tools (Ndscope, Diaggui, DataViewer, StripTool), only StripTool can be used without a functioning NDS server (which, in turn, requires a functioning DAQ server). StripTool connects directly to the EPICS server run by the RTS process. As such, StripTool is useful for basic DC tests of the fast channels, but it can only access the downsampled monitor channels. Ian and Anchal are going to carry out some simple DAC-to-ADC loopback tests to the furthest extent possible using StripTool (using DC signals) and will document their findings separately.

We don't yet have a working DAQ network because we are still missing one piece of critical hardware: a 10G switch compatible with the older Myricom network cards. In the older RCG version 3.x used by the 40m, the DAQ code is hardwired to interface with a Myricom 10G PCIe card. I was able to locate a spare Myricom card, pictured in Attachment 3, in the old fb machine. Since it looks like it is going to take some time to get an old 10G switch from the sites, I went ahead and ordered one this week. I have not been able to find documentation on our particular Myricom card, so it might be compatible with the latest 10G switches but I just don't know. So instead I bought exactly the same older (discontinued) model as is used in the 40m DAQ network, the Netgear GSM7352S. This way we'll also have a spare. The unit I bought is in "like-new" condition and will unfortunately take about a week to arrive.

Attachment 1: c1bhd.png  19.008 MB  Uploaded Sun Jun 6 12:46:56 2021  | Hide | Hide all
c1bhd.png
Attachment 2: 16bit_dacs.png  9.694 MB  Uploaded Sun Jun 6 12:47:15 2021  | Hide | Hide all
16bit_dacs.png
Attachment 3: myricom.png  16.184 MB  Uploaded Sun Jun 6 12:48:32 2021  | Hide | Hide all
myricom.png
ELOG V3.1.3-