40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Wed Feb 24 22:13:47 2021, Jon, Update, CDS, Planning document for front-end testing 
    Reply  Fri Mar 5 17:48:25 2021, Jon, Update, CDS, Front-end testing image_67203585.JPGimage_67216641.JPGimage_17185537.JPG
       Reply  Tue Mar 9 16:52:47 2021, Jon, Update, CDS, Front-end testing image_72192707.JPG
          Reply  Tue Mar 16 16:27:22 2021, Jon, Update, CDS, Front-end testing 
             Reply  Tue Mar 16 19:04:20 2021, gautam, Update, CDS, Front-end testing 
             Reply  Fri Mar 19 18:14:56 2021, Jon, Update, CDS, Front-end testing image_72192707_(1).JPGimage_50412545.JPG
                Reply  Wed Mar 24 19:02:21 2021, Jon, Update, CDS, Front-end testing 
                   Reply  Mon Mar 29 17:55:50 2021, Jon, Update, CDS, Front-end testing 
                      Reply  Tue Mar 30 18:21:34 2021, Jon, Update, CDS, Front-end testing 
                      Reply  Tue Apr 6 07:19:11 2021, Jon, Update, CDS, New SimPlant cymac 
                         Reply  Tue Apr 6 11:13:01 2021, Jon, Update, CDS, FE testing 
                            Reply  Sat Apr 10 08:51:32 2021, Jon, Update, CDS, I/O Chassis Assembly 
                               Reply  Thu Apr 29 10:51:35 2021, Jon, Update, CDS, I/O Chassis Assembly 
                                  Reply  Tue May 4 07:38:36 2021, Jon, Update, CDS, I/O Chassis Assembly Screen_Shot_2021-05-03_at_4.16.06_PM.png
                                     Reply  Tue May 11 16:29:55 2021, Jon, Update, CDS, I/O Chassis Assembly Screen_Shot_2021-05-11_at_3.03.42_PM.png
                                        Reply  Tue May 11 17:43:09 2021, Koji, Update, CDS, I/O Chassis Assembly 
                                           Reply  Fri May 28 11:16:21 2021, Jon, Update, CDS, Front-End Assembly and Testing c1bhd.pnggds_tp.pngteststand.jpegbench_supply.jpeg
                                              Reply  Sun Jun 6 08:42:05 2021, Jon, Update, CDS, Front-End Assembly and Testing c1bhd.png16bit_dacs.pngmyricom.png
                                                 Reply  Tue Jun 22 16:53:01 2021, Ian MacMillan, Update, CDS, Front-End Assembly and Testing 
                                                    Reply  Thu Jun 24 17:32:52 2021, Ian MacMillan, Update, CDS, Front-End Assembly and Testing C1-SU2_Channel_Responses.pdfC1-BHD_Channel_Responses.pdfCDS_Channel_Test.zip
                                                       Reply  Fri Jun 25 14:06:10 2021, Jon, Update, CDS, Front-End Assembly and Testing test_stand.JPG
Message ID: 16167     Entry time: Fri May 28 11:16:21 2021     In reply to: 16131     Reply to this: 16185
Author: Jon 
Type: Update 
Category: CDS 
Subject: Front-End Assembly and Testing 

An update on recent progress in the lab towards building and testing the new FEs.

1. Timing problems resolved / FE BIOS changes

The previously reported problem with the IOPs losing sync after a few minutes (16130) was resolved through a change in BIOS settings. However, there are many required settings and it is not trivial to get these right, so I document the procedure here for future reference.

The CDS group has a document (T1300430) listing the correct settings for each type of motherboard used in aLIGO. All of the machines received from LLO contain the oldest motherboards: the Supermicro X8DTU. Quoting from the document, the BIOS must be configured to enforce the following:

• Remove hyper-threading so the CPU doesn’t try to run stuff on the idle core, as hyperthreading simulate two cores for every physical core.
• Minimize any system interrupts from hardware, such as USB and Serial Ports, that might get through to the ‘idled’ core. This is needed on the older machines.
• Prevent the computer from reducing the clock speed on any cores to ‘save power’, etc. We need to have a constant clock speed on every ‘idled’ CPU core.

I generally followed the T1300430 instructions but found a few adjustments were necessary for diskless and deterministic operation, as noted below. The procedure for configuring the FE BIOS is as follows:

  1. At boot-up, hit the delete key to enter the BIOS setup screen.
  2. Before changing anything, I recommend photographing or otherwise documenting the current working settings on all the subscreens, in case for some reason it is necessary to revert.
  3. T1300430 assumes the process is started from a known state and lists only the non-default settings that must be changed. To put the BIOS into this known state, first navigate to Exit > Load Failsafe Defaults > Enter.
  4. Configure the non-default settings following T1300430 (Sec. 5 for the X8DTU motherboard). On the IPMI screen, set the static IP address and netmask to their specific assigned values, but do set the gateway address to all zeros as the document indicates. This is to prevent the IPMI from trying to initiate outgoing connections.
  5. For diskless booting to continue to work, it is also necessary to set Advanced > PCI/PnP Configuration > Load Onboard LAN 1 Option Rom > Enabled.
  6. I also found it was necessary to re-enable IDE direct memory access and WHEA (Windows Hardware Error Architecture) support. Since these machines have neither hard disks nor Windows, I have no idea why these are needed, but I found that without them, one of the FEs would hang during boot about 50% of the time.
    • Advanced > PCI/PnP configuration > PCI IDE BusMaster  > Enabled.
    • Advanced > ACPI Configuration > WHEA Support > Enabled.

After completing the BIOS setup, I rebooted the new FEs about six times each to make sure the configuration was stable (i.e., would never hang during boot).

2. User models created for FE testing

With the timing issue resolved, I proceeded to build basic user models for c1bhd and c1sus2 for testing purposes. Each one has a simple structure where M ADC inputs are routed through IIR filters to an output matrix, which forms linear signal combinations that are routed to N DAC outputs. This is shown in Attachment 1 for the c1bhd case, where the signals from a single ADC are conditioned and routed to a single 18-bit DAC. The c1sus2 case is similar; however the Contec BO modules still needed to be added to this model.

The FEs are now running two models each: the IOP model and one user model. The assigned parameters of each model are documented below.

Model Host CPU DCUID Path
c1x06 c1bhd 1 23 /opt/rtcds/userapps/release/cds/c1/models/c1x06.mdl
c1x07 c1sus2 1 24 /opt/rtcds/userapps/release/cds/c1/models/c1x07.mdl
c1bhd c1bhd 2 25 /opt/rtcds/userapps/release/isc/c1/models/c1bhd.mdl
c1sus2 c1sus2 2 26 /opt/rtcds/userapps/release/sus/c1/models/c1sus2.mdl

The user models were compiled and installed following the previously documented procedure (15979). As shown in Attachment 2, all the RTS processes are now working, with the exception of the DAQ server (for which we're still awaiting hardware). Note that these models currently exist only on the cloned copy of the /opt/rtcds disk running on the test stand. The plan is to copy these models to the main 40m disk later, once the new FEs are ready to be installed.

3. AA and AI chassis installed

I installed several new AA and AI chassis in the test stand to interface with the ADC and DAC cards. This includes three 16-bit AA chassis, one 16-bit AI chassis, and one 18-bit AI chassis, as pictured in Attachment 3. All of the AA/AI chassis are powered by one of the new 15V DC power strips connected to a bench supply, which is housed underneath the computers as pictured in Attachment 4.

These chassis have not yet been tested, beyond verifying that the LEDs all illuminate to indicate that power is present.

Attachment 1: c1bhd.png  8.790 MB  Uploaded Fri May 28 13:54:42 2021  | Show | Show all
Attachment 2: gds_tp.png  13.324 MB  Uploaded Fri May 28 14:14:06 2021  | Show | Show all
Attachment 3: teststand.jpeg  1.527 MB  Uploaded Fri May 28 14:19:38 2021  | Show | Show all
Attachment 4: bench_supply.jpeg  1.271 MB  Uploaded Fri May 28 14:22:58 2021  | Show | Show all
ELOG V3.1.3-