Kiwamu, Koji, Valera

We centered the beam on MC2 in pitch by moving the MC1,2,3 in the following combination [-9,+3,-7]. This actuation vector mostly moves the spot on MC2 vertically. The attached plot shows the dither before and after the centering. We monitored the demodulated signals and saw the reduction of the MC2 pit response from -1.0 to -0.22 which corresponds to the beam spot position change from 9 to 2 mm. Thus all the spots on MC mirrors are within 2 mm of the center. We estimate based on the distance between the MC1-MC3 of 20 cm, the distance from the center between MC1 and MC3 to the end of the Faraday isolator of 80 cm, and the aperture of the FI of 12 mm, the maximum angle out of MC of 3/200 rad. Which implies the maximum differential spot motion of 3 mm not to be limited by the FI aperture.

Yesterday we found that MC3 OSEM LL PD did not have a sensible signal - the readback was close to zero and it was making MC move around. I disabled the PD LL so that the damping is done with just three face plus side PDs. There still no signal from MC3 LL PD today. It needs debugging.

This is what was done in past two days:

- The ETMY and ITMY pitch and yaw dofs are modulated at 40, 44, 42, 46 Hz respectively (oscillator A=30). The c1ass lockin numbers are 12, 14, 27, 29.

- The NAS55I signal is demodulated at the above frequencies. The demodulated I/Q signal phase is set to shift all signal into I-phase. The lockin inputs are bandpassed around respective frequency f with butter("Bandpass",2,f-0.5,f+0.5). The demod signals are then additionally low passed with butter ("Lowpass",4,0.5) so the servo ugf has to be below 0.5 Hz. The servo filter is p:z 0.0001:0.1.

- The ETMY demodulated signal is fed back to ITMY and visa versa.

- With the above 2x2 servo running we moved the input beam PZTs by hand to follow the cavity.

- At the end we offloaded the servo control signals to the SUS biases again by hand.

- The beam spot centering was estimated by unbalancing the ETMY/ITMY pitch/yaw coil combinations intentionally by 5%, which produces 1.3 mm shift of the node, and comparing the response to the residual signals.

- The dof set up currently is: ETMY pitch lockin 12 -> dof2, ITMY pitch lockin 14 -> dof4, ETMY yaw lockin 27 -> dof7, ITMY yaw lockin 29 -> dof9

- The next step is to demodulate the TRY(X) and servo the input beam PZTs

Here the status of the dither alignment or c1ass:

- Both pitch and yaw centering on ETMY/ITMY were closed simultatenously with ugf of ~1/30 Hz.

- I made a medm screen with beam positions as measured by the dither system.The snapshot is attached. There are visual perimeter alarms (red box around the display) to warn about arm power being low or the dither lines not being on. The screen has a pull down menu with 4 scripts:

. assUp - sets up the gains, phases and matricies for the dither system (both the spot centering and the input beam alignment)

. assOn - turns on the dithers and servo - just the Y-arm centering part at the moment

. assOff - turns off the servo and dither lines

. assDitherOn - turns on the dither lines but does not turn on the servo

- All scripts are in scripts/ASS and the medm screen is in medm/c1ass/master/

Still to do:

- Commission the input beam and X-arm servos

- Make scripts for X-arm

I closed all 8 dither loops for the Y arm initial alignment: 2x2 centering servo (this worked before) and 2x2 input beam servo for both pitch and yaw.

So far it looks pretty good - the error points go to zero and the arm power goes up to 1.

The offloading to the alignment biases and the PZTs is not yet automated.

Today the PMC, MC, and Y arm were very cooperative and a pleasure to work with.

I changed optics in the ETMX transmon path to remove clipping (which made a false QPD signal).

During the weekend I found that there was an offset in X arm c1ass pitch servo, which derives the signal by demodulating the arm cavity power, coming from the beam clipping in the transmon path.

The clipping was on the pair of the 1" mirrors that steer the beam after the 2" lens (see attached picture). The beam is about 5-6 mm in diameter at this distance from the lens and was not well centered.

I moved the steering mirrors downstream by about 8" where the beam is about 2-3 mm (the attached picture shows the mirrors in the new location). The Y arm layout is different from X arm and I didn't find any obvious clipping in transmon path.

The max X arm buildup went up from 1.3 to 1.5. I changed the TRX gain from -0.003 to -0.002 to obtain the normalized X arm power of 1 in this state. The MC refl DC is 1.6 out of 4.9 V and the Y arm buildup is ~0.9 so the TRX(Y) gains will have to be adjusted once the MC visibility is maximized.

 The current status of the dither alignment system:

- Both Xarm and Yarm alignment are working. The scripts are: scripts/autoDither/alignX(Y). Each script sets up the respective arm, turns on the dither lines and servos for 66 sec, offloads the control signals to TM alignment biases and PZT sliders in case of  Yarm, and to TM and BS alignment biases in case of Xarm, and finally turns off and clears the servo filters and turns off the dither lines.

- Jammie witnessed the final tests of both scripts - both X and Y arm power went up from 0.6-0.7 to close to 1 and the AS beam became symmetric. Also Jammie wanted me to leave the ETMY oplev in its current non-nominal but more stable state i.e. the oplev signals go to the ADC from the D010033 card not the D020432 one. The scripts can now run from the CONFIGURE medm screen.

- Both arms use signals derived from modulating ITM and ETM in pitch and yaw dofs and demodulating the arm power (TRX or TRY) and the cavity length signal (AS55I). The Yarm actuation has 8 dofs - pitch and yaw of the ITM, ETM, and two input beam PZTs so all the sensed dofs are controlled. The Xarm actuation has only 6 dofs - pitch and yaw of the ITM, ETM, and BS. The Xarm servo is set up to servo the beam position on the ETMX and the relative alignment of the cavity and the input beam. The ITMX spot position is unconstrained and provides the null test. The residual displacement on the ITMX is 0.2-0.3 mm in yaw and 0.9-1.0 mm in pitch. The I phases of the beam centering lockins, which are also the error points of corresponding DOF filters, are calibrated in mm by unbalancing the TM coils by known amount. The attached snap shot of the medm screen now has both X and Y arm calibrated beam spot positions and uncalibrated input beam indicators. The input beam angle and position signals can/should be calibrated by tapping the signals digitally and applying the proper matrix transformation - this will require the model change.

- Currently there is no lock loss catching in the model. We should add a trigger on arm power (or an equivalent mechanism) to turn off the inputs to prevent the spurious inputs.

 We installed the ISS AOM in the PSL. The AOM was placed right after the EOM. The beam diameter is ~600 um at the AOM. The AOM aperture is 3 mm.

We monitored the beam size by scanning the leakage beam through the turning mirror after the AOM. The beam diameter changed from 525 um to 515 um at a fixed point. We decided that the AOM thermal lensing is not large enough to require a  new scan of the mode going into the PMC and we can proceed with PMC mode matching using the scan that was taken without the AOM (to be posted).

Hi Steve,

The mode cleaner was misaligned probably due to the earthquake (the drop in the MC transmitted value slightly after utc 7:38:52 as seen in the second plot). The plots show PMC transmitted and MC sum signals from 10th june 07:10:08 UTC over a duration of 17 hrs. The PMC was realigned at about 4-4:15 pm today by rana. This can be seen in the first plot.

Summary: I am implementing digital audio filtering on various interferometer signals in order to listen to the processed audio which will help in characterizing and noise reduction in the interferometer. following is a summary of the gui i have made towards a general purpose DAF module linked to the LSC. 

Details:  attachment 1 shows the top level overview of the daf module.

The "INPUTS" button shown redirects to the medm screen shown in attachment 2, which is a collection of inputs going into the module.

Each of the buttons shown in "C1DAFI_INPUTS.png" is further linked to various i/o boxes like adc1, adc2, lsc signal and exitation. An example is shown in attachment 3. This is the specific I/O box for the LSC signal.

The field labelled "INPUT_MTRX" is linked to a matrix which routes these 4 inputs to various DSP blocks. Similarly, the "OUTPUT_MTRX" tab is useful for choosing which output goes to the speaker. 

Time and computational load monitoring is done in the "GDS_TP" tab which links to the medm screen shown in attachment 4.

Currently the AGC is successfully implemented as one of the DSP block. The details of the AGC implementation were given in a previous elog: https://nodus.ligo.caltech.edu:8081/40m/12159

I need to make a few changes to the code for Frequency Shifting and Whitening before uploading them on the FE. I will put the details soon.

Some more things that I think need to be added: 

1) "Enable" buttons for each of the DSP blocks.

2) Labels for each of the matrix elements.

3) Further headers and other description for each of the tabs 

I have added Enable buttons for each of the DSP blocks, and labels for the matrix elements. The input matrix takes inputs from each of the 4 channels: ADC1, ADC2, LSC and EXC, and routes them to the audio processing blocks (attachment 2). The output matrix (attachment 3) takes the outputs of the various DSP blocks and routes them to the output and then to the speakers. 

I wish to have stereo audio output for the DAF module. Hence, there needs to be a second output from the DAF. I added this second output to the model. Following are the details:

FiBox: It consists of two analog inputs which are digitized and multiplexed and transmitted optically. (only 1 fiber is needed due to multiplexing). Attachment 1 shows the fibox with its 2 analog inputs (one of which, is connected), and 1 fiber output. The output of the DAF goes to the FiBox. Until today, the Fibox recieved only 1 analog input. This analog signal comes from the DAC-8 (count starting from 0), which is located at "CH 1 OUT" SMA output in the "MONITORS" bin on the racks (attachment 2).

I have added another output channel to the DAF model both in software and in hardware. The DAF now also uses DAC-9 analog output which goes to the second analog input of the FiBox. The DAC-9 output is located at "CH 2 OUT" SMA output in the "MONITORS" bin on the racks (attachment 4).

After making the changes, the Fibox is shown in attacment 3.

Testing: The LSC input on passing through the DAF block is given through two different DAC outputs, to the same Fibox channel (one after the other), and the output is heard. More concrete testing will be done tomorrow. It will be as follows:

1) Currently, I need to search for a suitable cable that would connect the second channel of the output fibox to the audio mixer. After doing this, end to end testing of both channels will be done.

2) I could not access the AWG, probably because the DAQ was offline today afternoon. Using a signal from the AWG will give a more concrete testing of the stereo output.

3) After this, I will separate the two channels of the stereo completely (currectly they are seperated only at the DAF output stage)

4) I also will edit the medm gui appropriately.

"medm: command not found" error when run through command line both in pianosa and rossa in both editing and execution modes. It however gets executed and edited through the sitemap button. Don't know the source of the problem. Gautam did check the .bashrc file. aliases for SITEMAP and m40m are intact in the .bashrc file.

I have updated the DAFI with the following changes:

1) Separated both the channels of stereo output completely, as well as in the GUI.

2) Added text monitors for the inputs and outputs.

The stereo output is now ready except for a cable going from the second channel of the output fibox to the audio mixer.

Attached is the main DAF_OVERVIEW screen and its link button from the LSC screen labelled "DAFI"

Using an RC to BNC connector from the inner drawer, I have added a second output cable going from the output Fibox in the control room to the audio mixer.

I am trying to design an antialiasing filter, which also has two switchable whitening stages. I have designed a first version of a PCB for this.

The board takes differential input through PCB mountable BNCs. It consists of an instrumentaiton amplifier made using quad opamp ADA4004, followed by two whitening blocks, also made using ADA4004, which can be bypassed if needed, depending upon a control input. The mux used for this purpose is Maxim MAX4158EUA. These two whitening blocks are followed by 2 the LPF stages. A third LPF stage could be added if needed. These use AD829 opamps. After the LPFs are two amplifiers for giving a differential output through two output BNCs. The schematic is shown in attachment 1: "AA.pdf". The top layers of the layout are shown in attachment 2 (AAtop.pdf), the bottom layers in attachment 3 (AAbottom.pdf), and the entire layout in attachment 4 (AAbrd.pdf). 

The board has 6 layers (in the order from top to bottom):

1) Top signal layer; 

2) Internal plane 1 (GND),

3) Internal plane 2 (+15V),

4) Internal plane 3 (-15V),

5) Internal plane 4 (GND),

6) Bottom signal layer. 

Power: +15, -15 and GND is given through a 4 pin header connector. 

The dimensions of the board are 1550 mil  6115 mil (38.1mm155.3mm) and the overall dimensions including the protruding BNC edges are 1550 mil  7675 mil (38.1mm194.9mm)

I would like to have inputs on the layout telling me if any component/trace needs to be changed/better placed, any other things about the board need to be changed, etc.

P.S.: I have also added a zipped folder "AA.zip" containing the schematic and board files, as well as the above pdfs.

Attached is a diagram, showing the entire (planned) signal flow of the DAF model. Some thoughts on the implementation after discussion with eric:

1) Since the LSC control signals and ASC signals are running on the c1lsc FE at the same rate as DAFI (16kHz), it would be wise to start from these.

   Current implementation: has a matrix at the end of the LSC PD signals, which selects one of the PD signals and outputs it to the DAFI via IPC communication.

    Proposed Changes: Add another matrix at the end of the LSC PD signals, to give to the second stereo output. Similarly, add two matrices each at the end of the LSC control signals and     the ASC signals. Each matrix must select one of the signals and output it to the DAF via IPC.

2) The PEM running on the c1sus FE system will have to be brought to DAFI in a similar fashon. However, since c1sus runs at 2kHz, there is a possibility of imaging while the signal is    transfered to the DAFI. This could be taken care of by an anti imaging filter, or inserting zeros between two samples coming at to the 16 kHz system from the 2kHz system and then low-passing it to remove the aliased parts. (similar to upsampling)

3) For the SUS control signals, input can be given from a matrix prepared for each optic seperately.

I have added the control signals DARM_CTRL, MICH_CTRL, PRCL_CTRL, SRCL_CTRL, CARM_CTRL, XARM_CTRL, YARM_CTRL, MC_CTRL to the DAFI model from the LSC model via IPC commn.

The changes done to the LSC model include addition of an extra block going to DAFI (attachment 2, red rectangle in attachment 1), and addition of an extra overall output from the LSC, called DAFI_OUT2, which goes to DAFI through IPC link C1:LSC-DAF_2 (attach. 3). Now two distinct inputs can be given to the DAFI, whose intended purpose is to act as two distinct audio signals in the stereo output, but can also be used for arbitrary math. 

I am going to add the following PEM channels as DAF inputs subsequently, in a similar 2 input fashon.

SEIS_GUR1_X_OUT
SEIS_GUR1_Y_OUT
SEIS_GUR1_Z_OUT
SEIS_GUR2_X_OUT
SEIS_GUR2_Y_OUT
SEIS_GUR2_Z_OUT
SEIS_STS_1_X_OUT
SEIS_STS_1_Y_OUT
SEIS_STS_1_Z_OUT
ACC_MC1_X_OUT 
ACC_MC1_Y_OUT
ACC_MC1_Z_OUT
ACC_MC2_X_OUT
ACC_MC2_Y_OUT
ACC_MC2_Z_OUT

I have added the PEM signals mentioned in the previous elog as DAF inputs through PCIE IPC, and compiled and restarted the c1pem and c1daf models. 

Attached are the pictures of the simulink diagram of the addition in the PEM and the DAF. 

Since the signals are moving from a 2kHz clock rate machine to a 16kHz clock rate machine, some imaging effects are possible, which I have to look into.

Today, at around 10:30, c1lsc machine froze and stopped responding to ping and ssh after I compiled and restarted c1daf. I think it is due to a large array in one of my codes. The daqd.log file shows the following:

..................................................................
CA.Client.Exception...............................................
    Warning: "Virtual circuit unresponsive"
    Context: "c1lsc.martian.113.168.192.in-addr.arpa:5064"
    Source File: ../tcpiiu.cpp line 945
    Current Time: Thu Jul 14 2016 22:27:42.102649102
..................................................................

I think the c1lsc FE may need a hard reboot.

c1lsc is up and running, Eric did a manual reboot today.

Summary: I am trying to implement frequency warping/pitch shifting on the real time FE. Here is a description long overdue:

Description: The overall idea is as follows:

The DFT of a frame   is given by . A matrix W containing all for k, n = 1, 2, ..., m can be calculated and predefined in the code. The input arrival rate is 16384 Hz, i.e. once in every 60 $\mu$s time window. Hence, the fourier coefficients can be updated cumulatively in each cycle using the current value of the input, previous value of the fourier coefficient and the components of the W matrix. This will distribute the computational load of the FFT into all the time windows. Similar operations can be carried out for the inverse STFT.

I have written and run a pseudo-real time code on my CPU. The following is the essence:
 Let the frame-length be M, and the intended scale factor of the frequency warping be 'r'. The frame overlap is 50%. At each clock cycle, the following tasks are performed:  (1 to 3 are routine tasks performed at every clock cycle, 4 is a special task performed only when a frame is filled.)
 1) Take input and apply hanning window to it.
 2) Cumulate for every k using the value of x_i[n] (the input) at that particular instant. Also start to cumulate X_{i+1}[k], which will be later transfered to X_i[k].
 3) Because of 4), we now have 'r+1' filled frames corresponding to output fft. Now take the ifft using two consecutive frames corresponding to only two time series points. The computations required for this task are the same as the computations required for calculation of the fourier coefficients iteratively, since the entire time series ifft is not computed.
 4) Do these special tasks after each frame gets filled:
      At this point, the ffts of the current frame and a previous frame is ready. Let us call them X1 and X2.
      Calculate phase difference between the two.
      Calculate all the interpolated |Y_i| in between these two frames depending upon the scale factor.
      Assign phase of X1 to first Y frame and assign increasing phase to all the other Y frames.
      and also do all the usual non-special tasks.

This code takes about 9-10 microseconds for a cycle with special tasks, and 5-6 microseconds for a cycle with routine tasks on my laptop (brought down from 100 microseconds peak time in the earlier offline implementation due to elemination of explicit dft and reduction in fft size), for a frame size of 32 samples. However, when fed into the c1lsc FE, it crashes, as it has done once again today evening, in the same fashon as yesterday. There could be 2 possible reasons:

1) Size of the array containing the matrix elements is too large for the FE memory,

2) the computations are taking up more than 60 microseconds.

Since there are already a few codes with similar array sizes, I am more inclined to think that 2) is more likely.

Another problem that I am anticipating is that for a 32 point dft and a sampling rate of 16kHz, the frequency resolution achieved is about 500 Hz, which is not sufficient if we need to represent seismic signals. The only way I can think of, for representing such signals with a small number of fft points, is to reduce the effective sampling rate, i.e. do DSP on inputs at a much lower rate than 16kHz (say 1kHz, which will give a resolution of ~30 Hz, or 2kHz giving a resolution of ~60Hz). Another advantage of this method is that it frees up more clock cycles for computation, thus the computational load can be further distributed.  The problem in this implementation is that it will increase the delays.

c1lsc FE is up and running.

Details:

2) The machine was manually rebooted.

3) c1daf was recompiled and installed, with the problematic piece of code removed.

4) NTP timing was adjusted.

5) Frame Builder was restarted.

6) All models on c1lsc machine were restarted.

Attachment 1 shows the CDS status after the recovery. I wont be trying to run frequency warping immediately, I will first finish implementing the other harmless modules first.

Summary: I have added an arbitrary math block to the DAFI model, which takes two inputs, say X and Y, and can perform various unary and binary operations on them:

Details:

• Unary Operations:

1. Delay - There exists a text-based input to specify the amount of delay to be given to a particular signal.
2. sin()
3. cos()

• Binary Operations:

1. Weighted addition and multiplication: The output is calculated according to the relation: A*X + B*Y + C*X*Y. A, B, C are constant inputs, which can be given through text-based inputs in the GUI.
2. MAX{X,Y}
3. MIN{X,Y}

Attachment 1 shows the existing DAFI gui, updated with cascading of various DSP blocks, upto three levels, button-based ENABLE and DISABLE controls for all blocks except arb. math (the control on arb. math. is achieved by clicking on the block.) On clicking the arb. math block one is taken to the dedicated arb. math screen, which has enable buttons for all the processes listed above. A screenshot of this screen is in attachment 2. There is one control input, which controls all the unary operations on X and the binary operations on X and Y, and another control input which controls the unary operations on Y. switching on a particular arb. math process gives a particular control input, which choses the appropriate section of the code. At a time, only one process from the top grey block (corresponding to unary operations on X and binary operations on X and Y) and one process from the bottom grey block (corresponding to unary operations on Y) can be selected. Thus, the outputs which go from the arb. math block to the intermediate matrices (MATRIX1L or MATRIX2L) are:

a) Either an output of unary operation on X or a binary operation on X and Y, the specific one depending upon the control input,

    and

b) Output of a unary operation on Y, again the specific one depending upon the control input

Thus there is apparent asymmetricity in the action of the arb. math block on the two inputs. However, this is done in order to reduce to total number of outputs and control signals, and this can be easily taken care of by interchanging the inputs before the block. 

While compiling this code, the c1lsc machine had crashed once, it was found that this was due to a stray "printf()" command in the c code. This glich in the code now stands rectified  There is a possibility that the previous incidents of the code crashing could also be due to the existence of a printf() command. 

Preliminary Testing: I have done a preliminary testing of the arb math block, i.e. verified that on enabling the sin and cos processes, the output is less that 1, on swithching on the process of weighted avarage and multiplication, the output looks like it is right, for a few simple values of A, B, C, like 0, 1, etc. The delay block however is giving zero output for delay of more than 6 samples.

1) I have added the status summary of the DAFI block to the main FE status overview screen in the c1lsc cloumn. (attachment 1)

2) I have edited all the kissel matrix buttons appropriately, and given them appropriate lables. (attachment 2)

The code for frequency warping contained a "printf()" command, which had caused the system to crash in one another instance (refer elog 12320) . Hence, I tried running the code tody by removing this line. Unfortunately, this did not work. the model still crashed. Attached is the screenshot of the FE status.

• The slotted disks that capture the wires do not have the alignment bore used to center the wire in the hole
  
• We didn't correctly route the far field QPD cable so it runs funny
  
• We didn't have a tool which could be used to get two of the DCPD preamp box mounting screws (which are M3's chub!)
  
• We don't have the cable clamps to tie off the electrical cables to the intermediate mass
  
• We don't have any of the cabling from the OMC-SUS top to the rack so we can't test anything
  
• We haven't uploaded pretty pictures for all to see
  

• The bottom plate of the EQ stops is too thick so that it bumps into the tombstones
  
• The vertical member on the "waist" EQ stops is too close to the breadboard, possibly interfering with the REFL beam
  
• The "waist" EQ stops are made from a thin plate that doesn't have enough thickness to mount helicoils in
  
• Helicoil weren't loaded in the correct bottom EQ stops
  
• The DCPD cable loops over the end EQ stop looking nasty but not actually making contact
  

• ND10 and ND20 neutral density filter
  
• EOM and mount set for 4 inch beam height
  
• Post for fiber launch to get to 4 inch
  
• Mode matching lens at 4in
  
• 3x steering mirror at 4in
  
• RF photodiode at 4in
  
• Post for camera to 4in
  
• Light sheild for camera
  
• Long BNC cable
  

• DitherLock_sweep: Sweep of the IN2/IN1 for the dither lock error point showing 600 Hz UGF
  
• HiResPZTDither_sweep: Sweep of the PZT dither input compared to the PDH error signal. I restarted the front end before the sweep was finished accounting for the blip.
  
• HiResPZTDither_sweep2: Finish of the PZT dither sweep
  

• FIgure 1: PZT dither path. Most of the features in this plot are understood: There is a 2kHz high pass filter in the PZT drive which is otherwise flat. The resonance features above 5 kHz are believed to be the tombstones. I don't understand the extra motion from 1-2 kHz.
  
• Figure 2: PZT dither path zoom in. Since I want to dither the PZT to get an error signal, it helps to know where to dither. The ADC Anti-aliasing filter is a 3rd order butterworth at 10 kHz, so I looked for nice flat places below 10 KHz and settled on 8 kHz as relatively harmless.
  
• Figure 3: PZT LSC path. This path has got a 1^2:10^2 de-whitening stage in the hardware which hasn't been digitally compensated for. You can see its effect between 10 and 40 Hz. The LSC path also has a 160 Hz low path which is visible causing a 1/f between 200 and 500 Hz. I have no idea what the 1 kHz resonant feature is, though I am inclined to point to the PDH loop since that is pretty close to the UGF and there is much gain peaking at that frequency.
  

• Save your file /cvs/cds/advLigo/src/epics/simLink/tpt.mdl
  
• go to /cvs/cds/advLIGO on the TPT machine
  
• do make clean-tpt tpt install-tpt
  
• do rm /cvs/cds/caltech/chans/daq/C2TPT.ini (this step is needed because the DAQ install code isn't quite right at the time of this writing.
  
• do make install-daq-tpt
  
• run starttpt to restart the tpt computer.
  

Rolf came by today morning. For now, we've restarted the FE machine and the expansion chassis (note that the correct order in which to do this is: turn off computer--->turn off expansion chassis--->turn on expansion chassis--->turn on computer). The debugging measures Rolf suggested are (i) to replace the old generation ADC card in the expansion chassis which has a red indicator light always on and (ii) to replace the PCIe fiber (2010 make) running from the c1lsc front-end machine in 1X6 to the expansion chassis in 1Y3, as the manufacturer has suggested that pre-2012 versions of the fiber are prone to failure. We will do these opportunistically and see if there is any improvement in the situation.

Another tip from Rolf: if the c1lsc FE is responsive but the models have crashed, then doing sudo reboot by ssh-ing into c1lsc should suffice* (i.e. it shouldn't take down the models on the other vertex FEs, although if the FE is unresponsive and you hard reboot it, this may still be a problem). I'll modify I've modified the c1lsc reboot script accordingly.

* Seems like this can still lead to the other vertex FEs crashing, so I'm leaving the reboot script as is (so all vertex machines are softly rebooted when c1lsc models crash).