I spent this afternoon trying to debug fb1, with very little to show for it.  We're back to running from fb.

The first thing I did was to recompile EPICS from source, so that all the libraries needed by daqd were compiled for the system at hand.  I compiled epics-3.14-12-2_long from source, and installed it at /opt/rtapps/epics on local disk, not on the /opt/rtapps network mount.  I then recompiled daqd against that, and the framecpp, gds, etc from the LSCSoft packages.  So everything has been compiled for this version of the OS.  The compilation goes smoothly.

There are two things that I see while running this new daqd on fb1:

instability with mx_streams

The mx stream connection between the front ends and the daqd is flaky.  Everything will run fine for a while, the spontaneously one or all of the mx_stream processes on the front ends will die.  It appears more likely that all mx_stream processes will die at the same time.  It's unclear if this is some sort of chain reaction thing, or if something in daqd or in the network itself is causing them all to die at the same time.  It is independent of whether or not we're using multiple mx "end points" (i.e. a different one for each front end and separate receiver threads in the daqd) or just a single one (all front ends connecting to a single mx receiver thread in daqd).

Frequently daqd will recover from this.  The monit processes on the front ends restart the mx_stream processes and all will be recovered.  However occaissionally, possibly if the mx_streams do not recover fast enough (which seems to be related to how frequently the receiver threads in daqd can clear themselves), daqd will start to choke and will start spitting out the "empty blocks" messages that are harbirnger of doom:

Aborted 2 send requests due to remote peer 00:30:48:be:11:5d (c1iscex:0) disconnected
00:30:48:d6:11:17 (c1iscey:0) disconnected
mx_wait failed in rcvr eid=005, reqn=182; wait did not complete; status code is Remote endpoint is closed
disconnected from the sender on endpoint 005
mx_wait failed in rcvr eid=001, reqn=24; wait did not complete; status code is Remote endpoint is closed
disconnected from the sender on endpoint 001
[Wed Dec  9 18:40:14 2015] main profiler warning: 1 empty blocks in the buffer
[Wed Dec  9 18:40:15 2015] main profiler warning: 0 empty blocks in the buffer
[Wed Dec  9 18:40:16 2015] main profiler warning: 0 empty blocks in the buffer

My suspicion is that this time of failure is tied to the mx stream failures, so we should be looking at the mx connections and network to solve this problem.

frame writing troubles

There's possibly a separate issue associated with writing the second or minute trend files to disk.  With fair regularity daqd will die soon after it starts to write out the trend frames, producing the similar "empty blocks" messages.

To get C1PEM data back into the frames, I removed the new BLRMS blocks, recompiled, reinstalled, re-enabled it in daqd, restarted.

We still really want more headroom in our framebuilder situation. 

I noticed what I thought was excessive movement of the beam spot on ITMX and ETMX on the control room monitors, and when I checked the CDS FE status overview MEDM screen, I saw that c1scx and c1asx had crashed. I ssh-ed into c1iscex and restarted both models, and then restarted fb as well. However, the DAQ-DCO_C1SCX_STATUS indicator remains red even after restarting fb (see attached screenshot). I am not sure how to fix this so I am leaving it as is for now, and the X arm looks to have settled down.

[ericq,gautam]

Forgot to submit this yesterday...

While we were trying to get the X-arm locked to IR using MC2, frame-builder mysteriously crashed, necessitating us having to go down to the computer and perform a hard reboot (after having closed the PSL shutter and turning all the watchdogs to "shutdown"). All the models restarted by themselves, and everything seems back to normal now..

I've set up two IPC channels that take the output from the digital frequency counters and send them to the end front-ends (via the RFM model). A summary of the steps I followed:

1. Set up two Dolphin channels in C1ALS to send the output of the frequency counter blocks to C1RFM (I initially used RFM blocks for these, but eric suggested using Dolphin IPC for the ALS->RFM branch, as they're faster.. Eric's removed the redundant channel names)
2. Set up two RFM channels in C1RFM to send the out put of the frequency counter blocks to C1SCX/Y (along with CDS monitor points to monitor the error rate and a filter module between the ALS->RFM and RFM->SCX/Y IPC blocks - I just followed what seemed to be the convention in the RFM model).
3. Set up the receiving channels in C1SCX and C1SCY
4. Re-compiled and re-started the models in the order C1ALS, C1RFM, C1SCX and C1SCY.

I've set things up such that we can select either the "PZT IN" or the frequency counter as the input to the slow servo, via means of a EPICS variable called "FC_SWITCH" (so C1:ALS-X_FC_SWITCH or C1:ALS-Y_FC_SWITCH). If this is 0, we use the default "PZT IN" signal, while setting it to 1 will change the input to the slow servo to be the frequency readout from the digital frequency counter. I've not updated the MEDM screens to reflect the two new paths yet, but will do so soon. It also remains to install appropriate filters for the servo path that takes the frequency readout as the input.

Tangentially related to this work: I've modified the FC library block so that it outputs frequency in MHz as opposed to Hz, just for convenience..

A few more related changes:

1. The couplers that used to sit on the green beat PDs on the PSL table have now been shifted to the IR broadband PDs in the FOL box so that I can get the IR beat frequency over to the frequency counters. The FOL box itself, along with the fibers that bring IR light to the PSL table, have been relocated to the corner of the PSL table where the green beat PDs sit because of cable length constraints.
2. I've updated the ALS slow control MEDM screen to allow for slow control of the beat frequency. The servo shape for now is essentially just an integrator with a zero at 1 Hz. The idea is to set an offset in the new filter module, which is the desired beat frequency, and let the integrator maintain this beat frequency. One thing I've not taken care of yet is automatically turning this loop off when the IMC loses lock. Screenshot of the modified MEDM screen is attached. 
3. I checked the performance by using the temperature sliders to introduce an offset. The integrator is able to bring the beat frequency back to the setpoint in a few seconds, provided the step I introduced was not two big (~20 counts, but this is a pretty large shift in beat frequency, nearly 20MHz).

To do:

1. Figure out how to deal with the IMC losing lock. I guess this is important if we want to use the IR beatnote as a diagnostic for the state of the X AUX laser.
2. Optimize the servo gains a little - I still see some ringing when I introduce an offset, this could be avoided...

Daily summary pages are blank today. Yesterday is ok

I came in to check the status of the nitrogen and noticed that the striptool panels in the control room were all blank.

• PMC was unlocked but I was able to relock it using the usual procedure
• FB seems to be down: I was unable to ssh into it (or any of the FEs for that matter). I checked the lights on the RAID array, they are all green. I am holding off on doing a hard reboot of FB in case there is some other debugging that can be done first
• None of the watchdogs were tripped, but judging by the green spots on the mirrors, all of them are moving quite a bit. I've shutdown the watchdogs on all the optics except the MC mirrors, but the ITMs and ETMs still seem to be moving quite a bit.

I am leaving things in this state for now. It is unclear why this should have happened, it doesn't seem like there was a power glitch?

We went and looked at the monitor plugged into FB. All kinds of messages were being spammed to the screen (maybe RAM errors), and nothing could be done to interrupt. Sadly, a hard reboot of FB was neccesary.

Video of error messages: https://youtu.be/7rea_kokhPY

After the reboot, it just took a couple of model restarts to get the CDS screen happy.

As discussed in a Wednesday meeting some time ago, we don't need to be writing channels from BLRMS filter modules to frames at 16k (we suspect this is leading to the frequent daqd crashes which were seen the last time we tried setting BLRMS up for all the suspensions). EricQ pointed out to me that there conveniently exists a library block that is much better suited to our purposes, called BLRMS_2k. I've replaced all the BLRMS library blocks in the sus_single_BLRMS library block that I made with there BLRMS_2k blocks. I need to check that the filters used by the BLRMS_2k block (which reside in /opt/rtcds/userapps/release/cds/common/src/BLRMSFILTER.c) are appropriate, after which we can give setting up BLRMS for all the suspensions a second try...

I noticed that the TRY button in the ASS main screen was linking to LSC_TRX instead of LSC_TRY. Gautam fixed it.

Some CDS related things:

Keith Thorne has told us about a potential fix for our framebuilder woes. Jamie is going to be at the 40m next week to implement this, which could interfere with normal interferometer operation - so plan accordingly. 

I spent a little time doing some plumbing in the realtime models for Varun's audio processing work. Specifically, I tried to spin up a new model (C1DAF), running on the c1lsc machine. This included:

• Removing the unused TT3 and TT4 parts from the IOO block in c1ass.mdl, freeing up some DAC outputs on the LSC rack
• Adding an output row to the LSC input matrix which pipes to a shared memory IPC block. (This seemed like the simplest way for the DAFI model to have access to lots of signals with minimal overhead).
• Removing two unused ADC inputs from c1lsc.mdl (that went to things like PD_XXX), to give c1daf.mdl the required two ADC inputs - and to give us the option of feeding in some analog signals.
• Editing the rtsystab file to include c1daf in the list of models that run on c1lsc
• Editing the existing DAFI .mdl file (which just looked like an old recolored cut-n-paste of c1ioo.mdl) to accept the IPC and ADC connections, and one DAC output that would go to the fibox. 

The simple DAFI model compiled and installed without complaint, but doesn't succesfully start. For some reason, the frontend never takes the CPU offline. Jamie will help with this next week. Since things aren't working, these changes have not been commited to the userapps svn. 

Barring objections, starting tomorrow morning, Jamie will be testing the new FB code. The IFO will not be available for other use while this is ongoing.

I am re-starting work on the daqd upgrade again now.  Expect the daqd to be offline for most of the day.  I will report progress.

Summary: new daqd code running overnight test on fb1.  Stability issues persist.

The code is from Keith's "tests/advLigoRTS-40m" branch, which is a branch of the current trunk.  It's supposed to include patches to fix the crashes when multiple frame types are written to disk at the same time.  However, the issue is not fixed:

2016-06-07_20:38:55 about to write frame @ 1149392336
2016-06-07_20:38:55 Begin Full WriteFrame()
2016-06-07_20:38:57 full frame write done in 2seconds
2016-06-07_20:39:11 about to write frame @ 1149392352
2016-06-07_20:39:11 Begin Full WriteFrame()
2016-06-07_20:39:13 full frame write done in 2seconds
2016-06-07_20:39:27 about to write frame @ 1149392368
2016-06-07_20:39:27 Begin Full WriteFrame()
2016-06-07_20:39:29 full frame write done in 2seconds
2016-06-07_20:39:43 about to write second trend frame @ 1149391800
2016-06-07_20:39:43 Begin second trend WriteFrame()
2016-06-07_20:39:43 about to write frame @ 1149392384
2016-06-07_20:39:43 Begin Full WriteFrame()
2016-06-07_20:39:44 full frame write done in 1seconds
2016-06-07_20:39:59 about to write frame @ 1149392400
2016-06-07_20:40:04 Begin Full WriteFrame()
2016-06-07_20:40:04 Second trend frame write done in 21 seconds
2016-06-07_20:40:14 [Tue Jun  7 20:40:14 2016] main profiler warning: 1 empty blocks in the buffer
2016-06-07_20:40:15 [Tue Jun  7 20:40:15 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:16 [Tue Jun  7 20:40:16 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:17 [Tue Jun  7 20:40:17 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:18 [Tue Jun  7 20:40:18 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:19 [Tue Jun  7 20:40:19 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:20 [Tue Jun  7 20:40:20 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:21 [Tue Jun  7 20:40:21 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:22 [Tue Jun  7 20:40:22 2016] main profiler warning: 0 empty blocks in the buffer
2016-06-07_20:40:23 [Tue Jun  7 20:40:23 2016] main profiler warning: 0 empty blocks in the buffer

This failure comes when a full frame (1149392384+16) is written to disk at the same time as a second trend (1149391800+600).  It seems like every time this happens daqd crashes.

I have seen other stability issues as well, maybe caused by mx flakiness, or some sort of GPS time synchronization issue caused by our lack of IRIG-B cards.  I'm going to look to see if I can get the GPS issue taken care of so we take that out of the picture.

For the last couple of hours I've only seen issues with the frame writing every 20 minutes, when the full and second trend frames happen to be written at the same time.  Running overnight to gather more statistics.

38 restarts overnight.  Problem definitely not fixed.  I'll be reverting back to old daqd and fb this morning.  Then regroup and evaluate options.

[EDIT: corrected name of installed card]

We just installed a Spectracom TSyc-PCIe timing card on fb1.  The hope is that this will help with the GPS timeing syncronization issues we've been seeing in the new daqd on fb1, hopefully elliminating some of the potential failure channels.

The driver, called "symmetricom" in the advLigoRTS source (name of product from competing vendor), was built/installed (from DCC T1500227):

Something is wrong with the timing we're getting out of the symmetricom driver, associated with the new spectracom card.

controls@fb1:~/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom 127$ lalapps_tconvert 
1149538884
controls@fb1:~/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom 0$ cat /proc/gps 
704637380.00
controls@fb1:~/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom 0$ 

The GPS time is way off, and it's counting up at something like 900 seconds/second.  Something is misconfigured, but I haven't figured out what yet.

The timing distribution module we're using is spitting out what appears to be an IRIG B122 signal (amplitude moduled 1 kHz carrier), which I think is what we expect.  This is being fed into the "AM IRIG input" connector on the card.

Not sure why the driver is spinning so fast, though, with the wrong baseline time.  Reboot of the machine didn't help.

I've restarted the old daqd on fb until I can figure out what's going on with the symmetricom driver on fb1.

Steve:  Jamie with hair.... long time ago
 

Looks like we might have a problem with the IRIG-B output of the GPS receiver.

Rolf came over this morning to help debug the strange symmetricom driver behavior on fb1 with the new Spectracom card.  We restarted the machine againt and this time when we loaded the drive rit was clocking at a normal rate (second/second).  However, the overall GPS time was still wrong, showing a time in October from this year.

The IRIG-B122 output is supposed to encode the time of year via amplitude modulation of a 1kHz carrier.  The current time of year is:

controls@fb1:~ 0$ TZ=utc date +'%j day, %T'
162 day, 18:57:35
controls@fb1:~ 0$ 

The absolute year is not encoded, though, so the symmetricon driver has the year offset hard coded into the driver (yuck), to which it adds the time of year from the IRIG-B signal to get the correct GPS time.

However, loading the symmetricom module shows the following:

...
[ 1601.607403] Spectracom GPS card on bus 1; device 0
[ 1601.607408] TSYNC PIC BASE 0 address = fb500000
[ 1601.607429] Remapped 0xffffc90017012000
[ 1606.606164] TSYNC NOT receiving YEAR info, defaulting to by year patch
[ 1606.606168] date = 299 days 18:28:1161455320
[ 1606.606169] bcd time = 1161455320 sec  959 milliseconds 398 microseconds  959398630 nanosec
[ 1606.606171] Board sync = 1
[ 1606.616076] TSYNC NOT receiving YEAR info, defaulting to by year patch
[ 1606.616079] date = 299 days 18:28:1161455320
[ 1606.616080] bcd time = 1161455320 sec  969 milliseconds 331 microseconds  969331350 nanosec
[ 1606.616081] Board sync = 1
controls@fb1:~ 0$ 

Apparently the symmetricom driver thinks it's the 299nth day of the year, which of course corresponds to some time in october, which jives with the GPS time the driver is spitting out.

Rolf then noticed that the timing module in the VME crate in the adjacent rack, which also receives an IRIG-B signal from the distribution box, was also showing day 299 on it's front panel display. We checked and confirmed that the symmetricom card and the VME timing module both agree on the wrong time of year, strongly suggesting that the GPS receiver is outputing bogus data on it's IRIG-B output, even though it's showing the correct time on it's front panel.  We played around with setting in the GPS receiver to no avail.  Finally we rebooted the GPS receiver, but it seemed to come up with the same bogus IRIG-B output (again both symmetricom driver and VME timing module agree on the wrong day).

So maybe our GPS receiver is busted?  Not sure what to try now.

The GPS receiver (EndRun Technologies box in 1Y5? (rack closest to door)) seems to not coming back up properly after the reboot.  The front pannel says that it's "LKD", but the "sync" LED is flashing instead of solid, and the time of year displayed on the front panel is showing day 6.  The fb1 symmetricom driver and VME timing module are still both seeing day 299, though.  So something may definitely be screwy with the GPS receiver.

Summary: I am implementing digital audio filtering on various interferometer signals in order to listen to the processed audio which will help in characterizing and noise reduction in the interferometer. following is a summary of the gui i have made towards a general purpose DAF module linked to the LSC. 

Details:  attachment 1 shows the top level overview of the daf module.

The "INPUTS" button shown redirects to the medm screen shown in attachment 2, which is a collection of inputs going into the module.

Each of the buttons shown in "C1DAFI_INPUTS.png" is further linked to various i/o boxes like adc1, adc2, lsc signal and exitation. An example is shown in attachment 3. This is the specific I/O box for the LSC signal.

The field labelled "INPUT_MTRX" is linked to a matrix which routes these 4 inputs to various DSP blocks. Similarly, the "OUTPUT_MTRX" tab is useful for choosing which output goes to the speaker. 

Time and computational load monitoring is done in the "GDS_TP" tab which links to the medm screen shown in attachment 4.

Currently the AGC is successfully implemented as one of the DSP block. The details of the AGC implementation were given in a previous elog: https://nodus.ligo.caltech.edu:8081/40m/12159

I need to make a few changes to the code for Frequency Shifting and Whitening before uploading them on the FE. I will put the details soon.

Some more things that I think need to be added: 

1) "Enable" buttons for each of the DSP blocks.

2) Labels for each of the matrix elements.

3) Further headers and other description for each of the tabs 

I'm running another overnight test with new daqd software on fb1.  The normal daqd process on fb has been shutdown, and the front ends are sending their signals to fb1.

fb1 is running separate data concentrator (dc) and  frame writer (fw) processes, to see if this is a more stable configuration than the all-in-one framebuilder (fb) that we have been trying to run with.  I'll report on the test tomorrow.

I have added Enable buttons for each of the DSP blocks, and labels for the matrix elements. The input matrix takes inputs from each of the 4 channels: ADC1, ADC2, LSC and EXC, and routes them to the audio processing blocks (attachment 2). The output matrix (attachment 3) takes the outputs of the various DSP blocks and routes them to the output and then to the speakers. 

Very encouraging results from the test last night.  The new configuration did not crash once overnight, and seemed to write out full, second trend, and minute trend frames without issue.  However, full validity of all the written out frames has not been confirmed.

overview

The configuration under test involves two separate daqd binaries instead of one.  We usually run with what is referred to as a "framebuilder" (fb) configuration:

• fb: a single daqd binary that:
  • collect the data from the front ends
  • coallate full data into frame file format
  • calculates trend data
  • writes frame files to disk.

The current configuration separates the tasks into multiple separate binaries: a "data concentrator" (dc) and a "frame writer" (fw):

• dc:
  • collect data from front ends
  • coallate full data into frame file format
  • broadcasts frame files over local network
• fw:
  • receives frame files from broadcast
  • calculates trend data
  • writes frame files to disk

This configuration is more like what is run at the sites, where all the various components are separate and run on separate hardware.  In our case, I tried just running the two binaries on the same machine, with the broadcast going over the loopback interface.  None of the systems that use separated daqd tasks see the failures that we've been seeing with the all-in-one fb configuration (and other sites like AEI have also seen).

My guess  is that there's some busted semaphore somewhere in daqd that's being shared between the concentrator and writer components.  The writer component probably aquires the lock while it's writing out the frame, which prevents the concentrator for doing what it needs to be doing while the frame is being written out.  That causes the concentrator to lock up and die if the frame writing takes too long (which it seems to almost necessarily do, especially when trend frames are also being written out).

results

The current configuration hasn't been tweaked or optimized at all.  There is of course basically no documentation on the meaning of the various daqdrc directives.  Hopefully I can get Keith Thorne to help me figure out a well optimized configuration.

There is at least one problem whereby the fw component is issuing an excessively large number of re-transmission requests:

2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 6 packets; port 7097
2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 8 packets; port 7097
2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 3 packets; port 7097
2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 5 packets; port 7097
2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 5 packets; port 7097
2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 5 packets; port 7097
2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 5 packets; port 7097
2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 6 packets; port 7097
2016-06-15_09:46:23 [Wed Jun 15 09:46:23 2016] Ask for retransmission of 1 packets; port 7097

It's unclear why.  Presumably the retransmissions requests are being honored, and the fw eventually gets the data it needs.  Otherwise I would hope that there would be the appropriate errors.

The data is being written out as expected:

 full/11500: total 182G
drwxr-xr-x  2 controls controls 132K Jun 15 09:37 .
-rw-r--r--  1 controls controls  69M Jun 15 09:37 C-R-1150043856-16.gwf
-rw-r--r--  1 controls controls  68M Jun 15 09:37 C-R-1150043840-16.gwf
-rw-r--r--  1 controls controls  68M Jun 15 09:37 C-R-1150043824-16.gwf
-rw-r--r--  1 controls controls  69M Jun 15 09:36 C-R-1150043808-16.gwf
-rw-r--r--  1 controls controls  69M Jun 15 09:36 C-R-1150043792-16.gwf
-rw-r--r--  1 controls controls  68M Jun 15 09:36 C-R-1150043776-16.gwf
-rw-r--r--  1 controls controls  68M Jun 15 09:36 C-R-1150043760-16.gwf
-rw-r--r--  1 controls controls  69M Jun 15 09:35 C-R-1150043744-16.gwf

 trend/second/11500: total 11G
drwxr-xr-x  2 controls controls 4.0K Jun 15 09:29 .
-rw-r--r--  1 controls controls 148M Jun 15 09:29 C-T-1150042800-600.gwf
-rw-r--r--  1 controls controls 148M Jun 15 09:19 C-T-1150042200-600.gwf
-rw-r--r--  1 controls controls 148M Jun 15 09:09 C-T-1150041600-600.gwf
-rw-r--r--  1 controls controls 148M Jun 15 08:59 C-T-1150041000-600.gwf
-rw-r--r--  1 controls controls 148M Jun 15 08:49 C-T-1150040400-600.gwf
-rw-r--r--  1 controls controls 148M Jun 15 08:39 C-T-1150039800-600.gwf
-rw-r--r--  1 controls controls 148M Jun 15 08:29 C-T-1150039200-600.gwf
-rw-r--r--  1 controls controls 148M Jun 15 08:19 C-T-1150038600-600.gwf

 trend/minute/11500: total 152M
drwxr-xr-x 2 controls controls 4.0K Jun 15 07:27 .
-rw-r--r-- 1 controls controls  51M Jun 15 07:27 C-M-1150023600-7200.gwf
-rw-r--r-- 1 controls controls  51M Jun 15 04:31 C-M-1150012800-7200.gwf
-rw-r--r-- 1 controls controls  51M Jun 15 01:27 C-M-1150002000-7200.gwf

The frame sizes look more or less as expected, and they seem to be valid as determined with some quick checks with the framecpp command line utilities.

Visual inspection of rooftop GPS antennae:

The 2 GPS antennas are located on the north west corner of CES roof. Their condition looks well weathered. I'd be surprised if they work.

The BNC connector of the 1553 head is inside of the conduit so it is more likely to have a better connection than the other one.

I have not had a chance to look into the "GPS Time Server" unit.

Just to be clear, there's still quite a bit of work to fully transition the 40m to this new system/configuration.  Once we determine a good configuration we need to complete the install, and modify the setup to run the two binaries instead of just the one.  The data is also being written to a raid on the new fb1, and we need to decide if we should use this new raid, or try to figure out how to move the old jetstor raid to the new fb1 machine.

I wish to have stereo audio output for the DAF module. Hence, there needs to be a second output from the DAF. I added this second output to the model. Following are the details:

FiBox: It consists of two analog inputs which are digitized and multiplexed and transmitted optically. (only 1 fiber is needed due to multiplexing). Attachment 1 shows the fibox with its 2 analog inputs (one of which, is connected), and 1 fiber output. The output of the DAF goes to the FiBox. Until today, the Fibox recieved only 1 analog input. This analog signal comes from the DAC-8 (count starting from 0), which is located at "CH 1 OUT" SMA output in the "MONITORS" bin on the racks (attachment 2).

I have added another output channel to the DAF model both in software and in hardware. The DAF now also uses DAC-9 analog output which goes to the second analog input of the FiBox. The DAC-9 output is located at "CH 2 OUT" SMA output in the "MONITORS" bin on the racks (attachment 4).

After making the changes, the Fibox is shown in attacment 3.

Testing: The LSC input on passing through the DAF block is given through two different DAC outputs, to the same Fibox channel (one after the other), and the output is heard. More concrete testing will be done tomorrow. It will be as follows:

1) Currently, I need to search for a suitable cable that would connect the second channel of the output fibox to the audio mixer. After doing this, end to end testing of both channels will be done.

2) I could not access the AWG, probably because the DAQ was offline today afternoon. Using a signal from the AWG will give a more concrete testing of the stereo output.

3) After this, I will separate the two channels of the stereo completely (currectly they are seperated only at the DAF output stage)

4) I also will edit the medm gui appropriately.

After poking at the new configuration more, it also started to show instability.  I couldn't figure out how to make test points or excitations available in this configuration, and adding in the full set of test point channels, and trying to do simple things like plotting channels with dtt, the frame writer (fw) would fall over, apparetnly unable to keep up with the broadcast from the dc.

I've revered everything back to the old semi-working fb configuration, and will be kicking this to the CDS group to deal with.

I called https://www.endruntechnologies.com/pdf/USM3014-0000-000.pdf  and they said it's very likely just needs a software update. They will email Jamie the details.

I got the email from them.  There was apparently a bug that manifested on February 14 2016.  I'll try to software update today.

http://endruntechnologies.com/pdf/FSB160218.pdf
http://endruntechnologies.com/upgradetemplx.htm

I just upgraded the EndRun Technologies Tempus LX GPS receiver timing unit, and it seems to have fixed all the problems.  

Thanks to Steve for getting the info from EndRun.  There was indeed a bug in the firmware that was fixed with a firmware upgrade.

I upgraded both the system firmware and the firmware of the GPS subsystem:

Tempus LX GPS(root@Tempus:~)-> gntpversion 
Tempus LX GPS 6010-0044-000 v 5.70 - Wed Oct 1 04:28:34 UTC 2014
Tempus LX GPS(root@Tempus:~)-> gpsversion 
F/W 5.10 FPGA 0416
Tempus LX GPS(root@Tempus:~)->

After reboot the system is fully functional, displaying the correct time, and outputting the correct IRIG-B data, as confirmed by the VME timing unit.

I added a wiki page for the unit: https://wiki-40m.ligo.caltech.edu/NTP

Steve added this picture

"medm: command not found" error when run through command line both in pianosa and rossa in both editing and execution modes. It however gets executed and edited through the sitemap button. Don't know the source of the problem. Gautam did check the .bashrc file. aliases for SITEMAP and m40m are intact in the .bashrc file.

The workstations' .bashrc is a symbolic link to /users/controls/.bashrc

In it, someone commented out the critical line:

#source /ligo/cdscfg/workstationrc.sh

I uncommented it. medm (and all of the other things like cdsutils) work again.

I blame jamie.

I have updated the DAFI with the following changes:

1) Separated both the channels of stereo output completely, as well as in the GUI.

2) Added text monitors for the inputs and outputs.

The stereo output is now ready except for a cable going from the second channel of the output fibox to the audio mixer.

Attached is the main DAF_OVERVIEW screen and its link button from the LSC screen labelled "DAFI"

Using an RC to BNC connector from the inner drawer, I have added a second output cable going from the output Fibox in the control room to the audio mixer.

The DAFI block was reviewed by Rana yesterday. The following changes/improvements were suggested: (Updated on 20th July 2016 with tasks taat remain in red)

1) include all the various channels like PEM, LSC, ASC, SUS, SEI, etc. as the inputs. Currently the inputs are only the LSC.
2) include all the control signals.
3) create a very detailed diagram of the entire signal flow and plan tasks accordingly.
4) Enable cascading of various DSP processes.
5) Adjusting the gain of the AGC such that the amplitude of the output signal comes to about half the peak amplitude offered by the ADC. This will help taking advantage of the entire dynamic range of the ADC.
6) change the enable button styles from a text input based controller to a button controller.
7) Currently, disabling a particular signal terminates the signal. Instead, it should turn into a unity gain block on disabling.
8) Check if the Fibox does AC coupling or not. If not, add an AC coupling arrangement in the DAFI.
9) Check the nature of the ADC1 and ADC2 inputs to the DAFI. I checked them yesterday, and they are channels 25 and 26 of ADC0, which are empty.

Attached is a diagram, showing the entire (planned) signal flow of the DAF model. Some thoughts on the implementation after discussion with eric:

1) Since the LSC control signals and ASC signals are running on the c1lsc FE at the same rate as DAFI (16kHz), it would be wise to start from these.

   Current implementation: has a matrix at the end of the LSC PD signals, which selects one of the PD signals and outputs it to the DAFI via IPC communication.

    Proposed Changes: Add another matrix at the end of the LSC PD signals, to give to the second stereo output. Similarly, add two matrices each at the end of the LSC control signals and     the ASC signals. Each matrix must select one of the signals and output it to the DAF via IPC.

2) The PEM running on the c1sus FE system will have to be brought to DAFI in a similar fashon. However, since c1sus runs at 2kHz, there is a possibility of imaging while the signal is    transfered to the DAFI. This could be taken care of by an anti imaging filter, or inserting zeros between two samples coming at to the 16 kHz system from the 2kHz system and then low-passing it to remove the aliased parts. (similar to upsampling)

3) For the SUS control signals, input can be given from a matrix prepared for each optic seperately.

I have added the control signals DARM_CTRL, MICH_CTRL, PRCL_CTRL, SRCL_CTRL, CARM_CTRL, XARM_CTRL, YARM_CTRL, MC_CTRL to the DAFI model from the LSC model via IPC commn.

The changes done to the LSC model include addition of an extra block going to DAFI (attachment 2, red rectangle in attachment 1), and addition of an extra overall output from the LSC, called DAFI_OUT2, which goes to DAFI through IPC link C1:LSC-DAF_2 (attach. 3). Now two distinct inputs can be given to the DAFI, whose intended purpose is to act as two distinct audio signals in the stereo output, but can also be used for arbitrary math. 

I am going to add the following PEM channels as DAF inputs subsequently, in a similar 2 input fashon.

SEIS_GUR1_X_OUT
SEIS_GUR1_Y_OUT
SEIS_GUR1_Z_OUT
SEIS_GUR2_X_OUT
SEIS_GUR2_Y_OUT
SEIS_GUR2_Z_OUT
SEIS_STS_1_X_OUT
SEIS_STS_1_Y_OUT
SEIS_STS_1_Z_OUT
ACC_MC1_X_OUT 
ACC_MC1_Y_OUT
ACC_MC1_Z_OUT
ACC_MC2_X_OUT
ACC_MC2_Y_OUT
ACC_MC2_Z_OUT

For some reason, all of the non-IOP models on the vertex frontends had crashed.

To get all of the bits green, I ended up restarting all models on all frontends. (This means Gautam's coil tests have been interrupted.)

I have added the PEM signals mentioned in the previous elog as DAF inputs through PCIE IPC, and compiled and restarted the c1pem and c1daf models. 

Attached are the pictures of the simulink diagram of the addition in the PEM and the DAF. 

Since the signals are moving from a 2kHz clock rate machine to a 16kHz clock rate machine, some imaging effects are possible, which I have to look into.

Today, at around 10:30, c1lsc machine froze and stopped responding to ping and ssh after I compiled and restarted c1daf. I think it is due to a large array in one of my codes. The daqd.log file shows the following:

..................................................................
CA.Client.Exception...............................................
    Warning: "Virtual circuit unresponsive"
    Context: "c1lsc.martian.113.168.192.in-addr.arpa:5064"
    Source File: ../tcpiiu.cpp line 945
    Current Time: Thu Jul 14 2016 22:27:42.102649102
..................................................................

I think the c1lsc FE may need a hard reboot.

c1lsc is up and running, Eric did a manual reboot today.

Summary: I am trying to implement frequency warping/pitch shifting on the real time FE. Here is a description long overdue:

Description: The overall idea is as follows:

The DFT of a frame   is given by . A matrix W containing all for k, n = 1, 2, ..., m can be calculated and predefined in the code. The input arrival rate is 16384 Hz, i.e. once in every 60 $\mu$s time window. Hence, the fourier coefficients can be updated cumulatively in each cycle using the current value of the input, previous value of the fourier coefficient and the components of the W matrix. This will distribute the computational load of the FFT into all the time windows. Similar operations can be carried out for the inverse STFT.

I have written and run a pseudo-real time code on my CPU. The following is the essence:
 Let the frame-length be M, and the intended scale factor of the frequency warping be 'r'. The frame overlap is 50%. At each clock cycle, the following tasks are performed:  (1 to 3 are routine tasks performed at every clock cycle, 4 is a special task performed only when a frame is filled.)
 1) Take input and apply hanning window to it.
 2) Cumulate for every k using the value of x_i[n] (the input) at that particular instant. Also start to cumulate X_{i+1}[k], which will be later transfered to X_i[k].
 3) Because of 4), we now have 'r+1' filled frames corresponding to output fft. Now take the ifft using two consecutive frames corresponding to only two time series points. The computations required for this task are the same as the computations required for calculation of the fourier coefficients iteratively, since the entire time series ifft is not computed.
 4) Do these special tasks after each frame gets filled:
      At this point, the ffts of the current frame and a previous frame is ready. Let us call them X1 and X2.
      Calculate phase difference between the two.
      Calculate all the interpolated |Y_i| in between these two frames depending upon the scale factor.
      Assign phase of X1 to first Y frame and assign increasing phase to all the other Y frames.
      and also do all the usual non-special tasks.

This code takes about 9-10 microseconds for a cycle with special tasks, and 5-6 microseconds for a cycle with routine tasks on my laptop (brought down from 100 microseconds peak time in the earlier offline implementation due to elemination of explicit dft and reduction in fft size), for a frame size of 32 samples. However, when fed into the c1lsc FE, it crashes, as it has done once again today evening, in the same fashon as yesterday. There could be 2 possible reasons:

1) Size of the array containing the matrix elements is too large for the FE memory,

2) the computations are taking up more than 60 microseconds.

Since there are already a few codes with similar array sizes, I am more inclined to think that 2) is more likely.

Another problem that I am anticipating is that for a 32 point dft and a sampling rate of 16kHz, the frequency resolution achieved is about 500 Hz, which is not sufficient if we need to represent seismic signals. The only way I can think of, for representing such signals with a small number of fft points, is to reduce the effective sampling rate, i.e. do DSP on inputs at a much lower rate than 16kHz (say 1kHz, which will give a resolution of ~30 Hz, or 2kHz giving a resolution of ~60Hz). Another advantage of this method is that it frees up more clock cycles for computation, thus the computational load can be further distributed.  The problem in this implementation is that it will increase the delays.

c1lsc FE is up and running.

Details:

2) The machine was manually rebooted.

3) c1daf was recompiled and installed, with the problematic piece of code removed.

4) NTP timing was adjusted.

5) Frame Builder was restarted.

6) All models on c1lsc machine were restarted.

Attachment 1 shows the CDS status after the recovery. I wont be trying to run frequency warping immediately, I will first finish implementing the other harmless modules first.

Summary: I have added an arbitrary math block to the DAFI model, which takes two inputs, say X and Y, and can perform various unary and binary operations on them:

Details:

• Unary Operations:

1. Delay - There exists a text-based input to specify the amount of delay to be given to a particular signal.
2. sin()
3. cos()

• Binary Operations:

1. Weighted addition and multiplication: The output is calculated according to the relation: A*X + B*Y + C*X*Y. A, B, C are constant inputs, which can be given through text-based inputs in the GUI.
2. MAX{X,Y}
3. MIN{X,Y}

Attachment 1 shows the existing DAFI gui, updated with cascading of various DSP blocks, upto three levels, button-based ENABLE and DISABLE controls for all blocks except arb. math (the control on arb. math. is achieved by clicking on the block.) On clicking the arb. math block one is taken to the dedicated arb. math screen, which has enable buttons for all the processes listed above. A screenshot of this screen is in attachment 2. There is one control input, which controls all the unary operations on X and the binary operations on X and Y, and another control input which controls the unary operations on Y. switching on a particular arb. math process gives a particular control input, which choses the appropriate section of the code. At a time, only one process from the top grey block (corresponding to unary operations on X and binary operations on X and Y) and one process from the bottom grey block (corresponding to unary operations on Y) can be selected. Thus, the outputs which go from the arb. math block to the intermediate matrices (MATRIX1L or MATRIX2L) are:

a) Either an output of unary operation on X or a binary operation on X and Y, the specific one depending upon the control input,

    and

b) Output of a unary operation on Y, again the specific one depending upon the control input

Thus there is apparent asymmetricity in the action of the arb. math block on the two inputs. However, this is done in order to reduce to total number of outputs and control signals, and this can be easily taken care of by interchanging the inputs before the block. 

While compiling this code, the c1lsc machine had crashed once, it was found that this was due to a stray "printf()" command in the c code. This glich in the code now stands rectified  There is a possibility that the previous incidents of the code crashing could also be due to the existence of a printf() command. 

Preliminary Testing: I have done a preliminary testing of the arb math block, i.e. verified that on enabling the sin and cos processes, the output is less that 1, on swithching on the process of weighted avarage and multiplication, the output looks like it is right, for a few simple values of A, B, C, like 0, 1, etc. The delay block however is giving zero output for delay of more than 6 samples.

1) I have added the status summary of the DAFI block to the main FE status overview screen in the c1lsc cloumn. (attachment 1)

2) I have edited all the kissel matrix buttons appropriately, and given them appropriate lables. (attachment 2)

The code for frequency warping contained a "printf()" command, which had caused the system to crash in one another instance (refer elog 12320) . Hence, I tried running the code tody by removing this line. Unfortunately, this did not work. the model still crashed. Attached is the screenshot of the FE status.

c1susaux (which controls watchdogs and alignments for all non-ETM optics) was down, the last BURT was done yesterday around 2PM. 

I restarted via keying the crate. I restored the BURT snapshot from yesterday.