40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 53 of 346  Not logged in ELOG logo
ID Date Author Type Categoryup Subject
  12158   Wed Jun 8 13:50:39 2016 jamieConfigurationCDSSpectracom IRIG-B card installed on fb1

[EDIT: corrected name of installed card]

We just installed a Spectracom TSyc-PCIe timing card on fb1.  The hope is that this will help with the GPS timeing syncronization issues we've been seeing in the new daqd on fb1, hopefully elliminating some of the potential failure channels.

The driver, called "symmetricom" in the advLigoRTS source (name of product from competing vendor), was built/installed (from DCC T1500227):

controls@fb1:~/rtscore/tests/advLigoRTS-40m 0$ cd src/drv/symmetricom/
controls@fb1:~/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom 0$ ls
Makefile  stest.c  symmetricom.c  symmetricom.h
controls@fb1:~/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom 0$ make
make -C /lib/modules/3.2.0-4-amd64/build SUBDIRS=/home/controls/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom modules
make[1]: Entering directory `/usr/src/linux-headers-3.2.0-4-amd64'
  CC [M]  /home/controls/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom/symmetricom.o
/home/controls/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom/symmetricom.c:59:9: warning: initialization from incompatible pointer type [enabled by default]
/home/controls/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom/symmetricom.c:59:9: warning: (near initialization for ‘symmetricom_fops.unlocked_ioctl’) [enabled by default]
/home/controls/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom/symmetricom.c: In function ‘get_cur_time’:
/home/controls/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom/symmetricom.c:89:2: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
/home/controls/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom/symmetricom.c: In function ‘symmetricom_init’:
/home/controls/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom/symmetricom.c:188:2: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
/home/controls/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom/symmetricom.c:222:3: warning: label ‘out_remove_proc_entry’ defined but not used [-Wunused-label]
/home/controls/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom/symmetricom.c:158:22: warning: unused variable ‘pci_io_addr’ [-Wunused-variable]
/home/controls/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom/symmetricom.c:156:6: warning: unused variable ‘i’ [-Wunused-variable]
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/controls/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom/symmetricom.mod.o
  LD [M]  /home/controls/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom/symmetricom.ko
make[1]: Leaving directory `/usr/src/linux-headers-3.2.0-4-amd64'
controls@fb1:~/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom 0$ sudo make install
#remove all old versions of the driver
find /lib/modules/3.2.0-4-amd64 -name symmetricom.ko -exec rm -f {} \; || true
find /lib/modules/3.2.0-4-amd64 -name symmetricom.ko.gz -exec rm -f {} \; || true
# Install new driver
install -D -m 644 symmetricom.ko /lib/modules/3.2.0-4-amd64/extra/symmetricom.ko
/sbin/depmod -a || true
/sbin/modprobe symmetricom
if [ -e /dev/symmetricom ] ; then \
        rm -f /dev/symmetricom ; \
    fi
mknod /dev/symmetricom c `grep symmetricom /proc/devices|awk '{print $1}'` 0
chown controls /dev/symmetricom
controls@fb1:~/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom 0$ ls /dev/symmetricom
/dev/symmetricom
controls@fb1:~/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom 0$ ls -al /dev/symmetricom
crw-r--r-- 1 controls root 250, 0 Jun  8 13:42 /dev/symmetricom
controls@fb1:~/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom 0$ 
  12161   Thu Jun 9 13:28:07 2016 jamieConfigurationCDSSpectracom IRIG-B card installed on fb1

Something is wrong with the timing we're getting out of the symmetricom driver, associated with the new spectracom card.

controls@fb1:~/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom 127$ lalapps_tconvert 
1149538884
controls@fb1:~/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom 0$ cat /proc/gps 
704637380.00
controls@fb1:~/rtscore/tests/advLigoRTS-40m/src/drv/symmetricom 0$ 

The GPS time is way off, and it's counting up at something like 900 seconds/second.  Something is misconfigured, but I haven't figured out what yet.

The timing distribution module we're using is spitting out what appears to be an IRIG B122 signal (amplitude moduled 1 kHz carrier), which I think is what we expect.  This is being fed into the "AM IRIG input" connector on the card.

Not sure why the driver is spinning so fast, though, with the wrong baseline time.  Reboot of the machine didn't help.

  12162   Thu Jun 9 15:14:46 2016 jamieUpdateCDSold fb restarted, test of new daqdon fb1 aborted for time being

I've restarted the old daqd on fb until I can figure out what's going on with the symmetricom driver on fb1.

Steve:  Jamie with hair.... long time ago
 

Attachment 1: Jamie.jpg
Jamie.jpg
  12166   Fri Jun 10 12:09:01 2016 jamieConfigurationCDSIRIG-B debugging

Looks like we might have a problem with the IRIG-B output of the GPS receiver.

Rolf came over this morning to help debug the strange symmetricom driver behavior on fb1 with the new Spectracom card.  We restarted the machine againt and this time when we loaded the drive rit was clocking at a normal rate (second/second).  However, the overall GPS time was still wrong, showing a time in October from this year.

The IRIG-B122 output is supposed to encode the time of year via amplitude modulation of a 1kHz carrier.  The current time of year is:

controls@fb1:~ 0$ TZ=utc date +'%j day, %T'
162 day, 18:57:35
controls@fb1:~ 0$ 

The absolute year is not encoded, though, so the symmetricon driver has the year offset hard coded into the driver (yuck), to which it adds the time of year from the IRIG-B signal to get the correct GPS time.

However, loading the symmetricom module shows the following:

...
[ 1601.607403] Spectracom GPS card on bus 1; device 0
[ 1601.607408] TSYNC PIC BASE 0 address = fb500000
[ 1601.607429] Remapped 0xffffc90017012000
[ 1606.606164] TSYNC NOT receiving YEAR info, defaulting to by year patch
[ 1606.606168] date = 299 days 18:28:1161455320
[ 1606.606169] bcd time = 1161455320 sec  959 milliseconds 398 microseconds  959398630 nanosec
[ 1606.606171] Board sync = 1
[ 1606.616076] TSYNC NOT receiving YEAR info, defaulting to by year patch
[ 1606.616079] date = 299 days 18:28:1161455320
[ 1606.616080] bcd time = 1161455320 sec  969 milliseconds 331 microseconds  969331350 nanosec
[ 1606.616081] Board sync = 1
controls@fb1:~ 0$ 

Apparently the symmetricom driver thinks it's the 299nth day of the year, which of course corresponds to some time in october, which jives with the GPS time the driver is spitting out.

Rolf then noticed that the timing module in the VME crate in the adjacent rack, which also receives an IRIG-B signal from the distribution box, was also showing day 299 on it's front panel display. We checked and confirmed that the symmetricom card and the VME timing module both agree on the wrong time of year, strongly suggesting that the GPS receiver is outputing bogus data on it's IRIG-B output, even though it's showing the correct time on it's front panel.  We played around with setting in the GPS receiver to no avail.  Finally we rebooted the GPS receiver, but it seemed to come up with the same bogus IRIG-B output (again both symmetricom driver and VME timing module agree on the wrong day).

So maybe our GPS receiver is busted?  Not sure what to try now.

 

  12167   Fri Jun 10 12:21:54 2016 jamieConfigurationCDSGPS receiver not resetting properly

The GPS receiver (EndRun Technologies box in 1Y5? (rack closest to door)) seems to not coming back up properly after the reboot.  The front pannel says that it's "LKD", but the "sync" LED is flashing instead of solid, and the time of year displayed on the front panel is showing day 6.  The fb1 symmetricom driver and VME timing module are still both seeing day 299, though.  So something may definitely be screwy with the GPS receiver.

  12173   Mon Jun 13 20:01:30 2016 varunUpdateCDSDAFI GUI update

Summary: I am implementing digital audio filtering on various interferometer signals in order to listen to the processed audio which will help in characterizing and noise reduction in the interferometer. following is a summary of the gui i have made towards a general purpose DAF module linked to the LSC. 

Details:  attachment 1 shows the top level overview of the daf module.

The "INPUTS" button shown redirects to the medm screen shown in attachment 2, which is a collection of inputs going into the module.

Each of the buttons shown in "C1DAFI_INPUTS.png" is further linked to various i/o boxes like adc1, adc2, lsc signal and exitation. An example is shown in attachment 3. This is the specific I/O box for the LSC signal.

The field labelled "INPUT_MTRX" is linked to a matrix which routes these 4 inputs to various DSP blocks. Similarly, the "OUTPUT_MTRX" tab is useful for choosing which output goes to the speaker. 

Time and computational load monitoring is done in the "GDS_TP" tab which links to the medm screen shown in attachment 4.

Currently the AGC is successfully implemented as one of the DSP block. The details of the AGC implementation were given in a previous elog: https://nodus.ligo.caltech.edu:8081/40m/12159

I need to make a few changes to the code for Frequency Shifting and Whitening before uploading them on the FE. I will put the details soon.

Some more things that I think need to be added: 

1) "Enable" buttons for each of the DSP blocks.

2) Labels for each of the matrix elements.

3) Further headers and other description for each of the tabs 

Attachment 1: C1DAF_OVERVIEW.png
C1DAF_OVERVIEW.png
Attachment 2: C1DAF_INPUTS.png
C1DAF_INPUTS.png
Attachment 3: C1DAF_LSC.png
C1DAF_LSC.png
Attachment 4: MONITOR.png
MONITOR.png
  12179   Tue Jun 14 19:37:40 2016 jamieUpdateCDSOvernight daqd test underway

I'm running another overnight test with new daqd software on fb1.  The normal daqd process on fb has been shutdown, and the front ends are sending their signals to fb1.

fb1 is running separate data concentrator (dc) and  frame writer (fw) processes, to see if this is a more stable configuration than the all-in-one framebuilder (fb) that we have been trying to run with.  I'll report on the test tomorrow.

  12180   Tue Jun 14 20:10:19 2016 varunUpdateCDSDAFI GUI update

I have added Enable buttons for each of the DSP blocks, and labels for the matrix elements. The input matrix takes inputs from each of the 4 channels: ADC1, ADC2, LSC and EXC, and routes them to the audio processing blocks (attachment 2). The output matrix (attachment 3) takes the outputs of the various DSP blocks and routes them to the output and then to the speakers. 

Attachment 1: C1DAF_OVERVIEW.png
C1DAF_OVERVIEW.png
Attachment 2: input_matrix.png
input_matrix.png
Attachment 3: output_matrix.png
output_matrix.png
  12181   Wed Jun 15 09:52:02 2016 jamieUpdateCDSVery encouraging results from overnight split daqd test

laughVery encouraging results from the test last night.  The new configuration did not crash once overnight, and seemed to write out full, second trend, and minute trend frames without issueyes.  However, full validity of all the written out frames has not been confirmed.

overview

The configuration under test involves two separate daqd binaries instead of one.  We usually run with what is referred to as a "framebuilder" (fb) configuration:

  • fb: a single daqd binary that:
    • collect the data from the front ends
    • coallate full data into frame file format
    • calculates trend data
    • writes frame files to disk.

The current configuration separates the tasks into multiple separate binaries: a "data concentrator" (dc) and a "frame writer" (fw):

  • dc:
    • collect data from front ends
    • coallate full data into frame file format
    • broadcasts frame files over local network
  • fw:
    • receives frame files from broadcast
    • calculates trend data
    • writes frame files to disk

This configuration is more like what is run at the sites, where all the various components are separate and run on separate hardware.  In our case, I tried just running the two binaries on the same machine, with the broadcast going over the loopback interface.  None of the systems that use separated daqd tasks see the failures that we've been seeing with the all-in-one fb configuration (and other sites like AEI have also seen).

My guess frown is that there's some busted semaphore somewhere in daqd that's being shared between the concentrator and writer components.  The writer component probably aquires the lock while it's writing out the frame, which prevents the concentrator for doing what it needs to be doing while the frame is being written out.  That causes the concentrator to lock up and die if the frame writing takes too long (which it seems to almost necessarily do, especially when trend frames are also being written out).

results

The current configuration hasn't been tweaked or optimized at all.  There is of course basically no documentation on the meaning of the various daqdrc directives.  Hopefully I can get Keith Thorne to help me figure out a well optimized configuration.

There is at least one problem whereby the fw component is issuing an excessively large number of re-transmission requests:

2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 6 packets; port 7097
2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 8 packets; port 7097
2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 3 packets; port 7097
2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 5 packets; port 7097
2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 5 packets; port 7097
2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 5 packets; port 7097
2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 5 packets; port 7097
2016-06-15_09:46:22 [Wed Jun 15 09:46:22 2016] Ask for retransmission of 6 packets; port 7097
2016-06-15_09:46:23 [Wed Jun 15 09:46:23 2016] Ask for retransmission of 1 packets; port 7097

It's unclear why.  Presumably the retransmissions requests are being honored, and the fw eventually gets the data it needs.  Otherwise I would hope that there would be the appropriate errors.

The data is being written out as expected:

 full/11500: total 182G
drwxr-xr-x  2 controls controls 132K Jun 15 09:37 .
-rw-r--r--  1 controls controls  69M Jun 15 09:37 C-R-1150043856-16.gwf
-rw-r--r--  1 controls controls  68M Jun 15 09:37 C-R-1150043840-16.gwf
-rw-r--r--  1 controls controls  68M Jun 15 09:37 C-R-1150043824-16.gwf
-rw-r--r--  1 controls controls  69M Jun 15 09:36 C-R-1150043808-16.gwf
-rw-r--r--  1 controls controls  69M Jun 15 09:36 C-R-1150043792-16.gwf
-rw-r--r--  1 controls controls  68M Jun 15 09:36 C-R-1150043776-16.gwf
-rw-r--r--  1 controls controls  68M Jun 15 09:36 C-R-1150043760-16.gwf
-rw-r--r--  1 controls controls  69M Jun 15 09:35 C-R-1150043744-16.gwf

 trend/second/11500: total 11G
drwxr-xr-x  2 controls controls 4.0K Jun 15 09:29 .
-rw-r--r--  1 controls controls 148M Jun 15 09:29 C-T-1150042800-600.gwf
-rw-r--r--  1 controls controls 148M Jun 15 09:19 C-T-1150042200-600.gwf
-rw-r--r--  1 controls controls 148M Jun 15 09:09 C-T-1150041600-600.gwf
-rw-r--r--  1 controls controls 148M Jun 15 08:59 C-T-1150041000-600.gwf
-rw-r--r--  1 controls controls 148M Jun 15 08:49 C-T-1150040400-600.gwf
-rw-r--r--  1 controls controls 148M Jun 15 08:39 C-T-1150039800-600.gwf
-rw-r--r--  1 controls controls 148M Jun 15 08:29 C-T-1150039200-600.gwf
-rw-r--r--  1 controls controls 148M Jun 15 08:19 C-T-1150038600-600.gwf

 trend/minute/11500: total 152M
drwxr-xr-x 2 controls controls 4.0K Jun 15 07:27 .
-rw-r--r-- 1 controls controls  51M Jun 15 07:27 C-M-1150023600-7200.gwf
-rw-r--r-- 1 controls controls  51M Jun 15 04:31 C-M-1150012800-7200.gwf
-rw-r--r-- 1 controls controls  51M Jun 15 01:27 C-M-1150002000-7200.gwf

The frame sizes look more or less as expected, and they seem to be valid as determined with some quick checks with the framecpp command line utilities.

  12182   Wed Jun 15 10:19:10 2016 SteveConfigurationCDSGPS antennas... debugging

Visual inspection of rooftop GPS antennae:

The 2 GPS antennas are located on the north west corner of CES roof. Their condition looks well weathered. I'd be surprised if they work.

The BNC connector of the 1553 head is inside of the conduit so it is more likely to have a better connection than the other one.

I have not had a chance to look into the "GPS Time Server" unit.

Attachment 1: GPStimeServer.jpg
GPStimeServer.jpg
Attachment 2: GPSsn.jpg
GPSsn.jpg
Attachment 3: GPSantennas.jpg
GPSantennas.jpg
Attachment 4: GPSantHead.jpg
GPSantHead.jpg
Attachment 5: GPSantHead2.jpg
GPSantHead2.jpg
Attachment 6: GPSantCabels.jpg
GPSantCabels.jpg
  12183   Wed Jun 15 11:21:51 2016 jamieUpdateCDSstill work to do to transition to new configuration/code

Just to be clear, there's still quite a bit of work to fully transition the 40m to this new system/configuration.  Once we determine a good configuration we need to complete the install, and modify the setup to run the two binaries instead of just the one.  The data is also being written to a raid on the new fb1, and we need to decide if we should use this new raid, or try to figure out how to move the old jetstor raid to the new fb1 machine.

  12185   Wed Jun 15 22:12:55 2016 varunUpdateCDSDAFI update: stereo output

I wish to have stereo audio output for the DAF module. Hence, there needs to be a second output from the DAF. I added this second output to the model. Following are the details:

FiBox: It consists of two analog inputs which are digitized and multiplexed and transmitted optically. (only 1 fiber is needed due to multiplexing). Attachment 1 shows the fibox with its 2 analog inputs (one of which, is connected), and 1 fiber output. The output of the DAF goes to the FiBox. Until today, the Fibox recieved only 1 analog input. This analog signal comes from the DAC-8 (count starting from 0), which is located at "CH 1 OUT" SMA output in the "MONITORS" bin on the racks (attachment 2).

I have added another output channel to the DAF model both in software and in hardware. The DAF now also uses DAC-9 analog output which goes to the second analog input of the FiBox. The DAC-9 output is located at "CH 2 OUT" SMA output in the "MONITORS" bin on the racks (attachment 4).

After making the changes, the Fibox is shown in attacment 3.

Testing: The LSC input on passing through the DAF block is given through two different DAC outputs, to the same Fibox channel (one after the other), and the output is heard. More concrete testing will be done tomorrow. It will be as follows:

1) Currently, I need to search for a suitable cable that would connect the second channel of the output fibox to the audio mixer. After doing this, end to end testing of both channels will be done.

2) I could not access the AWG, probably because the DAQ was offline today afternoon. Using a signal from the AWG will give a more concrete testing of the stereo output.

3) After this, I will separate the two channels of the stereo completely (currectly they are seperated only at the DAF output stage)

4) I also will edit the medm gui appropriately.

 

Quote:

I have added Enable buttons for each of the DSP blocks, and labels for the matrix elements. The input matrix takes inputs from each of the 4 channels: ADC1, ADC2, LSC and EXC, and routes them to the audio processing blocks (attachment 2). The output matrix (attachment 3) takes the outputs of the various DSP blocks and routes them to the output and then to the speakers. 

 

Attachment 1: IMG_20160615_145535907.jpg
IMG_20160615_145535907.jpg
Attachment 2: IMG_20160615_145413005_HDR.jpg
IMG_20160615_145413005_HDR.jpg
Attachment 3: IMG_20160616_101229499.jpg
IMG_20160616_101229499.jpg
Attachment 4: IMG_20160616_101157096.jpg
IMG_20160616_101157096.jpg
  12191   Thu Jun 16 16:11:11 2016 jamieUpdateCDSupgrade aborted for now

After poking at the new configuration more, it also started to show instability.  I couldn't figure out how to make test points or excitations available in this configuration, and adding in the full set of test point channels, and trying to do simple things like plotting channels with dtt, the frame writer (fw) would fall over, apparetnly unable to keep up with the broadcast from the dc.

I've revered everything back to the old semi-working fb configuration, and will be kicking this to the CDS group to deal with.

  12200   Mon Jun 20 11:11:20 2016 SteveConfigurationCDSGPS receiver not resetting properly

I called https://www.endruntechnologies.com/pdf/USM3014-0000-000.pdf  and they said it's very likely just needs a software update. They will email Jamie the details.

Quote:

The GPS receiver (EndRun Technologies box in 1Y5? (rack closest to door)) seems to not coming back up properly after the reboot.  The front pannel says that it's "LKD", but the "sync" LED is flashing instead of solid, and the time of year displayed on the front panel is showing day 6.  The fb1 symmetricom driver and VME timing module are still both seeing day 299, though.  So something may definitely be screwy with the GPS receiver.

 

  12201   Mon Jun 20 11:19:41 2016 jamieConfigurationCDSGPS receiver not resetting properly
Quote:

I called https://www.endruntechnologies.com/pdf/USM3014-0000-000.pdf  and they said it's very likely just needs a software update. They will email Jamie the details.

I got the email from them.  There was apparently a bug that manifested on February 14 2016.  I'll try to software update today.

http://endruntechnologies.com/pdf/FSB160218.pdf
http://endruntechnologies.com/upgradetemplx.htm

  12202   Mon Jun 20 14:03:04 2016 jamieConfigurationCDSEndRun GPS receiver upgraded, fixed

I just upgraded the EndRun Technologies Tempus LX GPS receiver timing unit, and it seems to have fixed all the problems.  cool

Thanks to Steve for getting the info from EndRun.  There was indeed a bug in the firmware that was fixed with a firmware upgrade.

I upgraded both the system firmware and the firmware of the GPS subsystem:

Tempus LX GPS(root@Tempus:~)-> gntpversion 
Tempus LX GPS 6010-0044-000 v 5.70 - Wed Oct 1 04:28:34 UTC 2014
Tempus LX GPS(root@Tempus:~)-> gpsversion 
F/W 5.10 FPGA 0416
Tempus LX GPS(root@Tempus:~)->

After reboot the system is fully functional, displaying the correct time, and outputting the correct IRIG-B data, as confirmed by the VME timing unit.

I added a wiki page for the unit: https://wiki-40m.ligo.caltech.edu/NTP

 

Steve added this picture

Attachment 1: GPSreceiverWorking.jpg
GPSreceiverWorking.jpg
  12207   Tue Jun 21 11:26:42 2016 varunFrogsCDSmedm command not working

"medm: command not found" error when run through command line both in pianosa and rossa in both editing and execution modes. It however gets executed and edited through the sitemap button. Don't know the source of the problem. Gautam did check the .bashrc file. aliases for SITEMAP and m40m are intact in the .bashrc file.

  12208   Tue Jun 21 11:49:29 2016 ericqFrogsCDSmedm command not working

The workstations' .bashrc is a symbolic link to /users/controls/.bashrc

In it, someone commented out the critical line:

#source /ligo/cdscfg/workstationrc.sh

I uncommented it. medm (and all of the other things like cdsutils) work again.

I blame jamie.

  12211   Wed Jun 22 10:15:45 2016 varunUpdateCDSDAFI update: stereo output

I have updated the DAFI with the following changes:

1) Separated both the channels of stereo output completely, as well as in the GUI.

2) Added text monitors for the inputs and outputs.

The stereo output is now ready except for a cable going from the second channel of the output fibox to the audio mixer.

Attached is the main DAF_OVERVIEW screen and its link button from the LSC screen labelled "DAFI"

Quote:

I wish to have stereo audio output for the DAF module. Hence, there needs to be a second output from the DAF. I added this second output to the model. Following are the details:

FiBox: It consists of two analog inputs which are digitized and multiplexed and transmitted optically. (only 1 fiber is needed due to multiplexing). Attachment 1 shows the fibox with its 2 analog inputs (one of which, is connected), and 1 fiber output. The output of the DAF goes to the FiBox. Until today, the Fibox recieved only 1 analog input. This analog signal comes from the DAC-8 (count starting from 0), which is located at "CH 1 OUT" SMA output in the "MONITORS" bin on the racks (attachment 2).

I have added another output channel to the DAF model both in software and in hardware. The DAF now also uses DAC-9 analog output which goes to the second analog input of the FiBox. The DAC-9 output is located at "CH 2 OUT" SMA output in the "MONITORS" bin on the racks (attachment 4).

After making the changes, the Fibox is shown in attacment 3.

Testing: The LSC input on passing through the DAF block is given through two different DAC outputs, to the same Fibox channel (one after the other), and the output is heard. More concrete testing will be done tomorrow. It will be as follows:

1) Currently, I need to search for a suitable cable that would connect the second channel of the output fibox to the audio mixer. After doing this, end to end testing of both channels will be done.

2) I could not access the AWG, probably because the DAQ was offline today afternoon. Using a signal from the AWG will give a more concrete testing of the stereo output.

3) After this, I will separate the two channels of the stereo completely (currectly they are seperated only at the DAF output stage)

4) I also will edit the medm gui appropriately.

 

Quote:

I have added Enable buttons for each of the DSP blocks, and labels for the matrix elements. The input matrix takes inputs from each of the 4 channels: ADC1, ADC2, LSC and EXC, and routes them to the audio processing blocks (attachment 2). The output matrix (attachment 3) takes the outputs of the various DSP blocks and routes them to the output and then to the speakers. 

 

 

Attachment 1: C1DAF_OVERVIEW.png
C1DAF_OVERVIEW.png
Attachment 2: DAF_link_from_LSC.png
DAF_link_from_LSC.png
  12215   Mon Jun 27 15:12:09 2016 varunUpdateCDSDAFI update: stereo output

Using an RC to BNC connector from the inner drawer, I have added a second output cable going from the output Fibox in the control room to the audio mixer.

Quote:

I have updated the DAFI with the following changes:

1) Separated both the channels of stereo output completely, as well as in the GUI.

2) Added text monitors for the inputs and outputs.

The stereo output is now ready except for a cable going from the second channel of the output fibox to the audio mixer.

Attached is the main DAF_OVERVIEW screen and its link button from the LSC screen labelled "DAFI"

Quote:

I wish to have stereo audio output for the DAF module. Hence, there needs to be a second output from the DAF. I added this second output to the model. Following are the details:

FiBox: It consists of two analog inputs which are digitized and multiplexed and transmitted optically. (only 1 fiber is needed due to multiplexing). Attachment 1 shows the fibox with its 2 analog inputs (one of which, is connected), and 1 fiber output. The output of the DAF goes to the FiBox. Until today, the Fibox recieved only 1 analog input. This analog signal comes from the DAC-8 (count starting from 0), which is located at "CH 1 OUT" SMA output in the "MONITORS" bin on the racks (attachment 2).

I have added another output channel to the DAF model both in software and in hardware. The DAF now also uses DAC-9 analog output which goes to the second analog input of the FiBox. The DAC-9 output is located at "CH 2 OUT" SMA output in the "MONITORS" bin on the racks (attachment 4).

After making the changes, the Fibox is shown in attacment 3.

Testing: The LSC input on passing through the DAF block is given through two different DAC outputs, to the same Fibox channel (one after the other), and the output is heard. More concrete testing will be done tomorrow. It will be as follows:

1) Currently, I need to search for a suitable cable that would connect the second channel of the output fibox to the audio mixer. After doing this, end to end testing of both channels will be done.

2) I could not access the AWG, probably because the DAQ was offline today afternoon. Using a signal from the AWG will give a more concrete testing of the stereo output.

3) After this, I will separate the two channels of the stereo completely (currectly they are seperated only at the DAF output stage)

4) I also will edit the medm gui appropriately.

 

Quote:

I have added Enable buttons for each of the DSP blocks, and labels for the matrix elements. The input matrix takes inputs from each of the 4 channels: ADC1, ADC2, LSC and EXC, and routes them to the audio processing blocks (attachment 2). The output matrix (attachment 3) takes the outputs of the various DSP blocks and routes them to the output and then to the speakers. 

 

 

 

Attachment 1: IMG_20160627_151753247.jpg
IMG_20160627_151753247.jpg
  12251   Wed Jul 6 10:58:29 2016 VarunUpdateCDSDAFI review

The DAFI block was reviewed by Rana yesterday. The following changes/improvements were suggested: (Updated on 20th July 2016 with tasks taat remain in red)

1) include all the various channels like PEM, LSC, ASC, SUS, SEI, etc. as the inputs. Currently the inputs are only the LSC.
2) include all the control signals.
3) create a very detailed diagram of the entire signal flow and plan tasks accordingly.
4) Enable cascading of various DSP processes.
5) Adjusting the gain of the AGC such that the amplitude of the output signal comes to about half the peak amplitude offered by the ADC. This will help taking advantage of the entire dynamic range of the ADC.
6) change the enable button styles from a text input based controller to a button controller.
7) Currently, disabling a particular signal terminates the signal. Instead, it should turn into a unity gain block on disabling.
8) Check if the Fibox does AC coupling or not. If not, add an AC coupling arrangement in the DAFI.
9) Check the nature of the ADC1 and ADC2 inputs to the DAFI. I checked them yesterday, and they are channels 25 and 26 of ADC0, which are empty.

  12266   Thu Jul 7 12:44:52 2016 varunUpdateCDSDAFI update

Attached is a diagram, showing the entire (planned) signal flow of the DAF model. Some thoughts on the implementation after discussion with eric:

1) Since the LSC control signals and ASC signals are running on the c1lsc FE at the same rate as DAFI (16kHz), it would be wise to start from these.

   Current implementation: has a matrix at the end of the LSC PD signals, which selects one of the PD signals and outputs it to the DAFI via IPC communication.

    Proposed Changes: Add another matrix at the end of the LSC PD signals, to give to the second stereo output. Similarly, add two matrices each at the end of the LSC control signals and     the ASC signals. Each matrix must select one of the signals and output it to the DAF via IPC.

2) The PEM running on the c1sus FE system will have to be brought to DAFI in a similar fashon. However, since c1sus runs at 2kHz, there is a possibility of imaging while the signal is    transfered to the DAFI. This could be taken care of by an anti imaging filter, or inserting zeros between two samples coming at to the 16 kHz system from the 2kHz system and then low-passing it to remove the aliased parts. (similar to upsampling)

3) For the SUS control signals, input can be given from a matrix prepared for each optic seperately.

Attachment 1: DAFI.pdf
DAFI.pdf
  12282   Fri Jul 8 22:26:03 2016 varunUpdateCDSDAFI Update: Changes in LSC model

I have added the control signals DARM_CTRL, MICH_CTRL, PRCL_CTRL, SRCL_CTRL, CARM_CTRL, XARM_CTRL, YARM_CTRL, MC_CTRL to the DAFI model from the LSC model via IPC commn.

The changes done to the LSC model include addition of an extra block going to DAFI (attachment 2, red rectangle in attachment 1), and addition of an extra overall output from the LSC, called DAFI_OUT2, which goes to DAFI through IPC link C1:LSC-DAF_2 (attach. 3). Now two distinct inputs can be given to the DAFI, whose intended purpose is to act as two distinct audio signals in the stereo output, but can also be used for arbitrary math. 

I am going to add the following PEM channels as DAF inputs subsequently, in a similar 2 input fashon.

SEIS_GUR1_X_OUT
SEIS_GUR1_Y_OUT
SEIS_GUR1_Z_OUT
SEIS_GUR2_X_OUT
SEIS_GUR2_Y_OUT
SEIS_GUR2_Z_OUT
SEIS_STS_1_X_OUT
SEIS_STS_1_Y_OUT
SEIS_STS_1_Z_OUT
ACC_MC1_X_OUT 
ACC_MC1_Y_OUT
ACC_MC1_Z_OUT
ACC_MC2_X_OUT
ACC_MC2_Y_OUT
ACC_MC2_Z_OUT

 

 

Attachment 1: lsc.png
lsc.png
Attachment 2: lsctodaf.png
lsctodaf.png
Attachment 3: lsctodaf1.png
lsctodaf1.png
  12284   Sun Jul 10 15:54:23 2016 ericqUpdateCDSAll models restarted

For some reason, all of the non-IOP models on the vertex frontends had crashed.

To get all of the bits green, I ended up restarting all models on all frontends. (This means Gautam's coil tests have been interrupted.)

  12287   Sun Jul 10 20:08:44 2016 varunUpdateCDSDAFI Update: Changes in LSC and PEM models

I have added the PEM signals mentioned in the previous elog as DAF inputs through PCIE IPC, and compiled and restarted the c1pem and c1daf models. 

Attached are the pictures of the simulink diagram of the addition in the PEM and the DAF. 

Since the signals are moving from a 2kHz clock rate machine to a 16kHz clock rate machine, some imaging effects are possible, which I have to look into.

Attachment 1: pemtodaf.png
pemtodaf.png
Attachment 2: pemindaf.png
pemindaf.png
  12303   Thu Jul 14 23:38:59 2016 varunUpdateCDSc1lsc FE unresponsive

Today, at around 10:30, c1lsc machine froze and stopped responding to ping and ssh after I compiled and restarted c1daf. I think it is due to a large array in one of my codes. The daqd.log file shows the following:


..................................................................
CA.Client.Exception...............................................
    Warning: "Virtual circuit unresponsive"
    Context: "c1lsc.martian.113.168.192.in-addr.arpa:5064"
    Source File: ../tcpiiu.cpp line 945
    Current Time: Thu Jul 14 2016 22:27:42.102649102
..................................................................

I think the c1lsc FE may need a hard reboot.

  12304   Fri Jul 15 12:21:28 2016 varunUpdateCDSc1lsc FE unresponsive

c1lsc is up and running, Eric did a manual reboot today.

Quote:

Today, at around 10:30, c1lsc machine froze and stopped responding to ping and ssh after I compiled and restarted c1daf. I think it is due to a large array in one of my codes. The daqd.log file shows the following:


..................................................................
CA.Client.Exception...............................................
    Warning: "Virtual circuit unresponsive"
    Context: "c1lsc.martian.113.168.192.in-addr.arpa:5064"
    Source File: ../tcpiiu.cpp line 945
    Current Time: Thu Jul 14 2016 22:27:42.102649102
..................................................................

I think the c1lsc FE may need a hard reboot.

 

  12307   Sat Jul 16 00:30:42 2016 varunUpdateCDSDAFI update: Frequency warping | c1lsc unresponsive

Summary: I am trying to implement frequency warping/pitch shifting on the real time FE. Here is a description long overdue:

Description: The overall idea is as follows:

The DFT of a frame x_i[n]  is given by X_i[k] = \sum_{n=0}^{m-1}x_i[n]e^{-j2\pi \frac{kn}{m}}. A matrix W containing all W_{kn} = e^{-j2\pi \frac{kn}{m}} for k, n = 1, 2, ..., m can be calculated and predefined in the code. The input arrival rate is 16384 Hz, i.e. once in every 60 $\mu$s time window. Hence, the fourier coefficients can be updated cumulatively in each cycle using the current value of the input, previous value of the fourier coefficient and the components of the W matrix. This will distribute the computational load of the FFT into all the time windows. Similar operations can be carried out for the inverse STFT.

I have written and run a pseudo-real time code on my CPU. The following is the essence:
 Let the frame-length be M, and the intended scale factor of the frequency warping be 'r'. The frame overlap is 50%. At each clock cycle, the following tasks are performed:  (1 to 3 are routine tasks performed at every clock cycle, 4 is a special task performed only when a frame is filled.)
 1) Take input and apply hanning window to it.
 2) Cumulate X_i[k] for every k using the value of x_i[n] (the input) at that particular instant. Also start to cumulate X_{i+1}[k], which will be later transfered to X_i[k].
 3) Because of 4), we now have 'r+1' filled frames corresponding to output fft. Now take the ifft using two consecutive frames corresponding to only two time series points. The computations required for this task are the same as the computations required for calculation of the fourier coefficients iteratively, since the entire time series ifft is not computed.
 4) Do these special tasks after each frame gets filled:
      At this point, the ffts of the current frame and a previous frame is ready. Let us call them X1 and X2.
      Calculate phase difference between the two.
      Calculate all the interpolated |Y_i| in between these two frames depending upon the scale factor.
      Assign phase of X1 to first Y frame and assign increasing phase to all the other Y frames.
      and also do all the usual non-special tasks.

This code takes about 9-10 microseconds for a cycle with special tasks, and 5-6 microseconds for a cycle with routine tasks on my laptop (brought down from 100 microseconds peak time in the earlier offline implementation due to elemination of explicit dft and reduction in fft size), for a frame size of 32 samples. However, when fed into the c1lsc FE, it crashes, as it has done once again today evening, in the same fashon as yesterday. There could be 2 possible reasons:

1) Size of the array containing the W_{kn} matrix elements is too large for the FE memory,

2) the computations are taking up more than 60 microseconds.

Since there are already a few codes with similar array sizes, I am more inclined to think that 2) is more likely.

Another problem that I am anticipating is that for a 32 point dft and a sampling rate of 16kHz, the frequency resolution achieved is about 500 Hz, which is not sufficient if we need to represent seismic signals. The only way I can think of, for representing such signals with a small number of fft points, is to reduce the effective sampling rate, i.e. do DSP on inputs at a much lower rate than 16kHz (say 1kHz, which will give a resolution of ~30 Hz, or 2kHz giving a resolution of ~60Hz). Another advantage of this method is that it frees up more clock cycles for computation, thus the computational load can be further distributed.  The problem in this implementation is that it will increase the delays.

  12309   Mon Jul 18 18:44:52 2016 varunUpdateCDSc1lsc FE recovered

c1lsc FE is up and running.

Details:

2) The machine was manually rebooted.

3) c1daf was recompiled and installed, with the problematic piece of code removed.

4) NTP timing was adjusted.

5) Frame Builder was restarted.

6) All models on c1lsc machine were restarted.

Attachment 1 shows the CDS status after the recovery. I wont be trying to run frequency warping immediately, I will first finish implementing the other harmless modules first.

Attachment 1: CDS_status160718.png
CDS_status160718.png
  12319   Thu Jul 21 12:03:35 2016 varunUpdateCDSDAFI update: Humming noise in DAFI output

 

Summary: There was always a constant humming noise in the output of speakers of both the audio channels. Tried to resolve the problem. Details are given below:

Details: The source of the noise was the typical 60 Hz (and harmonics), ~13 mV peak to peak output, in at least three channels of the DAC. (two coming from the DAF module, and one not related to the DAF.) Attachment 1 shows the noise in both the DAF channels. As compared to that, the signal coming through the AGC weak, about 6 mV RMS, about the same order as noise. In order to resolve this, the gain of the AGC was increased, so that the RMS output voltage of the Fibox (FBAO, the one at the output) was about 1.23 V RMS. It is approximately equal to +4 dBu, which is the typical expected output of the Fibox, according to the datasheet.

Attachment 1: New_Doc_13.pdf
New_Doc_13.pdf New_Doc_13.pdf
  12320   Thu Jul 21 14:27:24 2016 varunUpdateCDSDAFI Update: Arbitrary Math block

Summary: I have added an arbitrary math block to the DAFI model, which takes two inputs, say X and Y, and can perform various unary and binary operations on them:

Details:

  • Unary Operations:
  1. Delay - There exists a text-based input to specify the amount of delay to be given to a particular signal.
  2. sin()
  3. cos()
  • Binary Operations:
  1. Weighted addition and multiplication: The output is calculated according to the relation: A*X + B*Y + C*X*Y. A, B, C are constant inputs, which can be given through text-based inputs in the GUI.
  2. MAX{X,Y}
  3. MIN{X,Y}

Attachment 1 shows the existing DAFI gui, updated with cascading of various DSP blocks, upto three levels, button-based ENABLE and DISABLE controls for all blocks except arb. math (the control on arb. math. is achieved by clicking on the block.) On clicking the arb. math block one is taken to the dedicated arb. math screen, which has enable buttons for all the processes listed above. A screenshot of this screen is in attachment 2. There is one control input, which controls all the unary operations on X and the binary operations on X and Y, and another control input which controls the unary operations on Y. switching on a particular arb. math process gives a particular control input, which choses the appropriate section of the code. At a time, only one process from the top grey block (corresponding to unary operations on X and binary operations on X and Y) and one process from the bottom grey block (corresponding to unary operations on Y) can be selected. Thus, the outputs which go from the arb. math block to the intermediate matrices (MATRIX1L or MATRIX2L) are:

a) Either an output of unary operation on X or a binary operation on X and Y, the specific one depending upon the control input,

    and

b) Output of a unary operation on Y, again the specific one depending upon the control input

Thus there is apparent asymmetricity in the action of the arb. math block on the two inputs. However, this is done in order to reduce to total number of outputs and control signals, and this can be easily taken care of by interchanging the inputs before the block. 

While compiling this code, the c1lsc machine had crashed once, it was found that this was due to a stray "printf()" command in the c code. This glich in the code now stands rectified  There is a possibility that the previous incidents of the code crashing could also be due to the existence of a printf() command. 

Preliminary Testing: I have done a preliminary testing of the arb math block, i.e. verified that on enabling the sin and cos processes, the output is less that 1, on swithching on the process of weighted avarage and multiplication, the output looks like it is right, for a few simple values of A, B, C, like 0, 1, etc. The delay block however is giving zero output for delay of more than 6 samples.

 

 

 

 

Attachment 1: dafioverview.png
dafioverview.png
Attachment 2: arbmath.png
arbmath.png
  12321   Thu Jul 21 15:03:13 2016 varunUpdateCDSDAFI Update

1) I have added the status summary of the DAFI block to the main FE status overview screen in the c1lsc cloumn. (attachment 1)

2) I have edited all the kissel matrix buttons appropriately, and given them appropriate lables. (attachment 2)

Attachment 1: festatus.png
festatus.png
Attachment 2: matrices.png
matrices.png
  12324   Thu Jul 21 22:02:35 2016 varunUpdateCDSDAFI update: Frequency warping

The code for frequency warping contained a "printf()" command, which had caused the system to crash in one another instance (refer elog 12320) . Hence, I tried running the code tody by removing this line. Unfortunately, this did not work. the model still crashed. Attached is the screenshot of the FE status.

Attachment 1: 07212016.png
07212016.png
  12336   Tue Jul 26 09:56:34 2016 ericqUpdateCDSc1susaux restarted

c1susaux (which controls watchdogs and alignments for all non-ETM optics) was down, the last BURT was done yesterday around 2PM. 

I restarted via keying the crate. I restored the BURT snapshot from yesterday.

  12361   Mon Aug 1 20:09:37 2016 ranaUpdateCDSDAFI Update

I found the DAFI screen as a button inside of the LSC screen - I think its more logically found from the sitemap, so I'll move it into there as well.

Quote:

1) I have added the status summary of the DAFI block to the main FE status overview screen in the c1lsc cloumn. (attachment 1)

2) I have edited all the kissel matrix buttons appropriately, and given them appropriate lables. (attachment 2)

Gautam and I noticed a 60 Hz + harmonics hum which comes from the DAFI. Its the noisiest thing in the control room. It goes away when we unplug the fiber coming into the control room FiBox receiver, so its not a ground loop on this end. Probably a ground loop at the LSC rack.

Upon further investigation we notice that the Fibox at the LSC rack had its gain turned all the way up to +70 dB. This seemed too much, we reduced it to ~20 (?) so that we could use more of the DAC range.  Also, it is powered by a AC/DC converter plugged in to the LSC rack power strip. We cannot use this for a permanent install - must power the FiBox using the same power supplies as are used for the LSC electronics. Probably we'll have to make a little box that takes the fused rack power of 15 V and turns it into +12 V with a regulator (max current of 0.15 A). Making sure that the FiBox doesn't pollute the rest of the LSC stuff with its nasty internal DC-DC converters.

We also put a high pass in the output filter banks of DAFI. For the PEM channels we put in a 60 Hz comb. WE then routed the Y-end Guralp in through the boxes and out the output, mostly bypassing the frequency shifting and AGC. It seems that there is still a problem with GUR2.

Does anyone know which one is GUR1 and which one is GUR2? I don't remember the result of the Guralp cable switching adventures - maybe Koji or Steve does. According to the trend it was totally dead before March and in March it became alive enough for us to see ~30 ADC counts of action, so way smaller than GUR111 or GUR snoopy or whatever its called.

  12539   Fri Oct 7 20:25:14 2016 KojiUpdateCDSPower-cycled c1psl and c1iool0

Found the MC autolocker kept failing, It turned out that c1iool0 and c1psl went bad and did not accept the epics commands.

Went to the rack and power cycled them. Burt resotred with the snapshot files at 5:07 today.

The PMC lock was restored, IMC was locked, WFS turned on, and WFS output offloaded to the bias sliders.

The PMC seemed highly misaligned, but I didn't bother myself to touch it this time.

  12542   Mon Oct 10 11:48:05 2016 gautamUpdateCDSPower-cycled c1susaux, realigned PMC, spots centered on WFS1 and WFS2

[Koji, Gautam]

We did the following today morning:

  1. I re-aligned the PMC - transmission level on the scope on the PSL table is now ~0.72V which is around what I remember it being
  2. The spot had fallen off WFS 2 - so we froze the output of the MC WFS servo, and turned the servo off. Then we went to the table to re-center the spot on the WFS. The alignment had drifted quite a bit on WFS2, and so we had to change the scale on the grid on the MEDM screen to +/-10 (from +/- 1) to find the spot and re-center it using the steering mirror immediately before the WFS. It would appear that the dark offsets are different on WFS1 and WFS2, so the "SUM" reads ~2.5 on WFS1 and ~0.3 on WFS2 when the spots are well centered
  3. Coming back to the control room, we ran the WFSoffsets script and turned on the WFS servo again. Trying to run the relief servo, we were confronted by an error message that c1susaux needed to be power cycled (again). This is of course the slow machine that the ITMX suspension is controlled by, and in the past, power cycling c1susaux has resulted in the optic getting stuck. An approach that seems to work (without getting ITMX stuck)  is to do the following:
    • Save the alignment of the optic, turn off Oplev servo
    • Move the bias sliders on IFO align to (0,0) slowly
    • Turn the watchdog for ITMX off
    • Unplug the cables running from the satellite box to the vacuum feedthrough
    • Power cycle the slow machine. Be aware that when the machine comes back on, the offset sliders are reset to the value in the saved file! So before plugging the cables back in, it would be advisable to set these to (0,0) again, to avoid kicking the optic while plugging the cables back in
    • Plug in the cables, restore alignment and Oplev servos, check that the optic isn't stuck
  4. Y green beat touch up - I tweaked the alignment of the first mirror steering the PSL green (after the beam splitter to divide PSL green for X and Y beats) to maximize the beat amplitude on a fast scope. Doing so increased the beat amplitude on the scope from about 20mVpp to ~35mVpp. A detailed power budget for the green beats is yet to be done

It is unfortunate we have to do this dance each time c1susaux has to be restarted, but I guess it is preferable to repeated unsticking of the optic, which presumably applies considerable shear force on the magnets...


After Wednesday's locking effort, Eric had set the IFO to the PRMI configuration, so that we could collect some training data for the PRC angular feedforward filters and see if the filter has changed since it was last updated. We should have plenty of usable data, so I have restored the arms now.

  12592   Wed Nov 2 22:56:45 2016 gautamUpdateCDSc1pem revamped

installing the BLRMS 2k blocks turned out to be quite non-trivial due to a whole host of CDS issues that had to be debugged, but i've restored everything to a good state now, and the channels are being logged. detailed entry with all the changes to follow.

  12595   Thu Nov 3 12:38:42 2016 gautamUpdateCDSc1pem revamped

A number of changes were made to C1PEM and some library parts. Recall that the motivation was to add BLRMS channels for all our suspension coils and shadow sensor PDs, which we are first testing out on the IMC mirrors.

Here is the summary:

BLRMS_2k library block

  • The name of the custom C code block in this library part was named 'BLRMSFILTER' which conflicted with the name of the function call in the C code it is linked to, which lead to compilation errors
  • Even though the part was found in /opt/rtcds/userapps/release/cds/c1/models and not in the common repository, just to be safe, I made a copy of the part called BLRMS_2k_40m which lives in the above directory. I also made a copy of the code it calls in /opt/rtcds/userapps/release/cds/c1/src

C1PEM model + filter channels

  • Adding the updated BLRMS_2k_40m library part still resulted in some compilation errors - specifically, it was telling me to check for missing links around the ADC parts
  • Eric suggested that the error messages might not be faithfully reporting what the problem is - true enough, the problem lay in the fact that c1pem wasn't updated to follow the namespace convention that we now use in all the RT models - the compiler was getting confused by the fact that the BLRMS stuff was in a namespace block called 'SUS', but the rest of the PEM stuff wasn't in such a block
  • I revamped c1pem to add namespace blocks called PEM and DAF, and put the appropriate stuff in the blocks, after which there were no more compilation errors
  • However, this namespace convention messed up the names of the filter modules and associated channels - this was resolved with Eric's help (find and replace did the job, this is a familiar problem that we had encountered not too long ago when C1IOO was similarly revamped...)
  • There was one last twist in that the model would compile and install, but just would not start. I tried the usual voodo of restarting all the models, and even did a soft reboot of c1sus, to no avail. Looking at dmesg, I tracked the problem down to a burt restore issue - the solution was to press the little 'BURT' button next to c1pem on the CDS overview MEDM screen as soon as it appeared while restarting the model

All the channels seem to exist, and FB seems to not be overloaded judging by the performance overnight up till the power outage. I will continue to monitor this...

GV Edit 3 Nov 2016 7pm:

I had meant to check the suitability of the filters used - there is a detailed account of the filters implemented in BLRMSFILTER.c here, and I quickly looked at the file on hand to make sure the BP filters made sense (see Attachment #1). These the BP filters are 8th order elliptical filters and the lowpass filters are16th order elliptical filters scaled for the appropriate frequency band, which are somewhat different from what we use on the seismometer BLRMS channels, where the filters are order 4, but I don't think we are significantly overloaded on the computational aspect, and the lowpass filters have sufficiently steep roll-off, these should be okay...

Attachment 1: BLRMSresp.pdf
BLRMSresp.pdf
  12597   Thu Nov 3 13:36:16 2016 ericqUpdateCDSc1pem revamped

It seems that the EX and EY BLRMS banks were missing the BP and LP filters for the 0.03-0.1 and 0.1-0.3 bands. I've copied over the filters from the BS seismometer.

However, if it looks like the integrated C code BLRMS block works out well, we could replace the seismometers' filter module heavy BLRMS blocks and cut down on the PEM model bloat.

  12599   Fri Nov 4 18:31:05 2016 LydiaUpdateCDSc1auxex channels/pins for Acromag

Here are the channels we are planning to switch over from c1auxex to Acromag, and their current pin numbers on the existing VME boards. 

Analog inputs: 

C1:SUS-ETMX_UL_AIOut    #C0 S0
C1:SUS-ETMX_LL_AIOut    #C0 S1
C1:SUS-ETMX_UR_AIOut    #C0 S2
C1:SUS-ETMX_LR_AIOut    #C0 S3
C1:SUS-ETMX_Side_AIOut    #C0 S4
C1:SUS-ETMX_OL_SEG1    #C0 S5
C1:SUS-ETMX_OL_SEG2    #C0 S6
C1:SUS-ETMX_OL_SEG3    #C0 S7
C1:SUS-ETMX_OL_SEG4    #C0 S8
C1:SUS-ETMX_OL_X    #C0 S9
C1:SUS-ETMX_OL_Y    #C0 S10
C1:SUS-ETMX_OL_S    #C0 S11
C1:SUS-ETMX_ULPD    #C0 S12
C1:SUS-ETMX_LLPD    #C0 S13
C1:SUS-ETMX_URPD    #C0 S14
C1:SUS-ETMX_LRPD    #C0 S15
C1:SUS-ETMX_SPD    #C0 S16
C1:SUS-ETMX_ULV    #C0 S17
C1:SUS-ETMX_LLV    #C0 S18
C1:SUS-ETMX_URV    #C0 S19
C1:SUS-ETMX_LRV    #C0 S20
C1:SUS-ETMX_SideV    #C0 S21
C1:SUS-ETMX_ULPD_MEAN    #C0 S12
C1:SUS-ETMX_LLPD_MEAN    #C0 S13
C1:SUS-ETMX_SDPD_MEAN    #C0 S16

Analog Outputs:

C1:ASC-QPDX_S1WhiteGain    #C0 S0
C1:ASC-QPDX_S2WhiteGain    #C0 S1
C1:ASC-QPDX_S3WhiteGain    #C0 S2
C1:ASC-QPDX_S4WhiteGain    #C0 S3
C1:SUS-ETMX_ULBiasAdj    #C0 S4
C1:SUS-ETMX_LLBiasAdj    #C0 S5
C1:SUS-ETMX_URBiasAdj    #C0 S6
C1:SUS-ETMX_LRBiasAdj    #C0 S7
C1:LSC-EX_GREENLASER_TEMP    #C0 S0 This appears to have the same pin as another channel-- is it not being used? 

Binary Outputs:

C1:SUS-ETMX_UL_ENABLE    #C0 S0
C1:SUS-ETMX_LL_ENABLE    #C0 S1
C1:SUS-ETMX_UR_ENABLE    #C0 S2
C1:SUS-ETMX_LR_ENABLE    #C0 S3
C1:SUS-ETMX_SD_ENABLE    #C0 S4
C1:ASC-QPDX_GainSwitch1    #C0 S7
C1:ASC-QPDX_GainSwitch2    #C0 S8
C1:ASC-QPDX_GainSwitch3    #C0 S9
C1:ASC-QPDX_GainSwitch4    #C0 S10
C1:AUX-GREEN_X_Shutter2    #C0 S15

  12600   Sat Nov 5 15:45:44 2016 ranaUpdateCDSc1auxex channels/pins for Acromag

We don't need to record any of the AIOut channels, the OL channels (since we record them fast), or the _MEAN channels (I think they must be CALC records or just bogus).

  12604   Mon Nov 7 19:49:44 2016 JohannesUpdateCDSacromag chassis hooked up to PSL

[Lydia, Johannes]

We're waiting on the last couple electrical components to arrive that are needed to complete the acromag chassis, but it is essentially operational. Right now it is connected to the PSL Mephisto's diagnostics port, for which only a single XT1221 A/D unit is needed. We assigned the IP address 192.168.113.121 to it. For the time being I'm running a tmux session on megatron (named "acromag") that grabs and broadcasts the epics channels, with Lydia's original channel definitions. Since the chassis is 4U tall, there's not really any place in the rack for it, so we might want to move it to the X-end before we start shuffling rack components around. Once we finalize its location we can proceed with adding the channels to the frames.

For the eventual gradual replacement of the slow machines, we need to put some thought into the connectors we want in the chassis. If we want to replicate the VME crate connectors we probably need to make our own PCB boards for them, as there don't seem to be panel-mount screw terminal blocks readily available for DIN 41612 connectors. Furthermore, if we want to add whitening/AA filters, the chassis may actually be large enough to accomodate them, and arranging things on the inside is quite flexible. There are a few things to be considered when moving forward, for example how many XT units we can practically fit in the chassis (space availability, heat generation, and power requirements) and thus how many channels/connectors we can support with each.

Steve: 1X3 has plenty of room

Attachment 1: acromag_chassis_location.jpg
acromag_chassis_location.jpg
Attachment 2: acromag_chassis_top_view.jpg
acromag_chassis_top_view.jpg
  12607   Tue Nov 8 17:51:09 2016 LydiaUpdateCDSacromag chassis hooked up to PSL

We set up the chassis in 1X7 today. Steve is ordering a longer 25 pin cable to reach. Until then the PSL diagnostic channels will not be usable.

  12608   Wed Nov 9 11:40:44 2016 ericqUpdateCDSsafe.snap BURT files now in svn

This is long overdue, but our burt files for SDF now live in the LIGO userapps SVN, as they should.

The canonical files live in places like /opt/rtcds/userapps/release/cds/c1/burtfiles/c1x01_safe.snap and are symlinked to files like /opt/rtcds/caltech/c1/target/c1x01/c1x01epics/burt/safe.snap

  12610   Thu Nov 10 19:02:03 2016 gautamUpdateCDSEPICS Freezes are back

I've been noticing over the last couple of days that the EPICS freezes are occurring more frequently again. Attached is an instance of StripTool traces flatlining. Not sure what has changed recently in terms of the network to cause the return of this problem... Also, they don't occur coincidentally on multiple workstations, but they do pop up in both pianosa and rossa.

Not sure if it is related, but we have had multiple slow machine crashes today as well. Specifically, I had to power cycle C1PSL, C1SUSAUX, C1AUX, C1AUXEX, C1IOOL0 at some point today

Attachment 1: epicsFreezesBack.png
epicsFreezesBack.png
  12613   Mon Nov 14 14:21:06 2016 gautamSummaryCDSReplacing DIMM on Optimus

I replaced the suspected faulty DIMM earlier today (actually I replaced a pair of them as per the Sun Fire X4600 manual). I did things in the following sequence, which was the recommended set of steps according to the maintenance manual and also the set of graphics on the top panel of the unit:

  1. Checked that Optimus was shut down
  2. Removed the power cables from the back to cut the standby power. Two of the fan units near the front of the chassis were displaying fault lights, perhaps this has been the case since the most recent power outage after which I did not reboot Optimus
  3. Took off the top cover, removed CPU 6 (labelled "G" in the unit). The manual recommends finding faulty DIMMs by looking for an LED that is supposed to indicate the location of the bad card, but I couldn't find any such LEDs in the unit we have, perhaps this is an addition to the newer modules?
  4. Replaced the topmost (w.r.t the orientation the CPU normally sits inside the chassis) DIMM card with one of the new ones Steve ordered
  5. Put everything back together, powered Optimus up again. Reboot went smoothly, fan unit fault lights which I mentioned earlier did not light up on the reboot so that doesn't look like an issue.

I then checked for memory errors using edac-utils, and over the last couple of hours, found no errors (corrected or otherwise, see Praful's earlier elog for the error messages that we were getting prior to the DIMM swap)- I guess we will need to monitor this for a while more before we can say that the issue has been resolved.

Looking at dmesg after the reboot, I noticed the following error messages (not related to the memory issue I think):

[   19.375865] k10temp 0000:00:18.3: unreliable CPU thermal sensor; monitoring disabled
[   19.375996] k10temp 0000:00:19.3: unreliable CPU thermal sensor; monitoring disabled
[   19.376234] k10temp 0000:00:1a.3: unreliable CPU thermal sensor; monitoring disabled
[   19.376362] k10temp 0000:00:1b.3: unreliable CPU thermal sensor; monitoring disabled
[   19.376673] k10temp 0000:00:1c.3: unreliable CPU thermal sensor; monitoring disabled
[   19.376816] k10temp 0000:00:1d.3: unreliable CPU thermal sensor; monitoring disabled
[   19.376960] k10temp 0000:00:1e.3: unreliable CPU thermal sensor; monitoring disabled
[   19.377152] k10temp 0000:00:1f.3: unreliable CPU thermal sensor; monitoring disabled

I wonder if this could explain why the fans on Optimus often go into overdrive and make a racket? For the moment, the fan volume seems normal, comparable to the other SunFire X4600s we have running like megatron and FB...

  12615   Mon Nov 14 19:32:51 2016 ranaSummaryCDSReplacing DIMM on Optimus

I did apt-get update and then apt-get upgrade on optimus. All systems are nominal.

  12632   Mon Nov 21 19:54:13 2016 JohannesUpdateCDSacromag chassis hooked up to PSL

[Lydia, Johannes]

We connected and powered up the Acromag chassis today. It lives in 1X4 and is powered by the Sorensen +20V power supply in 1X5 via the fuse rail on the side of 1X4. For this we had to branch off the 20V path to the dewhitening and anti-image filter crate of the c1:susaux driven SOS optics. After confirming that none of the daughter modules in the crate draw from the 20V line, we added a wire leading to a new fuse we added for this unit and ran a power cable from there.

The diagnostic connector of the PSL laser is now connected to the unit and a tmux session was created on megatron that interfaces with the chassis and broadcasts the EPICS channels. We need to watch out in the coming days for epics freezes/outages, as in the past these seemed to occur around the same times we were toying with the Acromags.

Quote:

We set up the chassis in 1X7 today. Steve is ordering a longer 25 pin cable to reach. Until then the PSL diagnostic channels will not be usable.

 

Attachment 1: acromag_chassis.jpg
acromag_chassis.jpg
  12649   Wed Nov 30 11:56:56 2016 LydiaUpdateCDSWiring for Acromag auxex replacement

I've attached a schematic for how we will connect the Acromag mosules to the slow channel I/O curently going to c1auxex. The following changes are made:

  • We are getting rid of the slow readbacks from the Anti-Image and Oplev boards, as Rana says they are unnnecessary.
  • The whitening switching for the QPD is currently done by a Contec "fast" binary I/O module, but can be managed by acromag instead. This alllows CAB_1Y9_34 to  be fed directly into the Acromag box since all of its connections can now be managed slow. 
  • There's no need to change the PD whitening scheme around (since the signals never get huge), so we can set those to always be on and then lose those Contec channels. This means all of the necessary pins on CAB_1Y9_10 can go to Acromag. 
  • All the other backplane cables go the the fast machines only. 

 

Attachment 1: auxex_acromag.pdf
auxex_acromag.pdf
ELOG V3.1.3-