Interesting information from Alex.  We're limited to 2 Megabytes per second per front end model.  Assuming all your channels are running at a 2kHz rate, we can have at most 256 channels being set to the frame builder from the front end (assuming 4 byte data).  We're fine for the moment, but perhaps useful to keep in mind.

I talked to Alex this morning and he said the frame builder is being flaky (it crashed on us twice this morning, but the third time seemed to stay up when requesting data).  I've added a new wiki page called "New Computer Restart Prodecures" under Computers and Scripts, found here. It includes all the codes that need to be running, and also a start order if things seem to be in a particularly bad state.  Unfortunately, there were no fixes done to the frame builder but it is on Alex's list of things to do.

In regards to the timing out of the front ends, Alex came over to the 40m this morning and we sat down debugging.  We tried several things, such as removing all filters from C1MCS.txt file in the chans directory, and watching the timing as we pressed various medm control buttons. We traced it to a filters used by the DAC in the model talking to the IOP front end, which actually sends the data to the physical DAC card.  The filter is used when converting between sample rates, in this case between the 16 kHz of the front end model and the 64 kHz of the IOP.  Sending it raw zeros after having had real data seemed to cause this filter to eat up an usually large amount of CPU time. 

We modified the /opt/rtcds/caltech/c1/core/advLigoRTS/src/include/drv/fm10Gen.c file.

We reverted a change that was done between version 908 and 929, where underflows (really small numbers) were dealt with by adding and then subtracting a very small number.  We left the adding and subtracting, but also restored the hard limits on the history.

So instead of relying on just:

input += 1e-16;
junk = input;
input -= 1e-16;

we also now use

if((new_hist < 1e-20) && (new_hist > -1e-20)) new_hist = new_hist<0 ? -1e-20: 1e-20;

Thus any filter value who's absolute value is less than 1e-20 will be clamped to -1e-20 or 1e-20.  On the bright side, we no longer crash the front ends when we turn something off.

Since this morning, the fb's timing has been off.  Steve pointed it out to me earlier today, but I didn't have a chance to look at it until now. 

This was different from the more common problem of the mx stream needing to be restarted - that causes 3 red blocks per core, on all cores on a computer, but it doesn't have to be every computer.  This was only one red block per core in the CDS FE status screen, but it was on every core on every computer. 

The error message, when you click into the details of a single core, was 0x4000.  I elog searched for that, and found elog 6920, which says that this is a timing issue with the frame builder.  Since Jamie had already set things on nodus' config correctly, all I did was reconnect the fb to the ntp: 

fb$ sudo /etc/init.d/ntp-client restart

As in elog 6920, the daqd stopped, then restarted itself, and cleared the error message. It looks like everything is good again.

I suspect (without proof) that this may have to do with the campus network being down this morning, so the computers couldn't sync up with the outside world.

The above timing problem has been repeating (a couple of times this week so far). It does not seem to be related to the campus network.

The same solution was applied.

....fb timing issue happened again.

I thought that it was the thing that Koji and I saw the other day, where it was individual front end computers that had lost ntp sync, since it wasn't every core on every computer that was red, but reconnecting to the ntp server on c1lsc didn't do anything.  I then tried reconnecting to the ntp server on fb, and that fixed things right up.  Annoying.

...yet again.

lsc and sus needed mxstream restarts after I restarted the ntp on fb.

We need to correctly setup crontab or rc.local for the frontend machines.

fb timing was off again.

Off again. Restarted ntp on fb.

There currently seems to be a timing issue with  the frame builder.  We switched over to using a symmetricom card to get an IRIG-B signal into the fb machine, but the gps time stamp is way off (~80 years Alex said).

If there is a frame buiilder issue, its currently often necessary to kill the associated mx_stream processes, since they don't seem to restart gracefully.  To fix it the following steps should be taken:

Kill frame builder, kill the two mx_stream processes, then /etc/restart_streams/, then restart the frame builder (usual daqd -c ./daqdrc >& ./daqd.log in /opt/rtcds/caltech/c1/target/fb).

To restart (or start after a boot) the nds server, you need to go to /opt/rtcds/caltech/c1/target/fb and type

./nds /opt/rtcds/caltech/c1/target/fb/pipe

At this time, testpoints are kind of working, but timing issues seem to be preventing useful work being done with it.  I'm leaving with Alex working on the code.

Alex fixed the time issue with the IRIG-B signal being far off, apparently their IRIG-B signal in downs seems to be different.  He simply corrected for the difference in the two signals in the code.

For debugging purposes we uncommented the following line in the feCodeGen.pl script (in /opt/rtcds/caltech/c1/advLigoRTS/src/epics/util/):

print EPICS "test_points ONE_PPS $dac_testpoint_names $::extraTestPoints\n" 

This is to make every ADC testpoint available from the IOP (such as c1x02).

1) Need to check 1 PPS signal alignment

2) Figure out why 1PPS and ADC/DAC testpoints went away from feCodeGen.pl?

3) Fix 1PPS testpoint giving NaN data

4) Figure out why is daqd printing "making gps time correction" twice?

5) Need to investigate why mx_streams are still getting stuck

6) Epics channels should not go out on 114 network (seen messages when doing
burt restore/save).

7) Dataviewer leaves test points hanging, daqd does not deallocate them
(net_Writer.c shutdown_netwriter call)

8) Need to install wiper scripts on fb

9) Need to install newer kernel on fb to avoid loading myrinet firmware
(avoid boot delay)

Problem:

The front ends and fb computers were unresponsive this morning.

This was due to the fb machine having its ethernet cable plugged into the wrong input.   It should be plugged into the port labeled 0.

Since all the front end machines mount their root partition from fb, this caused them to also hang.

Solution:

The cable has been relabled to "fb" on both ends, and plugged into the correct jack.  All the front ends were rebooted.

Testing RFM for green locking:

I tested the RFM connection between c1ioo and c1scx.  Unfortunately, on the first test, it turns out the c1ioo machine had its gps time off by 1 second compared to c1sus and c1iscex.  A second reboot seems to have fixed the issue.

However, it bothers me that the code didn't come up with the correct time on the first boot.

The test was done using the c1gcv model and by modifying the c1scx model.  At the moment, the MC_L channel is being passed the MC_L input of the ETMX suspension.  In the final configuration, this will be a properly shaped error signal from the green locking.

The MC_L signal is currently not actually driving the optic, as the ETMX POS MATRIX currently has a 0 for the MC_L component.

/var on fb1 filled up today, which caused all sorts of CDS issues.  I found out about the problem by reading the logs of the services that were having trouble running, in which they complained about not being able to write to disk.  I looked at the filesystem status with 'df' and noticed that /var was full, which is where applications write temporary data, and will always cause problems if it's full.

I tracked the issue down to multiple multi-gigabyte log files: /var/log/messages and /var/log/messages.1.  They were full of lines like this one:

Seems like something related to the gpstime kernel module?

Anyway, I deleted the log files for now, which cleared up the space on /var.  Things should be back to normal now, until the logs fill up again...

Jonathan Hanks pointed me to this fix to the gpstime kernel module that was unfortunately put in after the 3.4 release that we're currently using:

https://git.ligo.org/cds/advligorts/-/commit/6f6d6e2eb1d3355d0cbfe9fe31ea3b59af1e7348

I hacked the source in place (/usr/src/gpstime-3.4/drv/gpstime/gpstime.c) to get the fix, and then rebuilt the kernel module with dkms :

sudo dkms uninstall gpstime/3.4
sudo dkms install gpstime/3.4

I then stopped daqd_dc, unloaded gpstime, reloaded it, restarted daqd_dc.  The messages are no longer showing up in /var/log/messages, so I think we're ok for the moment.

NOTE: the fix will be undone if we for some reason reinstall the advligorts-gpstime-dkms package.  There shouldn't be a need to do that, but we should be aware.  I'm discussing with Jonathan if we want to try to push out a new debian package to fix this issue...

[Tega, Chris]

We migrated fb1 OS from the teststand fb1 drive to the internal 2TB RAID of fb1. We then rebooted twice to check that we no longer have the fb1 booting issue. 

The next step is to set up software RAID and backup for chiara, which we plan to complete this week. Then we would work on nodus and workstation OS upgrade next week.

I fixed the JetStor 416S raid array IP address by plugging in my laptop to its ethernet port, setting my IP to be on the same subnet, and using the web interface.  (After finally tracking down the password, it has been placed in the usual place).

After this change, I powered up the fb40m2 machine and reboot the fb40m machine. This seems to have made all the associated lights green.

Data viewer is working such that is recording from the point I fixed the JetStor raid array and did the fb40m reboot.  It also can go back in time before the IP switch over.

Having determined that Rana (the computer) was having to many issues with testing the new Raid array due to age of the system, we proceeded to test on fb40m.

We brought it down and up several times between 11 and noon.  We  eventually were able to daisy chain the old raid and the new raid so that fb40m sees both.  At this time, the RAID arrays are still daisy chained, but the computer is setup to run on just the original raid, while the full 14 TB array is initialized (16 drives, 1 hot spare, RAID level 5 means 14 TB out of the 16 TB are actually available).  We expect this to take a few hours, at which point we will copy the data from the old RAID to the new RAID (which I also expect to take several hours).  In the meantime, operations should not be affected.  If it is, contact one of us.

This afternoon the alignment script chrashed after returning sysntax errors. We found that the tpman wasn't running on the framebuilder becasue it had probably failed to get restarted in one of the several reboots executed in the morning by Alex and Jo.

Restarting the tpman was then sufficient for the alignment scripts to get back to work.

The fb40m just went out of order with status indicator number 8

It recovered on its own five minutes later.

 Backup script restarted, backup of trend frames and /cvs/cds is up-to-date.

The frame builder was power cycled during the morning bootfest.  I have restarted the backup script once more.

The 40m frame builder is currently being patched to be able utilize the full 14 TB of the new raid array (as opposed to being limited to 2 TB).  This process is expected to take several hours, during which the frame builder will be unavailable.

Alex came over this morning and we began work on the frame builder change over.  This required fb40m be brought down and disconnected from the RAID array, so the frame builder is not available.

He brought a Netgear switch which we've installed at the top of the 1X7 rack.  This will eventually be connected, via Cat 6 cable, to all the front ends.  It is connected to the new fb machine via a 10G fiber.

Alex has gone back to Downs to pickup a Symmetricon (sp?) card for getting timing information into the frame builder.  He will also be bringing back a harddrive with the necessary framebuilder software to be copied onto the new fb machine.

He said he'd like to also put a Gentoo boot server on the machine.  This boot server will not affect anything at the moment, but its apparently the style the sites are moving towards.  So you have a single boot server, and diskless front end computers, running Gentoo.  However for the moment we are sticking with our current Centos real time kernel (which is still compatible with the new frame builder code).  However this would make a switch over to the new system possible in the future.

At the moment, the RAID array is doing a file system check, and is going slowly while it checks terabytes of data.  We will continue work after lunch. 

 Punchline: things still don't work.

FB40m up and running again after restarting the DAQ.

I received an e-mail from Alex indicating he found the testpoint problem and fixed it today:

Quote from Alex: "After we swapped the frame builder computer it has reconfigured all device files and I needed to create some symlinks on /dev/ to make tpman work again. I test the testpoints and they do work now."

Alex and Steve,

SunFire x4600 ( not  MEGATRON 2 , it is fb40m2 ) and JetStor ( 16 x 1 TB drives ) were installed on side rails at the bottom of 1Y6

We cleaned up the fibres and cabling in 1Y7 also

I did a simulation of linear quadratic gaussian (LQG) controller applied to local damping. The cost function was frequency shaped to have a peak at 1 Hz. This technique prevents the controller from adding sensor noise at high and very low frequencies.

Noise was simulated to have 1/f spectrum (seismic) multiplied by stack with a resonance at 4 Hz with Q=5.

New  Lumitek IR Sensor Cards are here. We got 2 pieces of Q-11-T (2" x 2"), 2 pieces of Q-11-T (0.75" x 0.75")  and one Q-11 (4" x 5")

 1, Vacuum envelope grounds must be connected all times!  After door removal reconnect both cables immediately.

 2, The crane folding had a new issue of getting cut as picture shows.

 3, Too much oplev light is scattered. This picture was taken just before we put on the heavy door.

 4, We were unprepared to hold the smaller side chamber door 29" od of the IOC

 5, Silicon bronze 1/2-13 nuts for chamber doors will be replaced. They are not smooth turning.

Can we get some panel mount FC/APC connectors and put them on a box?   Then we could have the whole setup inside of a box that is filled with foam and sits outside the PSL hut. 

 AP Armaflex  tube 7/8" ID X 1" wall insulation for the long fiber in wall mounted cable trays installed yesterday.

The 6 ft long sections are not glued. Cable tied into the tray pressed against one an other, so they are air tight. This will allow us adding more fibers later.

 Atm2: Fiber PSL ends  protection added on Friday.

Alex, Gautam and Steve,

Single mode fiber 50m long is layed out into cable tray that is attached to the beam tube of the Y arm.

It goes from ETMY to PSL enclosure. It is protected at both ends with " clear- pvc, slit corrugated loom tubing " 1.5" ID

The fiber is not protected between 1Y1 and 1Y4

 I positioned the fiber loaded protecting tubing and anchored them so they can do their job.

However, the area needs a good clean up.

 What I did today.

1. Collimation of a beam.

• I then practiced collimation of a 700 nm laser (output) beam after being coupled through a fibre.
• I put together the set-up as shown in the attached picture where I used ....... to couple 650nm light into the PM.... fiber.
• I kept shifting the focus of the output beam to an appreciable distance till it was approximately collimated.

2. Coupling of the IR light at the ETMY table to a fibre.

• The fibre coupler was put in place to couple light into the fiber.
• I put in the mirrors as planned to direct the IR beam exiting the doubling crystal towards the fiber coupler (input).
• The mirrors were aligned such that the beam falls on the input lens of the coupler.
• The far-end of the fibre originally would have gone to to PSL table but it has been put on this table to study the power of the IR beam transmitted through this set-up.  The output end of the fiber has been connected to another fiber optic coupler to collimate the exiting beam.
• The picture of the current status attached.

A Matlab script to calculate Wiener filter coefficients and convert fir to iir is ready. Input is a file with zero mean witness and desired signals, output is a Foton zpk command to specify iir filter.

The plot shows comparison of offline fir , iir and online iir filtering. Spectrum below 4 Hz is still oscillating due to acoustic coupling, this is not a filtering effect. At 1 Hz actuator is badly compensated, more work should be done. Other then that online and offline filtering are the same.

 We decided to write a script that will check online filters for digital noise. One method can be implemented using the following algorithm:

• calculate filter output using single precision
• calculate filter output using double precision and assume that it is precise
• find digital noise at the output of the filter when single precision is used
• extrapolate the result to the double precision filter dividing by 2^D-S ~ 10⁷, D - number of bits used in double precision mantissa, S - in single precision

Restriction: Single precision filter internal variables must be checked for overflows.

I applied this method to filtering a 1 Hz sine wave with a notch filter. Precise output should also be a 1 Hz wave => at other frequencies we see noise => digital noise spectrum should coincide with filter output. The plot shows the method worked out for this example.

Using this method I estimated digital noise of butter("LowPass", 2, 0.001) applied to white noise. Sampling frequency was 16 kHz. 

The script estimates digital noise produces by online filters. First version of Matlab files and complied c files are in scripts/digital_noise directory.

Algorithm for 1 filter bank (max number of filters = 10):

1.        extract sos - representation from Foton file for each filter (Matlab)
2.        download data from corresponding DQ channel using NDS (Matlab)
3.        find filters that are switched on (Matlab)
4.        filter signal using Df2 and BQF with single and double precision (C)
5.        estimate digital filter noise (Matlab)
6.        calculate power spectral density and plot the result (Matlab)

More details on (2)

Often DQ channels have reduced sampling rate. In this case the script will upsample data adding zeros.

AI filter is not applied. But in the end only the frequency range (0, DQ RATE / 2) is analyzed.

More details on (3):

• input switch ON / OFF => 4 / 0
• filters 1 - 6 ON /OFF
  • 1 => 48 / 0
  • 2 => 192 / 0
  • 3 => 768 / 0
  • 4 => 3072 / 0
  • 5 => 12288 / 0
  • 6 => 49152 / 0
  • If a switch is ON but there is no corresponding filter (one green and one red line under the switch) then the switch value is divided by 3

_SW2R channel value is the sum of the following numbers:

• decimation switch ON / OFF => 512 / 0
• output switch ON / OFF => 1024 / 0
• filters 7 - 10 ON /OFF
  
  • 7 => 3 / 0
  • 8 => 12 / 0
  • 9 => 48 / 0
  • 10 => 192 / 0
  • If a switch is ON but there is no corresponding filter (one green and one red line under the switch) then the switch value is divided by 3

Note: as for now Matlab script assumes that input, output and decimation filters are switched ON and there are no turned ON filter switches that do not correspond to any filters

More details on (5)

Digital noise using double precision is estimated by extrapolation of digital noise with single precision. The last is calculated by subtracting outputs of the filters with single and double precision. Then this noise is multiplied by 3 * 10^-7.

This extrapolation number was achieved by printf tests of the number 0.123456789012345678 with single and double precision on C. Using type 'float' variables 10 significant numbers show up, using type 'double' - 17.

I also did 'calibration tests' to achieve extrapolation number - signal was filters with an aggresive low-pass filter. At high frequencies filter output spectrum is flat => digital noise amplitude must be the same. The plot shows GUR1_X channel filtered with low-pass chebyshev type 1 filter.

However, extrapolation number is not the same for all cases. In the following example of analyzing BS_SUSPOS filter bank using extrapolation 3 * 10^-7 we get noise that is slightly overestimated. In some other examples we need to take a larger number. But in average, I think, this is a good approximation.

To avoid extrapolation problem we can use long double precision (~19 digits). I was able to do this with gcc compiler. However, in mex compiler using long double in filter calculations, I do not get any better precision then using double precision. I'll think more about it.

Online filter diverges. I did offline simulations with current c-code. Offline filter also diverges, even in the simplest case 

witness = randn(1e6, 1); target = witness + 0.01*randn(1e6, 1);

I tried to create a new implementation of FXLMS algorithm as a c code. Then with this c code I did offline filtering with MCL and GUR signals and compared the error signals depending on the length of the filter.

One can see the code at the svn

adaptOnline - start here and choose algorithm

adaptive_filtering - Matlab implementation of AF

current_version.c - current version of the Filter (Matt's)

fxlms_filter.c - new version of the FXLMS filter

oaf.c - agent between Matlab and C (edited Matt's file)

Data samples can be found at nodus /users/den/wiener_filtering/data

I say just fix the clipping. Don't worry about the PRM OSEM filters. We can do that next time when we put in the ITM baffles. No need for them on this round.

The LSC time had gone too high. I deleted ~20 filters and rebooted. CPU time came down to 50 usec.

The filters all looked like old trash to me, but its possible they were used.

I didn't delete anything from the DARM, CARM, etc. banks but did from the PD and TM filter banks. You can always go back in time by using the

filter_archive/

All fine, except ITMX_sensor_UL's  60  counts deep hoop for an hour.

I made sketches of the final setup. There will be a box in the rack that contains both the heater circuit and the temperature sensor boards. One of them is in the loop while the other isn't. Instead of having many cables leading to the can, there will only be these three, though they can be made into a single wire. It will be connected to the can through a D-9 connector. The second attachment is what will be inside of the box, with all the major wires and components labeled.

-----

Edit: I've canged the layout to (hopefully) make the labels easier to read. I've also added in a cable to the ADC that reads out the voltage across the 1 ohm resistor. I also attached the circuit diagrams for the heater circuit and the temperature sensors. The one for the heater circuit was made by Kevin and I used the same design, except I have LM7818 and LM7918, since the 15V ones were not available at the time I made the circuit. 

In addition, all the wires leading to the can will all be part of one bundle of wires (I didn't clearly indicate it as such). There will be a total of 6 wires: two are needed for the wire to supply power to the heater and will have a LEMO connector on the rack end and two are needed for each temperature sensor, which will be attached to the board directly on the rack end. 

Also, we don't need two voltage regulators for each temperature circuit. We can just have one of each of LM7815 and LM7915 to supply +/- 15V to the boards.

I've updated the sketches and added in front panels for the seismometer block and the 1U panel (attachments 3 and 4). There was an issue when it came to the panel on the block because the hole is only big enough for the cable that already exists there and there is no space to add in the D-9 connector. Not quite sure how to resolve this issue. Attachment 7 is the current panel on the seismometer block. Attachments 5 and 6 are the updated temperature circuit and the heater circuit.

The boxes will be located in the short racks at EX and EY to minimize cable length.