The rsync job to sync our frames over to the cluster has been on a 20 MB/s BW limit for awhile now.

Dan Kozak has now set up a cronjob to do this at 10 min after the hour, every hour. Let's see how this goes.

You can find the script and its logfile name by doing 'crontab -l' on nodus.

Still seems to be running without causing FB issues. One thought is that we could look through the FB status channel trends and see if there is some excess of FB problems at 10 min after the hour to see if its causing problems.

I also looked into our minute trend situation. Looks like the files are comrpessed and have checksum enabled. The size changes sometimes, but its roughly 35 MB per hour. So 840 MB per day.

According to the wiper.pl script, its trying to keep the minute-trend directory to below some fixed fraction of the total /frames disk. The comment in the scripts says 0.005%,

but I'm dubious since that's only 13TB*5e-5 = 600 MB, and that would only keep us for a day. Maybe the comment should read 0.5% instead...

I'm not so sure. I just was experiencing some severe network latency / EPICS channel freezes that was alleviated by killing the rsync job on nodus. It started a few minutes after ten past the hour, when the rysnc job started. 

Unrelated to this, for some odd reason, there is some weirdness going on with ssh'ing to martian machines from the control room computers. I.e. on pianosa, ssh nodus fails with a failure to resolve hostaname message, but ssh nodus.martian succeeds. 

So today, after an "rm" error while working with the autoburt.pl script and burt restores in general, I asked Dan Kozak how to actually look at the backup data.  He said there's no way to actually look at it at the moment.  You can reverse the rsync command or ask him to grab the data/file if you know what you want.  However, in the course of this, we realized there was no /cvs/cds data backup.

Turns out, the rsync command line in the script had a "-n" option.  This means do a dry run.  Everything *but* the actual final copying.

I have removed the -n from the script and started it on nodus, so we're backing up as of 5:22pm today.

I'm thinking we should have a better way of viewing the backup data, so I may ask Dan and Stewart about a better setup where we can login and actually look at the backed up files.

In addition, tomorrow I'm planning to add cron jobs which will put changes to files in the /chans and /scripts directories into the SVN on a daily basis, since the backup procedure doesn't really provide a history for those, just a 1 day back backup.

 Added TRX and TRY and POY11_I_ERR and POX11_I_ERR to the c1lsc.mdl using a new-style DAQ Channels block, recompiled, installed, started the model, all good.  Restarted the daqd on the framebuilder, and everything is green.  I can go back and get recorded data using dataviewer (for the last few minutes since I started fb), so it all looks good.

Note on the new DAQ Channels block:  Put the text block (from CDS_PARTS) at the same level as the channel you want to save, and name it exactly as it is in the model.  The code-generator will add the _DQ for you.  i.e. if you define a channel "TRY_OUT_DQ" in the lsc model, you'll end up with a channel "C1:LSC-TRY_OUT_DQ_DQ".

We can't compile any changes to the LSC or the GCV models since Jamie's new script / program isn't found.  I don't know where it is (I can't find it either), so I can't do the compiling by hand, or point explicitly to the script.  The old way of compiling models in the wiki is obsolete, and didn't work :(

This means we can't (a) record TRY or (b) add the Q quadrature of the beat PD to the real time system tonight.

We're going to try just using Yuta's pynds script to capture data in real time, so we can keep working for tonight.

Sorry about that.  I had modified the path environment that pointed to the rtcds util.  The rtcds util is now in /opt/rtcds/caltech/c1/scripts/rtcds, which is in the path.  Starting a new shell should make it available again.

Finally I found a company who can do Koji's improved  -hard to make-  specification on ruby or sapphire wire standoff.

NOT POLISHED excimer laser cut, wire groove radius R 0.0005" + - 0.0002"

$250 ea at 50 pieces of order

Atm 1 & 5, showing the ruby R ~10 mm as it is seated on Al SOS test mass

Atm. 2, 3 & 4  chipped long edges with SOS sus wire OD 43 micron as  calibration

Bluebean Optical Tech Limited of Shanghai delivered 50 pieces red ruby prisms with radius.  The first prism pictures were taken at June 5

and it was retaken again as BB#1 later

More samples were selected randomly as one from each bag of 5 and labeled as BB#2.......6    

 The R10 mm radius can be seen agains the  ruler edge.  The v-groove edge was labeled with blue marker and pictures were taken

from both side of this ridge. The top view is shown as the wire laying across on it.

SOS sus wire of 43 micron OD used as calibration as it was placed close to the side that it was focused on.

The V-groove ridge surface quality was evaluated based on as scale of 1 – 10 with 10 being the most positive.

Remaining thing to examin, take picture of the contacting ridge to SOS from the side.

3-4 hrs ago we run out of nitrogen. We are back to Vacuum Normal

I have attached the result of running the PID script on the seismometer with the can on. The daily fluctuations are no more than 0.07 degrees off from the setpoint of 39 degrees. Not really sure what happened in the past day to cause the strange behavior. It seems to have returned back to normal today.

I'm running a comsol job on optimus in a tmux session named cryocavs. Should be done in less than 24 hours, judging by past durations.

I'm running a test with daqd right now, so please do not disturb for the moment.

I'm temporarily writing frames into a tempfs, which is a filesystem that exists purely in memory.  There should be ZERO IO contention for this filesystem, so if the daqd failures are due to IO then all problems should disappear.  If they don't, then we're dealing with some other problem.

There will be no data saved during this period.

I have reverted daqd to the previous configuration, so that it's writing frames to disk.  It's still showing instability.

 In order to figure out what downsampling ratio we can take, we need to determine the running time of the fxlms_filter() function. If the filter length is equal to 5000, downsampling ratio is equal to 1, number of witness channels is 1 then with ordinary compilation without speed optimization one call runs for 0.054 ms (milli seconds). The test was done on the 3 GHz Intel processor. With speed optimization flags the situation is better

-O1 0.014 ms, -O3 0.012 ms

However, Alex said that speed optimization is not supported at RCG because it produce unstable execution time for some reason. However, by default the kernel should optimize for size -Os. With this flag the running time is also 0.012 ms. We should check if the front-end machine compilers indeed use -Os flag during the compilation and also play with speed optimization flags. Flags -O3 and -Os together might also give some speed improvement.

But for now we have time value = 0.012 ms as running time for 5000 coefficient filter, 1 witness channel and downsample ratio = 1. Now, we need to check how this time is scaled if we change the parameters.

5000 cofficients - 0.012 ms

10000 coefficients - 0.024 ms

15000 coefficients - 0.036 ms

20000 coefficients - 0.048 ms

We can see that filter length scaling is linear. Now we check downsampling ratio

ratio=1 - 0.048 ms

ratio=2 - 0.024 ms

ratio=4 - 0.012 ms

Running time on the dependance of downsample ratio is also linear as well as on the dependence of the number of witness channels and degrees of freedom. 

If we want to filter 8 DOF with approximately 10 witness channels for each DOF, then 5000 length filter will make 1 cycle for ~1 ms, that is good enough to make the downsample ratio equal to 4.

Things get a little bit complicated when the code is called for the first time. Some time is taken to initialize variables and check input data. As a result the running time of the first cycle is ~0.1 ms for 1 DOF that is ~10 times more then running time of an ordinary cycle. This effect takes place at this moment when one presses reset button in the c1oaf model - the filter becomes suspended for a while. To solve this problem the initialization should be divided by several (~10) parts.

This is long overdue, but our burt files for SDF now live in the LIGO userapps SVN, as they should.

The canonical files live in places like /opt/rtcds/userapps/release/cds/c1/burtfiles/c1x01_safe.snap and are symlinked to files like /opt/rtcds/caltech/c1/target/c1x01/c1x01epics/burt/safe.snap

The 40m safety audit will be at Wednesday afternoon, March 3

Please participate getting our lab to inspection grade safety level.

Correction list by visiting safety committee, Haick Issaian is not shown:

1,  update laser, crane operator list and post it

2,  check fire extinguishers monthly, date and initials must be on the tags

3,  move drinking water tower so it does not block fire extinguisher

4,  post updated crane doc at cranes

5,  post present phone lists at IFO room phones

6,  emergency laser shutoff at the south end must be mounted without C-clamp

7,  use heavy cable tie to insure position of  mag-fan on cabinet top

Additional to do list:

a,  safety glasses to be cleaned

b, let the electrical shop fix Rack-AC power to optical tables at the ends

c, measure transmission of  laser safety glasses

d, update  IFO outside door info signs

e, update laser inventory and post it

f,  schedule annual crane inspection and renew maintenance contract

g, PSL enclosure  inner shelf needs a good clean up so it is earthquake safe

 Completed with the exception of d and g

                                                 Recommended correction list:

1,  refill- upgrade first aid boxes

2,  maintain 18" ceiling to bookshelf clearance so the ceiling fire sprinklers are not blocked: room 101

3,  label chilled water supply & return valves in IFO room

4,  calibrate bake room hoods annually

5,  update safety sign at fenced storage

              40m still to do list:

1,   clean and measure all safety glasses

2,   annual crane inspection is scheduled for 8am March 19, 1013

3,   make PSL encloser shelf earthquake proof

Do you see something that is not safe? Add it to this list please.

 We had our annual safety inspection today.  Our SOPs are outdated. The full list of needed correction will be posted tomorrow.

The most useful found was that the ITMX-ISCT ac power is coming  from 1Y1 rack. This should actually go to 1Y2 LSC rack ?

 Please test this so we do not create more ground loops.

 Annual crane inspection is scheduled for 8-11am Monday, March 17, 2014

The control room Smart UPS has two red extension cords that has to be removed: Nodus and Linux1

 Last long extension cord removed from 1Y1 to ITMX-ISCT

The AC power strip at ITMX-ISCT is coming from wall L#26

Be aware that this may affect POP QPD and POP RF Thorlabs PD

 Late adition: CHECK all viewport covers.

A, transparent Lexan sheet is protecting glass windows in a horizontal position

B, metal housing protection is required on each viewport except signal ports

C, signal ports should be shielded by optical table enclosure

We have to cover this window-camera with implosion proof cover or just remove it and blank it.

Question number 2: Do our vertically positioned windows with flip able covers require protective lexan ? NO 5-5-2014

Safety audit went soothly. We thank all participients.

Correction list:

1, Bathroom water heater cable to be stress releived and connector replaced by twister lock type.

2, Floor cable bridge at the vacuum rack to be replaced. It is cracked.

3, Sprinkler head to be moved eastward 2 ft in room 101

4, Annual crane inspection is scheduled for 8am Marc 3, 2015

5, Annual safety glasses cleaning and transmission measurement will get done tomorrow morning.

Safety glasses were measured and they are all good. I'd like to measure your personal glass if it is not on this picture.

Safety audit went smothly.

Crane inspection is scheduled for March 4

Safety glasses will be measured before April 1

Bob cleaned the safety glasses. They were sonicated in warm 2% Liquinox  water for 10 minutes.  Steve checked them by transmission measurement of 1064 nm at 150 mW

 Linus-1, Nodus and others  ac cords can be moved over to new blank yellow extension cord with multiple recepticals.

 Remove two red extension cords going to Smart UPS

 Batteries replaced and cylinder recharged. Please clean up your experimental set up if it is blocking breakers or entry way etc.

I will start the final clean up 2pm today.

Emergency exit lights were inspected:  2 out of 13 batteries have to be replaced

One of the Halon fire extinguishers needs to be recharged out of 8

Please do participate in preparation for the upcoming safety audit on Feb 28

  1064 nm transmison were measured of 40m safety glasses as shown . Their performance did not degrade. They are as good as their labels.

Safety glasses 1064 nm transmission measured at ~200 mW level. They are all good.

This is the third safety glasses that I found laying around in the IFO lab lately. Safety glasses must be worn ALL times in the IFO room!

This rule is essential for your protection! Please do not enter if you can not put up with this regulation!

Lightwave M126-1064-700 lasers  sn415  at east end of the Y arm and  sn201 at the AP table  are connected individually  to one each EMERGENCY LASER SHUT OFF SWITCH. 

The PSL had one 1064 nm beam to be blocked around the north east side. The end enclosures are fine.

Nancy and Sharmila received introductory early bird surf safety training for the 40m lab.

 Rijuparna Chakraborty and Elli Elenora King received 40m specific  basic safety training in the 40mLab

Eric Quintero and Mike Jenson received 40m specific basic safety training.

Den Martynov received 40m specific safety training.

Ayaka Shoda, visiting graduate student received basic 40m specific safety training today