40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 42 of 341  Not logged in ELOG logo
ID Date Author Type Categoryup Subject
  6392   Fri Mar 9 11:59:38 2012 Zweizig the ELOG MavenSummaryCDSNDS2 restart

 Hi Rana,

It looks like the channe list file has a few blank lines that the channel list reader is choking on. I removed the lines and it is working now.. I have made the error message a bit more obvious (gave the file name  and line number) and allowed it to ignore empty lines so this won't cause problems with future versions (when installed). The bottom line is nds2 is now running on mafalda.
Best regards,

John   

  6397   Fri Mar 9 20:44:24 2012 Jim LoughUpdateCDSDAQ restart with new ini file

DAQ reload/restart was performed at about 1315 PST today. The previous ini file was backed up as c1pem20120309.ini in the /chans/daq/working_backups/ directory.

I set the following to record:

The two JIMS channels at 2048:
[C1:PEM-JIMS_CH1_DQ] Persistent version of JIMS channel. When bit drops to zero indicating something bad (BLRMS threshold exceeded) happens the bit stays at zero  for >= the value of the persist EPICS variable.
[C1:PEM-JIMS_CH2_DQ] Non-persistent version of JIMS channel.

And all of the BLRMS channels at 256:
Names are of the form:
[C1:PEM-RMS_ACC1_F0p1_0p3_DQ]
[C1:PEM-RMS_ACC1_F0p3_1_DQ]

On monday I intend to look at the weekend seismic data to establish thresholds on the JIMS channels.

256 was the lowest rate possible according to the RCG manual. The JIMS channels are recorded at 2048 because I couldn't figure out how to disable the decimation filter. I will look into this further.

  6404   Tue Mar 13 13:28:31 2012 Ryan FisherUpdateCDSDAQ restart with new ini file
Extra note: This was the ini file that was edited:
/cvs/cds/rtcds/caltech/c1/chans/daq/C1PEM.ini
  6422   Thu Mar 15 08:48:40 2012 RyanSummaryCDSSummary of Syracuse Visit to 40m Mar 5-9 2012

JIMS Channels in PEM Model

The PEM model has been modified now to include the JIMS(Joint Information Management System) channel processing. Additionally Jim added test points at the outputs of the BLRMS.

For each seismometer channel, five bands are compared to threshold values to produce boolean results.  Bands with RMS below threshold produce bits with value 1, above threshold results in 0.  These bits are combined to produce one output channel that contains all of the results.

A persistent version of the channel is generated by a new library block that called persist which holds the value at 0 for a number of time steps equal to an EPICS variable setting from the time the boolean first drops to zero. The persist allows excursions shorter than the timestep of a downsampled timeseries to be seen reliably.

The EPICS variables for the thresholds are of the form (in order of increasing frequency):
C1:PEM-JIMS_GUR1X_THRES1
C1:PEM-JIMS_GUR1X_THRES2
etc.
 
The EPICS variables for the persist step size are of the form:
C1:PEM-JIMS_GUR1X_PERSIST
C1:PEM-JIMS_GUR1Y_PERSIST
etc.

The JIMS Channels are being recorded and written to frames:

The two JIMS channels at 2048:
[C1:PEM-JIMS_CH1_DQ] Persistent version of JIMS channel. When bit drops to zero indicating something bad (BLRMS threshold exceeded) happens the bit stays at zero  for >= the value of the persist EPICS variable.
[C1:PEM-JIMS_CH2_DQ] Non-persistent version of JIMS channel.

And all of the BLRMS channels at 256:
Names are of the form:
[C1:PEM-RMS_ACC1_F0p1_0p3_DQ]
[C1:PEM-RMS_ACC1_F0p3_1_DQ]

For additional details about the JIMS Channels and the implementation, please see the previous elog entries by Jim.

 

Conlog

I have a working aLIGO Conlog/EPICS Log installed and running on megatron.

Please see this wiki page for the details of use:
https://wiki-40m.ligo.caltech.edu/aLIGO%20EPICs%20log%20%28conlog%29

I also edited this page with restart instructions for megatron:
https://wiki-40m.ligo.caltech.edu/Computer_Restart_Procedures#megatron

Please see Ryan's previous elog entries for installation details.

Future Work

  • Determine useful thresholds for each band
  • Generate MEDM Screens for JIMS Channels
  • Add a decimation option to channels
  • Add EPICS Strings in PEM model to describe bits in JIMS Channels
  • Add additional JIMS Channels: Testing additional characterization methods
  • Implement a State Log on Megatron: Will Provide a 1Hz index into JIMS Channels
  • Generate a single web page that allows access to aLIGO Conlog/EPICS Log and State Log

 

  6436   Thu Mar 22 16:45:06 2012 kiwamuUpdateCDSc1scx and c1scy not properly running

It seems that neither c1scx nor c1scy is working properly as their ADC counts are showing digital-zeros.

However the IOPs, c1gcx and c1gcy look running fine, and also the IOPs seem successfully recognizing the ADCs according to dmesg.

Also there is one more confusing fact : c1scx and c1scy are synchronizing to the timing signal somehow.

I restarted the c1scx front end model to see if this helps, but unfortunately it didn't work.

As this is not the top priority concern for now, I am leaving them as they are now with the watchgods off.

(I may try hardware rebooting them in this evening)

Quote from #6434

The power was turned back on at 4pm It took some time for Suresh to restart the computers. We have damping but things are not perfect yet. Auto BURTH did not work well.

 

  6438   Thu Mar 22 17:41:15 2012 sureshUpdateCDSc1scx and c1scy not properly running

Quote:

It seems that neither c1scx nor c1scy is working properly as their ADC counts are showing digital-zeros.

Quote from #6434

The power was turned back on at 4pm It took some time for Suresh to restart the computers. We have damping but things are not perfect yet. Auto BURTH did not work well.

 When Steve and I restarted the c1iscex and c1iscey computers after the power shutdown, the models within them did not start-up automatically.  I had to start them manually from a terminal in the control room. 

I also tried rebooting the FB a couple of times.  Did not make any difference.

Manually starting the c1x05, c1scy and c1x01, c1scx models (with the Burt Restore button ON) did not resolve the issue of zeros in the epics screens.  though it did re-establish timing. 

  6439   Thu Mar 22 23:43:56 2012 KojiUpdateCDSc1scx and c1scy not properly running

Did you guys checked if the simplant switch is set to "REAL WORLD" mode?

Edit by KI:

Bingo ! The input signals were bypassed to the simplant. I switched the simplant settings to REAL WORLD and now both end suspensions are working fine.

  6540   Tue Apr 17 11:05:04 2012 JamieUpdateCDSCDS upgrade in progress

I am continuing to attempt to upgrade the CDS system to RTS 2.5.  Systems will continue to be up and down for the rest of the day.

  6541   Tue Apr 17 19:03:09 2012 JamieUpdateCDSCDS upgrade in progress

Upgrade progresses, but not complete.  There are some relatively minor issues, and one potentially big issue.

All new software has been installed, including the new epics that supports long channel names.

I've been doing a LOT of cleanup.  It was REALLY messy in there.

The new framebuilder/daqd code is running on fb.

Models are compiling with the new RCG and I am able to get them running.  Some of them are not compiling for relatively minor reasons (the simulink models need updating).  I'm also running into compile problems with IOPs that are using the dolphin drivers.

The major issue is that the framebuilder and the models are not syncing their timing, so there's no data collection.  I've spoken to Alex and he and Rolf are going to come over tomorrow to sort it out.  It's possible that we're missing timing hardware that the new code is expecting.

There are still some stability issues I haven't sorted out yet, and I have a lot more cleanup to do.

At this rate I'm going to shoot for being done Thursday.

  6546   Wed Apr 18 19:59:48 2012 JamieUpdateCDSCDS upgrade success

The upgrade is nearly complete:

  • new daqd code is running on fb
  • the fe/daqd timing issue was resolved by adjusting the GPS offset in the daqdrc.  I will document this more later.
  • the power outage conveniently rebooted all the front-end machines, so they're all now running new caRepeater
  • all models have been successfully recompiled with RCG 2.5 (with only a couple small glitches)
  • all new models are running on all front-end machines (with a couple exceptions)
  • all suspension models seem to be damping under local control (PRM is having troubles that are likely unrelated to the upgrade).
  • a lot of cleanup has been done

Remaining tasks/issues:

  • more testing OF EVERYTHING needs o be done
  • I did not yet update the DIS dolphin code, so we're running with the old code.  I don't think this is a problem, but it would be nice to get us running what they're running at the sites
  • I tried to cleanup/simplify how front-end initialization is done.  However, there is a problem and models are not auto-starting after reboot.  This needs to be fixed.
  • the userapps directory is in a new place (/opt/rtcds/userapps).  Not everything in the old location was checked into the repository, so we need to check to make sure everything that needs to be is checked in, and that all the models are running the right code.
  • the c1oaf model seems to be having a dolphin issue that needs to be sorted
  • the c1gfd model causes c1ioo to crash immediately upon being loaded.  I have removed it from the rtsystab.  That model needs to be fixed.
  • general model cleanup is in order.
  • more front-end cleanup is needed, particularly in regards to boot-up procedure.
  • document the entire upgrade procedure.

I'll finish up these remaining tasks tomorrow.

  6547   Wed Apr 18 23:12:49 2012 DenUpdateCDSoaf

Adaptive filter outputs some non-zero signal in the OFF position

Screenshot.png

I turned "ON" one of them and c1lsc suspended, I've rebooted it and restarted models on c1lsc and c1sus.

Now it also outputs something non-zero though the first line of the adaptive code is "if(OFF) output=0.0; return;" May be another version of the code has been compiled.

Edit: Old version (~september) of the code and oaf model is running now. In the 2.1 code there was a link from src/epics/simLink to oaf code for each DOF. It seems that 2.5 version finds models and c codes in standard directories. I need to move working code to the proper directory.

  6548   Thu Apr 19 08:43:16 2012 JamieUpdateCDSoaf

Quote:

Edit: Old version (~september) of the code and oaf model is running now. In the 2.1 code there was a link from src/epics/simLink to oaf code for each DOF. It seems that 2.5 version finds models and c codes in standard directories. I need to move working code to the proper directory.

 Yes, things have changed.  Please wait until I have things cleaned up before working on models.  I'll explain what the new setup is.

  6552   Fri Apr 20 19:54:57 2012 JamieUpdateCDSCDS upgrade problems

I ran into a couple of snags today.

A big one is that the framebuilder daqd started going haywire when I told it to start writing frames.  After restart the logs started showing this:

[Fri Apr 20 17:23:40 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:41 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:42 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:43 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:44 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:45 2012] main profiler warning: 0 empty blocks in the buffer
GPS time jumped from 1019002442 to 1019003041
FATAL: exception not rethrown
FATAL: exception not rethrown
FATAL: exception not rethrown

and the network seemed like it started to get really slow.  I wasn't able to figure out what was going on, so I shut the frame writing off again.  I'll have to work with Rolf on that next week.

Another big problem is the workstation application upgrades.  The NDS protocol version has been incremented, which means that all the NDS client applications have to be upgraded.  The new dataviewer is working fine (on pianosa), but dtt is not:

controls@pianosa:~ 0$ diaggui
diaggui: symbol lookup error: /ligo/apps/linux-x86_64/gds-2.15.1/lib/libligogui.so.0: undefined symbol: _ZN18TGScrollBarElement11ShowMembersER16TMemberInspector
controls@pianosa:~ 127$ 

I don't know what's going on here.  All the library paths are ok.  Hopefully I'll be able to figure this out soon.  The old version of dtt definitely does not work with the new setup.

I might go ahead and upgrade some more of the workstations to Ubuntu in the next couple of days as well, so everything is more on the same page.

I also tried to cleanup the front-end boot process, which has it's own problems (models won't auto-start).  I haven't figured that out yet either.  It really needs to just be completely overhauled.

  6554   Sat Apr 21 17:38:19 2012 JamieUpdateCDSdtt, dataviewer working; problem with trend frames

Quote:

Another big problem is the workstation application upgrades.  The NDS protocol version has been incremented, which means that all the NDS client applications have to be upgraded.  The new dataviewer is working fine (on pianosa), but dtt is not:

controls@pianosa:~ 0$ diaggui
diaggui: symbol lookup error: /ligo/apps/linux-x86_64/gds-2.15.1/lib/libligogui.so.0: undefined symbol: _ZN18TGScrollBarElement11ShowMembersER16TMemberInspector
controls@pianosa:~ 127$

 dtt (diaggui) and dataviewer are now working on pianosa to retrieve realtime data and past data from DQ channels.

Unfortunately it looks like there may be a problem with trend data, though.  If I try to retrieve 1 minute of "full data" with dataviewer for channel C1:SUS-ITMX_SUSPOS_IN1_DQ around GPS 1019089138 everything works fine:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
T0=12-04-01-00-17-45; Length=60 (s)
60 seconds of data displayed

but if I specify any trend data (second, minute, etc.) I get the following:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
Server error 18: trend data is not available
datasrv: DataWriteTrend failed in daq_send().
T0=12-04-01-00-17-45; Length=60 (s)
No data output.

Alex warned me that this might have happened when I was trying to test the new daqd without first turning off frame writing.

I'm not sure how to check the integrity of the frames, though.  Hopefully they can help sort this out on Monday.

  6555   Sat Apr 21 20:40:28 2012 JamieUpdateCDSframebuilder frame writing working again

Quote:

A big one is that the framebuilder daqd started going haywire when I told it to start writing frames.  After restart the logs started showing this:

[Fri Apr 20 17:23:40 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:41 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:42 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:43 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:44 2012] main profiler warning: 0 empty blocks in the buffer
[Fri Apr 20 17:23:45 2012] main profiler warning: 0 empty blocks in the buffer
GPS time jumped from 1019002442 to 1019003041
FATAL: exception not rethrown
FATAL: exception not rethrown
FATAL: exception not rethrown

and the network seemed like it started to get really slow.  I wasn't able to figure out what was going on, so I shut the frame writing off again.  I'll have to work with Rolf on that next week.

So the framebuilder/daqd frame writing issue seems to have been a transient one.  Alex said he tried enabling frame writing manually and it worked fine, so I tried re-enabling the frame writing config lines and sure enough it worked fine.  So it's a mystery.  Alex said the "main profiler warning" lines tend to show up when data is backed up in the buffer, although he didn't explain why exactly that would have been the issue here.

daqdrc frame writing directives:

start frame-saver;
sync frame-saver;
start trender;
start trend-frame-saver;
sync trend-frame-saver;
start minute-trend-frame-saver;
sync minute-trend-frame-saver;
start raw_minute_trend_saver;
  6556   Sat Apr 21 21:10:34 2012 JamieUpdateCDStrend frame issue partially resolved

Quote:

Unfortunately it looks like there may be a problem with trend data, though.  If I try to retrieve 1 minute of "full data" with dataviewer for channel C1:SUS-ITMX_SUSPOS_IN1_DQ around GPS 1019089138 everything works fine:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
T0=12-04-01-00-17-45; Length=60 (s)
60 seconds of data displayed

but if I specify any trend data (second, minute, etc.) I get the following:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
Server error 18: trend data is not available
datasrv: DataWriteTrend failed in daq_send().
T0=12-04-01-00-17-45; Length=60 (s)
No data output.

Alex warned me that this might have happened when I was trying to test the new daqd without first turning off frame writing.

Alex told me that the "trend data is not available" message comes from the "trender" functionality not being enabled in daqd.  After re-enabling it (see #6555) minute trend data was available again.  However, there still seems to be an issue with second trends.  When I try to retrieve second trend data from dataviewer for which minute trend data *is* available I get the following error message:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
No data found

read(); errno=9
read(); errno=9
T0=12-04-04-02-14-29; Length=3600 (s)
No data output.

Awaiting more help from Alex...

  6560   Tue Apr 24 14:30:08 2012 JamieUpdateCDSIntroducing: rtcds, a utility for interacting with the CDS RTS/RCG

The new rtcds utility

I have written a new utility for interacting with the CDS RTS/RCG system: "rtcds".  It should be available on all workstations and front-end machines, but certain commands are restricted to run on certain front end machines (build, start, stop, etc.).  Here's the help:

controls@c1lsc ~ 0$ rtcds help
Usage: rtcds <command> [arg]

Available commands:

  build|make SYS      build model
  install SYS         install model
  start SYS|all       start model
  restart SYS|all     restart running model
  stop|kill SYS|all   stop running model
  list                list all models for current host

controls@c1lsc ~ 0$ 

Please use this new utility from now on when interacting with RTS.

I'll be improving and expanding it as we go.   Please let me know if you encounter any problems.

 

  6561   Tue Apr 24 14:35:37 2012 JamieUpdateCDSlimited second trend lookback

Quote:

Alex told me that the "trend data is not available" message comes from the "trender" functionality not being enabled in daqd.  After re-enabling it (see #6555) minute trend data was available again.  However, there still seems to be an issue with second trends.  When I try to retrieve second trend data from dataviewer for which minute trend data *is* available I get the following error message:

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
No data found

read(); errno=9
read(); errno=9
T0=12-04-04-02-14-29; Length=3600 (s)
No data output.

Awaiting more help from Alex...

It looks like this is actually just a limit of how long we're saving the second trends, which is just not that long.  I'll look into extending the second trend look-back.

  6562   Tue Apr 24 14:55:00 2012 JamieUpdateCDScds code paths (rtscore, userapps) have moved

NOTE: Unless you really care about what's going on under the hood, please ignore this entire post and go here: USE THE NEW RTCDS UTILITY

Quote:

  • the userapps directory is in a new place (/opt/rtcds/userapps).  Not everything in the old location was checked into the repository, so we need to check to make sure everything that needs to be is checked in, and that all the models are running the right code.

 An important new aspect of this upgrade is that we have some new directories and some of the code paths have moved to correspond with the "standard" LIGO CDS filesystem hierarchy:

  • rtscore:      /opt/rtcds/rtscore/release       -->  RTS RCG source code (svn: adcLigoRTS)
  • userapps: /opt/rtcds/userapps/release  -->  CDS userapps source for models, RCG C code, medm screens, scripts, etc (svn: cds_user_apps)
  • rtbuild:       /opt/rtcds/caltech/c1/rtbuild    -->  RCG build directory

All work should be done in the "userapps" directory, and all builds should be done in the build directory.  Some important points:

WARNING: DO NOT MODIFY ANYTHING IN RTSCORE

This is important.  The rtscore directory is now just where the RCG source code is stored.  WE NO LONGER BUILD MODELS IN THE RTS SOURCE  Please use the rtbuild directory instead.

NO MORE MODEL/CODE SYMLINKS

You don't need to link you model or code anywhere to get it to compile.  The RCG now uses proper search paths to source in the RCG_LIB_PATH to find the needed source.  This has been configured by the admin, so as long as you put your code in the right place it should be found.

ALL CODE/MODELS/ETC GO IN USERAPPS

All RTS code is now stored ENTIRELY in the userapps directory (e.g. /opt/rtcds/userapps/release/isc/c1/models/c1lsc.mdl).  This is more-or-less the same as before, except that symlinking is no longer necessary.  I have placed a symlink at the old location for convenience.

BUILD ALL MODELS IN RTSCORE

You can run "make c1lsc" in the rtbuild directory, just as you used to in the old core directory.  However, don't do that.  USE THE NEW RTCDS UTILITY instead.

  6573   Thu Apr 26 16:35:34 2012 JamieSummaryCDSrosalba now running Ubuntu 10.04

This morning I installed Ubuntu 10.04 on rosalba.  This is the same version that is running on pianosa.  The two machines should be identically configured, although rosalba may be missing some apt-getable packages.

  6574   Thu Apr 26 18:15:59 2012 JamieUpdateCDSpossible issue with mx_stream on front ends

I'm noticing what appears to be occasional failures of mx_stream on the front end machines.  It doesn't happen that frequently, but I've noticed it a couple of times already since the upgrade.

The symptom is that the DC Status goes to "0xbad" (red) and the "FE NET" goes red for all models on a given front end.

The solution seems to be restarting mx_stream on the given front end:    sudo  /etc/init.d/mx_stream restart"

There is nothing in the mx_stream log:

 controls@c1sus ~ 0$ cat /opt/rtcds/caltech/c1/target/fb/mx_stream_logs/c1sus.log 
 c1x02
 c1sus
 c1mcs
 c1rfm
 c1pem
 mmapped address is 0x7f43740ec000
 mapped at 0x7f43740ec000
 mmapped address is 0x7f43700ec000
 mapped at 0x7f43700ec000
 mmapped address is 0x7f436c0ec000
 mapped at 0x7f436c0ec000
 mmapped address is 0x7f43680ec000
 mapped at 0x7f43680ec000
 mmapped address is 0x7f43640ec000
 mapped at 0x7f43640ec000
 send len = 263596
 Connection Made

but I do see some funny messages in the front end dmesg:

 [200341.317912] DXH Adapter 0 : Heartbeat alive-check for node=12 failed (cnt=8387 state=0x1 deb=0 val=0).
 [200341.318670] DXH Adapter 0 : Session for node 12 is disabled - Status = 0x5
 [200341.319062] Session callback reason=1 status=5 target_node=12
 [200341.319069] Session callback reason=3 status=0 target_node=12
 [200341.359534] (map_table_check_access:752):my id 1 ->  remote id 2 : entry was valid - is now tentatively valid
 [200341.859584] DXH Adapter 0 : Probe failure for node=12 - disabling session probeStatus=0x40000f02
 [200341.860335] DXH Adapter 0 : Session for node 12 is disabled - Status = 0x3
 [200341.860728] Session callback reason=1 status=3 target_node=12
 [200374.006111] DXH Adapter 0 : Set reachable remote node list.
 [200409.020670] DXH Adapter 0 : Set reachable remote node list.
 [200409.021076] DXH Adapter 0 : Session for node 12 is deleted - Status = 0x0
 [200409.021468] Session callback reason=5 status=0 target_node=12
 [200412.362824] (map_table_insert:648):** successfully inserted **(valid unicast) inst 0 node 1->0 fwd 0 fwd_tp 4 egress 0
 [200418.025994] (map_table_check_access:752):my id 1 ->  remote id 0 : entry was valid - is now invalid
 [200418.025998] (map_table_insert:648):** successfully inserted **(valid unicast) inst 0 node 1->2 fwd 0 fwd_tp 4 egress 0
 [200421.743916] Session callback reason=0 status=0 target_node=12
 [200422.073776] DXH Adapter 0 : Set reachable remote node list.
 [200422.342446] Session callback reason=7 status=0 target_node=12
 [200422.342454] DXH Adapter 0 : Session for node 12 is ok.

I'm awaiting feedback from experts.

 

  6576   Thu Apr 26 20:44:23 2012 JamieUpdateCDSdaq network failure, c1ioo failing to start models

Den tried adding a SINGLE acquire channel to the c1ioo, which for some reason hung c1ioo and took down the entire DAQ network (at least all communication between the front ends and the fb).  We recovered by restarting c1ioo and restarting mx_stream on all the rest of the front-ends

After "recovering", though, c1ioo is failing to load models, or at least it's IOP. Here is the tail of dmesg when trying to start the IOP:

[ 1751.140283] c1x03: Initializing space for daqLib buffers
[ 1751.140284] c1x03: Initializing Network
[ 1751.140285] c1x03: Found 1 frameBuilders on network
[ 1751.250658] CPU 2 is now offline
[ 1751.250657] c1x03: Sync source = 4
[ 1751.250657] c1x03: Waiting for EPICS BURT Restore = 1
[ 1751.310008] c1x03: Waiting for EPICS BURT 0
[ 1751.310008] c1x03: BURT Restore Complete
[ 1751.310008] c1x03: Initialized servo control parameters.
[ 1751.311699] c1x03: DAQ Ex Min/Max = 1 3
[ 1751.311699] c1x03: DAQ XEx Min/Max = 3 53
[ 1751.311733] c1x03: DAQ Tp Min/Max = 10001 10007
[ 1751.311733] c1x03: DAQ XTp Min/Max = 10007 10507
[ 1751.311737] c1x03: DIRECT MEMORY MODE of size 64
[ 1751.311737] c1x03: daqLib DCU_ID = 33
[ 1751.311737] c1x03: Invalid num daq chans = 0
[ 1751.311737] c1x03: DAQ init failed -- exiting

The chan file for this model (/opt/rtcds/caltech/c1/chans/daq/C1X03.ini) looks totally fine, has two un-acquired channels uncommented, and has otherwise not been touched. The C1:FEC-33_MSGDAQ is also reading: "ERROR reading DAQ file!"

I'm at a loss for what is going on. I've tried restarting every CDS process on the machine, restarting the model multiple times, restarting fb, and even restarting the entire c1ioo machine, all to no affect.

  6577   Fri Apr 27 08:11:19 2012 steveUpdateCDSc1ioo condition

 

 

Attachment 1: c1ioo.png
c1ioo.png
  6578   Fri Apr 27 09:00:15 2012 JamieUpdateCDSsus watchdogs?

Why are all the suspension watchdogs tripped?  None of the suspension models are running on c1ioo, so they should be completely unaffected.  Steve, did you find them tripped, or did you shut them off?

In either event they should be safetly turned back on.

  6579   Fri Apr 27 09:27:48 2012 DenUpdateCDSsus watchdogs?

Quote:

Why are all the suspension watchdogs tripped?  None of the suspension models are running on c1ioo, so they should be completely unaffected.  Steve, did you find them tripped, or did you shut them off?

In either event they should be safetly turned back on.

 I've turned off the coils. Though non of them are on the c1ioo, who knows what can happen when we'll try to run the models again.

  6580   Fri Apr 27 12:12:14 2012 DenUpdateCDSc1ioo is back

Rolf came to the 40m today and managed to figure out what the problem is. Reading just dmesg was not enough to solve the problem. Useful log was in

>> cat /opt/rtcds/caltech/c1/target/c1x03/c1x03epics/iocC1.log

Starting iocInit
The CA server's beacon address list was empty after initialization?
iocRun: All initialization complete
sh: iniChk.pl: command not found
Failed to load DAQ configuration file

iniChk.pl checks the .ini file of the model.

>> cat /opt/rtcds/rtscore/release/src/drv/param.c


int loadDaqConfigFile(DAQ_INFO_BLOCK *info, char *site, char *ifo, char *sys)
{

  strcpy(perlCommand, "iniChk.pl ");
  .........
  strcat(perlCommand, fname); // fname - name of the .ini file
  ..........
}

So the problem was not in the C1X03.ini. The code could not find the perl script though it was in the /opt/rtcds/caltech/c1/scripts directory. Some environment variables are not set. Rolf added /opt/rtcds/caltech/c1/scripts/ to $PATH variable and c1ioo models (x03, ioo, gcv) started successfully. He is not sure whether this is a right way or not, because other machines also do not have "scripts" directory in their PATH variable.

>> cat /opt/rtcds/caltech/c1/target/c1x03/c1x03epics/iocC1.log

Starting iocInit
The CA server's beacon address list was empty after initialization?
iocRun: All initialization complete

Total count of 'acquire=0' is 2
Total count of 'acquire=1' is 0
Total count of 'acquire=0' and 'acquire=1' is 2

Counted 0 entries of datarate=256     for a total of 0
Counted 0 entries of datarate=512     for a total of 0
Counted 0 entries of datarate=1024     for a total of 0
Counted 0 entries of datarate=2048     for a total of 0
Counted 0 entries of datarate=4096     for a total of 0
Counted 0 entries of datarate=8192     for a total of 0
Counted 0 entries of datarate=16384     for a total of 0
Counted 0 entries of datarate=32768     for a total of 0
Counted 2 entries of datarate=65536     for a total of 131072

Total data rate is 524288 bytes - OK

Total error count is 0

Rolf mentioned about automatic set up of variables - kiis of smth like that - probably that script is not working correctly. Rolf will add this problem to his list.

  6582   Mon Apr 30 13:00:50 2012 SureshUpdateCDSFrame Builder is down

Frame builder is down.  PRM has tripped its watch dogs.  I have reset the watch dog on PRM and turned on the OPLEV. It has damped down.  Unable to check what happened since FB is not responding.

There was an minor earthquake yesterday morning which people could feel a few blocks away.  It could have caused the the PRM to unlock.

Jamie,Rolf,  is it okay or us to restart the FB?  

  6583   Mon Apr 30 13:58:25 2012 JamieUpdateCDSFrame Builder is down

Quote:

Frame builder is down.  PRM has tripped its watch dogs.  I have reset the watch dog on PRM and turned on the OPLEV. It has damped down.  Unable to check what happened since FB is not responding.

There was an minor earthquake yesterday morning which people could feel a few blocks away.  It could have caused the the PRM to unlock.

Jamie,Rolf,  is it okay or us to restart the FB?  

 If it's down it's alway ok to restart it.  If it doesn't respond or immediately crashes again after restart then it might require some investigation, but it should always be ok to restart it.

  6584   Mon Apr 30 16:56:05 2012 SureshUpdateCDSFrame Builder is down

Quote:

Quote:

Frame builder is down.  PRM has tripped its watch dogs.  I have reset the watch dog on PRM and turned on the OPLEV. It has damped down.  Unable to check what happened since FB is not responding.

There was an minor earthquake yesterday morning which people could feel a few blocks away.  It could have caused the the PRM to unlock.

Jamie,Rolf,  is it okay or us to restart the FB?  

 If it's down it's alway ok to restart it.  If it doesn't respond or immediately crashes again after restart then it might require some investigation, but it should always be ok to restart it.

I tried restarting the fb in two different ways.  Neither of them re-established the connection to dtt or epics.

1) I restarted the fb from the control room console with the 'shutdown' command.  No change.

2) I halted the machine with 'shutdown -h now' and restarted it with the hardware reset button on its front-panel. No change.

The console connected to the fb showed that the network file systems did not load.  Could this have resulted in failure to start several services since it could not find the files which are stored on the network file system?

The fb is otherwise healthy since I am able to ssh into it and browse the directory structure.

  6586   Mon Apr 30 20:43:33 2012 SureshUpdateCDSFrame Builder is down

Quote:

Quote:

Quote:

Frame builder is down.  PRM has tripped its watch dogs.  I have reset the watch dog on PRM and turned on the OPLEV. It has damped down.  Unable to check what happened since FB is not responding.

There was an minor earthquake yesterday morning which people could feel a few blocks away.  It could have caused the the PRM to unlock.

Jamie,Rolf,  is it okay or us to restart the FB?  

 If it's down it's alway ok to restart it.  If it doesn't respond or immediately crashes again after restart then it might require some investigation, but it should always be ok to restart it.

I tried restarting the fb in two different ways.  Neither of them re-established the connection to dtt or epics.

1) I restarted the fb from the control room console with the 'shutdown' command.  No change.

2) I halted the machine with 'shutdown -h now' and restarted it with the hardware reset button on its front-panel. No change.

The console connected to the fb showed that the network file systems did not load.  Could this have resulted in failure to start several services since it could not find the files which are stored on the network file system?

The fb is otherwise healthy since I am able to ssh into it and browse the directory structure.

[Mike, Rana]

The fb is okay.  Rana found that it works on Pianosa, but not on Allegra or Rossa.  It also works on Rosalba, on which Jamie recently installed Ubuntu.

The white fields on the medm  'Status' screen for fb are an unrelated problem.

 

 

  6591   Tue May 1 08:18:50 2012 JamieUpdateCDSFrame Builder is down

Quote:

 

I tried restarting the fb in two different ways.  Neither of them re-established the connection to dtt or epics.

 Please be conscious of what components are doing what.  The problem you were experiencing was not "frame builder down".  It was "dtt not able to connect to frame builder".  Those are potentially completely different things.  If the front end status screens show that the frame builder is fine, then it's probably not the frame builder.

Also "epics" has nothing whatsoever to do with any of this.  That's a completely different set of stuff, unrelated to DTT or the frame builder.

  6599   Thu May 3 19:52:43 2012 JenneUpdateCDSOutput errors from dither alignment (Xarm) script

epicsThreadOnceOsd epicsMutexLock failed.
Segmentation fault
Number found where operator expected at -e line 1, near "0 0"
        (Missing operator before  0?)
syntax error at -e line 1, near "0 0"
Execution of -e aborted due to compilation errors.
Number found where operator expected at (eval 1) line 1, near "*  * 50"
        (Missing operator before  50?)
epicsThreadOnceOsd epicsMutexLock failed.
Segmentation fault
Number found where operator expected at -e line 1, near "0 0"
        (Missing operator before  0?)
syntax error at -e line 1, near "0 0"
Execution of -e aborted due to compilation errors.
syntax error at -e line 1, at EOF
Execution of -e aborted due to compilation errors.
Number found where operator expected at (eval 1) line 1, near "*  * 50"
        (Missing operator before  50?)
epicsThreadOnceOsd epicsMutexLock failed.
status : 0
I am going to execute the following commands
ezcastep -s 0.6 C1:SUS-ETMX_PIT_COMM +-0,50
ezcastep -s 0.6 C1:SUS-ITMX_PIT_COMM +,50
ezcastep -s 0.6 C1:SUS-ETMX_YAW_COMM +,50
ezcastep -s 0.6 C1:SUS-ITMX_YAW_COMM +-0,50
ezcastep -s 0.6 C1:SUS-BS_PIT_COMM +0,50
ezcastep -s 0.6 C1:SUS-BS_YAW_COMM +0,50
hit a key to execute the commands above

  6609   Sun May 6 00:11:00 2012 DenUpdateCDSmx_stream

c1sus and c1iscex computers could not connect to framebuilder, I restarted it, did not help. Then I restarted mx_stream daemon on each of the computers and this fixed the problem.

sudo /etc/init.d/mx_stream restart

  6616   Mon May 7 21:05:38 2012 DenUpdateCDSbiquad filter form

I wanted to switch the implementation of IIR_FILTER from DIRECT FORM II to BIQUAD form in C1IOO and C1SUS models. I modified RCG file /opt/rtcds/rtscore/release/src/fe/controller.c by adding #define CORE_BIQUAD line:

#ifdef OVERSAMPLE
#define CORE_BIQUAD      
#if defined(CORE_BIQUAD)

C1IOO model compiled, installed and is running now. C1SUS model compiled, but during installation I've got an error:

controls@c1sus ~ 0$ rtcds install c1sus


Installing system=c1sus site=caltech ifo=C1,c1
Installing /opt/rtcds/caltech/c1/chans/C1SUS.txt
Installing /opt/rtcds/caltech/c1/target/c1sus/c1susepics
Installing /opt/rtcds/caltech/c1/target/c1sus
Installing start and stop scripts
/opt/rtcds/caltech/c1/scripts/killc1sus
Performing install-daq
Updating testpoint.par config file
/opt/rtcds/caltech/c1/target/gds/param/testpoint.par
/opt/rtcds/rtscore/branches/branch-2.5/src/epics/util/updateTestpointPar.pl -par_file=/opt/rtcds/caltech/c1/target/gds/param/archive/testpoint_120507_205359.par -gds_node=21 -site_letter=C -system=c1sus -host=c1sus
Installing GDS node 21 configuration file
/opt/rtcds/caltech/c1/target/gds/param/tpchn_c1sus.par
Installing auto-generated DAQ configuration file
/opt/rtcds/caltech/c1/chans/daq/C1SUS.ini
Installing EDCU ini file
/opt/rtcds/caltech/c1/chans/daq/C1EDCU_SUS.ini
Installing Epics MEDM screens
Running post-build script

ERROR: Could not find file: test.py
Searched path: :/opt/rtcds/userapps/release/cds/c1/scripts:/opt/rtcds/userapps/release/cds/common/scripts:/opt/rtcds/userapps/release/isc/c1/scripts:/opt/rtcds/userapps/release/isc/common/scripts:/opt/rtcds/userapps/release/sus/c1/scripts:/opt/rtcds/userapps/release/sus/common/scripts:/opt/rtcds/userapps/release/psl/c1/scripts:/opt/rtcds/userapps/release/psl/common/scripts
Exiting
make: *** [install-c1sus] Error 1

Jamie, what is this test.py?

  6618   Mon May 7 21:46:10 2012 DenUpdateCDSguralp signal error

GUR1_XYZ_IN1 and GUR2_XYZ_IN1 are the same and equal to GUR2_XYZ.  This is bad since GUR1_XYZ_IN1 should be equal to GUR1_XYZ.  Note that GUR#_XYZ are copies of GUR#_XYZ_OUT, so there may be (although there isn't right now) filtering between the _IN1's and the _OUT's.  But certainly GUR1 should look like GUR1, not GUR2!!!

Looks like CDS problem, maybe some channel-hopping going on? I'm trying a restart of the c1sus computer right now, to see if that helps.....

Figure:  Green and red should be the same, yellow and blue should be the same.  Note however that green matches yellow and blue, not red.  Bad.

guralps.png

 

 

  6619   Mon May 7 22:39:37 2012 DenUpdateCDSc1sus

[Jenne, Den]

We decided to reboot C1SUS machine in hope that this will fix the problem with seismic channels. After reboot the machine could not connect to framebuilder. We restarted mx_stream but this did not relp. Then we manually executed

/opt/rtcds/caltech/c1/target/fb/mx_stream -s c1x02 c1sus c1mcs c1rfm c1pem -d fb:0 -l /opt/rtcds/caltech/c1/target/fb/mx_stream_logs/c1sus.log

but c1sus still could not connect to fb. This script returned the following error:

controls@c1sus ~ 128$ cat /opt/rtcds/caltech/c1/target/fb/mx_stream_logs/c1sus.log


c1x02
c1sus
c1mcs
c1rfm
c1pem
mmapped address is 0x7fb5ef8cc000
mapped at 0x7fb5ef8cc000
mmapped address is 0x7fb5eb8cc000
mapped at 0x7fb5eb8cc000
mmapped address is 0x7fb5e78cc000
mapped at 0x7fb5e78cc000
mmapped address is 0x7fb5e38cc000
mapped at 0x7fb5e38cc000
mmapped address is 0x7fb5df8cc000
mapped at 0x7fb5df8cc000
send len = 263596
OMX: Failed to find peer index of board 00:00:00:00:00:00 (Peer Not Found in the Table)
mx_connect failed

Looks like CDS error. We are leaving the WATCHDOGS OFF for the night.

  6622   Tue May 8 09:47:53 2012 JamieUpdateCDSbiquad filter form

Quote:

I wanted to switch the implementation of IIR_FILTER from DIRECT FORM II to BIQUAD form in C1IOO and C1SUS models. I modified RCG file /opt/rtcds/rtscore/release/src/fe/controller.c by adding #define CORE_BIQUAD line:

#ifdef OVERSAMPLE
#define CORE_BIQUAD      
#if defined(CORE_BIQUAD)

 I am really not ok with anyone modifying controller.c.  If we're going to be messing around with that we need to change procedure significantly.  This is the code that runs all the models, and we don't currently have any way to track changes in the code.

Did you change it back?  If not, do so immediately and stop messing with it.  Please consult with us first before embarking on these kinds of severe changes to our code.  This is the kind of shit that other people have done that has bit us in the ass in the past.

Futhermore, there is already a way to enable biquad filters in the new version with out modifying the RCG source.  All you need to do is set biquad=1 in the cdsParameters block for you model.

DO NOT MESS WITH CONTROLLER.C!

  6623   Tue May 8 09:58:17 2012 DenUpdateCDSSUS -> FB

 [Alex, Den] 

It was in vain to restart mx_stream yesterday as C1SUS did not see FB

controls@c1sus ~ 0$ /opt/open-mx/bin/omx_info 

Open-MX version 1.3.901
 build: root@fb:/root/open-mx-1.3.901 Wed Feb 23 11:13:17 PST 2011
Found 1 boards (32 max) supporting 32 endpoints each:
 c1sus:0 (board #0 name eth1 addr 00:25:90:06:59:f3)
   managed by driver 'igb'
Peer table is ready, mapper is 00:60:dd:46:ea:ec
================================================
  0) 00:25:90:06:59:f3 c1sus:0
  1) 00:60:dd:46:ea:ec fb:0                           // this line was missing
  2) 00:14:4f:40:64:25 c1ioo:0
  3) 00:30:48:be:11:5d c1iscex:0
  4) 00:30:48:bf:69:4f c1lsc:0
  5) 00:30:48:d6:11:17 c1iscey:0
 
At the same time FB saw C1SUS:
 
controls@fb ~ 0$ /opt/mx/bin/mx_info
 
MX Version: 1.2.12
MX Build: root@fb:/root/mx-1.2.12 Mon Nov  1 13:34:38 PDT 2010
1 Myrinet board installed.
The MX driver is configured to support a maximum of:
8 endpoints per NIC, 1024 NICs on the network, 32 NICs per host
===================================================================
Instance #0:  299.8 MHz LANai, PCI-E x8, 2 MB SRAM, on NUMA node 0
Status: Running, P0: Link Up
Network: Ethernet 10G

MAC Address: 00:60:dd:46:ea:ec
Product code: 10G-PCIE-8AL-S
Part number: 09-03916
Serial number: 352143
Mapper: 00:60:dd:46:ea:ec, version = 0x00000000, configured
Mapped hosts: 6

                                                        ROUTE COUNT
INDEX    MAC ADDRESS     HOST NAME                        P0
-----    -----------     ---------                        ---
   0) 00:60:dd:46:ea:ec fb:0                              1,0
   1) 00:30:48:d6:11:17 c1iscey:0                         1,0
   2) 00:30:48:be:11:5d c1iscex:0                         1,0
   3) 00:30:48:bf:69:4f c1lsc:0                           1,0
   4) 00:25:90:06:59:f3 c1sus:0                           1,0
   5) 00:14:4f:40:64:25 c1ioo:0                           1,0
 
For that reason when I restarted mx_stream on c1sus, the script tried to connect to the standard 00:00:00:00:00:00 address, as the true address was not specified.
 
Alex restarted mx on FB. Note, DAQD process will not allow one to do that until it runs, at the same time, you can't just kill it, it will restart automatically. For that reason one should open /etc/inittab and replace respawn to stop in the line
 
daq:345:respawn:/opt/rtcds/caltech/c1/target/fb/start_daqd.inittab
 
then execute inittab using init q and restart mx on the FB
 
controls@fb ~ 0$ sudo /sbin/init q
controls@fb ~ 0$ sudo /etc/init.d/mx restart
 

After that C1SUS started to communicate with FB. But the reason why this happened and how to prevent from this in future Alex does not know.

Restarting DAQD process (or may be C1SUS) also solved the problem with guralp channels, now they are fine. Again, why this happened is unknown.

 

  6624   Tue May 8 10:43:42 2012 DenUpdateCDSbiquad filter form

Quote:

Quote:

I wanted to switch the implementation of IIR_FILTER from DIRECT FORM II to BIQUAD form in C1IOO and C1SUS models. I modified RCG file /opt/rtcds/rtscore/release/src/fe/controller.c by adding #define CORE_BIQUAD line:

#ifdef OVERSAMPLE
#define CORE_BIQUAD      
#if defined(CORE_BIQUAD)

 I am really not ok with anyone modifying controller.c.  If we're going to be messing around with that we need to change procedure significantly.  This is the code that runs all the models, and we don't currently have any way to track changes in the code.

Did you change it back?  If not, do so immediately and stop messing with it.  Please consult with us first before embarking on these kinds of severe changes to our code.  This is the kind of shit that other people have done that has bit us in the ass in the past.

Futhermore, there is already a way to enable biquad filters in the new version with out modifying the RCG source.  All you need to do is set biquad=1 in the cdsParameters block for you model.

DO NOT MESS WITH CONTROLLER.C!

 ok

  6625   Tue May 8 16:43:15 2012 JenneUpdateCDSDegenerate channels, potentially a big mess

Rana theorized that we're having problems with the MC error signal in the OAF model (separate elog by Den to follow) because we've named a channel "C1:IOO-MC_F", and such a channel already used to exist.  So, Rana and I went out to do some brief cable tracing.

MC Servo Board has 3 outputs that are interesting:  "DAQ OUT" which is a 4-pin LEMO, "SERVO OUT" which is a 2-pin LEMO, and "OUT1", which is a BNC->2pin LEMO right now.

DAQ OUT should have the actal MC_F signal, which goes through to the laser's PZT.  This is the signal that we want to be using for the OAF model.

SERVO OUT should be a copy of this actual MC_F signal going to the laser's PZT.  This is also acceptable for use with the OAF model.

OUT1 is a monitor of the slow(er) MC_L signal, which used to be fed back to the MC2 suspension.  We want to keep this naming convention, in case we ever decide to go back and feed back to the suspensions for freq. stabilization.

Right now, OUT1 is going to the first channel of ADC0 on c1ioo.  SERVOout is going to the 7th channel on ADC0.  DAQout is going to the ~12th channel of ADC1 on c1ioo.  OUT1 and SERVOout both go to the 2-pin LEMO whitening board, which goes to some new aLIGO-style ADC breakout boards with ribbon cables, which then goes to ADC0.  DAQout goes to the 4pin LEMO ADC breakout, (J7 connector) which then directly goes to ADC1 on c1ioo.

So, to sum up, OUT1 should be "adc0_0" in the simulink model, SERVOout should be "adc0_6" on the simulink model, and DAQout should be "adc1_12" (or something....I always get mixed up with the channel counting on 4pin ADC breakout / AA boards). 

In the current simulink setup, OUT1 (adc0_0) is given the channel name C1:IOO-MC_F, and is fed to the OAF model.  We need to change it to C1:IOO-MC_L to be consistent with the old regime.

In the current simulink setup, SERVOout (adc0_6) is given the channel name C1:IOO-MC_SERVO.  It should be called C1:IOO-MC_F, and should go to the OAF model.

In the current simulink setup,DAQout (~adc1_12) doesn't go anywhere.  It's completely not in the system.  Since the cable in the back of this AA / ADC breakout board box goes directly to the c1ioo I/O chassis, I don't think we have a degenerate MC_F naming situation.  We've incorrectly labeled MC_L as MC_F, but we don't currently have 2 signals both called MC_F.

Okay, that doesn't explain precisely why we see funny business with the OAF model's version of MCL, but I think it goes in the direction of ruling out a degenerate MC_F name.

Problem:  If you look at the screen cap, both simulink models are running on the same computer (c1ioo), so when they both refer to ADC0, they're really referring to the same physical card.  Both of these models have adc0_6 defined, but they're defined as completely different things.  Since we can trace / see the cable going from the MC Servo Board to the whitening card, I think the MC_SERVO definition is correct.  Which means that this Green_PH_ADC is not really what it claims to be.  I'm not sure what this channel is used for, but I think we should be very cautious and look into this before doing any more green locking.  It would be dumb to fail because we're using the wrong signals.

 

Attachment 1: DegenerateIOOadc0chan6.png
DegenerateIOOadc0chan6.png
  6626   Tue May 8 17:48:50 2012 JenneUpdateCDSOAF model not seeing MCL correctly

Den noticed this, and will write more later, I just wanted to sum up what Alex said / did while he was here a few minutes ago....

 

Errors are probably really happening.... c1oaf computer status 4-bit thing:  GRGG.  The Red bit is indicating receiving errors.  Probably the oaf model is doing a sample-and-hold thing, sampling every time (~1 or 2 times per sec) it gets a successful receive, and then holding that value until it gets another successful receive. 

Den is adding EPICS channels to record the ERR out of the PCIE dolphin memory CDS_PART, so that we can see what the error is, not just that one happened.

Alex restarted oaf model:  sudo rmmod c1oaf.ko, sudo insmod c1oaf.ko .  Clicked "diag reset" on oaf cds screen several times, nothing changed.  Restarted c1oaf again, same rmmod, insmod commands.

Den, Alex and I went into the IFO room, and looked at the LSC computer, SUS computer, SUS I/O chassis, LSC I/O chassis and the dolphin switch that is on the back of the rack, behind the SUS IO chassis.  All were blinking happily, none showed symptoms of errors.

Alex restarted the IOP process:  sudo rmmod c1x04, sudo insmod c1x04.  Chans on dataviewer still bad, so this didn't help, i.e. it wasn't just a synchronization problem.  oaf status: RRGG. lsc status: RGGG. ass status: RGGG.

sudo insmod c1lsc.ko, sudo insmod c1ass.ko, sudo insmod c1oaf.ko .  oaf status: GRGG. lsc status: GGGG. ass status: GGGG.  This means probably lsc needs to send something to oaf, so that works now that lsc is restarted, although oaf still not receiving happily.

Alex left to go talk to Rolf again, because he's still confused.

Comment, while writing elog later:  c1rfm status is RRRG, c1sus status is RRGG, c1oaf status is GRGG, both c1scy and c1scx are RGRG.  All others are GGGG.

  6627   Wed May 9 00:45:13 2012 JenneUpdateCDSNo signals for DTT from SUS

Upgrades suck.  Or at least making everything work again after the upgrade.

On the to-do list tonight:  look at OSEM sensor and OpLev spectra for PRM, when PRMI is locked and unlocked.  Goal is to see if the PRM is really moving wildly ("crazy" as Kiwamu always described it) when it's nicely aligned and PRMI is locked, or if it's an artifact of lever arm between PRM and the cameras (REFL and AS).

However, I can't get signals on DTT.  So far I've checked a bunch of signals for SUS-PRM, and they all either (a) are just digital 0 or (b) are ADC noise.  Lame.

Steve's elog 5630 shows what reasonable OpLev spectra should look like:  exactly what you'd expect.

Attached below is a small sampling of different SUS-PRM signals.  I'm going to check some other optics, other models on c1sus, etc, to see if I can narrow down where the problem is.  LSC signals are fine (I checked AS55Q, for example).

UPDATE:  SRM channels are same ADC noise.  MC1 channels are totally fine.  And Den had been looking at channels on the RFM model earlier today, which were fine.

ETMY channels - C1:SUS-ETMY_LLCOIL_IN1 and C1:SUS-ETMY_SUSPOS_IN1 both returned "unable to obtain measurement data".  OSEM sensor channels and OpLev _PERROR channel were digital zeros.

ETMX channels were fine

UPDATE UPDATE:  Genius me just checked the FE status screen again.  It was fine ~an hour ago when I sat down to start interferometer-izing for the night, but now the SUS model and both of the ETMY computer models are having problems connecting to the fb.  *sigh* 

Attachment 1: OpLevs_OSEMs_allSUSchannels_ADCnoiseOnly.pdf
OpLevs_OSEMs_allSUSchannels_ADCnoiseOnly.pdf
Attachment 2: Screenshot.png
Screenshot.png
  6628   Wed May 9 01:14:44 2012 JenneUpdateCDSNo signals for DTT from SUS

Quote:

UPDATE UPDATE:  Genius me just checked the FE status screen again.  It was fine ~an hour ago when I sat down to start interferometer-izing for the night, but now the SUS model and both of the ETMY computer models are having problems connecting to the fb.  *sigh* 

 Restarted SUS model - it's now happy. 

c1iscey is much less happy - neither the IOP nor the scy model are willing to talk to fb.  I might give up on them after another few minutes, and wait for some daytime support, since I wanted to do DRMI stuff tonight.

Yeah, giving up now on c1iscey (Jamie....ideas are welcome).  I can lock just fine, including the Yarm, I just can't save data or see data about ETMY specifically.  But I can see LSC data, so I can lock, and I can now take spectra of corner optics.

  6630   Wed May 9 08:21:42 2012 JamieUpdateCDSNo signals for DTT from SUS

Quote:

 c1iscey is much less happy - neither the IOP nor the scy model are willing to talk to fb.  I might give up on them after another few minutes, and wait for some daytime support, since I wanted to do DRMI stuff tonight.

Yeah, giving up now on c1iscey (Jamie....ideas are welcome).  I can lock just fine, including the Yarm, I just can't save data or see data about ETMY specifically.  But I can see LSC data, so I can lock, and I can now take spectra of corner optics.

 This is the mx_stream issue reported previously.  The symptom is that all models on a single front end loose contact with the frame builder, as opposed to *all* models on all front end loosing contact with the frame builder.  That indicates that the problem is a common fb communication issue on the single front end, and that's all handled with mx_stream.

ssh'ing into c1iscey and running "sudo /etc/init.d/mx_stream restart" fixed the problem.

  6632   Wed May 9 10:46:54 2012 DenUpdateCDSOAF model not seeing MCL correctly

Quote:

Den noticed this, and will write more later, I just wanted to sum up what Alex said / did while he was here a few minutes ago....

 From my point of view during rfm -> oaf transmission through Dolphin we loose a significant part of the signal. To check that I've created MEDM screen to monitor the transmission errors in the OAF model. It shows how many errors occurs per second. For MCL channel this number turned out to be 2046 +/- 1. This makes sense to me as the sampling rate is 2048 Hz => then we actually receive only 1-3 data points per second. We can see this in the dataviewer.

C1:OAF-MCL_IN follows C1:IOO-MC_F in the sense that the scale of 2 signals are the same in 2 states: MC locked and unlocked. It seems that we loose 2046 out of 2048 points per second.

oaf_rec.png

  6633   Wed May 9 11:31:50 2012 DenUpdateCDSRFM

I added PCIE memory cache flushing to c1rfm model by changing 0 to 1 in /opt/rtcds/rtscore/release/src/fe/commData2.c on line 159, recompiled and restarted c1rfm.

Jamie, do not be mad at me, Alex told me do that!

However, this did not help, C1RFM did not start. I decided to restart all models on C1SUS machine in hope that C1RFM uses some other models and can't connect to them but this suspended C1SUS machine. After reboot encounted the same C1SUS -> FB communication error and fixed it in the same was as in the previous case of C1SUS reboot. This happens already the second time (out of 2) after C1SUS machine reboot.

I changed /opt/rtcds/rtscore/release/src/fe/commData2.c back, recompiled and restarted c1rfm. Now everything is back. C1RFM -> C1OAF is still bad.

  6634   Wed May 9 14:32:31 2012 JenneUpdateCDSBurt restored

Den and Alex left things not-burt restored, and Den mentioned to me that it might need doing.

I burt restored all of our epics.snaps to the 1am today snapshot.  We lost a few hours of striptool trends on the projector, but now they're back (things like the BLRMS don't work if the filters aren't engaged on the PEM model, so it makes sense).

  6635   Wed May 9 15:02:50 2012 DenUpdateCDSRFM

Quote:

However, this did not help, C1RFM did not start. I decided to restart all models on C1SUS machine in hope that C1RFM uses some other models and can't connect to them but this suspended C1SUS machine.

 This happened because of the code bug -

// If PCIE comms show errors, may want to add this cache flushing
#if 1
if(ipcInfo[ii].netType == IPCIE)
          clflush_cache_range (&(ipcInfo[ii].pIpcData->dBlock[sendBlock][ipcIndex].data), 16); // & was missing - Alex fixed this
#endif
 

After this bug was fixed and the code was recompiled, C1:OAF_MCL_IN is OK, no errors occur during the transmission C1:OAF-MCL_ERR=0.

So the problem was in the PCIE card that could not send such amount of data and the last channel (MCL is the last) was corrupted. Now, when Alex added cache flushing, the problem is fixed.

We should spend some more attention to such problems. This time 2046 out of 2048 points were lost per second. But what if 10-20 points are lost, we would not notice that in the dataviewer, but this will cause problems.

  6639   Thu May 10 22:05:21 2012 DenUpdateCDSFB

Already for the second time today all computers loose connection to the framebuilder. When I ssh to framebuilder DAQD process was not running. I started it

controls@fb ~ 130$ sudo /sbin/init q

But I do not know what causes this problem. May be this is a memory issue. For FB

Mem:   7678472k total,  7598368k used,    80104k free

Practically all memory is used. If more is needed and swap is off, DAQD process may die.

  6640   Fri May 11 08:07:30 2012 JamieUpdateCDSFB

Quote:

Already for the second time today all computers loose connection to the framebuilder. When I ssh to framebuilder DAQD process was not running. I started it

controls@fb ~ 130$ sudo /sbin/init q

Just to be clear, "init q" does not start the framebuilder.  It just tells the init process to reparse the /etc/inittab.  And since init is supposed to be configured to restart daqd when it dies, it restarted it after the reloading of /etc/inittab.  You and Alex must have forgot to do that after you modified the inittab when you're were trying to fix daqd last week.

daqd is known to crash without reason.  It usually just goes unnoticed because init always restarts it automatically.  But we've known about this problem for a while.

Quote:

But I do not know what causes this problem. May be this is a memory issue. For FB

Mem:   7678472k total,  7598368k used,    80104k free

Practically all memory is used. If more is needed and swap is off, DAQD process may die.

This doesn't really mean anything, since the computer always ends up using all available memory.  It doesn't indicate a lack of memory.  If the machine is really running out of memory you would see lots of ugly messages in dmesg.

ELOG V3.1.3-