I was able to boot one of the 3 new Supermicro machines, which I christened c1bhd, in a diskless way (with the boot image hosted on fb, as is the case for all the other realtime FEs in the lab). This is just a first test, but it is reassuring that we can get this custom linux kernel to boot on the new hardware. Some errors about dolphin drivers are thrown at startup but this is to be expected since the server isn't connected to the dolphin network yet. We have the Dolphin adaptor card in hand, but since we have to get another PCIe card (supposedly from LLO according to the BHD spreadsheet), I defer installing this in the server chassis until we have all the necessary hardware on hand.

I also have to figure out the correct BIOS settings for this to really run effectively as a FE (we have to disable all the "un-necessary" system level services) - these machines have BIOS v3.2 as opposed to the older vintages for which there are instructions from K.T. et al.

There may yet be issues with drivers, but this is all the testing that can be done without getting an expansion chassis. After the vent and recovering the IFO, I may try experimenting with the c1ioo chassis, but I'd much prefer if we can do the testing offline on a subnet that doesn't mess with the regular IFO operation (until we need to test the IPC).

[JC, Paco]

I tried restarting all models this afternoon to see if the DC error 0x2000 on c1cal model cleared (currently no frame data is being written from c1cal). By running ./restartAllModels.sh on the scripts/cds/ folder, and then BURT restoring, but unfortunately this didn't work. I also tried restarting DAQD without success. Finally, I guess since the error has been related to channel inconsistencies in the model's .INI file (what where when?) I tried rtcds build c1cal and rtcds install c1cal and then another full restart. No success.

The 0x2000 error happens when the rtcds model can not acquire the requested number of channels at their data rates. Basically, there is a maximum total data acquisition that one model can do (which is unknown to me). To fix this, I removed 2048 Hz acquiring of C1:CAL-SENSMAT_<DOF>_<RFPD>_<I/Q>_DEMOD_SIG_IN1. This would not allow us to do software demodulation of calibration lines ourselves. but C1:CAL-SENSMAT_<DOF>_<RFPD>_<I/Q>_DEMOD_<I/Q>_IN1 are still acquired at 2048 Hz to do our own low pass filtering in software and C1:CAL-SENSMAT_<DOF>_<RFPD>_<I/Q>_DEMOD_<I/Q>_OUT are acquired at 256 Hz.

This removal worked, and restarting DAQD worked and now c1cal does not have any DC errors. Current total data acquisition C1:DAQ-FEC_50_TOTAL is 2521 which is less than our other heavy models like c1lsc, c1sus etc. So c1cal can probably acquire more in future, but care is required while adding new channels. This issue happened because we added BH44 to the calibration model.

I'm not sure why the c1cal model didn't come up the last time c1lsc was rebooted, but I did an "rtcds start c1cal" on the lsc machine, and it's up and running now.

[Anchal, Yuta]

We have modified c1cal model to support sensing matrix measurements for BHD PDs on Friday last week.
c1cal model now can inject dither to LO1, LO2, AS1, and AS4, and demodulate BH55_I, BH55_Q, BHDC_SUM and BHDC_DIFF signals.
Related models, c1lsc, c1hpc, and c1sus2 are also modifed accordingly.
MEDM screens are also edited accordingly.
Attachments highlight the modifications.

I experimented with removing somethings here and there to reduce the c1cal runtime. Eventually I deleted the LSC Sensing Matrix from it.

• Ever since the upgrade, the c1cal has gone from 60 to 68 usec run time, so its constantly over.
• Back when Jenne set it up back in Oct 2013, it was running at 39 usec.
• The purely CAL stuff had some wacko impossible filters in it: please don't try to invert the AA filters making a filter with multiple zeros in it Masayuki.
• I removed the weird / impossible / unstable filters.
• I'm guessing that the sensing matrix code had some hand-rolled C-code blocks which are just not very speedy, so we need to rethink how to do the lockin / oscillator stuff so that it doesn't overload the CPU. I bet its somewhere in the weird way the I/Q signals were untangled. My suggestion is to change this stuff to use the standard CDS lockin modules and just record the I/Q stuff. We don't need to try to make magnitude and phase in the front end.

After removing sensing matrix, the run time is now down to 6 usec.

I decided to rename the c1gcv model to be c1als.  This is in an ongoing effort to rename all the ALS stuff as ALS, and get rid of the various GC{V,X,Y} named stuff.

Most of what was in the c1gcv model was already in a subsystem with and ALS top names, but there were a couple of channels that were outside of that that had funky names, namely the "GCV_GREEN" channels.  This fixes that, and make things more consistent and simple.

Of course this required a bunch of other little changes:

• rename model in userapps svn
• target/fb/master had to be modified to point to the new chans/daq/C1ALS.ini channel file and gds/param/tpchn_c1als.par testpoint file
• rename RFM channels appropriately, and fix in receiver models (c1scx, c1scy, c1mcs)
• move custom medm screens in userapps svn (isc/c1/medm/c1als), and link to it at medm/c1als/master
• moved old medm/c1gcv directory into a subdirectory of medm/c1als
• update all medm screens that point to c1gcv stuff (mostly just ALS screens)

The above has been done.  Still todo:

• FIX SCRIPTS!  There are almost certainly scripts that point to GC{V,X,Y} channels.  Those will have to be fixed as we come across them.
• Fix the c1sc{x,y}/master/C1SC{X,Y}_GC{X,Y}_SLOW.adl screens.  I need to figure out a more consistent place for those screens.
• Fix the C1ALS_COMPACT screen
• ???

Yuta added channels so we can get the Q phase of all the beat PDs to the c1gcv model.  I showed him how to recompile/install/start.

During the install, it couldn't find: Unable to find the following file in CDS_MEDM_PATH: LOCKIN_FILTER.adl

On all the screens (ALS and SUS), lockin parts are white.  Someone changed something, then didn't go back to fix the screens.

Otherwise, things look to be working fine.

[Jamie, Yuta]

We recompiled c1gcv because the order of the channels were confusing. We found some change in the phase rotation module when we did this.

I did some cabling and checked each signals are actually going to the right channel. I labeled all the cables I know, which go into the AA chasis for ADC1 of c1ioo machine.

Below is the list of the channels. If you know anything about "unknown" channels, please let me know.

Current channel assignments for ADC1 of c1ioo machine:
  Red ones were added today. Green ones existed in the past, but channel assignment were changed.

Memorandum for me:
  Recompiling procedure;

ssh c1ioo

rtcds make c1gcv
rtcds install c1gcv
rtcds start c1gcv

I have edited c1scx.ini by hand in order to acquire some green locking related channels.

Somehow c1sus.ini, c1mcs.ini, c1scx.ini and c1scy.ini are not accessible via the daqconfig script.

As far as I remember it had been accessible via daqconfig a week ago when I edited c1scy.ini.

Anyway I had to edit it by hand. They need to be fixed at some point

[Anchal, Yuta]

To send BHD signals from c1hpc after unwhitening and taking sum/diff, c1hpc and c1lsc models are modified.
PDDC_DOF_MTRX medm screen was modified to reflect this change.
We don't need to unwhiten and take sum/diff again in c1lsc model anymore
 

c1hpc has option of dithering BS now (sending excitation to BS LSC port to c1sus over IPC). This is available for demodulating BHDC and BH55 signals. Also BS is a possible feedback point, however, we would stick to using LSC screen for any MICH locking.

c1sus underwent 2 changes. All suspension models were upgraded to the new suspension model (see 40m/16938 and 40m/17165). Now the channel data rates are set in simulink model and activateDQ script is not doing anything for any of the suspension models.

Many changes have been done to c1hpc to support dual demodulation at audio frequencies. We moved away with ASS style of lockin setup as the number of connections and screens required would become very large. Instead now, the demodulation is done for a selected oscillator, on a selected signal. Similarly, the demodulated signal can be further demodulated for another selected oscillator. Please familarize yourself with new screen and test the new model. The previous version of the model is kept as backup alogn with all it's medm screens, so nothing is lost. Shown as an example in the screenshot, AS1 and BS oscillators can be turned on, and BHDC_DIFF signal can be demodulated first with BS and next with AS oscillator to get the signal.

I moved the medm/c1ifo directory into the CDS userapps svn at cds/c1/medm/c1ifo, and then linked it back into the medm directory:

controls@rossa:~ 0$ ls -al /opt/rtcds/caltech/c1/medm/c1ifo
lrwxrwxrwx 1 controls controls 56 2012-07-31 11:53 /opt/rtcds/caltech/c1/medm/c1ifo -> /opt/rtcds/caltech/c1/userapps/release/cds/c1/medm/c1ifo
controls@rossa:~ 0$

I then committed whatever useful was in there.  We need to remember to commit when we make changes.

c1ioo computer can not connect to the framebuilder and everything is red in the status for this machine, C1:FEC-33_CPU_METER is not moving.

EDIT by KI:

 We rebooted the c1ioo machine, but none of the ftont end model came back. It looked like they failed the burt process for some reasons according to dmesg.

Then we restarted each front end model one by one, and every time after immediately we restarted it we hit the 'BURT' button in the GDS screen.

Everyone came back to the normal operation.

I was trying to set up a DAC channel to interface with the AOM driver on the PSL table.

• It would have been most convenient to use channels from c1ioo given proximity to the PSL table.
• Looking at the 1X2 rack, it looked like there were indeed some spare DAC channels available.
• So I thought I'd run a test by adding some TPs to the c1als model (because it seems to have the most head room in terms of CPU time used).
• I added the DAC_0 block from CDS_PARTS library to c1als model (after confirming that the same part existed in the IOP model, c1x03).
• Model recompiled fine (I ran rtcds make c1als and rtcds install c1als on c1ioo).
• However, I got a bunch of errors when I tried to restart the model with rtcds restart c1als. The model itself never came up.
• Looking at dmesg, I saw stuff like
  [4072817.132040] c1als: Failed to allocate DAC channel.
[4072817.132040] c1als: DAC local 0 global 16 channel 4 is already allocated.
[4072817.132040] c1als: Failed to allocate DAC channel.
[4072817.132040] c1als: DAC local 0 global 16 channel 5 is already allocated.
[4072817.132040] c1als: Failed to allocate DAC channel.
[4072817.132040] c1als: DAC local 0 global 16 channel 6 is already allocated.
[4072817.132040] c1als: Failed to allocate DAC channel.
[4072817.132040] c1als: DAC local 0 global 16 channel 7 is already allocated.
[4073325.317369] c1als: Setting stop_working_threads to 1
• Looking more closely at the log messages, it seemed like rtcds could not find any DAC cards on c1ioo.
• I went back to 1X2 and looked inside the expansion chassis. I could only find two ADC cards and 1 BIO card installed. The SCSI cable labelled ("DAC 0") running from the rear of the expansion chassis to the 1U SCSI->40pin IDE breakout chassis wasn't actually connected to anything inside the expansion chassis.
• I then undid my changes (i.e. deleted all parts I added in the simulink diagram), and recompiled c1als.
• This time the model came back up but I saw a "0x2000" error in the GDS overview MEDM screen.
• Since there are no DACs installed in the c1ioo expansion chassis, I thought perhaps the problem had to do with the fact that there was a "DAC_0" block in the c1x03 simulink diagram - so I deleted this block, recompiled c1x03, and for good measure, restarted all (three) models on c1ioo.
• Now, however, I get the same 0x2000 error on both the c1x03 and c1als GDS overview MEDM screens (see Attachment #1).
• An elog search revealed that perhaps this error is related to DAQ channels being specified without recording rates (e.g. 16384, 2048 etc). There were a few DAQ channels inside c1als which didn't have recording rates specified, so I added the rates, and restarted the models, but the errors persist.
• According to the RCG runtime diagnostics document, T1100625 (which admittedly is for RCG v 2.7 while we are running v3.4), this error has to do with a mismatch between the DAQ config files read by the RTS and the DAQD system, but I'm not sure how to debug this further.
• I also suspect there is something wrong with the mx processes:
  controls@c1ioo:~ 130$ sudo systemctl status mx
● open-mx.service - LSB: starts Open-MX driver
   Loaded: loaded (/etc/init.d/open-mx)
   Active: failed (Result: exit-code) since Tue 2017-10-03 00:27:32 UTC; 34min ago
  Process: 29572 ExecStop=/etc/init.d/open-mx stop (code=exited, status=1/FAILURE)
  Process: 32507 ExecStart=/etc/init.d/open-mx start (code=exited, status=1/FAILURE)
  Oct 03 00:27:32 c1ioo systemd[1]: Starting LSB: starts Open-MX driver...
Oct 03 00:27:32 c1ioo open-mx[32507]: Loading Open-MX driver (with  ifnames=eth1 )
Oct 03 00:27:32 c1ioo open-mx[32507]: insmod: ERROR: could not insert module /opt/3.2.88-csp/open-mx-1.5.4/modules/3.2.88-csp/open-mx.ko: File exists
Oct 03 00:27:32 c1ioo systemd[1]: open-mx.service: control process exited, code=exited status=1
Oct 03 00:27:32 c1ioo systemd[1]: Failed to start LSB: starts Open-MX driver.
Oct 03 00:27:32 c1ioo systemd[1]: Unit open-mx.service entered failed state.
• Not sure if this is related to the DC error though.

From page 21 of T1100625, DAQ status "0x2000" means that the channel list is out of sync between the front end and the daqd.  This usually happens when you add channels to the model and don't restart the daqd processes, which sounds like it might be applicable here.

It looks like open-mx is loaded fine (via "rtcds lsmod"), even though the systemd unit is complaining.  I think this is because the open-mx service is old style and is not intended for module loading/unloading with the new style systemd stuff.

[Suresh / Kiwamu]

HELP US Jamieeeeeeee !! We are unable to compile c1ioo.

It looks like something wrong with Makefile.

We ran make c1ioo -- this was successful every time. However make install-c1ioo doesn't run.

The below is the error messages we got.

        make install-target-c1ioo
        make[1]: Entering directory `/opt/rtcds/caltech/c1/core/branches/branch-2.1'
        Please make c1ioo first

Then we looked at Makefile and tried to find what was wrong. Then found the sentence (in 36th line from the top) saying

        if test $(site)no = no; then echo Please make $$system first; exit 1; fi;\

We thought the lack of the site-name specification caused the error.

So then we tried the compile it again with the site name specified by typing

     export site=c1

in the terminal window.

It went ahead a little bit further, but it still doesn't run all through the Make commands.

> It looks like something wrong with Makefile.

Sorry, this was my bad.  I was making a patch to the makefile to submit back upstream and I forgot to revert my changes.  I've reverted them now, so everything should be back to normal.

I had noticed for a while that the c1ioo frontend model had much higher variability than any of the other other 16k models, and would run longer than 60us multiple times an hour. This struck me as odd, since all it does is control the WFS loops. (You can see this on the Nov17 Summary page. (But somehow, the CDS tab seems broken since then, and I'm not sure why...))

This has happily now been solved! While poking around the model, I noticed that the MC2 transmission QPD data streams being sent over from c1mcs were using RFM blocks. This seemed weird to me, since I wasn't even aware that c1ioo had an RFM card. Since the c1sus and c1ioo frontends are both on the Dolphin network, I changed them to Dolphin blocks and voila! The cylce time now holds steady at 21usec. 

Update: I think I figured out the problem with the CDS summary pages. Looking at the .err files in /home/40m/public_html/summary/logs on the 40m LDAS account showed that C1:FEC-33_CPU_METER wasn't found in the frame files. Indeed, this channel was commented out in c1/chans/daq/C0EDCU.ini. I enabled it and restarted daqd. Hopefully the CDS tab will come back soon...

I have been editing and reloading the c1ioo model last two days. I have restarted the frame builder several times. After one of the restarts on Sunday evening the fb started having problems which initially showed up as dtt reporting synchronization error. This morning Kiwamu and I tried to restart the fb again and it stopped working all together. We called Joe and he fixed the fb problem by fixing the time stamps (Joe will add details to describe the fix when he sees this elog).

The following changes were made to c1ioo model:

- The angular dither lockins were added for each optics to do the beam spot centering on MC mirrors. The MCL signal is demodulated digitally at 3 pitch and 3 yaw frequencies. (The MCL signal was reconnected to the first input of the ADC interface board).

- The outputs of the lockins go through the sensing matrix, DOF filters, and control matrix to the MC1,2,3 SUS-MC1(2,3)_ASCPIT(YAW) filter inputs where they sum with dither signals (CLOCK output of the oscillators).

- The MCL_TEST_FILT was removed

The arm cavity dither alignment (c1ass) status:

- The demodulated signals were minimized by moving the ETMX/ITMX optic biases and simultaneously keeping the arm buildup (TRX) high by using the BS and PZT2. The minimization of the TRX demodulated signals has not been successful for some reason.

- The next step is to close the servo loops REFL11I demodulated signals -> TMs and TRX demodulated signals -> combination of BS and PZTs.

The MC dither alignment (c1ioo) status:

- The demodulated signals were obtained and sensing matrix (MCs -> lockin outputs) was measured for pitch dof.

- The inversion of the matrix is in progress.

- The additional c1ass and c1ioo medm screens and up and down scripts are being made.

In order to have the c1mcs model run, we're running with only 3 RFM channels between c1ioo and c1mcs at the moment.  This leaves the model at around 45 microseconds, and at least lets us damp.

Alex and I still need to track down why the RFM read calls are taking so much time to execute.

 I put label  at the dolphin fiber end at 1X2 today.   After this I had to reset it, but it failed.

 If by "fail" you're talking about the c1oaf model being off-line, I did that yesterday (see log 9910).  That probably has nothing to do with whatever you did today, Steve.

Steve and I nicely routed the dolphin fiber from c1ioo in the 1X2 rack to the dolphin switch in the 1X4 rack.  I shutdown c1ioo before removing the fiber, but still all the dolphin connected models crashed.  After the fiber was run, I brought back c1ioo and restarted all wedged models.  Everything is green again:

This C1IOO business seems to be wiping out the MC2_TRANS QPD servo settings each day.   What kind of BURT is being done to recover our settings after each of these activities?

(also we had to do mxstream restart on c1sus twice so far tonight -- not unusual, just keeping track)

I don't see how the work I did would affect this stuff, but I'll look into it.  I didn't touch the MC2 trans QPD signals.  Also nothing I did has anything to do with BURT.  I didn't change any channels, I only swapped out the IPCs.

Rolf came to the 40m today and managed to figure out what the problem is. Reading just dmesg was not enough to solve the problem. Useful log was in

>> cat /opt/rtcds/caltech/c1/target/c1x03/c1x03epics/iocC1.log

Starting iocInit
The CA server's beacon address list was empty after initialization?
iocRun: All initialization complete
sh: iniChk.pl: command not found
Failed to load DAQ configuration file

iniChk.pl checks the .ini file of the model.

>> cat /opt/rtcds/rtscore/release/src/drv/param.c

int loadDaqConfigFile(DAQ_INFO_BLOCK *info, char *site, char *ifo, char *sys)
{

  strcpy(perlCommand, "iniChk.pl ");
  .........
  strcat(perlCommand, fname); // fname - name of the .ini file
  ..........
}

So the problem was not in the C1X03.ini. The code could not find the perl script though it was in the /opt/rtcds/caltech/c1/scripts directory. Some environment variables are not set. Rolf added /opt/rtcds/caltech/c1/scripts/ to $PATH variable and c1ioo models (x03, ioo, gcv) started successfully. He is not sure whether this is a right way or not, because other machines also do not have "scripts" directory in their PATH variable.

>> cat /opt/rtcds/caltech/c1/target/c1x03/c1x03epics/iocC1.log

Starting iocInit
The CA server's beacon address list was empty after initialization?
iocRun: All initialization complete

Total count of 'acquire=0' is 2
Total count of 'acquire=1' is 0
Total count of 'acquire=0' and 'acquire=1' is 2

Counted 0 entries of datarate=256     for a total of 0
Counted 0 entries of datarate=512     for a total of 0
Counted 0 entries of datarate=1024     for a total of 0
Counted 0 entries of datarate=2048     for a total of 0
Counted 0 entries of datarate=4096     for a total of 0
Counted 0 entries of datarate=8192     for a total of 0
Counted 0 entries of datarate=16384     for a total of 0
Counted 0 entries of datarate=32768     for a total of 0
Counted 2 entries of datarate=65536     for a total of 131072

Total data rate is 524288 bytes - OK

Total error count is 0

Rolf mentioned about automatic set up of variables - kiis of smth like that - probably that script is not working correctly. Rolf will add this problem to his list.

I tried to restart c1ioo becuase I can't live without him.

I couldn't ssh or ping c1ioo, so I did hardware reboot.
c1ioo came back, but now ADC/DAC stats are all red.

c1ioo was OK until 3am when I left the control room last night. I don't know what happened, but StripTool from zita tells me that MC lock went off at around 4pm.

 c1ioo was still all red on the CDS status screen, so I tried a couple of things.

mxstreamrestart (which aliases on the front ends to sudo /etc/init.d/mx_stream restart) didn't help

sudo shutdown -r now didn't change anything either....c1ioo came back with red everywhere and 0x2bad on the IOP

eventually doing as Jamie did for c1sus in elog 6742, rtcds stop all, then rtcds start all fixed everything.  Interestingly, when I tried rtcds start iop, I got the error
Cannot start/stop model 'iop' on host c1ioo, so I just tried rtcds start all, and that worked fine....started with c1x03, then c1ioo, then c1gcv.

There seems to be a problem with how models are coming up on boot since the upgrade.  I think the IOP isn't coming up correctly for some reason, which is then preventing the rest of the models from starting since they depend on the IOP.

The simple way to fix this is to run the following:

ssh c1ioo
rtcds restart all

The "restart" command does the smart thing, by stopping all the models in the correct order (IOP last) and then restarting them also in the correct order (IOP first).

I've updated c1ioo model with adding WFS sensor to optic angle matrix and output filter module option. The output filter modules are named like EST_MC1_PIT to signify that that these are "estimated" angles of the optic. We can change this naming convention if we don't like it. I've also started DQ on the outputs of these filter moduels at 512 Hz sampling rate.

No medm screens have been made for these changes yet. One can still access them through:

For SENS_TO_OPT_P Matrix

For SENS_TO_OPT_Y Matrix

For filter modules:

Problem:

We couldn't set testpoints on the c1ioo machine.

Cause:

Awgtpman was getting into some strange race condition.  Alex added an additional sleep statement and shifted boot order slightly in the rc.local file.  This apparently is only a problem on c1ioo, which is a Sun X4600.  It was using up 100% of a single CPU for the awgtpman process.

Status:

We now have c1ioo test points working.

Future:

Need to examine the startc1ioo script and see if needs a similar modification, as that was tried at several points but yielded a similar state of non-functioning test points. For the moment, reboot of c1ioo is probably the best choice after making modifications to the c1ioo or c1x03 models.

Current CDS status:

The c1ioo host is now fully on the dolphin network!

After the mx stream issue from two weeks ago was resolved and determined to not be due to the introduction of dolphin on c1ioo, I went ahead and re-installed the dolphin host adapter card on c1ioo.  The Dolphin network configurations changes I made during the first attempt (see previous log in thread) were still in place.  Once I rebooted the c1ioo machine, everything came up fine:

We then tested the interface by making a cdsIPCx-PCIE connection between the c1ioo/c1als model and the c1lsc/c1lsc model for the ALS-X beat note fine phase signal.  We then locked both ALS X and Y, and compared the signals against the existing ALS-Y beat note phase connection that passes through c1sus/c1rfm via an RFM IPC:

The signal is perfectly coherent and we've gained ~25 degrees of phase at 1kHz.  EricQ calculates that the delay for this signal has changed from:

122 us -> 61 us 

I then went ahead and made the needed modifications for ALS-Y as well, and removed ALS->LSC stuff in the c1rfm model.

Next up: move the RFM card from the c1sus machine to the c1lsc machine, and eliminate c1sus/c1rfm model entirely.

For reference, here are the new IPC entries that were made for the ALS X/Y phase between c1als and c1lsc:

controls@fb ~ 0$ egrep -A5 'C1:ALS-(X|Y)_PHASE' /opt/rtcds/caltech/c1/chans/ipc/C1.ipc
[C1:ALS-Y_PHASE]
ipcType=PCIE
ipcRate=16384
ipcHost=c1ioo
ipcNum=114
desc=Automatically generated by feCodeGen.pl on 2014_Apr_17_14:27:41
--
[C1:ALS-X_PHASE]
ipcType=PCIE
ipcRate=16384
ipcHost=c1ioo
ipcNum=115
desc=Automatically generated by feCodeGen.pl on 2014_Apr_17_14:28:53
controls@fb ~ 0$ 

After all this IPC cleanup is done we should go through and clean out all the defunct entries from the C1.ipc file.

Problem:

c1ioo was not able to talk to c1sus because of timing issues.  This prevented the mode cleaner length signal (MC_L) from getting to c1sus.

Solution:

The replacement c1ioo chassis from Downs with a more recent revision of the IO backplane works. 

The c1ioo is now talking to c1sus and transmitting a signal.

We connected the cable hanging off the DAQ interface board labeled MC OUT1 to the MC Servo board's output labeled OUT1.

During debugging I modified the c1x02, c1x03, c1mcs and c1ioo codes to print debugging messages.  This was done by modifying the /opt/rtcds/caltech/c1/advLigoRTS/src/fe/commData2.c file.  I have since reverted those changes.

Future:

We still need to check that everything is connected properly and that the correct signal is being sent to the MC2 suspension.

I've been trying to figure out why the c1ioo machine crashes when I try to run the c1ioo front end.

I tried removing some RFM components from the c1ioo model, and then the c1gpt model (Kiwamu's green locking model) as an easier test case.  Both cause the machine to lock up once they start running.  Lastly, I tried running the c1x02 and c1sus models on the c1ioo machine instead of the c1sus machine, after first turning off all models on c1sus.  This led to the same lockup. 

Since those models run fine on the c1sus machine, I could only conclude that a recent change in the fe code generator or the Gentoo kernel and the Sun X4600 computer don't work together at the moment.

After talking to Alex, he got the idea to check if the monitor() and mwait() were supported on the c1ioo machine.  These function calls were added relatively recently, and are   used to poll a memory location to see if something has been written there, and then do something when it is.  Apparently the Sun X4600 computers do not support this call.  Alex has modified the code to not use these functions calls, at least for now.  He'd like to add a check to the code so it does use those calls on machines which have them supported.

After this change, the c1ioo and c1gpt front end codes do in fact run.

Problem:

We can't access any of the c1ioo computer testpoints.  Dataviewer and DTT both fail to read any data from them.

According to diag -l (when run on Rosalba in /opt/apps), the testpoints are not being set.  Also at some point during the day when debugging, we also somehow messed up all the front end connections to the framebuilder.

Errors reported by dataviewer:

Server error 13: no data found
datasrv: DataWriteRealtime failed: daq_send: Illegal seek
Server error 12: no such net-writer
Server error 13: no data found
datasrv: DataWriteRealtime failed: daq_send: Illegal seek
Server error 12: no such net-writer
read(); errno=0
Server error 6532: invalid channel name
Server error 16080: unknown error
datasrv: DataWriteRealtime failed: daq_send: Illegal seek

Error reported by daqd.log:

[Thu Nov  4 13:29:35 2010] About to request `C1:IOO-MC1_PIT_IN1' 10022
on node 34
[Thu Nov  4 13:29:35 2010] Requesting 1 testpoints; tp[0]=10022; tp[1]=32531

[Thu Nov  4 13:29:35 2010] About to request `C1:SUS-MC2_SUSPOS_IN1'
10094 on node 36
[Thu Nov  4 13:29:35 2010] Requesting 1 testpoints; tp[0]=10094; tp[1]=32531

[Thu Nov  4 13:29:38 2010] ETIMEDOUT: test point `C1:IOO-MC1_PIT_IN1'
(tp_num=10022) was not set by the test point manager; request failed
[Thu Nov  4 13:29:38 2010] About to clear `C1:IOO-MC1_PIT_IN1' 10022 on node 34

Attempted Fixes:

Remove all daq related files: /opt/rtcds/caltech/c1/target/gds/param/tpchn_c1ioo.par and /opt/rtcds/caltech/c1/chans/daq/C1IOO.ini.  Rebuilt the front end code.

Double checked ethernet connection between c1ioo and the daq router and fb. 

Confirmed open mx was running on c1ioo. Confirmed awgtpman was running (although at one point I did find duplicate awgtpman running for c1x03, the IOP associated with c1ioo).

Rebooted the c1ioo machine. Confirmed all necessary codes came back.

Restarted the daqd process several times on the fb machine.

Current Status:

Framebuilder is running, and c1sus testpoints are available.  c1ioo test points are not.  Waiting to hear back from Alex on possible ideas.

Now the c1ioo in on the Dolphin network, I changed the c1ioo MC{1,2,3}_{PIT,YAW} and MC{L,F} outputs to go out over the Dolphin network rather than the old RFM network.

Two models, c1mcs and c1oaf, are ultimately the consumers of these outputs.  Now they are picking up the new PCIE IPC channels directly, rather than from any sort of RFM/PCIE proxy hops.  This should improve the phase for these channels a bit, as well as reduce complexity and clutter.  More stuff was removed from c1rfm as well, moving us to the goal of getting rid of that model entirely.

c1ioo, c1mcs, and c1rfm were all rebuild/installed/restarted, and all came back fine.  The mode cleaner relocked once we reenabled the autolocker.

c1oaf, on the other hand, is not building.  It's not building even before the changes I attempted, though.  I tried reverting c1oaf back to what is in the SVN (which also corresponds to what is currently running) and it doesn't compile either:

controls@c1lsc ~ 2$ rtcds build c1oaf
buildd: /opt/rtcds/caltech/c1/rtbuild
### building c1oaf...
Cleaning c1oaf...
Done
Parsing the model c1oaf...
YARM_BLRMS_SEIS_CLASS TP
YARM_BLRMS_SEIS_CLASS_EQ TP
YARM_BLRMS_SEIS_CLASS_QUIET TP
YARM_BLRMS_SEIS_CLASS_TRUCK TP
YARM_BLRMS_S_CLASS EpicsOut
YARM_BLRMS_S_CLASS_EQ EpicsOut
YARM_BLRMS_S_CLASS_QUIET EpicsOut
YARM_BLRMS_S_CLASS_TRUCK EpicsOut
YARM_BLRMS_classify_seismic FunctionCall
Please check the model for missing links around these parts.
make[1]: *** [c1oaf] Error 1
make: *** [c1oaf] Error 1
controls@c1lsc ~ 2$ 

I've been trying to debug it but have had no success.  For the time being I'm shutting off the c1oaf model, since it's now looking for bogus signals on RFM, until we can figure out what's wrong with it. 

This afternoon, while I was trying to add the StochMon channels to the frames, I rebooted the c1ioovme and c1iool0.

I had to do it twice because of a mispelling in the C1IOO.INI file that the first time prevented the computer to restart properly.

Eventually I restored the old .ini file, as it was before the changes.

After rebooting I also burtrestored.

During the process the mode cleaner got unlocked. Later on the autoclokcer couldn't engage. I had to run the MC_down and MC_up scripts.

P_carrier   J_0(Gamma)^2 
--------- = ------------
P_SB        J_1(Gamma)^2

We rebooted c1iovme because the lines stopped responding to inputs on C1:I00-MC_DRUM1. This fixed the problem.

 Alastair and I restarted the c1iovme around the time of my last elog entry (~3:20).