We took lo phase noise spectra actuating on the for different optics-- LO1, LO2, AS1, and AS4. The servo was not changed during this time with a gain of 0.2, and we also took a noise spectrum without any light on the DCPDs. The plot is shown in Attachment #1, calibrated in rad/rtHz, and shown along with the rms values for the different suspension actuation points. The best one appears to be AS1 from this measurement, and all the optics seem to show the same 270 Hz (actually 268 Hz) resonant peak.
Koji suspected the observed noise peak belongs to some servo oscillation, perhaps of mechanical origin so we first monitored the amplitude in an exponentially averaging spectrum. The noise didn't really seem to change too much, so we decided to try adding a bandstop filter around 268 Hz. After the filter was added in FM6, we turned it on and monitored the peak height as it began to fall slowly. We measured the half-decay time to be 264 seconds, which implies an oscillation with Q = 4.53 * f0 * tau ~ 3.2e5. This may or may not be mechanical, further investigation might be needed, but if it is mechanical it might explain why the peak persisted in Attachment #1 even when we change the actuation point; anyways we saw the peak drop ~ 20 dB after more than half an hour... After a while, we noticed the 536 Hz peak, its second harmonic, was persisting, even the third harmonic was visible.
So this may be LO1 violin mode & friends -
We should try and repeat this measurement after the oscillation has stopped, maybe looking at the spectra before we close the LO_PHASE control loop, then closing it carefully with our violin output filter on, and move on to other optics to see if they also show this noise.
svn import tds https://40m.ligo.caltech.edu/svn/40m/tds --username rana
svn checkout https://40m.ligo.caltech.edu/svn/40m/tds --username rana
See Adhikari eLOG entry: http://nodus.ligo.caltech.edu:8080/AdhikariLab/194
Peter and Koji,
We are constructing a setup for the new 40m CDS using Realtime Code Generator (RCG).
We are trying to put simulated suspensions and test suspension controllers on a different processors of megatron
in order to create a virtual control feedback loop. Those CDS processes are communicating
each other via a shared memory, not via a reflective memory for now.
After some struggles with tremendous helps of Alex, we succeeded to have the communication between the two processes.
Also we succeeded to make the ADC/DAC cards recognized by megatoron, using the PCI express extension card replaced by Jay.
(This card runs multi PCI-X cards on the I/O chasis.)
- Establish a firewall between the 40m network and megatron (Remember this)
- Make DTT and other tools available at megatron
- Try virtual feedback control loops and characterize the performance
- Enable reflective memory functionalities on megatron
- Construct a hybrid system by the old/new CDSs
- Controllability tests using an interferometer
Note on MATLAB/SIMULINK
o Each cdsIPC should have a correct shared memory address spaced by 8 bytes. (i.e. 0x1000, 0x1008, 0x1010, ...)
Note on MEDM
o At the initial state, garbage (e.g. NaN) can be running all around the feedback loops. They are invisible as MEDM shows them as "0.0000".
To escape from this state, we needed to disconnect all the feedback, say, by turning off the filters.
Note on I/O chasis
o We needed to pull all of the power plugs from megatron and the I/O chasis once so that we can activate
the PCI-e - PCI-X extension card. When it is succeeded, all (~30) LEDs turn to green.
I took the output of the OMC DAC and plugged it directly into an OMC ADC channel to see if I could isolate the OMC DAC weirdness I'd been seeing. It looks like it may have something to do with DTT specifically.
Attachment 1 is a DTT transfer function of a BNC cable and some connectors (plus of course the AI and AA filters in the OMC system). It looks like this on both linux and solaris.
Attachment 2 is a transfer function using sweepTDS (in mDV), which uses TDS tools as the driver for interfacing with testpoints and DAQ channels.
Attachment 3 is a triggered time series, taken with DTT, of the same channels as used in the transfer functions, during a transfer function. I think this shows that the problem lies not with awg or tpman, but with how DTT is computing transfer functions.
I've tried soft reboots of the c1omc, which didn't work. Since the TDS version appears to work, I suspect the problem may actually be with DTT.
For the CDS upgrade preparation I put and moved those stuff at the rack 1Y9:
Placed 1Y9-12 ADC to DB44/37 Adapter LIGO D080397
Placed 1Y9-14 DAC to IDC Adapter LIGO D080303
Moved the ethernet switch from 1Y9-16 to 1Y9-24
Wiki has also been updated.
Joe, Peter, Jay, Koji, Rana
We put the new CDS stuff at Y end 1Y9 rack.
I've added the side coil to the model controller and plant, and the oplev quad to the model controller and plant. After the megatron wipe, the code now lives in /home/controls/cds/advLigo/src/epics/simLink. The files are mdc.mdl (controller) and mdp.mdl (plant). These RCG modules go at 16K with no decimation (no_oversampling=1 in the cdsParameters block) so hopefully will work with the old (16K) timing.
I've loaded many of the filters, there are some eft to do. These filters are simply copied from the current frontend.
Next I will port to the SUS module (which talks to the IO chassis). This means channel names will match with the current system, which will be important when we plug in the RFM.
The .mdl code for the mdc and mdp development modules is finished. These modules need more filters, and testing. Probably the most interesting piece left to do is putting in the gains and filters for the oplev model in mdp. It might be OK to simply ignore oplevs and first test damping of the real optic without them. However, it shouldn't be hard to get decent numbers for oplevs, add them to the mdp (plant) module, and make sure the mdc/mdp pair is stable. In mdp, the oplev path starts with the SUSPIT and SUSYAW signals. Kakeru recently completed calibration of the oplevs from oplev cts to radians: 1403 . From this work we should find the conversion factors from PIT and YAW to oplev counts, without making any new measurements. (The measurements wouldn't be hard either, if we can't simply pull numbers from a document.) These factors can be added to mdp as appropriate gains.
I've also copied mdc to a new module, which I've named "sas" to address fears of channel name collisions in the short term, and replaced the cpu-to-cpu connections with ADC and DAC connections. sas can be the guy for the phase I ETMY test. When we're happy with mdc/mdp, we hopefully can take the mdc filter file from chans, replace all the "MDC" strings with "SAS", and use it.
We started the test of the new CDS system at ETMY.
The plan is as follows:
We do the ETMY test from 9:30 to 15:00 at ETMY from Nov 12~17. This disables the ETMY during this period.
From 15:00 of the each day, we restore the ETMY configuration and confirm the ETMY work properly.
Today we connected megatron to the existing AA/AI modules via designated I/F boxes. The status of the test was already reported by the other entry.
During the test, c1iscey was kept running. We disabled the ETMY actuation by WatchDog. We did not touch the RFM network.
After the test we disconnected our cables and restored the connection to ICS110B and the AI/AA boards.
The WatchDog switches were released.
The lock of the ETMY was confirmed. The full interferometer was aligned one by one. Left in the full configuration with LA=off.
Tobin & Keith pointed out in the LLO ilog that there was a code bug in the autoburt.pl script for autoburts.
I edited the autoburt.pl script so that it will work from now until 2099 (by which time we may no longer be using this version of perl):
nodus:autoburt>diff autoburt.pl~ autoburt.pl
< $thisyear = "200".$timestamp;
> $thisyear = "20".$timestamp;
The autoburt has not been working ever since 11PM on New Year's eve.
I ran it by hand and it seems to run fine. I noticed along the way that it was running on op340m (our old Sun Blade 150 machine). The autoburt.pl was pointing at /cvs/cds/bin/perl
which is Perl v5.0. I changed it to use '/usr/bin/env' and now points at '/usr/bin/perl' which is perl 5.8. It runs fine with the new perl:
op340m:scripts>time perl /cvs/cds/scripts/autoburt.pl >> /cvs/cds/caltech/logs/autoburtlog.log
5.37u 6.29s 2:13.41 8.7%
Also ran correctly, via cron, at 9AM.
This is mostly a reminder to myself about what I discussed with Jay and Alex this morning.
The big black IO chassis are "almost" done. Except for the missing parts. We have 2 Dolphin, 1 Large and 1 Small I/O Chassis due to us. One Dolphin is effectively done and is sitting in the test stand. However, 2 are missing timing boards, and 3 are missing the boards necessary for the connection to the computer. The parts were ordered a long time ago, but its possible they were "sucked to one of the sites" by Rolf (remember this is according to Jay). They need to either track them down in Downs (possibly they're floating around and were just confused by the recent move), get them sent back from the sites, or order new ones (I was told by one person that the place they order from them notoriously takes a long time, sometimes up to 6 weeks. I don't know if this is exaggeration or not...). Other than the missing parts, they still need to wire up the fans and install new momentary power switches (apparently the Dolphin boards want momentary on/off buttons). Otherwise, they're done.
We are due another CPU, just need to figure out which one it was in the test stand.
6 more BIO boards are done. When I went over the plans with Jay, we realized we needed 7 more, not 6, so they're putting another one together. Some ADC/DAC interface boards are done. I promised to do another count here, to determine how many we have, how many we need, and then report that back to Jay before I steal the ones which are complete. Unfortunately, he did not have a new drawing for the ASC/vertex wiring, so we don't have a solid count of stuff needed for them. I'll be taking a look at the old drawings and also looking at what we physically have.
I did get Jay to place the new LSC wiring diagram into the DCC (which apparently the old one never was put in or we simply couldn't find it). Its located at: https://dcc.ligo.org/cgi-bin/private/DocDB/ShowDocument?docid=10985
I talked briefly with Alex, reminded him of feature requests and added a new one:
1) Single part representing a matrix of filter banks
2) Automatic generation of Simulated shared memory locations and an overall on/off switch for ADC/DACs
3) Individual excitation and test point pieces (as opposed to having to use a full filter bank). He says these already exist, so when I do the CVS checkout, I'll see if they work.
I also asked where the adl default files lived, and he pointed me at ~/cds/advLigo/src/epics/util/
In that directory are FILTER.adl, GDS_TP.adl, MONITOR.adl. Those are the templates. We also discovered the timing signal at some point was changed from something like SYS-DCU_ID to FEC-DCU_ID, so I basically just need to modify the .adl files to fix the time stamp channel as well. I basically need to do a CVS checkout, put the fixes in, then commit back to the CVS. Hopefully I can do that sometime today.
I also brought over 9 Contec DO-32L-PE boards, which are PCIe isolated digital output boards which do into the IO chassis. These have been placed above the 2 new computers, behind the 1Y6 rack.
Alberto and myself went to downs and acquired the 3rd 4x processor (Dual core, so 8x cores total) computer. We also retrieved 6 BIO interface boards (blue front thin boxes), 4 DAC interface boards, and 1 ADC interface boards. The tops have not been put on yet, but we have the tops and a set of screws for them. For the moment, these things have been placed behind the 1Y6 rack and under the table behind the 1Y5 rack
The 6 BIO boards have LIGO travelers associated with them: SN LIGO-S1000217 through SN LIGO-S1000222.
I've added a new page in the wiki which describes the current naming scheme for the .mdl model files used for the real time code generator. Note, that these model names do not necessarily have to be the names of the channels contained within. Its still possible to make all suspension related channels start with C1:SUS- for example. I'm also allocating 1024 8 byte channels for shared memory address space for each controller and each simulated plant.
The wiki page is here
Name suggestions, other front end models that are needed long term (HEPI is listed for example, even though we don't have it here, since in the long run we'd like to port the simulated plant work to the sites) are all welcome.
Awhile back we had requested a feature for the RCG code where a single file would define a memory location's name as well as its explicit hex address. Alex told me it had been implemented in the latest code in SVN. After being unable to find said file, I went back and talked to him and Rolf. Rolf said it existed, but had not been checked into the SVN yet.
I now have a copy of that file, called G1.ipc. It is supposed to live in /cvs/cds/caltech/chans/ipc/ , so I created the ipc directory there. The G1.ipc file is actually for a geo install, so we'll eventually make a C1.ipc file.
The first couple lines look like:
There are also section using ipcType IPC:
Effectively the ipcNum tells it which memory location to use, starting with 0x2000 (at least thats how I'm interpreting it. Every entry of a given ipcType has a different ipcNum which seems to be correlated to its description (at least early on - later in the file many desc= lines repeat, which I think means people were copy/pasting and got tired of editing the file. Once I get a C1.ipc file going, it should make our .mdl files much more understandable, at least for communicating between models. It also looks like it somehow interacts with the ADCs/DACs with ipcType PCI, although I'm hoping to get a full intro how to use the file tomorrow from Rolf and Alex.
I've added a diagram in the wiki under IFO Upgrade 2009-2010->New CDS->Diagram section Joe_CDS_Plan.pdf (the .svg file I used to create it is also there). This was mostly an exercise in me learning inkscape as well as putting out a diagram with which lists control and model names and where they're running.
A direct link is: CDS_Plan.pdf
Talked with Jay briefly today. Apparently there are 3 IO chassis currently on the test stand at Downs and undergoing testing (or at least they were when Alex and Rolf were around). They are being tested to determine which slots refer to which ADC, among other things. Apparently the numbering scheme isn't as simple as 0 on the left, and going 1,2,3,4, etc. As Rolf and Alex are away this week, it is unlikely we'll get them before their return date.
Two other chassis (which apparently is one more than the last time I talked with Jay), are still missing cards for communicating between the computer and the IO chassis, although Gary thinks I may have taken them with me in a box. I've done a look of all the CDS stuff I know of here at the 40m and have not seen the cards. I'll be checking in with him tomorrow to figure out when (and if) I have the the cards needed.
I've updated the LSC and IFO models that Rana created with new shared memory locations. I've used the C1:IFO- for the ifo.mdl file outputs, which in turn are read by the lsc.mdl file. The LSC outputs being lsc control signals are using C1:LSC-. Optics positions would presumably be coming from the associated suspension model, and am currently using SUP, SPX, and SPY for the suspension plant models (suspension vertex, suspension x end, suspension y end).
I've updated the web view of these models on nodus. They can be viewed at: https://nodus.ligo.caltech.edu:30889/FE/
I've also created a C1.ipc file in /cvs/cds/caltech/chans/ipc which assigns ipcNum to each of these new channels in shared memory.
I got around to actually try building the LSC and IFO models on megatron. Turns out "ifo" can't be used as a model name and breaks when trying to build it. Has something to do with the find and replace routines I have a feeling (ifo is used for the C1, H1, etc type replacements throughout the code). If you change the model name to something like ifa, it builds fine though. This does mean we need a new name for the ifo model.
Also learned the model likes to have the cdsIPCx memory locations terminated on the inputs if its being used in a input role (I.e. its bringing the channel into the model). However when the same part is being used in an output role (i.e. its transmitting from the model to some other model), if you terminate the output side, it gives errors when you try to make.
Its using the C1.ipc file (in /cvs/cds/caltech/chans/ipc/) just fine. If you have missing memory locations in the C1.ipc file (i.e. you forgot to define something) it gives a readable error message at compile time, which is good. The file seems to be being parsed properly, so the era of writing "0x20fc" for block names is officially over.
I suggest "ITF" for the model name.
I'm currently working on a set of scripts which will be able to parse a "template" mdl file, replacing certain key words, with other key words, and save it to a new .mdl file.
For example you pass it the "template" file of scx.mdl file (suspension controller ETMX), and the keyword ETMX, followed by an output list of scy.mdl ETMY, bs.mdl BS, itmx.mdl ITMX, itmy.mdl ITMY, prm.mdl PRM, srm.mdl SRM. It produces these new files, with the keyword replaced, and a few other minor tweaks to get the new file to work (gds_node, specific_cpu, etc). You can then do a couple of copy paste actions to produce a combined sus.mdl file with all the BS, ITM, PRM, SRM controls (there might be a way to handle this better so it automatically merges into a single file, but I'd have to do something fancy with the positioning of the modules - something to look into).
I also have plans for a script which gets passed a mdl file, and updates the C1.ipc file, by adding any new channels and incrementing the ipcNum appropriately. So when you make a change you want to propagate to all the suspensions, you run the two scripts, and have an already up to date copy of memory locations - no additional typing required.
Similar scripts could be written for the DAQ screens as well, so as to have all the suspension screens look the same after changing one set.
So I finished writing a script which takes an .ipc file (the one which defines channel names and numbers for use with the RCG code generator), parses it, checks for duplicate channel names and ipcNums, and then parses and .mdl file looking for channel names, and outputs a new .ipc file with all the new channels added (without modifying existing channels).
The script is written in python, and for the moment can be found in /home/controls/advLigoRTS/src/epics/simLink/parse_mdl.py
I still need to add all the nice command line interface stuff, but the basic core works. And already found an error in my previous .ipc file, where I used the channel number 21 twice, apparently.
Right now its hard coded to read in C1.ipc and spy.mdl, and outputs to H1.ipc, but I should have that fixed tonight.
This IPC stuff looks really a nice improvement of CDS.
Please just maintain the wiki updated so that we can keep the latest procedures and scripts to build the models.
1) What is c1asc doing? What is ascaux used for? What are the cables labeled "C1:ASC_QPD" in the 1X2 rack really going to?
2) Put the 4600 machine (megatron) in the 1Y3 (away from the analog electronics) This can be used as an OAF/IO machine. We need a dolphin fiber link from this machine to the IO chassis which will presumably be in 1Y1, 1Y2 (we do not currently have this fiber at the 40m, although I think Rolf said something about having one).
3) Merge the PSL and IOOVME crates in 1Y1/1Y2 to make room for the IO chassis.
4) Put the LSC and SUS machines into 1Y4 and/or 1Y5 along with the SUS IO chassis. The dolphin switch would also go here.
5) Figure out space in 1X3 for the LSC chassis. Most likely option is pulling asc or ascaux stuff, assuming its not really being used.
6) Are we going to move the OMC computer out from under the beam tube and into an actual rack? If so, where?
Rolf will likely be back Friday, when we aim to start working on the "New" Y end and possibly the 1X3 rack for the LSC chassis.
I modified the /etc/rc.d/rc.local file on megatron removing a bunch of the old test module names and added the new lsc and lsp modules, as well as a couple planned suspension models and plants, to shared memory so that they'll work. Basically I'm trying to move forward into the era of working on the actual model we're going to use in the long term as opposed to continually tweaking "test" models.
The last line in the file is now: /usr/bin/setup_shmem.rtl lsc lsp spy scy spx scx sus sup&
I removed mdp mdc mon mem grc grp aaa tst tmt.
I modified /cvs/cds/caltech/target/fb and changed the line "set controller_dcu=10" to "set controller_dcu=13" (where 13 is the lsc dcu_id number).
I also changed the set gds_server line from having 10 and 11 to 13 and 14 (lsc and lsp).
The file /cvs/cds/caltech/fb/master was modified to use C1LSC.ini and C1LSP.ini, as well as tpchn_C2.par (LSC) and tpchn_C3.par (LSP)
testpoint.par in /cvs/cds/caltech/target/gds/param was modified to use C-node1 and C-node2 (1 less then the gds_node_id for lsc and lsp respectively).
Note all the values of gds_node_id, dcu_id, and so forth are recorded at http://lhocds.ligo-wa.caltech.edu:8000/40m/Electronics/Existing_RCG_DCUID_and_gds_ids
I had a chat with Alex this morning and discovered that the dcu_ids 13,14,15,16 are reserved currently, and should not be used. I was told 9-12 and 17-26 were fine to use. I pointed out that we will eventually have more modules than that. His response was he is currently working on the framebuilder code and "modernizing" it, and that those restrictions will hopefully be lifted in the future although he isn't certain at this time what the real maximum gds_id number is (he was only willing to vouch for up to 26 - although the OMC seems to be currently working and set to 30).
Alex also suggested running an iop module to provide timing (since we are using adcSlave=1 option in the models). Apparently these are x00.mdl, x01.mdl, x11.mdl files in the /home/control/cds/advLigoRTS/src/epics/simLink/ directory. I saved x00.mdl as io1.mdl (I didn't want to use io0 as its a pain to differentiate between a zero and 'O'. This new IOP is using gds_node=1, dcu_id=9. I modified the approriate files to include it.
I modified /etc/rc.d/rc.local and added io1 to shmem line. I modified /cvs/cds/caltech/target/fb/daqdrc to use dcu_id 9 as the controller (this is the new iop model dcu_id number). In that same directory I modifed the file master by adding /cvs/cds/caltech/chans/daq/C1IO1.ini as well as uncommenting tpchn_C1 line. I modified testpoint.par in /cvs/cds/caltech/target/gds/param to include C-node0, and modified the prognum for lsc and lsp to 0x31001003 and 0x31001005.
So I started the 3 processes with startio1, startlsc, startlsp, then went to the fb directory and started the framebuilder. However, the model lsc.mdl is still having issues, although lsp and io1 seem to be working. At this point I just need to track down what fundamentally is different between lsc and lsp and correct it in the lsc model. I'm hoping its not related to the fact that we actually had a previous lsc front end and there's some legacy stuff getting in the way. One thing I can test is changing the name and see if that runs.
I'm currently in the process of tracking down what legacy code is interfering with the new lsc model.
It turns out if you change the name of lsc file to something else (say scx as a quick test for example), it runs fine. In fact, the lsc and scx GDS_TP screens work in that case (since they're looking at the same channels). As one would expect, running them both at the same time causes problems. Note to self, make sure the other one is killed first. It does mean the lsc code gets loaded part way, but doesn't seem to communicate on EPICs or to the other models. However, I don't know what existing code is interfering. Currently going trhough the target directories and so forth.
We placed 3 new computers in the racks. One in 1X4 (machine running SCX) and 2 in 1Y4 (LSC and SUS). These are 1U chassis, 4 core machines for the CDS upgrade. I will be bringing over 2 IO chassis and their rails over tomorrow, one to be placed in 1Y4, and 1 in 1X4.
We still need some more 40 pin adapter cables and will send someone over this week to make them. However, once we have those, we should be able to get two to three machines going, one end computer/chassis and the SUS computer/chassis.
After tomorrow we are still going to be owed 1 computer, another dolphin fiber, a couple of blue boxes, and the LSC, IO, and Y end IO chassis. We also realized we need further fiber for the timing system. We're going to need to get and then run fiber to both ends, as well as to 1X3, where the LSC IO chassis will be.
After having checked old possibilities and deciding I wasn't imagining the lsc.mdl file not working, but working as another name, I tracked Alex down and asked for help.
After scratching our heads, we finally tracked it down to the RCG code itself, as opposed to any existing code.
Apparently, the skeleton.st file (located in /home/controls/cds/advLigoRTS/src/epics/util/) has special additional behavior for models with the following names: lsc, asc, hepi, hepia, asc40m, ascmc, tchsh1, tchsh2.
Alex was unsure what this additional code was for. To disable it, we went into the skeleton.st file, and changed the name "SEQUENCER_NAME_lsc" to "SEQUENCER_NAME_lsc_removed" where ever it occured. These names were in #ifdef statements, so now these codes will only be used if the model is named lsc_removed. This apparently fixed the problem. Running startlsc now runs the code as it should, and I can proceed to testing the communication to the lsp model.
Alex said he'd try to figure out what these special #ifdef code pieces are intended for and hopefully completely remove them once we've determined we don't need it.
We have 2 new IO chassis with mounting rails and necessary boards for communicating to the computers. Still need boards to talk to the ADCs, DACs, etc, but its a start. These two IO chassis are currently in the lab, but not in their racks.
They will installed into 1X4 and 1Y5 tomorrow. In addition to the boards, we need some cables, and the computers need the approriate real time operating systems setup. I'm hoping to get Alex over sometime this week to help work on that.
So I discovered the hard way that the racks are not standard width, when I was unable to place a new IO chassis into the racks with rails attached. The IO chassis is narrow enough to fit through without the rails however.
I've talked to Steve and we decided on having some shelves made. I've asked Steve to get us 6. 1 for each end (2), 1 for SUS, 1 for LSC, 1 for IO, and 1 extra.
In /cvs/cds/caltech/target/fb modified:
master: cleaned up so only io1 (IO processor), LSC, LSP, SCY, SPY were listed, along with their associated tpchan files.
daqdrc: fixed "dcu_rate 9 = 32768" to "dcu_rate 9 = 65536" (since the IO processor is running at 64k)
Added "dcu_rate 21 = 16384" and "dcu_rate 22 = 16384"
Changed "set gds_server = "megatron" "megatron" "megatron" 9 "megatron" 10 "megatron" 11;" to
set gds_server = "megatron" "megatron" 9 9;
The above change was made after reading Rolf's Admin guide: http://lhocds.ligo-wa.caltech.edu:8000/40m/Upgrade_09/CDS?action=AttachFile&do=get&target=RCG_admin_guide.pdf
The set gds_server is simply telling which computer the gds daemons are running on, and we don't need to do it 5 times.
In /cvs/cds/caltech/gds/params modified:
testpoint.par: added C-node7 and C-node8 for SCY and SPY respectively.
Alex brought over a ADC, DAC, and PCIe card which goes into the computer and talk to the IO chassis. We tried installing the new "small" IO chassis in 1X4, but initially it couldn't find the ADC/DAC boards, just the Contec Binary out board.
We tried several different configurations (different computer, different IO chassis, the megatron chassis, the megatron IO chassis with new cards, a new IO chassis with megatron cards.
The two things were concluded once we got a new IO chassis talking with the new computer.
1) Its possible one of the slots is in the IO chassis as we didn't see the ADC until we moved it to a different slot in the new chassis
2) The card that Alex had brought over to put in the computer doesn't behave well with these IO chassis. He went back to downs to try and figure out what it is, and test it with the chassis over there.
Currently, Megatron's IO chassis is sitting on a table behind racks 1Y5 and 1Y6, along with the new "large" IO chassis. Megatron's PCIe card for talking to the IO chassis was moved to the computer in 1X4. The computer in 1X4 is currently being called c1iscex with IP 192.168.113.80.
The problem with the new models using the new shared memory/dolphin/RFM defined as names in a single .ipc file.
The first is the no_oversampling flag should not be used. Since we have a single IO processor handling ADCs and DACs at 64k, while the models run at 16k, there is some oversampling occuring. This was causing problems syncing between the models and the IOP.
It also didn't help I had a typo in two channels which I happened to use as a test case to confirm they were talking. However, that has been fixed.
Now that we have multiple machines we'd like to run the new front end code on, I'm finding it annoying to have to constantly copy files back and forth to have the latest models on different machines. So I've come to the conclusion that Rana was right all along, and I should working somewhere in /cvs/cds/caltech which gets mounted by everyone.
However, this leads to the svn problem: I.e. I need recent code checked out from the RCG repository, but our current /cvs/cds/caltech/cds/advLigo directory is covered by the 40m SVN. So for the moment, I've checked out the advLigoRTS from https://redoubt.ligo-wa.caltech.edu/svn/advLigoRTS/trunk into /cvs/cds/caltech/cds/advLigoRTS. This directory will be kept as up to date as I can keep it, both by running svn update to get Alex/Rolf's changes and on my end by keeping the new and updated models. It will remain linked the RCG repository and not the 40m repository. At some point a better solution is needed, but its the best I can come up with for now.
Also, because we are starting to compile on different machines sometimes, you may run into a problem where a code won't run on a different machine. This can be fixed by commenting out some lines in the startup script. Go to the /cvs/cds/caltech/scripts directory. Then edit the associated startSYS file by commenting out the lines that look like:
if [ `hostname` != megatron ]; then
echo Cannot run `basename $0` on `hostname` computer
Unfortunately, this gets reverted each time "make SYS" and "make install-SYS" gets run.
The other issue this leads to is that some machines don't have as many CPUs available as others. For example our new thin 1U machines have only 4 dual cores (8 CPUs total). This means the specific_cpu setting of any of the codes cannot be higher than 7 (cores being numbered 0 through 7). Core 0 is reserved for the real time kernel, and Core 1 will be used on all machines for the IO processor. This leaves only cores 2 through 7 available for models to use which include LSC, LSP, SUS, SUP, SPY, SCY, SPX, SCX, OMC, OMP, OAF, OAP?, IOC, IOP. Since there are more than 6 models, duplication in final production code of specific_cpus will be necessary. Codes which are all running on Megatron at one point will have to be rebuilt with new specific_cpu values when run on the actual final machine.
I created the sus model, which is the suspension controller for ITMX, ITMY, BS, PRM, SRM. I also created sup, which is the suspension plant model for those same optics.
Updated /cvs/cds/caltech/target/fb master and daqdrc files to add SUS, SUP models. Megatron's /etc/rc.d/rc.local file has been updated to include all the necessary models as well.
The suspension controller needs the Binary IO outputs need to be checked and corrected if wrong by changing the constant connected to the exclusive or gates. Right now its using the end suspension binary output values which may not be correct.
I've modified the lsc.mdl and lsp.mdl files back to an older configuration, where we do not use an IO processor. This seems to let things work for the time being on megatron while I try to figure out what the is wrong with the "correct" setup which includes the IO processor.
Basically I removed the adcSlave = 1 line in the cdsParameters block.
I've attached a screen shot of the desktop showing one filter bank in the LSP model passing its output correctly to a filter block in the LSC. I also put in a quick test filter (an integrator) and you can see it got to 80 before I turned off the offset.
So far this is only running on megatron, not the new machine in the new Y end.
The models being use for this are located in /cvs/cds/caltech/cds/advLigoRTS/src/epics/simLink
Talked with Alex and tracked down why the codes were not working on the new c1iscex finally. The .bashrc and .cshrc files in /home/controls/ on c1iscex has the following lines:
setenv EPICS_CA_ADDR_LIST 184.108.40.206
setenv EPICS_CA_AUTO_ADDR_LIST NO
This was interfering with channel access and preventing read and writes from working properly. We simply commented them out. After logging out and back in, the things like ezcaread and write started working, and we were able to get the models passing data back and forth.
Next up, testing RFM communications between megatron on c1iscex. To do this, I'd like to move Megatron down to 1Y3, and setup a firewall for it and c1iscex so I can test the frame builder and testpoints at the same time on both machines.
Alex updated the awg.par file to handle all the testpoints. Basically its very similar to the testpoint.par, but the prognum lines have to be 1 higher than the corresponding prognum in testpoint.par. A entry looks like:
After running "diag -i" and seeing some RPC number conflicts, we went into /cvs/cds/caltech/cds/target/gds/param/diag_C.conf and changed the line from
&chn * * 192.168.1.2 822087685 1
&chn * * 192.168.1.2 822087700 1
The number represents an RPC number. This was conflicting with the RPC number associated with the awgtpman processes. We then had to update the /etc/rpc file as well. At the end we changed chnconf 822087685 to chnconf 822087700. We then run /usr/sbin/xinetd reload
Lastly we edited the /etc/xinetd.d/chnconf file line
server_args = /cvs/cds/caltech/target/gds/param/tpchn_C4.par /cvs/cds/caltech/target/gds/param/tpchn_C5.par
server_args = /cvs/cds/caltech/target/gds/param/tpchn_C1.par /cvs/cds/caltech/target/gds/param/tpchn_C2.par /cvs/cds/caltech/target/gds/param/tpchn_C3.par /cvs/cds/caltech/target/gds/param/tpchn_C4.par /cvs/cds/caltech/target/gds/param/tpchn_C5.par /cvs/cds/caltech/target/gds/param/tpchn_C6.par /cvs/cds/caltech/target/gds/param/tpchn_C7.par /cvs/cds/caltech/target/gds/param/tpchn_C8.par /cvs/cds/caltech/target/gds/param/tpchn_C9.par
Alex also recompiled the frame builder code to be able to handle more than 7 front ends. This involved tracking down a newer version of libtestpoint.so on c1iscex and moving it over to megatron, then going in and by hand adding the ability to have up to 10 front ends connected.
Alex has said he doesn't like this code and would like it to dynamically allocate properly for any number of servers rather than having a dumb hard coded limit.
Other changes he needs to make:
1) Get rid of set dcu_rate ## = 16384 type lines in the daqrc file. That information is available from the /caltech/chans/C1LSC.ini type files which are automatically generated when you compile a model. This means not having to go in by hand to update these in daqrc.
2) Get some awg.par and testpoint.par rules, so that these are automatically updates when you build a model. Make it so it automatically assigns a prognum when read in rather than having to hard code them in by hand.
3)Slave the awgtpmans to a single clock running from the IO processor x00. This ensures they are all in sync.