40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 244 of 341  Not logged in ELOG logo
ID Date Author Type Categorydown Subject
  16963   Wed Jun 29 18:53:38 2022 ranaUpdateElectronicsElectronics noise

this is just the CDS error signal, but is not the electronics noise. You have to go into the lab and measure the noise at several points. It can't be done from the control room. You must measure before and afte the whitening.

Quote:

I measured electronics noise of WFSs and QPD (of the WFS/QPD, whitening, ADC...) by closing PSL and measuring the error signal. It was needed to put the offset in C1:IOO-MC_TRANS_SUMFILT_OFFSET to 14000 cts (without offset the sum of quadrants would give zero, and 14000 cts is the value when the cavity is locked). For WFS that are RF, if there is intensity noise at low frequencies, it is not affecting the measurement.

In the attachment please find the power spectrum of the error signal when the PSL shutter is on and off.

 

  16964   Thu Jun 30 17:19:55 2022 DeekshaSummaryElectronicsMeasured Transfer Functions of the Control Loop, Servo (OLTF); got Vectfit working

[Cici, Deeksha]

We were able to greatly improve the quality of our readings by changing the parameters in the config file (particularly increasing the integration and settle cycles, as well as gradually increasing our excitation signals' amplitude). Attached are the readings taken from the same (the files directly printed by ssh'ing the SR785 (apologies)) - Attachment 1 depicts the graph w/ 30 data points and attachment 2 depicts the graph with 300 data points. 

Cici successfully vectfit to the data, as included in Attachment 3. (This is the vectfit of the entire control loop's OLTF). There are two main concerns that need to be looked into, firstly, the manner in which to get the poles and zeros to input into the vectfit program. Similarly, the program works best when the option to enforce stable poles is disabled, once again it may be worth looking into how the program works on a deeper level in order to understand how to proceed. 

Just as the servo's individual transfer function was taken, we also came up with a  plan to measure the PZT's individual transfer function (using the MokuLab). The connections for the same have been made and the Moku is at the Xend (disconnected). We may also have to build a highpass filter (similar to the one whose signal enters the PZT) to facilitate taking readings at high frequencies using the Moku. 

Attachment 1: TFSR785_29-06-2022_114042.pdf
TFSR785_29-06-2022_114042.pdf
Attachment 2: TFSR785_29-06-2022_114650.pdf
TFSR785_29-06-2022_114650.pdf
Attachment 3: TF_OLG_vectfit.png
TF_OLG_vectfit.png
  16972   Tue Jul 5 20:05:06 2022 TomislavUpdateElectronicsWhitening electronics noise

For whitening electronics noise for WFS1, I get (attachment). This doesn't seem right, right?

Attachment 1: whitening_noises.png
whitening_noises.png
  16974   Wed Jul 6 18:51:20 2022 DeekshaUpdateElectronicsMeasuring the Transfer Function of the PZT

Yesterday, we set up the loop to measure the PZT of the transfer function - the MokuLab sends an excitation (note - a swept sine of 1.0 V) to the PZT. The cavity is locked to the PSL and the AUX is locked to the cavity. In order to measure the effect of our excitation, we take the beat note of the PSL and the AUX. This gives us a transfer function as seen in Attachment 1. The sampling rate of the MokuLab is set to 'ultrafast' (125kHz), so we can expect accurate performance upto 62.5kHz, however, in order to improve our readings beyond this frequency, modifications must be made to the script (MokuPhaseMeterTF) to avoid aliasing of the signal. A script should also be written to obtain and plot the coherence between the excitation and our output. 

Also attached are - Attachment 2 -  the circuit diagram of the setup, and Attachment 3 - the TF data calculated.

Edit - the SR560 as shown in the circuit diagram has since been replaced by a broadband splitter (Minicircuits ZFRSC-42-S+).

Attachment 1: pzt_transfer_fn.png
pzt_transfer_fn.png
Attachment 2: ckt_diagram.jpeg
ckt_diagram.jpeg
Attachment 3: MokuPhaseMeterTFData_20220706_174753_TF_Data.txt
2.000000000000000364e+04 1.764209350625748560e+07 2.715833132756984014e+00
1.928351995884991265e+04 1.695301366919569671e+07 1.509398637395631626e+00
1.859270710016814337e+04 1.647055321367538907e+07 -2.571975165101855865e+00
1.792664192275710593e+04 1.558169995329630189e+07 6.272729335836754183e-01
1.728443786563210961e+04 1.500850042360494658e+07 -1.500422400597591466e+00
1.666524012797089381e+04 1.456986577652360499e+07 2.046163000975175894e+00
1.606822453133765885e+04 1.376167843637173250e+07 1.736835046956476614e+00
1.549259642266657283e+04 1.326192932667389885e+07 -1.272425049850132606e+00
1.493758961654484847e+04 1.283127345074228011e+07 -2.026149685362535369e+00
1.440246537538758821e+04 1.208854709974890016e+07 -3.248352694840740407e-01
... 11 more lines ...
  16983   Mon Jul 11 11:16:45 2022 JCSummaryElectronicsStartup after Shutdown

[Paco, Yehonathan, JC]

We began starting up all the electronics this morning beginning in the Y-end. After following the steps on the Complete_Power_Shutdown_Procedures on the 40m wiki, we only came across 2 issues.

  1. The Green beam at the Y-End : Turn on the controller and the indicator light began flashing. After waiting until the blinking light becomes constant, turn on the beam. 
  2. C1lsc "could not find operating system"-unable to SSH from Rossa : We found an Elog of how to restart Chiara and this worked. We proceeded by adding this to the procedures of startup.
  16992   Tue Jul 12 14:56:17 2022 TomislavSummaryElectronicsElectronics noise measurements

[Paco, Tomislav]

We measured the electronics noise of the demodulation board, whitening board, and ADC for WFSs, and OPLEV board and ADC for DC QPD in MC2 transmission. We were using SR785.

Regarding the demodulation board, we did 2 series of measurements. For the first series of measurements, we were blocking WFS (attachment 1) and measuring noise at the output of the demod board (attachment 2a). This measurement includes dark noise of the WFS, electronics noise of demod board, and phase noise from LO. For the second series of the measurements, we were unplugging input to the demod board (attachment 2b & 2c is how they looked like before unplugging) (the mistake we made here is not putting 50-ohm terminator) and again measuring at the output of the demod board. This measurement doesn't include the dark noise of the WFS. We were measuring it for all 8 segments (I1, I2, I3, I4, Q1, Q2, Q3, Q4). The dark noise contribution is negligible with respect to demod board noise. In attachments 3 & 4 please find plots that include detection and demodulation contributions for both WFSs.

For whitening board electronics noise measurement, we were terminating the inputs (attachment 5) and measuring the outputs (attachment 6). Electronics noise of the whitening board is in the attachments 7 & 8.

For ADC electronics noise we terminated ADC input and measured noise using diaggui (attachments 9 & 10). Please find these spectra for WFS1, WFS2, and MC TRANS in attachments 11, 12 & 13.

For MC2 TRANS we measured OPLEV board noise. We did two sets of measurements, as for demod board of WFSs (with and without QPD dark noise) (attachments 14, 15 & 16). In the case of OPLEV board noise without dark noise, we were terminating the OPLEV input. Please find the electronics noise of OPLEV's segment 1 (including dark noise which is again much smaller with respect to the OPLEV's electronics noise) in attachment 17.

For the transfer functions, demod board has flat tf, whitening board tf please find in attachment 18, ADC tf is flat and it is (2**16 - 1)/20 [cts/V], and dewhitening tf please find in attachment 19. Also please find the ASD of the spectral analyzer noise (attachment_20).

Measurements for WFS1 demod and whitening were done on 5th of July between 15h and 18h local time. Measurements for WFS2 demod and whitening were done on 6th of July between 15h and 17h local time. All the rest were done on July 7th between 14h and 19h. In attachment 21 also find the comparison between electronics noise for WFSs and cds error signal (taken on the 28th of June between 17h and 18h). Sorry for bad quality of some pictures.

Attachment 1: attachment_1.jpg
attachment_1.jpg
Attachment 2: attachment_2a.jpg
attachment_2a.jpg
Attachment 3: attachment_2b.jpg
attachment_2b.jpg
Attachment 4: attachment_2c.jpg
attachment_2c.jpg
Attachment 5: attachment_3.png
attachment_3.png
Attachment 6: attachment_4.png
attachment_4.png
Attachment 7: attachment_5.jpg
attachment_5.jpg
Attachment 8: attachment_6.jpg
attachment_6.jpg
Attachment 9: attachment_7.png
attachment_7.png
Attachment 10: attachment_8.png
attachment_8.png
Attachment 11: attachment_9.jpg
attachment_9.jpg
Attachment 12: attachment_10.jpg
attachment_10.jpg
Attachment 13: attachment_11.png
attachment_11.png
Attachment 14: attachment_12.png
attachment_12.png
Attachment 15: attachment_13.png
attachment_13.png
Attachment 16: attachment_14.jpg
attachment_14.jpg
Attachment 17: attachment_15.jpg
attachment_15.jpg
Attachment 18: attachment_16.jpg
attachment_16.jpg
Attachment 19: attachment_17.png
attachment_17.png
Attachment 20: attachment_18.png
attachment_18.png
Attachment 21: attachment_19.png
attachment_19.png
Attachment 22: attachment_20.png
attachment_20.png
Attachment 23: attachment_21.png
attachment_21.png
  16995   Wed Jul 13 07:16:48 2022 JCUpdateElectronicsChecking Sorensen Power Supplies

[JC]

I went around 40m picking up any Sorensens that were laying around to test if they worked, or in need of repair. I gathered up a total of 7 Sorensens and each one with a Voltmeter. I made sure the voltage would rise on the Sorenson as well as the voltmeter, maxing out at ~33.4 Volts. For the current, the voltmeter can only rise to 10 Amps before it is fused. Many of the Sorensons that I found did not have their own wall connection, so I had to use the same one for multiple.

From these 7, I have found 5 that are well. One Sorenson I have tested has a output shortage above 20V and the other has yet to be tested.

Attachment 1: 658C5D39-11BD-4EE3-90E2-34CBBC1DBD3C.jpeg
658C5D39-11BD-4EE3-90E2-34CBBC1DBD3C.jpeg
Attachment 2: 5328312A-7918-44CC-82B7-54B57840A336.jpeg
5328312A-7918-44CC-82B7-54B57840A336.jpeg
  16998   Wed Jul 13 13:26:44 2022 ranaSummaryElectronicsElectronics noise measurements

as I said to you yesterday, I don't think image 2a shows the output of the demod board. The output of the demod board is actually the output connector ON the demod board. What you are showing in 2a, is the signal that goes from the whitening board to the ADC I believe. I may be msitaken, so please check with Tega for the signal chain.

  17005   Fri Jul 15 12:21:58 2022 JCUpdateElectronicsChecking Sorensen Power Supplies

Of the 7 Sorenson Power Supplies I tested, 5 are working fine, 1 cannot output voltage more than 20 Volts before shorting, and other does not output current. Six Sorensons are behind the X-Arm.

Quote:

[JC]

I went around 40m picking up any Sorensens that were laying around to test if they worked, or in need of repair. I gathered up a total of 7 Sorensens and each one with a Voltmeter. I made sure the voltage would rise on the Sorenson as well as the voltmeter, maxing out at ~33.4 Volts. For the current, the voltmeter can only rise to 10 Amps before it is fused. Many of the Sorensons that I found did not have their own wall connection, so I had to use the same one for multiple.

From these 7, I have found 5 that are well. One Sorenson I have tested has a output shortage above 20V and the other has yet to be tested.

 

Attachment 1: 50DF21D7-D61A-4674-B0DA-463378B00ADB.jpeg
50DF21D7-D61A-4674-B0DA-463378B00ADB.jpeg
Attachment 2: FA4CF579-6C1E-48D5-B152-74F35B4EE90B.jpeg
FA4CF579-6C1E-48D5-B152-74F35B4EE90B.jpeg
  17019   Tue Jul 19 17:18:34 2022 JCUpdateElectronicsNew Coil Driver on Rack 1X3

[Yehonathan, JC]

Yehonathan and I began to put the electronics on Rack 1X3. To do this, we had to move the monitor over the the PD testing table. Before mounting the Coil Drivers, we added numbers to the spaces to follow the rack plan Koji has provided. The drivers which have been mounted are PRM (Slots 10,11), BS (Slots 15, 16), ITMX (Slots 26, 27), and ITMY (34, 35).

Attachment 1: 22DC1767-6073-4D82-BEED-915318B57C03.jpeg
22DC1767-6073-4D82-BEED-915318B57C03.jpeg
  17031   Mon Jul 25 09:37:39 2022 DeekshaUpdateElectronicsUsing the DFD to measure PZT TF

The DFD was setup to measure the change in beatnote when excited. A long long (128in) cable goes from the SR785 near the DFD all the way to the Xend AUX which it accordingly excites and the DFD is monitored by the oscilloscope at the other end. This was completed on Friday. The wires and stand have been moved to the side but the setup is still a bit chaotic. As of writing this post, there is still atleast some minor issue with the setup as we aren't getting the expected output. 

[I will shortly update this elog with more pictures]

Edit: the SR785 was replaced by the AG 4395, and pictures added

 

Attachment 1: ag4395.jpeg
ag4395.jpeg
Attachment 2: dfd.jpeg
dfd.jpeg
  17039   Wed Jul 27 14:39:04 2022 DeekshaUpdateElectronicsNew and improved PZT TF data from the DFD

Paco and I messed around with the attenuation of the scope and bandwidth of the IF. We also replaced the BNC T's in the circuits with RF splitters. We saw some decent improvements to the data. The data is attached and a diagram of the experiment. [We analytically calculated the impedances to avoid any mismatch taking place]. Working on fitting the data.

We also moved around the wires so that the AG4395 is closer to the PZT.

 

Attachment 1: tf_data.png
tf_data.png
Attachment 2: latest_data.zip
Attachment 3: dfd_-_new.drawio.png
dfd_-_new.drawio.png
  17104   Thu Aug 25 15:24:06 2022 PacoHowToElectronicsRFSoC 2x2 board -- fandango

[Paco, Chris Stoughton, Leo -- remote]

This morning Chris came over to the 40m lab to help us get the RFSoC board going. After checking out our setup, we decided to do a very basic series of checks to see if we can at least get the ADCs to run coherently (independent of the DACs). For this I borrowed the Marconi 2023B from inside the lab and set its output to 1.137 GHz, 0 dBm. Then, I plugged it into the ADC1 and just ran the usual spectrum analyzer notebook on the rfsoc jupyter lab server. Attachment #1 - 2 shows the screen captured PSDs for ADCs 0 and 1 respectively with the 1137 MHz peaks alright.

The fast ADCs are indeed reading our input signals.


Before this simple test, we actually reached out to Leo over at Fermilab for some remote assistance on building up our minimally working firmware. For this, Chris started a new vivado project on his laptop, and realized the rfsoc 2x2 board files are not included in it by default. In order to add them, we had to go into Tools, Settings and add the 2020.1 Vivado Xilinx shop board repository path to the rfsoc2x2 v1.1 files. After a little bit of struggling, uninstalling, reinstalling them, and restarting Vivado, we managed to get into the actual overlay design. In there, with Leo's assistance, we dropped the Zynq MPSoC core (this includes the main interface drivers for the rfsoc 2x2 board). We then dropped an rf converter IP block, which we customized to use the right PLL settings. The settings, from the System Clocking tab were changed to have a 409.6 MHz Reference Clock (default was 122.88 MHz). This was not straightforward, as the default sampling rate of 2.00 GSPS was not integer-related so we had to also update that to 4.096 GSPS. Then, we saw that the max available Clock Out option was 256 MHz (we need to be >= 409.6 MHz), so Leo suggested we dropped a Clocking Wizard block to provide a 512 MHz clock input for the rfdc. The final settings are captured in Attachment # 3. The Clocking Wizard was added, and configured on its Output Clocks tab to provide a Requested Output Freq of 512 MHz. The finall settings of the Clocking wizard are captured in Attachment #4. Finally, we connected the blocks as shown in Attachment #5.

We will continue with this design tomorrow.

Attachment 1: adc0_1137MHz.png
adc0_1137MHz.png
Attachment 2: adc1_1137MHz.png
adc1_1137MHz.png
Attachment 3: rfdc_PLLsettings.png
rfdc_PLLsettings.png
Attachment 4: clockingwiz_settings.png
clockingwiz_settings.png
Attachment 5: blockIPdiag.png
blockIPdiag.png
  12692   Fri Dec 30 10:27:46 2016 ranaUpdateDetCharsummary pages dead again

Dead again. No outputs for the past month. We really need a cron job to check this out rather than wait for someone to look at the web page.

  12693   Thu Jan 5 21:43:16 2017 ranaUpdateDetCharsummary pages dead again

Max tells us that soem conf files were bad and that he did something and now some pages are being made. But the PEM and MEDM pages are bank. Also the ASC tab looks bogus to me.

  12910   Mon Mar 27 20:29:05 2017 ranaSummaryDetCharSummary pages broken again

Going to the summary pages and looking at 'Today' seems to break it and crash the browser. Other tabs are OK, but 'summary' is our default page.

I've noticed this happening for a couple of days now. Today, I moved the .ini files which define the config for the pages from the old chans/ location into the /users/public_html/detcharsummary/ConfigFiles/ dir. Somehow, we should be maintaining version control of detcharsummary, but I think right now its loose and free.

  13834   Fri May 11 18:17:07 2018 gautamUpdateDetCharAUX laser PLL setup

[koji, gautam]

As discussed at the meeting earlier this week, we will use some old *MOPA* channels for interfacing with the PLL system Jon is setting up. He is going to put a sketch+photos up here shortly, but in the meantime, Koji helped me identify a channel that can be used to tune the temperature of the Lightwave NPRO crystal via front panel BNC input. It is C1:PSL-126MOPA_126CURADJ, and is configured to output between +/-10V, which is exactly what the controller can accept. The conversion factor from EPICS value to volts is currently set to 1 (i.e. EPICS value of +1 corresponds to +1V output from the DAC). With the help of the wiring diagram, we identified pins 3 and 4 on cross-connect #J7 as the differential outputs corresponding to this channel. Not sure if we need to also setup a TTL channel for servo ENABLE/DISABLE, but if so, the wiring diagram should help us identify this as well. 

The cable from the DAC to the cross-connect was wrongly labelled. I fixed this now.

  15309   Wed Apr 22 13:52:05 2020 gautamUpdateDetCharSummary page revival

Covid 19 motivated me to revive the summary pages. With Alex Urban's help, the infrastructure was modernized, the wiki is now up to date. I ran a test job for 2020 March 17th, just for the IOO tab, and it works, see here. The LDAS rsync of our frames is still catching up, so once that is up, we can start the old condor jobs and have these updated on a more regular basis.

  15370   Wed Jun 3 11:20:19 2020 gautamUpdateDetCharSummary pages

Summary:

The 40m summary pages have been revived. I've not had to make any manual interventions in the last 5 days, so things seem somewhat stable, but I'm sure there will need to be multiple tweaks made. The primary use of the pages right now are for vacuum, seismic and PSL diagnostics.

Resources:

Caveats:

  • Intermittent failures of cron jobs
    • The status page relies on the condor_q command executing successfully on the cluster end. I have seen this fail a few times, so the status page may say the pages are dead whereas they're actually still running.
    • Similarly, the rsync of the pages to nodus where they're hosted can sometimes fail.
    • Usually, these problems are fixed on the next iteration of the respective cron jobs, so check back in ~half hour.
  • I haven't really looked into it in detail, but I think our existing C1:IFO-STATE word format is not compatible with what gwsumm wants - I think it expects single bits that are either 0 or 1 to indicate a particular state (e.g. MC locked, POX and POY locked etc). So if we want to take advantage of that infrastructure, we may need to add a few soft EPICS channels that take care of some logic checking (several such bits could also be and-ed together) and then assume either 0 or 1 value. Then we can have the nice duty cycle plots for the IMC (for example).
  • I commented out the obsolete channels (e.g. PEM MIC channels). We can add them back later if we so desire.
  • For some reason, the jobs that download the data are pretty memory-heavy: I have to request for machines on the cluster with >100 GB (yes 💯GB) ! of memory for the jobs to execute to completion. The frame-data certainly isn't so large, so I wonder what's going on here - is GWPy/GWsumm so heavy? The site summary pages run on a dedicated cluster, so probably the code isn't built for efficiency...
  • Weather tab in PEM is still in there but none of those channels mean anything right now.
  • The MEDM screenshot stuff is commented out for now too. This should be redone in a better way with some modern screen grab utilities, I'm sure there are plenty of python based ones.
  • There seems to be a problem with the condor .dag lockfile / rescue file not being cleared correctly sometimes - I am looking into this.
  15696   Wed Dec 2 18:35:31 2020 gautamUpdateDetCharSummary page revival

The summary pages were in a sad state of disrepair - the daily jobs haven't been running for > 1 month. I only noticed today because Jordan wanted to look at some vacuum trends and I thought summary pages is nice for long term lookback. I rebooted it just now, seems to be running. @Tega, maybe you want to set up some kind of scripted health check that also sends an alert.

  15812   Wed Feb 17 13:59:35 2021 gautamUpdateDetCharSummary pages

The summary pages had failed because of a conda env change. We are dependent on detchar's conda environment setup to run the scripts on the cluster. However, for some reason, when they upgraded to python3.9, they removed the python3.7 env, which was the cause of the original failure of the summary pages a couple of weeks ago. Here is a list of thoughts on how the pipeline can be made better.

  1. The status checking is pretty hacky at the moment.
    • I recommend not using shell via python to check if any condor jobs are "held".
    • Better is to use the dedicated python bindings. I have used this to plot the job durations, and it has worked well.
    • One caveat is that sometimes, there is a long delay between making a request via a python command, and condor actually returning the status. So you may have to experiment with execution times and build some try/except logic to catch "failures" that are just the condor command timing out and not an actual failure of the summare jobs.
  2. The status check should also add a mailer which emails the 40m list when the job is held. 
    • With htcondor and python, I think it's easy to also get the "hold reason" for the job and add that to the mailer.
  3. The job execution time command is not executing correctly anymore - for whatever reason, the condor_history command can't seem to apply the constraint of finding only jobs run by "40m", although running it without the constraint reveals that these certainly exist. Probably has to do with some recent upgrade of condor version or something. This should be fixed.
  4. We should clear the archive files regularly. 
    • The 40m home directory on the cluster was getting full. 
    • The summary page jobs generate a .h5 archive of all the data used to generate the plots. Over ~1 year, this amounts to ~1TB.
    • I added the cleanArchive job to the crontab, but it should be checked.
    • Do we even need these archives beyond 1 day? I think they make the plotting faster by saving data already downloaded locally, but maybe we should have the cron delete all archive 
  5. Can we make our own copy of the conda env and not be dependent on detchar conda env? The downside is that if something dramatic changes in gwsumm, we are responsible for debugging ourselves.

Remember that all the files are to be edited on nodus and not on the cluster.

  16893   Mon Jun 6 16:09:23 2022 ranaConfigurationDetCharSummary Pages: seis BLRMS

I updated the config file c1pem.ini in /users/public_html/detcharsummary/ConfigFiles, and commited it so I hope it works, but I did not have git push permissions. Does anyone know what is the idea here? Should we do our own personal git clone and modify that way or shoudl we do it with the control account.

Wiki needs to clear out all the outdated information on this workflow.

The changes are to make the y-scales useful. Currently, all of the past seis BLRMS plots are not so useful because the scales have not been set based on the actual signal levels. Let's see if this works, and we  can re-evaluate after a few weeks.

  17126   Thu Sep 1 09:00:02 2022 JCConfigurationDaily ProgressLocked both arms and aligned Op Levs

Each morning now, I am going to try to align both arms and lock. Along with that, sometime at towards the end of each week, we should align the OpLevs. This is a good habit that should be practiced more often, not only by me. As for the Y Arm, Yehonathan and I had to adjust the gain to 0.15 in order to stabilize the lock.

Attachment 1: Daily.pdf
Daily.pdf
Attachment 2: Daily.pdf
Daily.pdf
  280   Mon Jan 28 15:11:38 2008 robHowToDMFrunning compiled matlab DMF tools

I compiled Rana's seisBLRMS monitor, and it's now running on mafalda. To start your own DMF tools, here is a procedure:

1) build your tool in mDV, get it working the way you'd like.
2) Make a new directory /cvs/cds/caltech/apps/DMF/compiled_matlab/{your_new_directory} and copy the *.m file there.
3) Make the *.m in your new directory into a function with no args (just add a function line at the top)
4) compile it (from within a fully mDV-configured matlab) with mcc -m -R -nojvm {yourfile}.m at the matlab command line.
5) add a line corresponding to your new tool to the script /cvs/cds/caltech/apps/DMF/scripts/start_all
6) Run the start_all script referenced in part (5).

NB: Steps (4) and (6) must be carried out on mafalda.
  282   Mon Jan 28 18:56:47 2008 ranaUpdateDMFseisBLRMS 1.0
I made all of the updates I aludded to before:

  • Expanded the dmf.db file on c1aux to include all accelerometers and the seismometer.
  • Added the channels to the C0EDCU.ini file and restarted the framebuilder daqd.
  • Modified seisBLRMS.m to use .conf files for the channels. The .conf files are now residing in the compiled matlab directory that Rob made.

I still have yet to compile and test the new version. Its running on linux3 right now, but feel free to kill it and compile it to run on Mafalda.

It should be making trends overnight and so we can finally see what the undergrads are really up to.
  284   Tue Jan 29 14:56:39 2008 robUpdateDMFseisBLRMS 1.0

The seisBLRMS 1.0 program crashed at ~7:20 pm last night, so we didn't get data from overnight. It crashed when framecaching failed. I added
a try-catch-end statement around the call to dttfft2 to let the program survive this, then compiled it and started it on mafalda. After ~45 minutes, the compiled version encountered the same error, and while it didn't crash per se, after 20 minutes it still wasn't able to read data. We may have to dig deeper into the guts of mDV to make this stuff run more robustly.
  287   Wed Jan 30 20:39:31 2008 robUpdateDMFseisBLRMS


In order to reduce the probability of seisBLRMS crashing due to unavailability of data, I edited seisBLRMS.m so that it displays data from 6 minutes in the past, rather than 3. After compiling, this version ran for ~8 hours without crashing. I've killed the process now because it seems to interfere with alignment scripts that use ezcademod, causing "DATA RECEIVING ERROR 4608" messages. These don't cause ezcademod to crash, but so many of them are spit out that the scripts don't work very well. I guess running DMF constantly is just making the framebuilder work too hard with disk access. In the near term, we can maybe work around this by having DMF programs check the AutoDithering bit of the IFO state vector, and just not try to get data when we're running these sorts of scripts.
  291   Fri Feb 1 12:37:39 2008 robUpdateDMFseisBLRMS trends

Here are DV trends of the output of seisBLRMS over the last ~36 hours (which is how long it's been running), and another of the last 2 hours (which show the construction crew taking what appears to be a lunch break).
Attachment 1: seis36hours.png
seis36hours.png
Attachment 2: seis2hours.png
seis2hours.png
  312   Tue Feb 12 16:34:07 2008 robDAQDMFseisBLRMS 1.1
The compiled version of seisBLRMS had been running ~2 weeks without crashing as of last night, when I killed it
so it wouldn't interfere with alignment scripts.  I added an EPICS channel C1:DMF-ENABLE, and updated the DMF
executables to check this channel while running.  So far it seems to work.  When you're running alignment acripts,
simply click the DISABLE button on the C1DMF_MASTER.adl screen, and then re-ENABLE when the scripts finish. 
It's not clear why this is necessary though.  Theories include the constant disk access is keeping the
framebuilder busy, reducing its ability to deal with ezcademod commands and DMF programs just flooding the
network with so much traffic that ezcademod-related packets run late and get ignored.

Also, for reasons of aesthetics, I changed the data delay from 6 minutes to 5 minutes.  We'll see if that's enough.
  316   Thu Feb 14 15:04:53 2008 robDAQDMFDMF delay

Sometime ago I edited seisBLRMS to keep of track of how long it was taking to write RMS data (that is, the delay between the accelerometer data and the write of the EPICS rms data). Here's a plot of that info, showing how the delay increases over time. I think this indicates a logical flaw in the timing of the seisBLRMS program, which sort of relies on everything running well consistently; this should not be difficult to fix. I'll maybe try increasing the delay to ~10 minutes, and making it relatively inflexible.
Attachment 1: DMFdelay.png
DMFdelay.png
  317   Thu Feb 14 15:05:18 2008 robDAQDMFseisBLRMS 1.1
> 
> Also, for reasons of aesthetics, I changed the data delay from 6 minutes to 5 minutes.  We'll see if that's enough.

5 minutes didn't work.  
  390   Fri Mar 21 17:01:21 2008 ranaConfigurationDMFLocale change on Mafalda & seisBLRMS restart
Ever since we moved the accelerometers to be around the MC and changed their names, the seisBLRMS
has not been working. I tried to restart it today after fixing the channel names in the par file
but I ran into a PERL / UBUNTU bug.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LC_TIME = "en_US.ISO8859-1",
        LC_MONETARY = "en_US.ISO8859-1",
        LC_CTYPE = "en_US.ISO8859-1",
        LC_COLLATE = "en_US.ISO8859-1",
        LC_MESSAGES = "C",
        LC_NUMERIC = "en_US.ISO8859-1",
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

I don't know how this crept up or when it started. There were a bunch of fixes on the Ubuntu
forums which didn't work.

In the end I just set the 'unset' environment variables via our cshrc.40m file and this seemed
to make ligotools/perl happy. Lets hope this lasts...I love Linux.
  392   Fri Mar 21 23:17:47 2008 ranaConfigurationDMFseisBLRMS restarted
I updated the seisBLRMS par file with the new channel names of the accelerometers and the seismometer and then
recompiled the code and restarted it according to Rob's elog entry. It went fine and the seisBLRMS is now back in
action
.
  1230   Thu Jan 15 22:30:32 2009 ranaConfigurationDMFDMF start script
I tried to restart the DMF using the start_all script: http://dziban.ligo.caltech.edu:40/40m/280

it didn't work Frown
  1232   Fri Jan 16 11:33:59 2009 robConfigurationDMFDMF start script

Quote:
I tried to restart the DMF using the start_all script: http://dziban.ligo.caltech.edu:40/40m/280

it didn't work Frown


It should work soon. The PATH on mafalda does not include ".", so I added a line to the start_DMF subscript, which sets up the DMF ENV, to prepend this to the path before starting the tools. I didn't put it in the primary login path (such as in the .cshrc file) because Steve objects on philosophical grounds.

Also, the epics tools in general (such as tdsread) on mafalda were not working, due to PATH shenanigans and missing caRepeaters. Yoichi is harmonizing it.
  1359   Thu Mar 5 01:09:29 2009 rana, albertoUpdateDMFstill not working
We tried to run DMF on mafalda, but it didn't work. I tried to compile it using Rob's elog instructions.
On mafalda, I started matlab and ran the mdv_config.m to set up mDV. I tested that the seisBLRMS.m
script ran and correctly produced changes in the seisBLRMS strip tool. but when I tried to compile it I got:
>> mcc -v -m -R -nojvm seisBLRMS.m
Warning: Duplicate directory name: /cvs/cds/caltech/apps/linux/matlab/toolbox/local.
Compiler version: 4.6 (R2007a)
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/matlab/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/signal/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/control/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/filterdesign/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/shared/controllib/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/ident/mcc.enc
Warning: an error occurred while parsing class FilterDesignDialog.AbstractEditor:
Undefined function or variable 'DAStudio.Object'.

> In /cvs/cds/caltech/apps/linux/matlab/toolbox/shared/filterdesignlib/@FilterDesignDialog/@CoeffEditor/schema.p>schema at 9
Warning: an error occurred while parsing class FilterDesignDialog.CoeffEditor:
Invalid superclass handle.

Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/fixedpoint/mcc.enc
terminate called after throwing an instance of 'ApplicationRedefinedException*'
Abort (core dumped)
"/cvs/cds/caltech/apps/linux/matlab/bin/mcc" -E "/tmp/fileRnU5Qj_31324": Aborted
??? Error executing mcc, return status = 134.

In the meantime, I've started up a green terminal on allegra which is ssh'd into megatron.
On megatron there is a regular matlab session which is running seisBLRMS.m as a matlab script
and the seis DMF channels are getting updated.
  1379   Mon Mar 9 19:33:10 2009 ranaUpdateDMFseisBLRMS in temp condition

The seisBLRMS has been running on megatron via an open terminal ssh'd into there from allegra with matlab running. This

is because I couldn't get the compiled matlab functionality to work.

Even so, this running script has been dying lately because of some bogus 'NDS' error. So for today I

have set the NDS server for mDV on megatron to be fb40m:8088 instead of nodus.ligo.caltech.edu. If this seems to fix the problem

I will make this permanent by putting in a case statement to check whether or not the mDV'ing machine is a 40m-martian or not.

Attachment 1: Untitled.png
Untitled.png
  1392   Thu Mar 12 00:29:39 2009 JenneOmnistructureDMFDMF being whiny again

Quote:
The seisBLRMS has been running on megatron via an open terminal ssh'd into there from allegra with matlab running.


[Yoichi, Jenne]

seisBLRMS was down again. I assumed it was just because the DMF Master Enable was in the 'Disabled' state, but enabling it didn't do the trick. Rana's green terminal window was complaining about not being able to find nodus.ligo.caltech.edu. Yoichi and I stopped it, closed and restarted Matlab, ran mdv_config, then ran seisBLRMS again, and it seems happy now.

On the todo list still is making the DMF / seisBLRMS stuff happy all the time.
  1400   Fri Mar 13 19:26:09 2009 YoichiUpdateDMFseisBLRMS compiled

 I compiled seisBLRMS.

The tricks were the following:

(1) Don't add path in a deployed command.
It does not make sense to add paths in a compiled command because it may be moved to anywhere. Moreover, it can cause some weird side effects. Therefore, I enclosed the addpath part of mdv_config.m in a "if ~isdeployed ... end" clause to avoid adding paths when deployed. Instead of adding paths in the code, we have to add paths to necessary files with -I options at the compilation time. This way, mcc will add all the necessary files into the CTF archive.

(2) Add mex files to the CTF archive by -a options.
For some reason, mcc does not add necessary mex files into the CTF archive even though those files are called in the m-file which is being compiled. We have to add those files by -a options.

(3) NDS_GetData() is slow for nodus when compiled.
NDS_GetData(), which is called by get_data() stops for a few minutes when using nodus as an NDS server.
This problem does not happen when not compiled. I don't know the reason. To avoid this, I modified seisBLRMS.m so that when an environmental variable $NDS is defined, it will use an NDS server defined in this variable.

I wrote a Makefile to compile seisBLRMS. You can read the file to see the details of the tricks.
I also wrote a script start_seisBLRMS, which can be found in /cvs/cds/caltech/apps/DMF/compiled_matlab/seisblrms/. To start seisBLRMS, you can just call this script.
At this moment, seisBLRMS is running on megatron. Let's see if it continues to run without crashing.

Quote:

The seisBLRMS has been running on megatron via an open terminal ssh'd into there from allegra with matlab running. This

is because I couldn't get the compiled matlab functionality to work.

Even so, this running script has been dying lately because of some bogus 'NDS' error. So for today I

have set the NDS server for mDV on megatron to be fb40m:8088 instead of nodus.ligo.caltech.edu. If this seems to fix the problem

I will make this permanent by putting in a case statement to check whether or not the mDV'ing machine is a 40m-martian or not.

  1416   Sun Mar 22 22:47:58 2009 ranaUpdateDMFseisBLRMS compiled but still dying
Looks like seisBLRMS was restarted ~1 AM Friday morning but only lasted for 5 hours. I just restarted it on megatron;
let's see how it does. I'm not optimistic.
  1484   Wed Apr 15 02:20:46 2009 rana, yoichiUpdateDMFDMF now working copy

We found that DMF/ was not an SVN working copy, so I wiped out the SVN version, imported the on-disk copy, moved it to DMFold/ and then checked out the SVN version.

We can delete DMFold/ whenever we are happy with the SVN copy.

  1485   Wed Apr 15 03:52:27 2009 ranaUpdateDMFNDS client32 updated for DMF
 Since our seisBLRMS.m complains about 'can't find hostname' after a few hours, even though matlab is able to ping fb40m, 
I have recompiled the NDS mex client for 32-bit linux on mafalda and stuck it into the nds_mexs/ directory. This time I 
 

 compiled using the 'gcc' compiler instead of the 'ANSI C' compiler that is recommended in the README (which, I notice,

 

 is now missing from Ben Johnsons web page!). Let's see how long this runs.

  1977   Tue Sep 8 19:36:52 2009 JenneOmnistructureDMFDMF restarted

I (think I) restarted DMF.  It's on Mafalda, running in matlab (not the complied version which Rana was having trouble with back in the day).  To start Matlab, I did "nohup matlab", ran mdv_config, then started seisBLRMS.m running.  Since I used nohup, I then closed the terminal window, and am crossing my fingers in hopes that it continues to work.  I would have used Screen, but that doesn't seem to work on Mafalda.

  1979   Tue Sep 8 20:25:03 2009 JenneOmnistructureDMFDMF restarted

Quote:

I (think I) restarted DMF.  It's on Mafalda, running in matlab (not the complied version which Rana was having trouble with back in the day).  To start Matlab, I did "nohup matlab", ran mdv_config, then started seisBLRMS.m running.  Since I used nohup, I then closed the terminal window, and am crossing my fingers in hopes that it continues to work.  I would have used Screen, but that doesn't seem to work on Mafalda.

 Just kidding. That plan didn't work.  The new plan: I started a terminal window on Op540, which is ssh-ed into Mafalda, and started up matlab to run seisBLRMS.  That window is still open. 

Because Unix was being finicky, I had to open an xterm window (xterm -bg green -fg black), and then ssh to mafalda and run matlab there.  The symptoms which led to this were that even though in a regular terminal window on Op540, ssh-ed to mafalda, I could access tconvert, I could not make gps.m work in matlab.  When Rana ssh-ed from Allegra to Op540 to Mafalda and ran matlab, he could get gps.m to work.  So it seems like it was  a Unix terminal crazy thing. Anyhow, starting an xterm window on Op540m and ssh-ing to mafalda from there seemed to work.

Hopefully this having a terminal window open and running DMF will be a temporary solution, and we can get the compiled version to work again soon.

  2072   Thu Oct 8 22:17:15 2009 ranaConfigurationDMFinput channels changed

I changed the input channels of the DMF recently so that it now uses 3 Guralp channels in addition to the 3 ACC and 1 Ranger.

op440m:seisblrms>diff seisBLRMS-datachans.txt~ seisBLRMS-datachans.txt
4,7c4,7
< C1:PEM-ACC_MC2_X
< C1:PEM-ACC_MC2_Y
< C1:PEM-ACC_MC2_Z
< C1:PEM-SEIS_MC1_Y
---
> C1:PEM-SEIS_GUR1_X
> C1:PEM-SEIS_GUR1_Y
> C1:PEM-SEIS_GUR1_Z
> C1:PEM-SEIS_RANGER_Y
op440m:seisblrms>pwd
/cvs/cds/caltech/apps/DMF/compiled_matlab/seisblrms

The seisBLRMS channels still have the wrong names of IX and EX, but I have chosen to keep them like this so that we have a long trend. When looking at the historical seisBLRMS trend, we just have to remember that all of the sensors have been around the MC since last summer.

  3385   Sat Aug 7 21:57:56 2010 ranaSummaryDMFseisBLRMS restarted

The green xterm on op540m which is running the seisBLRMS DMF got stuck somehow ~3 days ago and lost its NDS connection. I closed the matlab session and restarted it. Seismic trends are now back online.

Attachment 1: Untitled.png
Untitled.png
  3563   Mon Sep 13 02:45:59 2010 ranaConfigurationDMFseisBLRMS restarts

I restarted the seisBLRMS DMF monitor by ssh'ing into mafalda and starting up a matlab session. I also have started a StripTool session on rossa by forwarding the process from op440m.

We need to get the modern EPICS installation onto these linux machines by copying what K. Thorne has done at LLO.

  12543   Mon Oct 10 17:27:29 2016 ranaUpdateDMFsummar pages dead again

Been non-functional for 3 weeks. Anyone else notice this? images missing since ~Sep 21.

  12544   Mon Oct 10 17:42:47 2016 Max IsiUpdateDMFsummar pages dead again

I've re-submitted the Condor job; pages should be back within the hour.

Quote:

Been non-functional for 3 weeks. Anyone else notice this? images missing since ~Sep 21.

 

  12548   Tue Oct 11 08:09:46 2016 Max IsiUpdateDMFsummar pages dead again

Summary pages will be unavailable today due to LDAS server maintenance. This is unrelated to the issue that Rana reported.

Quote:

I've re-submitted the Condor job; pages should be back within the hour.

Quote:

Been non-functional for 3 weeks. Anyone else notice this? images missing since ~Sep 21.

 

 

ELOG V3.1.3-