40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 26 of 330  Not logged in ELOG logo
ID Date Author Typeup Category Subject
  15425   Tue Jun 23 17:54:56 2020 ranaConfigurationVACVac maintenance complete

I propose we go for all CAPS for all channel names. The lower case names is just a holdover from Steve/Alan from the 90's. All other systems are all CAPS.

It avoids us having to force them all to UPPER in the scripts and channel lists.

  15446   Wed Jul 1 18:03:04 2020 JonConfigurationVACUPS replacements

​I looked into how the new UPS devices suggested by Chub would communicate with the vac interlocks. There are several possible ways, listed in order of preference:

  • Python interlock service directly queries the UPS via a USB link using the (unofficial) tripplite package. Direct communication would be ideal because it avoids introducing a dependency on third-party software outside the monitoring/control capability of the interlock manager. However the documentation warns this package does not work for all models...
  • Configure Tripp Lite's proprietary software (PowerAlert Local) to send SYSLOG event messages (UDP packets) to a socket monitored by the Python interlock manager.
  • Configure the proprietary software to execute a custom script upon an event occurring. The script would, e.g., set an EPICS flag channel which the interlock manager is continually monitoring.

I recommend we proceed with ordering the Tripp Lite 36HW20 for TP1 and Tripp Lite 1AYA6 for TP2 and TP3 (and other 120V electronics). As far as I can tell, the only difference between the two 120V options is that the 6FXN4 model is TAA-compliant.

  15465   Thu Jul 9 18:00:35 2020 JonConfigurationVACUPS replacements

Chub has placed the order for two new UPS units (115V for TP2/3 and a 220V version for TP1).

They will arrive within the next two weeks.

Quote:

​I looked into how the new UPS devices suggested by Chub would communicate with the vac interlocks. There are several possible ways, listed in order of preference:

  • Python interlock service directly queries the UPS via a USB link using the (unofficial) tripplite package. Direct communication would be ideal because it avoids introducing a dependency on third-party software outside the monitoring/control capability of the interlock manager. However the documentation warns this package does not work for all models...
  • Configure Tripp Lite's proprietary software (PowerAlert Local) to send SYSLOG event messages (UDP packets) to a socket monitored by the Python interlock manager.
  • Configure the proprietary software to execute a custom script upon an event occurring. The script would, e.g., set an EPICS flag channel which the interlock manager is continually monitoring.

I recommend we proceed with ordering the Tripp Lite 36HW20 for TP1 and Tripp Lite 1AYA6 for TP2 and TP3 (and other 120V electronics). As far as I can tell, the only difference between the two 120V options is that the 6FXN4 model is TAA-compliant.

  15510   Sat Aug 8 07:36:52 2020 Sanika KhadkikarConfigurationCalibration-RepairBS Seismometer - Multi-channel calibration

Summary : 

I have been working on analyzing the seismic data obtained from the 3 seismometers present in the lab. I noticed while looking at the combined time series and the gain plots of the 3 seismometers that there is some error in the calibration of the BS seismometer. The EX and the EY seismometers seem to be well-calibrated as opposed to the BS seismometer.

The calibration factors have been determined to be :

BS-X Channel: \small {\color{Blue} 2.030 \pm 0.079 }

BS-Y Channel: \small {\color{Blue} 2.840 \pm 0.177 }

BS-Z Channel: \small {\color{Blue} 1.397 \pm 0.182 }


Details :

The seismometers each have 3 channels i.e X, Y, and Z for measuring the displacements in all the 3 directions. The X channels of the three seismometers should more or less be coherent in the absence of any seismic excitation with the gain amongst all the similar channels being 1. So is the case with the Y and Z channels. After analyzing multiple datasets, it was observed that the values of all the three channels of the BS seismometer differed very significantly from their corresponding channels in the EX and the EY seismometers and they were not calibrated in the region that they were found to be coherent as well. 


Method :

Note: All the frequency domain plots that have been calculated are for a sampling rate of 32 Hz. The plots were found to be extremely coherent in a certain frequency range i.e ~0.1 Hz to 2 Hz so this frequency range is used to understand the relative calibration errors. The spread around the function is because of the error caused by coherence values differing from unity and the averages performed for the Welch function. 9 averages have been performed for the following analysis keeping in mind the needed frequency resolution(~0.01Hz) and the accuracy of the power calculated at every frequency. 

  1. I first analyzed the regions in which the similar channels were found to be coherent to have a proper gain analysis. The EY seismometer was found to be the most stable one so it has been used as a reference. I saw the coherence between similar channels of the 2 seismometers and the bode plots together. A transfer function estimator was used to analyze the relative calibration in between all 3 pairs of seismometers. In the given frequency range EX and EY have a gain of 1 so their relative calibration is proper. The relative calibration in between the BS and the EY seismometers is not proper as the resultant gain is not 1. The attached plots show the discrepancies clearly : 
  • BS-X & EY-X Transfer Function : Attachment #1
  • BS-Y & EY-Y Transfer Function : Attachment #2

          The gain in the given frequency range is ~3. The phase plotting also shows a 180-degree phase as opposed to 0 so a negative sign would also be required in the calibration factor. Thus the calibration factor for the Y channel of the BS seismometer should be around ~3. 

  • BS-Z & EY-Z Transfer Function : Attachment #3

The mean value of the gain in the given frequency range is the desired calibration factor and the error would be the mean of the error for the gain dataset chosen which is caused due to factors mentioned above.

Note: The standard error envelope plotted in the attached graphs is calculated as follows :

         1. Divide the data into n segments according to the resolution wanted for the Welch averaging to be performed later. 

         2. Calculate PSD for every segment (no averaging).

         3. Calculate the standard error for every value in the data segment by looking at distribution formed by the n number values we obtain by taking that respective value from every segment.

Discussions :

The BS seismometer is a different model than the EX and the EY seismometers which might be a major cause as to why we need special calibration for the BS seismometer while EX and EY are fine. The sign flip in the BS-Y seismometer may cause a lot of errors in future data acquisitions. The time series plots in Attachment #4 shows an evident DC offset present in the data. All of the information mentioned above indicates that there is some electrical or mechanical defect present in the seismometer and may require a reset. Kindly let me know if and when the seismometer is reset so that I can calibrate it again. 

Attachment 1: BS_X-EY_X.png
BS_X-EY_X.png
Attachment 2: BS_Y-EY_Y.png
BS_Y-EY_Y.png
Attachment 3: BS_Z-EY_Z.png
BS_Z-EY_Z.png
Attachment 4: timeseries.png
timeseries.png
  15526   Fri Aug 14 10:10:56 2020 JonConfigurationVACVacuum repairs today

The vac system is going down now for planned repairs [ELOG 15499]. It will likely take most of the day. Will advise when it's back up.

  15527   Sat Aug 15 02:02:13 2020 JonConfigurationVACVacuum repairs today

Vacuum work is completed. The TP2 and TP3 interlocks have been overhauled as proposed in ELOG 15499 and seem to be performing reliably. We're now back in the nominal system state, with TP2 again backing for TP1 and TP3 pumping the annuli. I'll post the full implementation details in the morning.

I did not get to setting up the new UPS units. That will have to be scheduled for another day.

Quote:

The vac system is going down now for planned repairs [ELOG 15499]. It will likely take most of the day. Will advise when it's back up.

  15528   Sat Aug 15 15:12:22 2020 JonConfigurationVACOverhaul of small turbo pump interlocks

Summary

Yesterday I completed the switchover of small turbo pump interlocks as proposed in ELOG 15499. This overhaul altogether eliminates the dependency on RS232 readbacks, which had become unreliable (glitchy) in both controllers. In their place, the V4(5) valve-close interlocks are now predicated on an analog controller output whose voltage goes high when the rotation speed is >= 80% of the nominal setpoint. The critical speed is 52.8 krpm for TP2 and 40 krpm for TP3. There already exist hardware interlocks of V4(5) using the same signals, which I have also tested.

Interlock signal

Unlike the TP1 controller, which exposes simple relays whose open/closed states are sensed by Acromags, what the TP2(3) controllers output is an energized 24V signal for controlling such a relay (output circuit pictured below). I hadn't appreciated this difference and it cost me time yesterday. The ultimate solution was to route the signals through a set of new 24V Phoenix Contact relays installed inside the Acromag chassis. However, this required removing the chassis from the rack and bringing it to the electronics bench (rather than doing the work in situ, as I had planned). The relays are mounted to the second DIN rail opposite the Acromags. Each TP2(3) signal controls the state of a relay, which in turn is sensed using an Acromag XT1111.

Signal routing

The TP2(3) "normal-speed" signals are already in use by hardware interlocks of V4(5). Each signal is routed into the main AC relay box, where it controls an "interrupter" relay through which the Acromag control signal for the main V4(5) relay is passed. These signals are now shared with the digital controls system using a passive DB15 Y-splitter. The signal routing is shown below.

Interlock conditions

The new turbo-pump-related interlock conditions and their channel predicates are listed below. The full up-to-date channel list and wiring assignments for c1vac are maintained here.

Channel Type New? Interlock-triggering condition
C1:Vac-TP1_norm BI No Rotation speed < 90% nominal setpoint (29 krpm)
C1:Vac-TP1_fail BI No Critical fault occurrence
C1:Vac-TP1_current AI No Current draw > 4 A
C1:Vac-TP2_norm BI Yes Rotation speed < 80% nominal setpoint (52.8 krpm)
C1:Vac-TP3_norm BI Yes Rotation speed < 80% nominal setpoint (40 krpm)

There are two new channels, both of which provide a binary indication of whether the pump speed is outside its nominal range. I did not have enough 24V relays to also add the C1:Vac-TP2(3)_fail channels listed in ELOG 15499. However, these signals are redundant with the existing interlocks, and the existing serial "Status" readback will already print failure messages to the MEDM screens. All of the TP2(3) serial readback channels remain, which monitor voltage, current, operational status, and temperature. The pump on/off and low-speed mode on/off controls remain implemented with serial signals as well.

The new analog readbacks have been added to the MEDM controls screens, circled below:

Other incidental repairs

  • I replaced the (dead) LED monitor at the vac controls console. In the process of finding a replacement, I came across another dead spare monitor as well. Both have been labeled "DEAD" and moved to Jordan's desk for disposal.
  • I found the current TP3 Varian V70D controller to be just as glitchy in the analog outputs as well. That likely indicates there is a problem with the microprocessor itself, not just the serial communications card as I thought might be the case. I replaced the controller with the spare unit which was mounted right next to it in the rack [ELOG 13143]. The new unit has not glitched since the time I installed it around 10 pm last night.
Attachment 1: small_tp_signal_routing.png
small_tp_signal_routing.png
Attachment 3: small_tp_signal_routing.png
small_tp_signal_routing.png
Attachment 4: medm_screen.png
medm_screen.png
  15738   Fri Dec 18 22:59:12 2020 JonConfigurationCDSUpdated CDS upgrade plan

Attached is the layout for the "intermediate" CDS upgrade option, as was discussed on Wednesday. Under this plan:

  • Existing FEs stay where they are (they are not moved to a single rack)

  • Dolphin IPC remains PCIe Gen 1

  • RFM network is entirely replaced with Dolphin IPC

Please send me any omissions or corrections to the layout.

Attachment 1: CDS_2020_Dec.pdf
CDS_2020_Dec.pdf
Attachment 2: CDS_2020_Dec.graffle
  15742   Mon Dec 21 09:28:50 2020 JamieConfigurationCDSUpdated CDS upgrade plan
Quote:

Attached is the layout for the "intermediate" CDS upgrade option, as was discussed on Wednesday. Under this plan:

  • Existing FEs stay where they are (they are not moved to a single rack)

  • Dolphin IPC remains PCIe Gen 1

  • RFM network is entirely replaced with Dolphin IPC

Please send me any omissions or corrections to the layout.

I just want to point out that if you move all the FEs to the same rack they can all be connected to the Dolphin switch via copper, and you would only have to string a single fiber to every IO rack, rather than the multiple now (for network, dolphin, timing, etc.).

  15746   Wed Dec 23 23:06:45 2020 gautamConfigurationCDSUpdated CDS upgrade plan
  1. The diagram should clearly show the host machines and the expansion chassis and the interconnects between them.
  2. We no longer have any Gentoo bootserver or diskless FEs.
  3. The "c1lsc" host is in 1X4 not 1Y3.
  4. The connection between c1lsc and Dolphin switch is copper not fiber. I don't know how many Gbps it is. But if the switch is 10 Gbps, are they really selling interface cables that have lower speed? The datasheet says 10 Gbps.
  5. The control room workstations - Debian10 (rossa) is the way forward I believe. it is true pianosa remains SL7 (and we should continue to keep it so until all other machines have been upgraded and tested on Debian 10).
  6. There is no "IOO/OAF". The host is called "c1ioo".
  7. The interconnect between Dolphin switch and c1ioo host is via fiber not copper.
  8. It'd be good to have an accurate diagram of the current situation as well (with the RFM network).
  9. I'm not sure if the 1Y1 rack can accommodate 2 FEs and 2 expansion chassis. Maybe if we clear everything else there out...
  10. There are 2 "2GB/s" Copper traces. I think the legend should make clear what's going on - i.e. which cables are ethernet (Cat 6? Cat 5? What's the speed limitation? The cable? Or the switch?), which are PCIe cables etc etc. 

I don't have omnigraffle - what about uploading the source doc in a format that the excellent (and free) draw.io can handle? I think we can do a much better job of making this diagram reflect reality. There should also be a corresponding diagram for the Acromag system (but that doesn't have to be tied to this task). Megatron (scripts machine) and nodus should be added to that diagram as well.

Please send me any omissions or corrections to the layout.

  15771   Tue Jan 19 14:05:25 2021 JonConfigurationCDSUpdated CDS upgrade plan

I've produced updated diagrams of the CDS layout, taking the comments in 15476 into account. I've also converted the 40m's diagrams from Omnigraffle ($150/license) to the free, cloud-based platform draw.io. I had never heard of draw.io, but I found that it has most all the same functionality. It also integrates nicely with Google Drive.

Attachment 1: The planned CDS upgrade (2 new FEs, fully replace RFM network with Gen 1 Dolphin IPC)
Attachment 2: The current 40m CDS topology

The most up-to-date diagrams are hosted at the following links:

Please send me any further corrections or omissions. Anyone logged in with LIGO.ORG credentials can also directly edit the diagrams.

Attachment 1: 40m_CDS_Network_-_Planned.pdf
40m_CDS_Network_-_Planned.pdf
Attachment 2: 40m_CDS_Network_-_Current.pdf
40m_CDS_Network_-_Current.pdf
  15772   Tue Jan 19 15:43:24 2021 gautamConfigurationCDSUpdated CDS upgrade plan

Not sure if 1Y1 can accommodate both c1sus2 and c1bhd as well as the various electronics chassis that will have to be installed. There may need to be some distribution between 1Y1 and 1Y3. Does Koji's new wiring also specify which racks hold which chassis?

Some minor improvements to the diagram:

  1. The GPS receiver in 1X7 should be added. All the timing in the lab is synced to the 1pps from this.
  2. We should add hyperlinks to the various parts datasheets (e.g. Dolphin switch, RFM switch, etc etc) so that the diagram will be truly informative and self-contained.
  3. Megatron and nodus, but especially chiara (NFS server), should be added to the diagram. 
  15921   Mon Mar 15 20:40:01 2021 ranaConfigurationComputersinstalled QTgrace on donatella for dataviewer

I installed QTgrace using yum on donatella.angel Both Grace and XMgrace are broken due to some boring fight between the Fedora package maintainers and the (non existent) Grace support team. So I have symlinked it:

controls@donatella|bin> sudo mv xmgrace xmgrace_bak
controls@donatella|bin> sudo ln -s qtgrace xmgrace
controls@donatella|bin> pwd
/usr/bin

I checked that dataviewer works now for realtime and playback.cool Although the middle click paste on the mouse doesn't work yet.angry

Attachment 1: cutiegrace.png
cutiegrace.png
  15928   Wed Mar 17 09:05:01 2021 Paco, AnchalConfigurationComputers40m Control Room Changes
  • Switched positions of allegra and donatella.
  • While doing so, the hdmi cable previously used by donatella snapped. We replaced this cable by another unused cable we found connected only on one end to rossa. We should get more HDMI cables if that cable was in use for some other purpose.
  • Paco bought a bluetooth speaker/mic that is placed infront of allegra and it's usb adapter is connected to iMac's keyboard in the bottom. With the new camera installed, the 40m video call environment is now complete.
  • Again, we have placed allegra's monitor for place holder but it is not working and we need new monitors for it in future whenever it is going to be used.
  16027   Wed Apr 14 13:16:20 2021 AnchalConfigurationComputers40m Control Room Changes
  • I have confirmed that the old two monitors' backlighting is not working. One can see the impression of the display without any brightness on them. Both old monitors are on the shelf behind.
  • Today we got a monitor and mouse from Mike. I had to change /etc/default/grub GRUB_GFXMODE to 1920x1200@30 on allegra for it to work with the(any) monitor.
  • Allegra is Debian 10 with latest cds-workstation installed on it. It is a good test station to migrate our existing scripts to start using updated cds-workstation configuration.
Quote:
  • Again, we have placed allegra's monitor for place holder but it is not working and we need new monitors for it in future whenever it is going to be used.

 

  16163   Wed May 26 11:45:57 2021 Anchal, PacoConfigurationIMCMC2 analog camera

[Anchal, Paco]

We went near the MC2 area and opened the lid to inspect the GigE and analog video monitors for MC2. Looked like whatever image is coming through the viewport is split into the GigE (for beam tracking) and the analog monitor. We hooked the monitor found on the floor nearby and tweaked the analog video camera around to get a feel for how the "ghost" image of the transmission moves around. It looks like in order to try and remove this "extra spots" we would need to tweak the beam tracking BS. We will consult the beam tracking authorities and return to this.

  16302   Thu Aug 26 10:30:14 2021 JamieConfigurationCDSfront end time synchronization fixed?

I've been looking at why the front end NTP time synchronization did not seem to be working.  I think it might not have been working because the NTP server the front ends were point to, fb1, was not actually responding to synchronization requests.

I cleaned up some things on fb1 and the front ends, which I think unstuck things.

On fb1:

  • stopped/disabled the default client (systemd-timesyncd), and properly installed the full NTP server (ntp)
  • the ntp server package for debian jessie is old-style sysVinit, not systemd.  In order to make it more integrated I copied the auto-generated service file to /etc/systemd/system/ntp.service, and added and "[install]" section that specifies that it should be available during the default "multi-user.target".
  • "enabled" the new service to auto-start at boot ("sudo systemctl enable ntp.service") 
  • made sure ntp was configured to serve the front end network ('broadcast 192.168.123.255') and then restarted the server ("sudo systemctl restart ntp.service")

For the front ends:

  • on fb1 I chroot'd into the front-end diskless root (/diskless/root) and manually specifed that systemd-timesyncd should start on boot by creating a symlink to the timesyncd service in the multi-user.target directory:
$ sudo chroot /diskless/root
$ cd /etc/systemd/system/multi-user.target.wants
$ ln -s /lib/systemd/system/systemd-timesyncd.service
  • on the front end itself (c1iscex as a test) I did a "systemctl daemon-reload" to force it to reload the systemd config, and then restarted the client ("systemctl restart systemd-timesyncd")
  • checked the NTP synchronization with timedatectl:
controls@c1iscex:~ 0$ timedatectl 
      Local time: Thu 2021-08-26 11:35:10 PDT
  Universal time: Thu 2021-08-26 18:35:10 UTC
        RTC time: Thu 2021-08-26 18:35:10
       Time zone: America/Los_Angeles (PDT, -0700)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: yes
 Last DST change: DST began at
                  Sun 2021-03-14 01:59:59 PST
                  Sun 2021-03-14 03:00:00 PDT
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2021-11-07 01:59:59 PDT
                  Sun 2021-11-07 01:00:00 PST
controls@c1iscex:~ 0$ 

Note that it is now reporting "NTP enabled: yes" (the service is enabled to start at boot) and "NTP synchronized: yes" (synchronization is happening), neither of which it was reporting previously.  I also note that the systemd-timesyncd client service is now loaded and enabled, is no longer reporting that it is in an "Idle" state and is in fact reporting that it synchronized to the proper server, and it is logging updates:

controls@c1iscex:~ 0$ sudo systemctl status systemd-timesyncd
â— systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled)
   Active: active (running) since Thu 2021-08-26 10:20:11 PDT; 1h 22min ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 2918 (systemd-timesyn)
   Status: "Using Time Server 192.168.113.201:123 (ntpserver)."
   CGroup: /system.slice/systemd-timesyncd.service
           â””─2918 /lib/systemd/systemd-timesyncd

Aug 26 10:20:11 c1iscex systemd[1]: Started Network Time Synchronization.
Aug 26 10:20:11 c1iscex systemd-timesyncd[2918]: Using NTP server 192.168.113.201:123 (ntpserver).
Aug 26 10:20:11 c1iscex systemd-timesyncd[2918]: interval/delta/delay/jitter/drift 64s/+0.000s/0.000s/0.000s/+26ppm
Aug 26 10:21:15 c1iscex systemd-timesyncd[2918]: interval/delta/delay/jitter/drift 128s/-0.000s/0.000s/0.000s/+25ppm
Aug 26 10:23:23 c1iscex systemd-timesyncd[2918]: interval/delta/delay/jitter/drift 256s/+0.001s/0.000s/0.000s/+26ppm
Aug 26 10:27:40 c1iscex systemd-timesyncd[2918]: interval/delta/delay/jitter/drift 512s/+0.003s/0.000s/0.001s/+29ppm
Aug 26 10:36:12 c1iscex systemd-timesyncd[2918]: interval/delta/delay/jitter/drift 1024s/+0.008s/0.000s/0.003s/+33ppm
Aug 26 10:53:16 c1iscex systemd-timesyncd[2918]: interval/delta/delay/jitter/drift 2048s/-0.026s/0.000s/0.010s/+27ppm
Aug 26 11:27:24 c1iscex systemd-timesyncd[2918]: interval/delta/delay/jitter/drift 2048s/+0.009s/0.000s/0.011s/+29ppm
controls@c1iscex:~ 0$ 

So I think this means everything is working.

I then went ahead and reloaded and restarted the timesyncd services on the rest of the front ends.

We still need to confirm that everything comes up properly the next time we have an opportunity to reboot fb1 and the front ends (or the opportunity is forced upon us).

There was speculation that the NTP clients on the front ends (systemd-timesyncd) would not work on a read-only filesystem, but this doesn't seem to be true.  You can't trust everything you read on the internet.

  16614   Mon Jan 24 12:33:41 2022 ranaConfigurationWikiAIC Wiki: txz files allowed

I updated the mime.local.conf file for the AIC Wiki so as to allow attachments with the .txz format. THis should be persistent over upgrades, since its a local file.

  94   Mon Nov 12 14:09:19 2007 robDAQGeneraltpman dead on fb40m
The testpoint manager was dead on fb40m. I know I re-started it sometime after the power outage, so something must have killed it. If you get an error from DTT like
"diagnostic kernel does not support: testpoints", then log into fb40m as root, check for the tpman with a ps -ef | grep tpman. If it's not there, then run /usr/controls/tpman & and close the terminal window.
  152   Fri Nov 30 21:27:24 2007 ranaDAQPEMweather / stacis / c1pem1
I was trying to add some Seis BLRMS channels to the c1pem1 processor so that we could have DMT trends.

Then I found that none of the Weather channels have been working for a year or so. I could also not
telnet into it. I tried resetting it but no luck. There was no entry in the Wiki for it so I added
a place holder.

Have the weather channels ever worked? Do we have those sensors? I think I've never actually looked
for this. Seems like a fine ugrad job.
  157   Mon Dec 3 00:10:42 2007 ranaDAQComputer Scripts / Programslinemon
I've started up one of our first Matlab based DMT processes as a test.

There's a matlab script running on Mafalda which is measuring the height of the 60 Hz peak
in the MC1 UL SENSOR and writing it to an unused EPICS channel (PZT1_PIT_OFFSET).

The purpose of this is just to see if such a thing is stable over long periods of time. Its
open on a terminal on linux3 so it can be killed at any time if it runs amok.

Right now the code just demods the channel and tracks the absolute value of the peak. The
next upgrade will have it track the actual frequency once per minute and then report that
as well. We also have to figure out how to make it a binary and then make a single script
that launches all of the binaries.

For now you can watch its progress on the StripTool on op540m; its cheap and easy DMT viewer.
  160   Mon Dec 3 19:06:49 2007 ranaDAQComputer Scripts / Programslinemon
I turned up my nose at Matlab's special tools. I modified the linetracker to use the
relationship phase = 2*pi*f*t to estimate the frequency each minute. The
code uses 'polyfit' to get the mean and trend of the unwrapped phase and then determines
how far the initial frequency estimate was off. It then uses the updated number as the
initial guess for the next minute.

I looked at a couple hours of data before letting it run. It looks like the phase of the
'60 Hz' peak varies at 20 second time scales but not much faster or rather anything faster
would be a glitch and not a monotonic frequency drift.

From the attached snapshot you can see that the amplitude (PZT1_PIT) varies by ~10 %
and the frequency by ~40 mHz in a couple hour span.
Attachment 1: spd64d1.jpg
spd64d1.jpg
  170   Wed Dec 5 19:25:07 2007 ranaDAQCDSDMF
I made a database file on C1AUX called dmf.db. It has 9 DMF EPICS channels which are also trended
so that one can now write data to those channels from a DMF Monitor and the data will be records.

New channels:
[C1:DMF-SEIS_1]
[C1:DMF-SEIS_2]
[C1:DMF-SEIS_3]
[C1:DMF-LINE_1]
[C1:DMF-LINE_2]
[C1:DMF-LINE_3]
[C1:DMF-MC_1]
[C1:DMF-MC_2]
[C1:DMF-MC_3]

I added these to C1AUX because it doesn't do much and can be booted without having much effect.
(it controls Mech Shutters, Video, and Illuminators. It used to also do the EO Shutter but I
removed that from its startup.cmd and it will no longer load those records).
  204   Wed Dec 19 20:28:27 2007 AndreyDAQPEMNames for all 6 accelerometers have been changed

I eventually changed the names for all 6 accelerometers (see my ELOG entry # 172 from Dec. 05 about my intent to do that).

I removed the word "BS" from their names,
and I changed the word combination "ACC_BS_EAST" in the old name for "ACC_ITMX" in the new name;
as well "ACC_BS_WEST" is now replaced by "ACC_ETMX".
(the reasoning behind such a change should become clear from my ELOG entry #172).

New accelerometer names are:
(note: there are no spaces (nowhere!) in the names of accelerometers, but ELOG replaces ": P" written without a space by a strange symbol Tongue)

C1 : PEM - ACC _ ETMX _ X ;
C1 : PEM - ACC _ ETMX _ Y ;
C1 : PEM - ACC _ ETMX _ Z ;
C1 : PEM - ACC _ ITMX _ X ;
C1 : PEM - ACC _ ITMX _ Y ;
C1 : PEM - ACC _ ITMX _ Z .

One can find them in "C1 : PEM - ACC" in Dataviewer.

  312   Tue Feb 12 16:34:07 2008 robDAQDMFseisBLRMS 1.1
The compiled version of seisBLRMS had been running ~2 weeks without crashing as of last night, when I killed it
so it wouldn't interfere with alignment scripts.  I added an EPICS channel C1:DMF-ENABLE, and updated the DMF
executables to check this channel while running.  So far it seems to work.  When you're running alignment acripts,
simply click the DISABLE button on the C1DMF_MASTER.adl screen, and then re-ENABLE when the scripts finish. 
It's not clear why this is necessary though.  Theories include the constant disk access is keeping the
framebuilder busy, reducing its ability to deal with ezcademod commands and DMF programs just flooding the
network with so much traffic that ezcademod-related packets run late and get ignored.

Also, for reasons of aesthetics, I changed the data delay from 6 minutes to 5 minutes.  We'll see if that's enough.
  316   Thu Feb 14 15:04:53 2008 robDAQDMFDMF delay

Sometime ago I edited seisBLRMS to keep of track of how long it was taking to write RMS data (that is, the delay between the accelerometer data and the write of the EPICS rms data). Here's a plot of that info, showing how the delay increases over time. I think this indicates a logical flaw in the timing of the seisBLRMS program, which sort of relies on everything running well consistently; this should not be difficult to fix. I'll maybe try increasing the delay to ~10 minutes, and making it relatively inflexible.
Attachment 1: DMFdelay.png
DMFdelay.png
  317   Thu Feb 14 15:05:18 2008 robDAQDMFseisBLRMS 1.1
> 
> Also, for reasons of aesthetics, I changed the data delay from 6 minutes to 5 minutes.  We'll see if that's enough.

5 minutes didn't work.  
  358   Tue Mar 4 23:22:32 2008 robDAQComputersc1susvme1&2 rebooted

I found that some channels from c1susvme1 and c1susvme2 were not being recording by the DAQ (and were not showing up in DV). I rebooted these processors, which fix the problem. If you see other cases of this (signal exactly zero, but not a testpoint problem), just reboot the corresponding processor.
  440   Wed Apr 23 22:39:54 2008 AndreyDAQComputer Scripts / ProgramsProblem with "get_data" and slow PEM channels

It turns out that I cannot read minute trends for the slow weather channels for more than 1000 seconds back (roughly more than 15 minutes ago) using "get_data" script.

For comparison, I tried MC1 slow channels, and similar problem did not arise there. Probably, something is wrong with the memory of slow weather channels. At the same time, I can see minute-trends in Dataviewer as long ago as I want.

In response to
>>get_data('C1: PEM-weather_outsideTemp', 'minute', gps('now') - 3690, 3600);
I get the error message:
"Warning: Missong C1: PEM-weather_outsideTemp M data at 893045156".
  456   Sun Apr 27 18:11:58 2008 robDAQComputersbr40m?

The testpoint manager (which runs on fb40m) crashed this afternoon. Upon re-starting it, I found there was a rogue dtt process on op440m and also a daqd daemon running on br40m. One or both of these caused the tpman to crash. br40m is the frame broadcaster, which is never used here as we don't run DMT. I killed the daqd process there.

The way to find if there is a rogue process is to watch the output to the console from the tpman when you start it:

Allocate new TP handle 56 by 131.215.113.203
Allocate new TP handle 57 by 131.215.113.203
Allocate new TP handle 58 by 131.215.113.203
Allocate new TP handle 59 by 131.215.113.203
Allocate new TP handle 60 by 131.215.113.203
Allocate new TP handle 61 by 131.215.113.203
Allocate new TP handle 62 by 131.215.113.203
Allocate new TP handle 63 by 131.215.113.203
Allocate new TP handle 64 by 131.215.113.203
Allocate new TP handle 65 by 131.215.113.203
Allocate new TP handle 66 by 131.215.113.203
Allocate new TP handle 67 by 131.215.113.203
Allocate new TP handle 68 by 131.215.113.203


If you see something like this, with a new TP handle being allocated every few seconds, you need to log in to the corresponding host and kill whatever process has run away.
  457   Sun Apr 27 22:57:15 2008 ajwDAQComputersbr40m?

Quote:

The testpoint manager (which runs on fb40m) crashed this afternoon. Upon re-starting it, I found there was a rogue dtt process on op440m and also a daqd daemon running on br40m. One or both of these caused the tpman to crash. br40m is the frame broadcaster, which is never used here as we don't run DMT. I killed the daqd process there.

The way to find if there is a rogue process is to watch the output to the console from the tpman when you start it:

Allocate new TP handle 56 by 131.215.113.203
Allocate new TP handle 57 by 131.215.113.203
Allocate new TP handle 58 by 131.215.113.203
Allocate new TP handle 59 by 131.215.113.203
Allocate new TP handle 60 by 131.215.113.203
Allocate new TP handle 61 by 131.215.113.203
Allocate new TP handle 62 by 131.215.113.203
Allocate new TP handle 63 by 131.215.113.203
Allocate new TP handle 64 by 131.215.113.203
Allocate new TP handle 65 by 131.215.113.203
Allocate new TP handle 66 by 131.215.113.203
Allocate new TP handle 67 by 131.215.113.203
Allocate new TP handle 68 by 131.215.113.203


If you see something like this, with a new TP handle being allocated every few seconds, you need to log in to the corresponding host and kill whatever process has run away.


I *think* Alex is responsible for the daqd daemon running on br40m (he set up some new stuff recently, a data concentrator and broadcaster); I'll make sure he sees this post.
  459   Tue Apr 29 21:09:12 2008 ranaDAQCDSFE Filters
These are new FE filters for downsampling and upsampling. We will be going from native hardware sampling rates of 64k down to 32k, 16k, and 2k.

The attached plot shows these filters. They are 3dB ripple, 40 dB stopband, 4th order elliptic filters in which I have moved the zeros around
into good places (e.g. to the Nyquist frequency).

I'm also attaching the .txt file containg the filter coefficients and the design strings. The filters are called x2, x4, and x32, for the
D2, D4, and D32 downsampling, respectively.
Attachment 1: fefilters.jpg
fefilters.jpg
Attachment 2: fefilters.txt
# FILTERS FOR ONLINE SYSTEM
#
# Computer generated file: DO NOT EDIT
#
# MODULES ULYAW
#
################################################################################
### ULYAW                                                                    ###
################################################################################
# SAMPLING ULYAW 65536
... 28 more lines ...
  583   Fri Jun 27 15:20:52 2008 robDAQLSC.ini file change

I removed C1:LSC-XARM_CTRL from the frames and added C1:LSC-CARM_ERR
  640   Mon Jul 7 13:58:37 2008 Eric, josephbDAQPEMUsing unused PEM channels to test camera code
Joe and I have taken control of the EPICS channels C1:PEM-Stacis_EEEX_geo and C1:PEM-Stacis_EEEY_geo since we heard that they are no longer in use.  We are currently 
using them to test the ability for the Snap camera code to read and write from EPICS channels.  Thus, the information being written to these channels is completely unrelated
to their names or previous use.  This is only temporary; we'll create our own channels for the camera code shortly (probably within the next couple of days).

- Eric
  660   Fri Jul 11 20:16:01 2008 EricDAQCamerasTaking data from the GC 750 Camera
Mafalda has been set up with a background process to constantly take data from the GC 750 camera (at the end of the x-arm) for the weekend. This camera will otherwise be inoperable until then.

In the small chance that this slows either Mafalda or the network to a crawl, the process to kill should have PID 26265.
  671   Tue Jul 15 10:09:42 2008 EricDAQCamerasDid anyone kill the picture taking process on Mafalda?
Did anyone kill the process on Mafalda that was taking pictures of the end mirror of the x-arm last Friday? I need to know whether or not it crashed of its own accord.
  673   Tue Jul 15 11:47:56 2008 JenneDAQPEMAccelerometer channels in ASS Adapt MEDM screen
Jenne, Sharon

We have traced which accelerometers correspond to which channels in the C1ASS_TOP MEDM screen.

Accelerometer Channel
------------- --------------------------
MC1-X C1:ASS-TOP_PEM_2_ADAPT_IN1
MC1-Y C1:ASS-TOP_PEM_3_ADAPT_IN1
MC1-Z C1:ASS-TOP_PEM_4_ADAPT_IN1
MC2-X C1:ASS-TOP_PEM_5_ADAPT_IN1
MC2-Y C1:ASS-TOP_PEM_6_ADAPT_IN1
MC2-Z C1:ASS-TOP_PEM_7_ADAPT_IN1

SEISMOMETER C1:ASS-TOP_PEM_1_ADAPT_IN1
  684   Wed Jul 16 17:36:51 2008 JohnDAQPSLFSS input offset
I changed the nominal FSS input offset to 0 from 0.3. Tolerance remains unchanged at +/-0.05.
  700   Fri Jul 18 19:43:55 2008 YoichiDAQComputersPSL fast channels cannot be read by dataviewer
At this moment only the PSL fast channels have trouble.
Rob restarted fb40m, c1IOVME, but no effect.
  826   Mon Aug 11 19:09:28 2008 JenneDAQPEMSeismometer DAQ is being funny
While looking at the Ranger seismometer's output to figure out what our max typical ground motion is, Rana and I saw that the DAQ output is at a weird level. It looks like even though the input to the DAQ channel is being saturated, the channel isn't outputing as many counts as expected to Dataviewer.

Sharon and I checked that the output of the seismometer looks reasonable - sinusoidal when I tap on the seismometer, and the the output of the SR560 (preamp) is also fine, and not clipping. If I stomp on the floor, the output of the SR560 goes above 2V (to about 3V ish), so we should be saturating the DAQ, and getting the max number of counts out. However, as you can see in the first figure, taken when I was tapping the seismometer, the number of counts at saturation is well beneath 32768counts. (16 bit machine, so the +-2V of the DAQ should have a total range of 65536. +2V should correspond to +32768counts.) The second figure shows 40 days of seismometer data. It looks like we saturate the DAQ regularly.

I did a check of the DAQ using an HP6236B power supply. I sent in 1V, 2V and 2.2V (measuring the output of the power supply with a 'scope), and measured the number of counts output on the DAQ.

Input Voltage [V]Counts on DataviewerExpected counts from 16 bit machine
11898316384
22933132768
2.22934732768


I'm not sure why the +1V output more than the expected number of counts (unless I mis-measured the output from the power supply).

Moral of the story is...when the DAQ is saturated, it is not outputting the expected number of counts. To be explored further tomorrow...
Attachment 1: SeisDAQ.png
SeisDAQ.png
Attachment 2: SeisData.png
SeisData.png
  917   Wed Sep 3 19:09:56 2008 YoichiDAQComputersc1iovme power cycled
When I tried to measure the sideband power of the FSS using the scan of the reference cavity, I noticed that the RC trans. PD signal was not
properly recorded by the frame builder.
Joe restarted c1iovme software wise. The medm screen said c1iovme is running fine, and actually some values were recorded by the FB.
Nonetheless, I couldn't see flashes of the RC when I scanned the laser frequency.
I ended up power cycling the c1iovme and run the restart script again. Now the signals recorded by c1iovme look fine.
Probably, the DAQ boards were not properly initialized only by the software reset.
I will re-try the sideband measurement tomorrow morning.
  1028   Mon Oct 6 12:45:41 2008 AlbertoDAQLSCC1LSC in coma
Alberto, Joe,

The C1LSC medm screen is frozen and the C1LSC computer is down. We tried to reboot and to restart it first turning off the power and then just rebooting remotely. None of them worked. We check whether any of the cable was unplugged but they were ok. Also all the led turned on to green after rebooting.
Trying to reboot we get the following error message: init_module: device or resource busy.

We called Alex who first suggested to check all the connection and then to swap the timing cable between two Pentek boards but the computer was still down.
It is possible that the board is dead. Alex and Rolf are going to look into this problem and for any spare board.

by now we can't lock any DOF of the IFO.
  1029   Mon Oct 6 16:41:33 2008 AlbertoDAQLSCC1LSC in coma

Quote:
Alberto, Joe,

The C1LSC medm screen is frozen and the C1LSC computer is down. We tried to reboot and to restart it first turning off the power and then just rebooting remotely. None of them worked. We check whether any of the cable was unplugged but they were ok. Also all the led turned on to green after rebooting.
Trying to reboot we get the following error message: init_module: device or resource busy.

We called Alex who first suggested to check all the connection and then to swap the timing cable between two Pentek boards but the computer was still down.
It is possible that the board is dead. Alex and Rolf are going to look into this problem and for any spare board.

by now we can't lock any DOF of the IFO.


Alex, Rob, Alberto,

Alex replaced the Pentek board used by C1LSC with a spare one that they had at the Wilson house. That fixed the DMA failure but since the board had a channel broken, one of the channels (POY) was still not available.

Looking at the wiring diagram of the ASC crate, we found that one of the Pentek boards in it was actually not used by anything and thus available to replace the bad one in LSC. We switched the two boards so that now the one that Alex installed is mounted in the ASC crate and it is connected to the cable labeled 1x2-ASC 6.

C1LSC is up again and all the channels in the C1LSC screen, including POY, now seem to be working properly.
  1066   Wed Oct 22 09:42:41 2008 AlbertoDAQComputersc1iool0 rebooted and MC autolocker restarted
This morning I found the MC unlocked. The MC-Down script didn't work because of network problems in communicating with scipe7, a.k.a. c1iool0. Telneting to the computer was also impossible so I power cycled it from its key switch. The first time it failed so I repeated it a second time and then it worked.
Yoichi then restarted c1iovme. It was also necessary to restart the MC autolocker script according to the following procedure:
- ssh into op440m
- from op440m, ssh into op340m
- restart /cvs/cds/caltech/scripts/scripts/MC/autolockMCmain40
  1114   Tue Nov 4 17:58:42 2008 AlbertoDAQPSLMC temperature sensor
I added a channel for the temperature sensor on the MC1/MC3 chamber: C1:PSL-MC_TEMP_SEN.
To do that I had to reboot the frame builder. The slow servo of the FSS had to get restarted, the reference cavity locked and so the PMC and MZ.
  1124   Fri Nov 7 18:38:19 2008 AlbertoDAQPSLMC temperature sensor hooked up
Alberto, Rana,
we found that the computer handling the signals from ICS-110B was C1IOVME so we restarted it. We changed the name of the channel to C1:PEM_TEMPS and the number to 16349. We tracked it up to the J14 connector of the DAQ.
We also observed the strange thing that both of the differential pairs on J13 are read by the channle. Also, if you connect a 50 Ohm terminator to one of the pairs, the signal even get amplified.

(The name of the channel is PEM-MC1_TEMPS)
  1130   Wed Nov 12 11:14:59 2008 CarynDAQPSLMC temp sensor hooked up incorrectly
MC Temperature sensor was not hooked up correctly. It turns out that for the 4 pin LEMO connections on the DAQ like J13, J14, etc. the channels correspond to horizontal pairs on the 4 pin LEMO. The connector we used for the temp sensor had vertical pairs connected to each BNC which resulted in both the differential pairs on J13 being read by the channel.
To check that a horizontal pair 4 pin LEMO2BNC connector actually worked correctly we unlocked the mode cleaner, and borrowed a connector that was hooked up to the MC servo (J8a). We applied a sine wave to each of the BNCs on the connector, checked the J13 signal and only one of the differential pairs on J13 was being read by the channel. So, horizontal pairs worked.
  1143   Tue Nov 18 13:28:08 2008 CarynDAQIOOnew channel for MC drum modes
Alberto has added a channel for the Mode Cleaner drum modes.
C1:IOO-MC_DRUM1
sample rate-2048
chnum-13648
  1228   Wed Jan 14 15:53:32 2009 steveDAQPSLMC temperature sensor

Quote:
I added a channel for the temperature sensor on the MC1/MC3 chamber: C1:PSL-MC_TEMP_SEN.
To do that I had to reboot the frame builder. The slow servo of the FSS had to get restarted, the reference cavity locked and so the PMC and MZ.


Where is this channel?
  1246   Thu Jan 22 14:38:41 2009 carynDAQPSLMC temperature sensor

Quote:

Quote:
I added a channel for the temperature sensor on the MC1/MC3 chamber: C1:PSL-MC_TEMP_SEN.
To do that I had to reboot the frame builder. The slow servo of the FSS had to get restarted, the reference cavity locked and so the PMC and MZ.


Where is this channel?


That's not the name of the channel anymore. The channel name is PEM-MC1_TEMPS. It's written in a later entry.
ELOG V3.1.3-