Rough summary of today's progress:
I didn't really see anything out of the ordinary on the ETMX suspension. Earthquake stops had clearance, OSEMS were secure, no visible glue degredation on face magnets. Inspection with green LED flashlight didn't reveal any obscene dirtieness on either face, just a few particles here and there. The top of the opic barrel unsurprisingly has a good amount of particulate. The wire grooves are way too small to resolve anything at this point, other than that they exist.
The suspension footprint is already marked, tomorrow we can move the suspension closer to the door to get an even closer look at it, before removing it from the chamber.
After hardware errors prevented me from using optimus, I switched my generation of summary pages back to the clusters. A day's worth of data is still too much to process using one computer, but I have successfully made summary pages for a timescales of a couple of hours on this site: https://ldas-jobs.ligo.caltech.edu/~praful.vasireddy/
Currently, I'm working on learning the current plot-generation code so that it can eventually be modified to include an interactive component (e.g., hovering over a point on a timeseries would display the GPS time). Also, the 40m summary pages have been down for the past 3 weeks but should be up and working soon as the clusters are now alive.
I'd suggest clamping and moving it to the flow bench so you can inspect with a bright light. Then remove the wire and inspect the standoff, but hurry up with getting it in the soak bath so you can start on the cleaning of the other ones.
I wonder if we're really sure that its a mechanical problem with ETMX.
Gautam tells me that the local damping was always ON when looking for the jumps. This means that the coil driver was still hooked up and we can't rule out glitches in the DAC or the coil driver.
The UL OSEM shows the biggest movement (10 microns). The LR shows the second most (6-7 microns). The others are 2x less. So its consistent with a voltage change on UL,
Is this consistent with a slip in one of the wire standoffs? I think no.
One glitch was seen to occur without a change in the output voltage monitors in ELOG 11744
With Koji's help, I've hacked together an arrangement that will allow us to monitor the output of the coil driver to the UL coil.
The arrangement consists of a short custom ribbon cable with female DB25 connectors on both ends - the particular wire sending the signal to the UL coil has a 100 ohm resistor wired in series, because the coil has resistance ~20ohm, and the output of the coil driver board has a series 200(?) ohm resistor, so by directly monitoring the voltage at this point, we may not see a glitch as it may register too small. Tangentially related: the schematic of the coil driver board suggests that the buffered output monitor has a gain of 0.5.
To monitor the voltage, I use the board to which the 4 Oplev signals are currently hooked up. Channel 7 on this particular board (corresponding to ADC channel 30 on c1scx) was conveniently wired up for some prior test, so I used this channel. Then, I modified the C1SCX model to add a testpoint to monitor the output of this ADC. Then, I turned OFF the input on the coil output filter for the UL Coil (i.e. C1:SUS-ETMX_ULCOIL_SW1) so that we can send a known, controlled signal to the UL Coil by means of awggui. Next, I added an excitation at 5 Hz, amplitude 20 counts (as the signal to the coil under normal conditions was approximately of this amplitude) to the excitation channel of the same filter module, which is the state I am leaving the setup in for the night. I have confirmed that I see this 5Hz oscillation on the monitor channel I set up. Oddly, the 0 crossings of the oscillations happen at approximately -1000 counts and not at 0 counts. I wonder where this offset is coming from? The two points I am monitoring the voltage across is shown in the attached photograph - the black clip is connected to the lead carrying the return signal from the coil.
I also wanted to set up a math block in the model itself that monitors, in addition to the raw ADC channel, a copy from which the known applied signal has been cancelled, as presumably a glitch would be more obvious in such a record. However, I was unable to access the excitation channel to the ULCOIL filter from within the SCX model. So I am just recording the raw output for tonight...
The last RGA scan of this pumpdown 78
Pressure plot of 640 days long pd 78
CC1 cold cathode gauge was jump started with an accidental pressure glitch, that you can see on P1 plot
Yesterday Q noticed that PRM_sensor_LR was 0.098V This actually went to ~ zero on 7-3
Vacuum Status: Chamber Open
All chamber annuloses are vented. Vac Monitor screen is not communicating with gauges. The valve position indicator are working.
RGA is pumped by Maglev through VM2
The DAFI block was reviewed by Rana yesterday. The following changes/improvements were suggested: (Updated on 20th July 2016 with tasks taat remain in red)
1) include all the various channels like PEM, LSC, ASC, SUS, SEI, etc. as the inputs. Currently the inputs are only the LSC.
2) include all the control signals.
3) create a very detailed diagram of the entire signal flow and plan tasks accordingly.
4) Enable cascading of various DSP processes.
5) Adjusting the gain of the AGC such that the amplitude of the output signal comes to about half the peak amplitude offered by the ADC. This will help taking advantage of the entire dynamic range of the ADC.
6) change the enable button styles from a text input based controller to a button controller.
7) Currently, disabling a particular signal terminates the signal. Instead, it should turn into a unity gain block on disabling.
8) Check if the Fibox does AC coupling or not. If not, add an AC coupling arrangement in the DAFI.
9) Check the nature of the ADC1 and ADC2 inputs to the DAFI. I checked them yesterday, and they are channels 25 and 26 of ADC0, which are empty.
I've added a new tab for VMon under the SUS parent tab. I'm still working out the scale and units, but let me know if you think this is a useful addition. Here's a link to my summary page that has this tab: https://ldas-jobs.ligo.caltech.edu/~praful.vasireddy/1151193617-1151193917/sus/vmon/
I'll have another tab with VMon BLRMS up soon.
Also, the main summary pages should be back online soon after Max fixed a bug. I'll try to add the SUS/VMon tab to the main pages as well.
I am using AD592CNZ temperature transducer ICs for measuring temperature inside as well as outside the enclosure. It is a current output IC which outputs current proportional to temperature. As mentioned in the data sheet of AD592, I am using the following two schematics:
Though I still need to calibrate these temperature transducers, I did some measurements. I have temperature readings, and now my goal in few days is to find a transfer function of temperature fluctuations inside the enclosure to outside the enclosure.
About data acquisition:
We have re-configured the raspberry pi(B8:27:EB:70:D0:D8) on martian network. It's new ip address is 192.168.113.107(domenica.martian). Also, we have added the Acromag Busworks card(00:01:C3:00:9F:C8) on the martian network and its ip address is 192.168.113.237(acroey.martian).
The main C1 summary pages are back online now thanks to Max and Duncan, with a gap in pages from June 8th to July 4th. Also, I've added my new VMon and Sensors tabs to the SUS parent tab on the main pages. These new tabs are now up and running on the July 7th summary page.
Here's a link to the main nodus pages with the new tabs: https://nodus.ligo.caltech.edu:30889/detcharsummary/day/20160707/sus/vmon/
And another to my ldas page with the tabs implemented: https://ldas-jobs.ligo.caltech.edu/~praful.vasireddy/1150848017-1150848317/sus/vmon/
Let me know if you have any suggestions or see anything wrong with these additions, I'm still working on getting the scales to be right for all graphs.
I wanted to know what this Vmon exactly is. D010001 is telling us that the Vmon channels are HPFed with fc=30Hz (Attachment 1). Is this true?
I checked the quiscent noise spectrum of the ITMX UL coil output (C1:SUS-ITMX_ULCOIL_OUT) and the corresponding VMON (C1:SUS-ITMX_ULVmon). (Attachment 2 Ref curves). I did not find any good coherence. So the nominal quiscent Vmon output is carrying no useful information.
Question: How much do we need to excite the coil output in order to see any meaningful signal?
As I excite the ITMX UL coil (C1:SUS-ITMX_ULCOIL_EXC) with uniform noise of 100-300 counts below 0.3Hz, I eventually could see the increase of the power spectrum and the coherence (Attachment 2). Below 0.1 Hz the coherence was ~1 and the transfer function was measured to be -75dB and flat. But wait, why is the transfer function flat?
In fact, if I inject broadband noise to the coil, I could increase the coil output and Vmon at the same time without gaining the coherence. (Attachment 3). After some more investigation, I suspect that this HPF is diabled (= bypassed) and aliasing of the high freq signal is causing the noise in Vmon.
In order to check this hypothesis, we need to visit the board.
Circuit1: It is nice to receive the voltage across the transimpedance resistor with a high impedance buffer (or amplifier), as close to the resister as possible. This amplifier needs to have low numbers for input bias current, input offset current, and input current noise. These current noise becomes the noise of the temperature reading. On the top of that, the input voltage noise of the buffer will be added to the output. The typical noise model can be found in http://www.analog.com/media/en/technical-documentation/application-notes/AN-940.pdf
The good candidates for the buffer is LT1128, ADA4004, OPA140, and LT1012. If the application is not too sensitive to the total noise, OPA604 is a good choise with easier handling.
Circuit2: With the same reason, AD741 is an old generic amp that is not a great choise for this purpose. The current noise is more significant because of the higher transimpedance here. The same noise model as above can be used to analyze the performance.
I started to receive emails from cron every 15min. Is the email related to this? And is it normal? I never received these cron emails before when the sum-page was running.
I don't know much about how the cron job runs, I'll forward this to Max.
Max says it should be fixed now. Have the emails stopped?
This should be fixed now—apologies for the spam.
It seemed something has been done. And I got cron emails.
Then, it seemed something has been done. And the emails stopped.
I've made a few changes to the monitoring setup in the hope we catch a glitch in the DAC output/ sus coil driver electronics. Summary of important changes:
It remains to see if we will actually be able to see the glitch in long stretches of data - it is unclear to me how big a glitch will be in terms of ADC counts.
The relevant channels are : C1:SCX-UL_DIFF_MON and C1:SCX-UL_DIFF_MON_EPICS (pardon the naming conventions as the setup is only temporary after all). Both these should be hovering around 0 in the absence of any glitching. The noise in the measured signal seems to be around 2 ADC counts. I am leaving this as is overnight, hopefully the ETMX coil drive signal chain obliges and gives us some conclusive evidence...
I have not committed any of the model changes to the SVN.
One of the pianosa monitors has ceased to function For now, it has been set up to operate with just the one monitor.
One of Donatella's monitors has a defective display as well. Maybe we should source some replacements. Koji has said we will talk to Larry Wallace about this..
It may be advantageous to look at the coil output data from when the OSEM damping is on, to try and reproduce the real output signal amplitude that gets sent to the coils.
Atm 1, It's right arm is perfect.
Atm 3-4, The left one has bended (dropped) end.
Atm 2, Our ruby wire stanoffs will fit the jig. Ruby OD 1.27 mm vs. old Aluminum OD 1.0 mm. Length ruby 6.4 mm vs Al 4.8 mm
Atm 5, The fixture translation stages are a bit loooose. Careful use of the micrometer is needed to be precise
Betsy agreed that the 40m will keep SOS fixtures.
The amplitude of the applied signal (20) was indeed chosen to roughly match what goes to the coils normally when the OSEM damping is on.
There appears to be no evidence of a detectable glitch in the last 10 hours or so (see attachment #1 - of course this is a 16Hz channel and the full data is yet to be looked at)... I guess the verdict on this is still inconclusive.
Attached is a diagram, showing the entire (planned) signal flow of the DAF model. Some thoughts on the implementation after discussion with eric:
1) Since the LSC control signals and ASC signals are running on the c1lsc FE at the same rate as DAFI (16kHz), it would be wise to start from these.
Current implementation: has a matrix at the end of the LSC PD signals, which selects one of the PD signals and outputs it to the DAFI via IPC communication.
Proposed Changes: Add another matrix at the end of the LSC PD signals, to give to the second stereo output. Similarly, add two matrices each at the end of the LSC control signals and the ASC signals. Each matrix must select one of the signals and output it to the DAF via IPC.
2) The PEM running on the c1sus FE system will have to be brought to DAFI in a similar fashon. However, since c1sus runs at 2kHz, there is a possibility of imaging while the signal is transfered to the DAFI. This could be taken care of by an anti imaging filter, or inserting zeros between two samples coming at to the 16 kHz system from the 2kHz system and then low-passing it to remove the aliased parts. (similar to upsampling)
3) For the SUS control signals, input can be given from a matrix prepared for each optic seperately.
We obtained two monitors of the same type from Larry.
Based on Koji's observation of a flat TF, it seems more likely the Vmon channels are looking at the path I've highlighted in green (named "EPICS V Mon"), rather than the path in red (named "DAQ Mon") that Koji initially suspected. This path still lacks any AA for the 16Hz EPICS sampling.
Ah, thanks. That makes sense. In that case, we should remove the texts "30Hz HPF" from the suspension screens.
Now we just need AA LPFs for these channels, or hook them up to the RT system.
I performed a visual inspection of ITMY in its natural habitat today. I did not get any great pictures from the HR side because it's located very towards the edge of the table towards the arm. Before that I checked the levelness of the table. East-west direction was fine, north-south was slightly off but still within the marks for 'level'.
The AR side had several speckles, a few of them located somewhat near the geometrical center of ITMY. The top of the barrel was worse of, as expected. The HR side was a little better, but there were a few pieces of dust? near near the center. Sample pictures are attached, I uploaded all the good ones to Picasa.
Clamps that mark the position of ITMY were already in place. I did not move the optic just yet, and we will have to move a cable block out of the way to bring ITMY near the opening for us to work on it. We will markt the position of that to preserve the weight distribution. Then we can probably take some better before/after pictures. Tomorrow I will be looking at ETMY.
Yesterday, I expanded the extent of the ETMX suspension coil driver investigation. I set up identical monitors for two more coils (so now we are monitoring the voltage sent to UL, UR and LL - I didn't set one up for LR because it is on a second DB25 connector). Furthermore, I increased the excitation amplitude from ~20 to ~2000 (each coil had an independent oscillator at slightly different frequency between 5Hz and 8.5 Hz), the logic being that during LSC actuation we send signals of approximately this amplitude to the coils and we wanted to see if a larger amplitude signal somehow makes the system more prone to glitches.
Over ~10 hours of observation, there is no clear evidence of any glitch. About 2 hours ago (~930am PDT Fri Jul 8), the watchdog tripped - but this was because even though I had increased the trip threshold to ~800 for the course of this investigation, megatron runs this script every 20 minutes or so that automatically reduces this threshold by 17 counts - so at some point, the threshold went lower than the coil voltage, causing the watchdog to trip. So this was not a glitch. The other break around 2am PDT earlier today was an FB crash.
Do we now go ahead and pull the suspension out, and proceed with the swap?
Move the suspension on the south clean bench and make more close inspection. We need to remove the OSEMs.
Then unmount the mirror. Bring it to the clean room and work on the bond removal.
Meanwhile, set up all suspension components inclusing the alignment test setup.
Acromag is talking now, after few changes to the original EPICS configuration and cross compile configuration. Modbus config files also were changed and compiled again to run it on linux-arm architecture. I have made use of pyModbus for the final work and I am planning to use the same for grabbing channels. Though I am unable to grab channel data right now, I am able to communicate to it over ethernet and send and receive data.
ETMX is currently in the clean room, the barrel is the tiniest bit submerged in acetone that will remove a guide rod, standoff, and side OSEM.
Additional inpsection of the standoffs on the flow bench did not provide any insight, pictures are in picasa. Here is a cropped version of a picture we took:
We should look at them under a microscope.
Optimus' memory errors are back so I found the exact DIMM model needed to replace: http://www.ebay.com/itm/Lot-of-10-Samsung-4GB-2Rx4-PC2-5300P-555-12-L0-M393T5160QZA-CE6-ECC-Memory-/201604698112?hash=item2ef0939000:g:EgEAAOSwqBJXWFZh I'm not sure what website would be the best for buying new DIMMs but this is the part we need: Samsung 4GB 2Rx4 PC2-5300P-555-12-L0 M393T5160QZA-CE6.
The magnet, guide rod, and standoff came off without too much force. However, some epoxy residue remains on the barrel. I didn't really want to scrape it off, so I've opted for more soaking. Much of the acetone had evaporated already, so I put some more - just to the point where the residue is submerged.
A new MEDM tab has been added to the summary pages (https://nodus.ligo.caltech.edu:30889/detcharsummary/day/20160708/medm/), although some of the screens are not updated when /cvs/cds/projects/statScreen/cronjob.sh is run. In /cvs/cds/projects/statScreen/log.txt, the following error is given for those files: import: unable to read X window image `0x20011f': Resource temporarily unavailable @ error/xwindow.c/XImportImage/5027. If anyone has seen this error before or knows how to fix it, please let me know.
In the meantime, I'll be working on creating an archive of MEDM screens for every hour to be displayed on the summary pages.
In the evening, I went into the clean room to check how it goes.
- The air around the table is quite warm like a hell. Is this normal?
- I checked how the scattered epoxy spots look like. They were not touching the bath anymore due to evaporation.
- I scraped the spots with the tweezers there. They were easily removed. The particlates on the side barrel were wiped by a wipe with aceton. (Result: Attachment 3)
- Then looked at the other side. I poked the standoff with the tweezer. It was easily removed. I don't think the bond was too weak. Just the area of the bond was so tiny.
- Also residue was scraped by a tweezer and wiped with a cloth. (Attachment 2)
- The removed stand off is in the stainless bowl together with the parts that Eric removed.
- I didn't want to leave the optic in the aceton fume. It was placed on a metal donuts for a 3" optic. (Attachment 4)
- I couldn't find a vacant clean glass jar for the lid. So, a foil hut was built. We should be very careful not to scratch the optic when we remove the hut. (Attachment 5)
- The aceton bath was covered with the foil as it was. (Attachment 6)
Some of the screens are up-to-date, and some are not. Are the errors associated with the screens that failed to get updated?
Thanks! Yes, only the screens that are not updated when the script is executed show this error. I'll try to keep debugging over the weekend.
While ETMX is out, I'm leaving the larger amplitude excitations to the coils on over the weekend, in case any electronic glitch decides to rear its head over the weekend. The watchdog should be in no danger of tripping now that we have removed the ETM.
Unrelated to this work: while removing the ETMX suspension from the chamber, I also removed the large mirror that was placed inside to aid photo taking, so that there is no danger of an earthquake knocking it over and flooding the chamber with dust.
I have added the control signals DARM_CTRL, MICH_CTRL, PRCL_CTRL, SRCL_CTRL, CARM_CTRL, XARM_CTRL, YARM_CTRL, MC_CTRL to the DAFI model from the LSC model via IPC commn.
The changes done to the LSC model include addition of an extra block going to DAFI (attachment 2, red rectangle in attachment 1), and addition of an extra overall output from the LSC, called DAFI_OUT2, which goes to DAFI through IPC link C1:LSC-DAF_2 (attach. 3). Now two distinct inputs can be given to the DAFI, whose intended purpose is to act as two distinct audio signals in the stereo output, but can also be used for arbitrary math.
I am going to add the following PEM channels as DAF inputs subsequently, in a similar 2 input fashon.
I was hoping to glue a standoff and guide rod today, but some problems have reared their heads. Story follows:
Upon first placng the optic into the standoff gluing fixture, I was presented with a geometric problem. In the assembly procedure, one glues the rods before the magnets, which prevents a situation like this:
When what you want to do is this:
So, I spun the optic around such that the magnet is on the far side of the scribe line from the side arm, and instead of extending the side arm past the scribe line, will bring it back towards the near side. I also swapped the arms of the fixture such that the guide rod will be glued on the opposite side of the optic than the side magnet, so the side magnet won't get in the way when doing the pitch adjustment of the second standoff.
Then, I found the scribed ruby rods, and took a look at one under a microscope. The groove looks nice and sharp. I placed the standoff in the side arm of the fixture.
However, the fact that the groove does not go all the way around the standoff leads to problem #1: when adjusting the position of the side arm, the standoff seems to roll around unpredictably, making it hard to deterministically position it while keeping the groove facing outwards.
Problem #2 is not too surprising give Steve's finding about the guide rod holding arm in ELOG 12264. Given that the tip is banged up, the guide rod does not sit straight in the arm, making it crooked. This would lead to the second standoff's groove not being well aligned to the suspension wire.
I will meditate on solutions to these problems... I have covered the optic and fixture with the same foil hut Koji made on Friday.
Also, I peeked at the aluminum standoffs under the microscope. Since the groove goes all the way around, we don't really know where the wire was seated before. Still, there are some places where the groove looks kind of worn:
For some reason, all of the non-IOP models on the vertex frontends had crashed.
To get all of the bits green, I ended up restarting all models on all frontends. (This means Gautam's coil tests have been interrupted.)
It took a little time, but I relocked the IMC and realigned to the point where the PRC is flashing, visible on REFL and AS, and tiny flashes are visible in TRY.
Comments on the schematic:
I have added the PEM signals mentioned in the previous elog as DAF inputs through PCIE IPC, and compiled and restarted the c1pem and c1daf models.
Attached are the pictures of the simulink diagram of the addition in the PEM and the DAF.
Since the signals are moving from a 2kHz clock rate machine to a 16kHz clock rate machine, some imaging effects are possible, which I have to look into.
I have obtained 2x100cc bottles of in-date first contact from Garilynn (use before date is 09/14/2016) for cleaning of our test-masses. They are presently wrapped in foil in the plastic box with all the other first contact supplies.
I found a note on Steve's desk that R. Abbott left yesterday afternoon about an unidentified slippery substance being present on the floor by cabinet S12, along the X arm. (Steve is away this week)
Just now, I found no trace of the substance in the vicinity of that cabinent (which is one of the cabinets for clean objects). Maybe the janitor cleaned it already?
I've noticed the spot that Rich means before, too. I think you only notice this when you're wearing the shoe covers, not sneakers or crocs. I didn't see any 'substance', it seems more like the floor finish (wax?) seems to be more slippery in that area than others.
We have positioned and applied epoxy to one ruby standoff on ETMX, for overnight curing according to the SOS standoff gluing procedure. This included:
Instead of trying to fix up a way of gluing the guiderod with the proper alignment, we chose to be more conservative and glue the standoff today, then switch the gluing fixture's arms tomorrow to glue the guide rod with the good fixture arm.
Additionally, we chose to glue one of the more assymetric standoffs on this first side. What I mean by this is: We have 3 ruby standoffs with grooves. Two of them have the groove about 1/8th of the way along their length, and one has it about 1/4 of the way. Since the second standoff is going to be glued while suspended, after pitch balancing, we figure that we want to use the more centered groove on that side, meaning we used one of the 1/8th standoffs today.
Unfortunately, we neglected to take any pictures :/
We replaced the right N2 bottle as it was empty.