I think I solved the problem (as you can probably see).
The cause was that this WYSIWYG interface for HTML is provided by an independent text editor called FCKeditor which is included in the elog. Although the elog installer has a bug and does not unzip properly the relative package. One has to do it by hand. (going to /elog/scripts/ and unzipping fckeditor.zip by hand in the same directory).
As a reference. The elog runs on background in nodus.
To kill the process:
1) pkill -3 elogd
2) rm -f /var/run/elogd.pid
To restart it:
elogd -p 8080 -c /export/elog/elog-2.7.5/elogd.cfg -D
I notice that Megatron is slower than any other computer in running code that invokes optickle or looptickle (i.e. three times slower than Ottavia). Even without the graphics.
Has anyone ever experienced that?
Today I checked out the SR560 around the lab. I confirmed that the one with the label "channel A noisy" is effectively mulfuctioning.
It was coonected to the lock-in amplifier set up for the drum mode peak readout.
I repleaced that with a working one.
The c1lsc has been unstable since last night. Its status on the DAQ screen was oscillating from green to red every minute.
Yesterday, I power recycled it. That brought it back but the MC got unclocked and the autolocker could not get engaged. I think it's because the power recycling also turned c1iscaux2 off which occupies the same rack crate.
Killing the autolocker on op340 e restarting didn't work. So I rebooted also c1dcuepis and burt-restored almost all snapshot files. To do that, as usual, I had to edit the snapshot files of c1dcuepics to move the quotes from the last line.
After that I restarted the autolocker that time it worked.
This morning c1lsc was again in the same unstable status as yesterday. This time I just reset it (no power recycling) and then I restarted it. It worked and now everything seems to be fine.
Both the Upgrade and the Old40m's signals look anomalous since the zero-crossing point does not change with the demodulation phases.
I suspect there's is a problem with the optickle model of the 40m.
fb:controls>VMIC RFM 5565 (0) found, mapped at 0x2868c90
VMIC RFM 5579 (1) found, mapped at 0x2868c90
Could not open 5565 reflective memory in /dev/daqd-rfm1
16 kHz system
Spawn testpoint manager
Channel list length for node 0 is 4168
Test point manager (31001001 / 1): node 0
this afternoon we centered the optical levers for all the optics.
To do that we first ran the alignment scripts for all the cavities.
New flipping mirror installed on the AP table on the beam path to the REFL199 PD.
If you're missing the double demod signal, please check that it is actually down.
I really don't understand why my programs that I used to use to get data from the HP Spectrum Analyzer and the Marconi frequency generator don't work anymore.
I spent hours trying to debug the code but I can't sort the problem out.
The main problem seem to be with the function recv from the socket library. Somehow it can't anymore get any data from the instruments. The thing I can't understand, though, is that if called directly from the python terminal it works fine!
In particular the problem is with the following lines in my code:
tmp = netSock.recv(1024)
Tried a lot of tickering but it didn't work.
I attach the two scripts I've been using. One (sweepfrequencyPRC.py) calls the other (HP4395PRC.py).
They worked egregiously for weeks in the past. Don't know what happened since then.
tmp = netSock.recv(1024)
This morning Joe looked at my code and made me notice that for some reason the query to the Spectrum Analyzer made by netSock.recv(1024) contained two answers. It was like the buffer contained the answer two different queries.
After some experiment I found that basically the GPIB interface wasn't switching from the "auto 1" to the "auto 0" mode as it should. I rewrote part of the code and that seemed have solved the problem.
Still don't understand why it used to work in the past and then it stopped.
It takes 18 months to double the computational power of microprocessors but it took man thousands of years to invent the zipper. I never really understood that till these days.
Here is a sample of my latest results from Optickle simulations of the locking signal for the Power Recycling Cavity.
Thanks also to Rob's revolutionary bidimensional rotating matrix idea (I can see entire books of linear algebra going to be rewritten now because of that) I could find the way to determine the optimal demodulation phases for the demod signals.
There were also an other couple of missing details. But that came easily along.
The parfor function for the parallel computation in Matlab sped up some loops by a factor of 100.
In these particular plots there's still no CARM offset scan. That's what I'm going to post next on the elog, together with the signals for the other degrees of freedom.
Just to show that I'm confident I'm getting reasonable results, I'll post two PRC scans for different CARM. One set of plots is for the current 40m with -19.78 deg of SRM detuning phase, the other is for the Old Upgrade (9 Mhz vs the 11 currently planned) with no detuning phase.
I'm going to put together the results and get some conclusion about the 3f locking scheme for the current 40m and the upgrade.
It's crucial that I get a stable transmitted power to have an accurate measurement of the PRC transmissivity and thus of its macroscopic length.
Earthquake 4.4 Leo Carrillo Beach.
Some of the watchdogs tripped out.
As PSL-126MOPA_DTEC went up, the power out put went down yesterday
[From Jenne: When we first opened up the MOPA box, the NPRO's cooling fins were HOT. This is a clear sign of something badbadbad. They should be COLD to the touch (cooler than room temp). After jiggling the needle valve, and hearing the water-rushing sounds, the NPRO radiator fins started getting cooler. After ~10min or so, they were once again cool to the touch. Good news. It was a little worrisome however that just after our needle-valve machinations, the DTEC was going down (good), but the HTEMP started to rise again (bad). It wasn't until after Alberto's tinkering that the HTEMP actually started to go down, and the power started to go up. This is probably a lot to do with the fact that these temperature things have a fairly long time constant.
Also, when we first went out to check on things, there was a lot more condensation on the water tubes/connections than I have seen before. On the outside of the MOPA box, at the metal connectors where the water pipes are connected to the box, there was actually a little puddle, ~1cm diameter, of water. Steve didn't seem concerned, and we dried it off. It's probably just more humid than usual today, but it might be something to check up on later.]
The laser power seems to have become more stable after fixing the laser chiller. The power is lower than it used to be (MOPA amplitude 2.5 versus 2.7) but, as shown in the attchement, it became more steady.
Tonight I centered the oplevs for ITMX/Y, SRM, PRM, BS.
After doing that I noticed that the BS drifted a little from where I had set it.
I just added two slow channels to C0EDCUEPICS to monitor the input of PD11. The names are:
Pete, Rob, Alberto,
yesterday we thought that some of the problems we were having in locking the IFO might be related to a change of the length of the mode cleaner. So today we decided to measure it again.
We followed the Sigg-Frolov technique (see 40m Wiki, Waldman, Fricke). For the record, the MC_AO input corresponds to IN2 on the MC Servo board.
We obtained: L = 27.092 +/- 0.001 m
From the new measurement we reset the frequencies of the Marconis to the following values:
tonight we worked on the tuning of the double demod phases for the handoff of the short DOFs control signals.
Only MICH can now undergo the handoff. PRC can't make it.
Basically, we tuned the PD6 demod phase and reduced the offset in PD6_I. Then we tuned the relative gain of PD6_I and PD2_I so that the two open loop transfer function of the control loops would match. We tried that in several ways and several times but without success.
I guess we're missing to do/check something.
This afternoon I tuned the handoff script for the SRC, after that Rob eralier during the day had already adjusted that for PRC. To do that, I followed the procedure in the Wiki.
After that the SRC could get locked with the double demod signals. the open loop transfer function emasurement on the PRC loop showed that it was nearly unstable. Rob reduced a little its gain to improve the stability.
The DD handoff is now working and we can get back to locking the interferometer.
For the 40m Upgrade, we plan to eliminate the Mach-Zehnder and replace it with a single EOM driven by all three modulation frequencies that we'll need: f1=11MHz, f2=5*f1=55MHz, fmc=29.5MHz.
A frequency generator will produce the three frequencies and with some other electronics we'll properly combine and feed them to the EOM.
The frequency generator will have two crystals to produce the f1 and fmc signals. The f2 modulation will be obtained by a frequency multiplier (5x) from the f1.
The frequency multiplier, for the way it works, will inevitably introduce some unwanted harmonics into the signals. These will show up as extra modulation frequencies in the EOM.
In order to quantify the effects of such unwanted harmonics on the interferometer and thus to let us set some limits on their amplitude, I ran some simulations with Optickle. The way the EOM is represented is by three RF modulators in series. In order to introduce the unwanted harmonics, I just added an RF modulator in series for each of them. I also made sure not to leave any space in between the modulators, so not to introduce phase shifts.
To check the effect at DC I looked at the sensing matrix and at the error signals. I considered the 3f error signals that we plan to use for the short DOFs and looked at how they depend on the CARM offset. I repeated the simulations for several possible amplitude of the unwanted harmonics. Some results are shown in the plots attached to this entry. 'ga' is the amplitude ratio of the unwanted harmonics relative to the amplitude of the 11 & 55 MHz modulations.
Comparing to the case where there are no unwanted harmonics (ga = 0), one can see that not considerable effect on the error signals for amplitudes 40dB smaller than that of the main sidebands. Above that value, the REFL31I signals, that we're going to use to control PRCL, will start to be distorted: gain and linearity range change.
So 40 dB of attenuation in the unwanted harmonics is probably the minimum requirement on the frequency multiplier, although 60dB would provide a safer margin.
I'm still thinking how to evaluate any AC effect on the IFO.
** TODO: Plot DC sweeps with a wider range (+/- 20 pm). Also plot swept sines to look for changes in TFs out to ~10 kHz.
Today I found the elog down, so I rebooted it following the instructions in the wiki.
I have the impression that Nodus has been rebooted since last night, hasn't it?
Tonight I tried to lock the interferometer. At the first attempts the arm power didn't go above about 4. The mode cleaner seemed to be not well aligned and it lost lock or got stuck on a wrong mode. I had to run the MC_UP and MC_DOWN scripts to lock it again.
After that the locking proceed more smoothly; at least till a power level in the arms of about 60. Then again the mode cleaner lost lock and I had to run the scripts again. Without the MCWFS servo off the MC reflected power is still rather high (about 1.7); also even when the WFS servo is engaged the reflected power is about 0.5, versus 0.3 that it should be.
Those are both signs of a not very good alignment. Tomorrow I'll have to work on the injection periscope on the PSL table to try to fix that.
Once again, this morning I found the wireless router disconnected from the LAN cable. No martian WiFi was available.
I wonder who is been doing that and for what reason.
Today I tuned the periscope on the PSL table to align the beam to the Mode Cleaner. With the Wave Front Sensor control off, I minimized the reflection from the MC and maximized the transmission. While doing that I also checked that the transmitted beam after the MC didn't lose the alignment with the interferometer's main Faraday isolator.
In this way, I've got a reflection, as read from the MC_REFLPD_MC, of about 0.6. Then I centered the WFS on the AS table. After that the WFS alignment control brought the reflection to 0.25 and a nice centered bull-eye spot showed on the monitor.
I popped by the 40m, and was dismayed to find that all of the front end computers are red (only framebuilder, DAQcontroler, PEMdcu, and c1susvmw1 are green....all the rest are RED).
I keyed the crates, and did the telnet.....startup.cmd business on them, and on c1asc I also pushed the little reset button on the physical computer and tried the telnet....startup.cmd stuff again. Utter failure.
I have to pick someone up from the airport, but I'll be back in an hour or two to see what more I can do.
I think the problem was caused by a failure of the RFM network: the RFM MEDM screen showed frozen values even when I was power recycling any of the FE computers. So I tried the following things:
After Alberto's bootfest which was more successful than mine, I tried powercycling the AWG crate one more time. No success. Just as Alberto had gotten, I got the DAQ screen's AWG lights to flash green, then go back to red. At Alberto's suggestion, I also gave the physical reset button another try. Another round of flash-green-back-red ensued.
When I was in a few hours ago while everything was hosed, all the other computer's 'lights' on the DAQ screen were solid red, but the two AWG lights were flashing between green and red, even though I was power cycling the other computers, not touching the AWG at the time. Those are the lights which are now solid red, except for a quick flash of green right after a reboot.
I poked around in the history of the curren and old elogs, and haven't found anything referring to this crazy blinking between good and bad-ness for the AWG computers. I don't know if this happens when the tpman goes funky (which is referred to a lot in the annals of the elog in the same entries as the AWG needing rebooting) and no one mentions it, or if this is a new problem. Alberto and I have decided to get Alex/someone involved in this, because we've exhausted our ideas.
Today Alex came over, performed his magic rituals on the DAQAWG computer and fixed it. Now it's up and running again.
I asked him what he did, but he's not sure of what fixed it. He couldn't remember exactly but he said that he poked around, did something somewhere somehow, maybe he tinkered with tpman and eventually the computer went up again.
Now everything is fine.
Since lately the alignment of the input beam to the interferometer has changed, I went checking the alignment of the beam on the photodiodea. They were all fine except for pd9, that is AS DD 199. Here the DC is totally null. The beam seems to go right on the diode but the scope on the PD's DC output shows no power. This is really strange and bad.
After inspecting PD9 with the viewer and the cards, the beam looks like it is aligned to the photodiode althought there is no signal at the DC output of the photodetector. So I checked the spectrum for PD9_i and Q (see attachments) and it seems that those channels are actually seeing the beam. I'm going to check the alignemtn again and see the efefct on the spectra to make sure that the beam is really hitting the PD.
I aligned PD9. Here are the spectra confirming that.
All suspentions are kicked up. Sus dampings and oplev servos turned off.
c1iscey and c1lsc are down. c1asc and c1iovme are up-down
The computers and RFM network are up working again. A boot fest was necessary.Then I restored all the parameters with burtgooey.
The mode cleaner alignment is in a bad state. The autolocker can't get it locked. I don't know what caused it to move so far from the good state that it was till this afternoon. I went tuning the periscope but the cavity alignment is so bad that it's taking more time than expected. I'll continue working on that tomorrow morning.
I now suspect that after the reboot the MC mirrors didn't really go back to their original place even if the MC sliders were on the same positions as before.
Trying to track the MC positions back for a few days, it seems that the data hasn't been recorded properly for a while. Something happened yesterday after my boot fest and then the record got restored. Attached here are the readbacks showing the event for MC1.
Is anything wrong with the data record?
we diagnosed the problem. It was related with sticky sliders. After a reboot of C1:IOO the actual output of the DAC does not correspond anymore to the values read on the sliders. In order to update the actual output it is necessary to do a change of the values of the sliders, i.e. jiggling a bit with them.
Chronicles of periscope and MC alignment
Yesterday morning I started aligning the periscope but it turned out to be trickier than usual. With the ASC (Alignment Sensing Control) off and only the length controls on, the Mode Cleaner didn't lock easily, although I knew I wasn't very far from the sweet spot.
In the afternoon the struggle continued and the matching of the the beam to the MC cavity became just worse. At some point I noticed that the ASC inputs somehow had got on - although the ASC still looked disabled from the MClock MEDM main screen. So I was actually working against the Wave Front Sensors and further worsening the periscope alignment.
That hurled me to the weeds. After hours of rowing across the stormy waters of a four-dimensional universe I got to have occasional TEM00 flashes at the transmission but still, surprisingly, no MC locking. Confused, I kept tuning the periscope but that just kicked me off road again.
Then at about 7pm Koji came to my rescue and suggested a more clever and systematic way to solve the problems. He suggested to keep record of the MC mirrors alignment state and re-align the cavity to the periscope. Then we would gradually bring the cavity back to the original good position changing the periscope alignment at the same time.
That would have worked straight away, if we hadn't been fighting against a subtle and cruel enemy: the 40m computer network. But I (as John Connor), and Koji (as the Terminator) didn't pull back.
Here's a short list of the kinds of weapons that the computers threw to us:
We then proceeded with Koji's plan. In an iterative process, we aligned the MC cavity maximizing the transmission and tuned the periscope in order to match the Faraday input of the interferometer. The last thing we did it by looking at the camera pointing at the Faraday isolator.
We found that we didn't have to tune the periscope much. That means that all afternoon I didn't really go too far, but the autolocker wasn't working properly, or it wasn't working at all.
Then we ran the alignment script for the X arm but it didn't work before we aligned the steering mirrors.
Then we ran it three times but could not get more than 0.87 at TRX. That means that there we still have to work on the alignment to the Faraday. That's job for today in the trenches of the lab.
After yesterday's changes in the MC cavity state today it was necessary to optimize the alignment to the Faraday.
The way I did it was by tuning the PSL periscope in pitch and yaw trying to maximize TRX with the arm locked. After a small change in either one of the two directions I first maximized the MC transmitted power and then I ran the alignment script for the X arm.
I explored the space for both pitch and yaw and the max that I could get from TRX was 0.91. I'm not sure whether the increase in TRX is entirely due to a better alignment to the Farady rather than to a higher MC transmitted power.
Also I'm not sure I'm well interpreting the image from the camera pointing at the Farady. I guess I need someone more familiar with it to tell me if it shows any sign of clipping.
Anyway, last week, even before the MC got misaligned, TRX didn't go above 0.90. So now I wonder whether it's the MC's fault or something else's if we have that value..
This morning I found the elog down. I restarted it using the procedure in the wiki.
This afternoon I kept working on the alignment of the beam so that it matches at the same the PSL periscope, the Mode Cleaner and the Faraday isolator at the input of the IFO.
The camera looking at the Farady showed a beam quite low from the center of the Faraday's entrance. I wanted to move it up.
After working on the periscope alignment and on the MC mirrors, I think I managed to moved it up a bit. To know whether that was enough or not I wanted to evaluate the alignment to the X arm by checking the value of TRX.
In order for the MC to be finely matched to the input beam from the periscope, the WFS controls have to be on. Before turning them on, I centered the beam on their QPDs and run the WFS_zero_offset script.
When I turned them on, the control signal in Pitch from WFS2 started going up with no stop. It was like the integrator in the loop was fed with a DC bias. The effect of that was to misalign the MC cavity from the good state in which it was with the only length control on (that is, transmission ~2.7, reflection ~ 0.4).
I don't know why that is happening. To exclude that it was due to a computer problem I first burtrestored C1IOO to July the 18th, but since that did not help, I even restarted it. Also that didn't solve the problem.
Flashes at ETMX show at least that the beam is going through the Farady. How well, I can't tell untill the MC is under full control.
I have to leave the lab now, but I can be back tomorrow to keep working on that.
I set the MC back to its good alignment (June 21st) using this procedure. The trend of the OSEM values over the last 40 days and 40 nights is attached.
Then I aligned the periscope to that beam. This took some serious periscope knob action. Without WFS, the transmission went to 2.7 V and the reflection down to 0.6V.
Then I re-aligned the MC_REFL path as usual. The beam was far enough off that I had to also re-align onto the MC LSC PD as well as the MC REFL camera (~2 beam radii).
Beams are now close to their historical positions on Faraday and MC2. I then restored the PZT sliders to their April snapshot and the X-arm locked.
Steve - please recenter the iris which is on the periscope. It has been way off for a long time.
So it looks OK now. The main point here is that we can trust the MC OSEMs.
Afterwards I rebooted c1susvme1 and c1susvme2 because they were skewed.
It is really surprising that we now have again the data from the MC OSEMs since up to two days ago the record looked corrupted (see the attachments in my entry 1774).
The reason I ended up severely misaligning the the MC is exactly that there wasn't anymore a reference position that I could go back to and I had to use the camera looking a the Faraday.