I have prepared several diagrams outlining the current state of the RF System.
These are uploaded into the svn40m here and will be kept uptodate as we complete various parts of the task. These plans have taken into account
the new priorities of the LSC (set out by Koji here )
We (Koji, Kiwamu and I) took stock of the RF cables which we have inherited from the earlier RF system and have made new plans for them.
I took stock of the filters purchased for the modifying the demod boards. We have pretty much everything we need so I will start modifying the boards right away. The following table summarises the modifications
LP Filter (U5)
We seem to have a spare SHP-175. I was wondering where that is supposed to go.
This is the status and tentative schedule for completing the various tasks. I have put the dates based on priority and state of the hardware.
The RF Cable layout plans are drawn on top of a Lab Layout. The various subsystems are drawn (not to scale) on separate layers. The graffle files are located here . I thought they might come in handy for others as well.
* Temporary strain relief for the heliax cables on 1X2 (Steve)
* RF diagrams and check lists (Suresh)
=> In the lunch meeting we will discuss the details about what we will do for the RF installation.
* Electronics design and plan for Green locking (Aidan / Kiwamu)
=> In the lunch meeting we will discuss the details.
* LSC model (Koji)
* Video cable session (team)
* LPF for the laser temperature control (Larisa)
For bode plot:
USE LOG-LOG plot for the amplitude
USE LOG-LINEAR plot for the phase
Search "Bode Plot" on web
This is the continuation of http://nodus.ligo.caltech.edu:8080/40m/4402
The first picture is of the actual component, where the resistor is 1M and capacitor is 10uF.
But before the component can be put into place, its transfer function had to be checked to make sure it was doing what we calculated it would do. The results of these are in the graphs generated: frequency vs. gain, and frequency vs. phase.
According to these graphs, we are not achieving the targeted cutoff frequency: need to recalculate and compensate for the extra 100k resistance being encountered.
This is the log of the work on Wednesday 23rd.
1. Power Supply of the freq divider box
Kiwamu claimed that the comparator output of the freq div box only had small output like ~100mV.
The box worked on the electronics bench, we track down the power supply and found the fuse of the +15V line
brew out. It took sometime to notice this fact as the brown-out-LED of the fuse was not on and the power
supply terminal had +15V without the load. But this was because of the facts 1) the fuse is for 24V, and 2)
the large resistor is on the fuse for lighting the LED when the fuse is brown out.
I found another 24V fuse and put it there. Kiwamu is working on getting the correct fuses.
2. MC locking problem
After the hustle of the freq divider, the MC didn't lock. I tracked down the problem on the rack and found
there was no LO for the MC. This was fixed by pushing the power line cable of the AM Stabilizer for the MC LO, which was a bit loose.
Its going to need some kind of way to locate the PMC on the top. In the previous design, we had the 3 balls to decouple the body from the base. That design was flawed due to the roughness of the holes in the PMC body.
Hmmm, so, this was just meant to be a riser that goes underneath the old PMC mount, to raise it from 3" beam height to 4" beam height. I will make another one that is a complete mount, designed for 4" beam height. Please hold........... .......... ....... ..... ... .
Also probably need some kind of relief on the bottom. Its possible that it would be OK like this, but I am unsure if the shop can maintain the flatness we want over the whole length and/or the flatness of any given (OLD) optical table over ~8". Its probably not a good idea to have to torque this (aluminum?) to make it conform to the optical table's shape.
I (think) I have finished the new PMC base riser. The eDrawing of it (so you can view it on any computer) has been uploaded to the PMC wiki page.
I also attach it here, for comments.
PLEASE DO NOT DISMANTLE THE SETUP !
Actually we tried looking for a level-3 or a smaller mixer, but we didn't find them at that moment. That's why we kept the level-7 mixer for the coarse path.
As you pointed out we can try an RF amplifier for it.
Using 2 dBm for a Level 7 mixer is so bogus, that I will dismantle this as soon as I come over.
In the last week Matt and I modified the MFD configuration because the mixer had been illegally used.
Since the output from the comparator is normally about 10 dBm, a 4-way power splitter reduced the power down to 4 dBm in each output port.
In order to reserve a 7 dBm signal to a level-7 mixer, we decided to use an asymmetric power splitter, which is just a combination of 2-way and 3-way splitter shown in the diagram above.
With this configuration we can reserve a 7 dBm signal for a mixer in the fine path.
However on the other hand we sacrificed the coarse path because the power going to the mixer is now 2.2 dBm in each port.
According to the data sheet for the mixer, 1 dB compression point for the RF input is 1dBm. Therefore we put a 1 dB attenuator for the RF port in the coarse system.
In the delay line of the fine path we found that the delay cable was quite lossy and it reduced the power from 2.2 dBm to about 0 dBm.
I measured some laser powers associated with the beat-note detection system on the PSL table.
The diagram below is a summary of the measurement. All the data were taken by the Newport power meter.
The reflection from the beat-note PD is indeed significant as we have seen.
In addition to it the BS has a funny R/T ratio maybe because we are using an unknown BS from the Drever cabinet. I will replace it by a right BS.
During my work for making a noise budget I noticed that we haven't carefully characterize the beat-note detection system.
The final goal of this work is to draw noise curves for all the possible noise sources in one plot.
To draw the shot noise as well as the PD dark noise in the plot, I started collecting the data associated with the beat-note detection system.
* Estimation and measurement of the shot noise
* measurement of the PD electrical noise (dark noise)
* modeling for the PD electrical noise
* measurement of the doubling efficiency
* measurement of an amplitude noise coupling in the frequency discriminators
Suresh is captivating his audience with gravity waves on last Friday, March 25
Koji was unable to build his c1lst model first thing this morning. Turns out there was a bug with RCG parser that was introduced on Friday when we did the RCG updates. We talked Alex who did a quick comment fix. The diff is as follows:
--- Parser3.pm (revision 2328)
+++ Parser3.pm (working copy)
@@ -1124,8 +1124,8 @@
print "Flattening the model\n";
print "Finished flattening the model\n";
- CDS::Tree::do_on_nodes($root, \&remove_tags, 0, $root);
- print "Removed Tags\n";
+ #CDS::Tree::do_on_nodes($root, \&remove_tags, 0, $root);
+ #print "Removed Tags\n";
CDS::Tree::do_on_nodes($root, \&remove_busses, 0, $root);
This was some code to remove TAGs from the .mdl file for some reason which I do not understand at this time. I will ask tommorrow in person so I can understand the full story.
Koji then rebuilt and started the c1lst process. This is his new test version of the LSC code. We descovered (again) that when you activate too many DAQ channels (simply uncommenting them, not even recording them with activate=1 in the .ini file) that the frame builder crashes. In addition, the c1lsc machine, which the code was running on, also hard crashed.
When a channel gets added to the .ini file (or uncommented) it is sent to the framebuilder, irregardless of whether its recorded or not by the frame builder. There is only about 2 megabytes per second bandwidth per computer. In this case we were trying to do something like 200 channels * 16384 Hz * 4 bytes = 13 megabytes per second.
The maximium number of 16384 channels is roughly 30, with little to no room for anything else. In addition, test points use the same allocated memory structure, so that if you use up all the capacity with channels, you won't be able to use testpoints to that computer (or thats what Alex has led me to believe).
The daqd process then core dumped and was causing all sorts of martian network slowdowns. At the same time, the c1lsc computer crashed hard, and all of the front end processes except for the IOP on c1sus crashed.
We rebooted c1lsc, and restarted the c1sus processes using the startc1SYS scripts. However, the c1susfe.ko apparently got stuck in a wierd state. We were completely unable to damp the optics and were in general ringing them up severely. We tried debugging, including several burt restores and single path checks.
Eventually we decided to reboot the c1sus machine after a bit of debugging. After doing a burt restore after the reboot, everything started to damp and work happily. My best guess is the kernel module crashed in a bad way and remained in memory when we simply did the restart scripts.
Last Friday, we discovered a bug in the RCG where the delay part was not actually delaying. We reported this to Alex who promptly put a fix in the same day. This allowed Matt's newly proposed frequency discriminator to work properly.
It also required a checkout of the latest RCG code (revision 2328), and rebuild of the various codes. We backed up all the kernel and executables first such as mbuf.ko and awgtpman.
We did the following:
1) Log into the fb machine.
2) Go to /opt/rtcds/caltech/c1/core/advLigoRTS/src/drv/mbuf and run make. Copy the newly built mbuf.ko file to /diskless/root/modules/18.104.22.168/kernel/drivers/mbuf/mbuf.ko on the fb machine.
3) Use "sudo cp" to copy the newly built mbuf.ko file to /diskless/root/modules/22.214.171.124/kernel/drivers/mbuf/
4) Go to /cvs/cds/rtcds/caltech/c1/core/advLigoRTS/src/gds and run make.
5) Copy the newly built awgtpman executable to /opt/rtcds/caltech/c1/target/gds/bin/
6) Go to /opt/rtcds/caltech/c1/core/advLigoRTS/src/mx_stream/ and run make.
7) Copy the newly built mx_stream executable to /opt/rtcds/caltech/c1/target/fb/
I modified the c1gfd.mdl simulink model. I made a backup as c1gfd_20110325.mdl.
The first change was to use a top_names block to put everything in. The block is labeled ALS. So all the channels will now be C1:ALS-GFD_SOMETHING. This means medm channel names will need to be updated. Also, the filter modules need to be updated in foton because of this.
I then proceeded to add the suggested changes made by Matt. To avoid a divide by zero case, I added a saturation part which saturates at 1e-9 (note this is positive) and 1e9.
Today we tried the Schmitt trigger DFD, and while it works it does not improve the noise performance. At least part of our problem is coming from the discrete nature of our DFD algorithm, so I would propose that an industrious day job person codes up a new DFD which avoids switching. We can probably do this by mixing the input signal (after high-passing) with a time-delayed copy of itself... as we do now, but without the comparator. This has the disadvantage of giving an amplitude dependent output, but since we are working in the digital land we can DIVIDE. If we mix the signal with itself (without delay) to get a rectified version, and low-pass it a little, we can use this for normalization. The net result should be something like:
output = LP2[ s(t) * s(t - dt) / LP1[ s(t) * s(t) ]],
where s(t) is the high-passed input and LP is a low-pass filter. Remember not to divide by zero.
I tidied up some of the stuff that was on the SP table. The ISS box that has been sitting on there for months is now underneath the X-arm on top of the spare Marconi which is stored there.
Kiwamu and I looked at all the electronics that are currently in place for the green locking on the X-arm and have made a set of block diagrams of the rack mounted units that we should build to replace the existing ... "works of art" that sprawl around out there at the moment.
1. "ETM Green Oscillator/PDH support box". Not a great name but this would provide the local oscillator signal for the end PDH (with a controllable phase rotator) as well as the drive oscillator for the end laser PZT. Since we need to hit a frequency of 216.075kHz with a precision that Kiwamu needs to determine, we'd need to be able to tune the oscillator ... it needs to be a VCO. It'd be nice to be able to measure the output frequency so I've suggested dividing it down by N times to put it into the DAQ - maybe N = 2^7 = 128x to give a measured frequency of around 1.7kHz. Additionally this unit will sum the PDH control signal into the oscillation. This box would support the Universal PDH box that is currently at the X-end.
2. "Vertex X-arm beatnote box" - this basically takes the RF and DC signals from the beatnote PD and amplifies them. It provides a monitor for the RF signal and then converts the RF signal into a square wave in the comparator.
3. "Mixer Frequency Discriminator" - just the standard MFD setup stored in a box. For temperature stability reasons, we want to be careful about where we store this box and what it is made of. That's also the reason that this stage is separated from the X-arm beatnote box with it's high-power amps.
4. RS232 and EPICS control of the doubling ovens
5. Intensity stabilization of the End Laser
P.S. I used Google Diagrams for the pictures.
X_arm and Y_arm vs PSL comparison.
Just a quick check of the performance of the X arm and Y arm lasers in comparison to the PSL. Plotting the data from the X arm vs PSL and Y arm vs PSL on the same plot shows that the X arm vs the PSL has no observable trending of mode-hopping in the laser, while the Y arm vs the PSL does. Suspect this is due to the fact that the X arm and PSL are both Innolight lasers with essentially identical geometry and crystals and they'll tend to mode-hop at roughly the same temperatures - note that the Xarm data is rough grained resolution so it's likely that any mode-hop transitions have been skipped over. The Lightwave on the other hand is a very different beast and has a different response, so won't hop modes at the same temperatures.
Given how close the PSL is to one of the mode-transition regions where it's currently operating (32.26 degC) it might be worth considering shifting the operating temperature down one degree or so to around 31 degC? Just to give a bit more headroom. Certainly worth bearing in mind if problems are noticed in the future.
The good news is that we seem to be running in a linear region of the PSL laser with a degree or so of range before the PSL Innolight laser starts to run multi-mode. On the attached graph we are currently running the PSL at 32.26degrees (measured) which puts us in the lower left corner of the plot. The blue data is the Lightwave set temperature (taken from the display on the laser controller) and the red data is the Lightwave laser crystal measured temperature (taken from the 10V/degC calibrated diagnostic output on the back of the laser controller - between pins 2 and 4).
The other good news is that we can see the transition between the PSL laser running in one mode and running in the next mode along. The transition region has no data points because the PMC has trouble locking on the multi-mode laser output - you can tell when this is happening because, as we approach the transition the PMC transmitted power starts to drop off and comes back up again once we're into the next mode region (top left portion of the plot).
The fitted lines for the region we're operating in are:
Y_arm_Temp_meas = 0.95152*T_PSL + 3.8672
Y_arm_Temp_set = 0.87326*T_PSL + 6.9825
Just a quick update... the Lightwave laser has now been moved up to the end of the Y arm. It's also been mounted on the new mounting block and heatsinks attached with indium as the heat transfer medium.
A couple of nice piccies...
There was a 2" mirror mount among the spares on the PSL table. It has a window LW-3-2050 UV mounted in it. I
have moved it to the Y-end table. We seem to have run out of 2" mirror mounts ...
With the exception of a 2" mirror mount, I've confirmed that we have everything for the Y-end green production and mode-matching.
We need to calculate a mode-matching solution for the Lightwave laser so that it gives the correct beam size in the doubling crystal.
Additionally, Rana has suggested that we change the pedestals from the normal 1" diameter pedestal+fork combo to the 3/4" diameter posts and wider bases that are used on the PSL table (as shown in the attached image).
[Steve, Suresh, Larisa]
The following cables were laid today: ETMYT, ETMY, IFOPO, MC1, OMCR, AS Spare, and MC2T.
Though the paper suggested 135' for the MC2T, we used a 110'. This is too short: need at least another 15' for the MC2T.
The RCR cable wasn't crossed off on the list, but a cable exists at the RCR cable which is black and is labeled (old label, 75 ohms)
There was no indication of which length was needed for MC1, so a 95' cable was used.
This is the grand plan we talked about in the beginning of the meeting.
The following is a message from the LIGO 40m Chief Recycling Officer:
Please get up off your (Alignment Stabilization Servo)es and recycle your bottles and cans! There is a recycling bin in the control room. Recent weeks have seen an increase in number of bottles/cans thrown away in the regular garbage. This is not cool.
Yesterday during the day, Alex ran a script to fix the time stamps in the trends files we had messed up back during the daqd change overs around Feb 17th and 23rd. See this elog for more information on the trend problem.
Due to how the script runs, basically taking all the data and making a new copy with the correct time stamps, the data collected while the script was running didn't get converted over. So when he did the final copy of the corrected data, it created a several hour gap in the data from yesterday during the day time.
The original files still exist on the fb machine in /frames/trend/minute_raw_22mar2011 directory.
The 40m lab was visited by ~ 30 LSC members the end of last week.
Solid door, numbered 4 at south west corner of PSL enclosure was replaced by laser protective window.
The carpenter shop's Mark is making 4 more identical ones for the east side.
The Lightwave NPRO126 of 700mW was moved from the AP-table into the PSL-enclosure temporarily.
It's emergency shutdown switch can be seen at the center bottom picture
Succeeded in handing off the servo from the green to the red.
This time we found that the fluctuation in the IR signals became lesser as the gain of the ALS servo increased.
Therefore I increased the UGF from 40 Hz to 180 Hz to have less noise in the IR PDH signal.
Here is a preliminary plot for today's noise spectra.
The blue curve is the ALS in-loop spectrum, that corresponds to the beat fluctuation.
The red curve is an out-of-loop spectrum taken by measuring the IR PDH signal.
Since the UGF is at about 180 Hz the rms is integrated from 200 Hz.
The residual displacement noise in the IR PDH signal is now 1.2 kHz in rms.
I am going to analyze this residual noise by comparing with the differential noise that I took yesterday (see the last entry ).
- Plan for tomorrow
* Video cable session (I need ETMY_TRNAS) (team)
* Characterization of the Y end laser (Bryan / Suresh)
* LPF for the X end laser temperature control (Larisa)
* Frequency Divider (Matt)
* X end mechanical shutter (Kiwamu)
I'm going to take the easy question - What are the pink data points??
And I'm going to answer the easy question - they're additional beat frequency temperature pair positions which seem to correspond to additional lines of beat frequencies other than the three highlighted, but that we didn't feel we had enough data points to make it worthwhile fitting a curve.
It's still not entirely clear where the multiple lines come from though - we think they're due to the lasers starting to run multi-mode, but still need a bit of thought on that one to be sure...
A comparator has been installed before the MFDs (mixer-based frequency discriminator) to eliminate the effect from the amplitude fluctuation (i.e. intensity noise).
As a result we reached an rms displacement of 580 Hz or 80 pm.
As a result we reached an rms displacement of 580 Hz or 80 pm.
(differential noise measurement)
Here is the resultant plot of the usual differential noise measurement.
The measurement has been done when the both green and red lasers were locked to the X arm.
In the blue curve I used only MFD. In the black curve I used the combination of the comparator and the MFD.
Noise below 3 Hz become lower by a factor of about 4, resulting in a better rms integrated from 40 Hz.
Note that the blue and the black curve were taken while I kept the same lock.
A calibration was done by injecting a peak at 311 Hz with an amplitude of 200 cnt on the ETMX_SUS_POS path.
Yesterday Koji modified his comparator circuit such that we can take a signal after it goes thorough the comparator.
The function of this comparator is to convert a sinusoidal signal to a square wave signal so that the amplitude fluctuation doesn't affect the frequency detection in the MFD.
I installed it and put the beat-note signal to it. Then the output signal from the comparator box is connected to the MFDs.
The input power for the comparator circuit has been reduced to -5 dBm so that it doesn't exceeds the maximum power rate.
OK. Today we did the same type of measurement for the Y arm laser as was done for the X arm laser here: http://nodus.ligo.caltech.edu:8080/40m/3759
And attached here is a preliminary plot of the outcome - oddities with adding on the fitted equations, but they go as follows
(Red) T_yarm = 1.4435*T_PSL - 14.6222
(Blue) T_yarm = 1.4223*T_PSL - 10.9818
(Green) T_yarm = 1.3719*T_PSL - 6.3917
It's a bit of a messy plot - should tidy it up later...
It's a bit of a messy plot - should tidy it up later...
Some tasks for the daytime tomorrow.
* Beam profile measurements of the Y end laser (Suresh / Bryan)
* Taking care of CDS and the simulated plant (Jamie / Joe)
* Reconnect the X end mechanical shutter to 1X9 (Kiwamu)
* LPF for the X end temperature feedback (Larisa)
I added a new ADC channel for a DC signal from the X end green PD.
It is called C1:GCX-REFL_DC and connected to adc_0_1, which is the second channel of ADC_0.
By the way, when I tried connecting it to an ADC I found that most of the channels on the AA board on 1X9 were not working.
Since the outputs form the board are too small the circuits may have benn broken. See the picture below.
In addition to that I realized that the signal from the PDH box for the temperature actuation is limited by +/- 2V due to the range of this AA board.
In fact the signal is frequently saturated due to this small voltage range.
We have to enlarge the range of this AA board like Valera did before for the suspensions (see this entry).
- Plan for this week
* Intensity stabilization for the end green laser (Matt / Kiwamu)
* Hand off the servo from Green to Red (Matt / Kiwamu)
* Y end green locking (Suresh / Bryan) (rough schedule)
* Reconnect the X end mechanical shutter to 1X9 (Kiwamu)
* Connect the end DCPD signal to a DAC (done)
* Make a LPF in a Pomona box for the temperature (Larisa)
* Clean up and finalize the X end setup (Kiwamu)
* Make a item lists for electronics. Order the electronics. (Aidan / Kiwamu)
Bryan Barr is visiting us from Glasgow for a month. He received 40m specific safety training on Friday.
PMC TRANS/REFL on MEDM showed red values for long time.
TRANS (a.k.a C1:PSL-PSL_TRANSPD) was the issue of the EPICS db.
REFL (a.k.a. C1:PSL-PMC_RFPDDC) was not physically connected.
There was an unknown BNC connected to the PMC DC output instead of dedicated SMA cable.
So they were swapped.
Now I run the following commands to change the EPICS thresholds:
ezcawrite C1:PSL-PMC_PMCTRANSPD.LOLO 0.8
ezcawrite C1:PSL-PMC_PMCTRANSPD.LOW 0.85
ezcawrite C1:PSL-PMC_PMCTRANSPD.HIGH 0.95
ezcawrite C1:PSL-PMC_PMCTRANSPD.HIHI 1
ezcawrite C1:PSL-PMC_RFPDDC.HIHI 0.05
ezcawrite C1:PSL-PMC_RFPDDC.HIGH 0.03
ezcawrite C1:PSL-PMC_RFPDDC.LOW 0.0
ezcawrite C1:PSL-PMC_RFPDDC.LOLO 0.0
As these commands only give us the tempolary fix, /cvs/cds/caltech/target/c1psl/psl.db was accordingly modified for the permanent one.
field(DESC,"RFPDDC- RFPD DC output")
field(INP,"#C0 S32 @")
field(DESC,"PMCTRANSPD- pre-modecleaner transmitted light")
field(INP,"#C0 S10 @")
A rough time-table and the various tasks are given below:
Note: 700mW NPRO sitting on AP table (Model No: 126-1064-700, Sl No. 415) = Alberto's laser
Temperature dependence of frequency of Alberto's laser:
a) Shifting Alberto's Laser (AL) to the PSL table and setting up a beat frequency measurement between AL and PSL
b) Determining the frequency vs Temperature curve for the AL
Repositioning the optics on the Y-end table and relocating Alberto's laser ( at this point it will be rechiristened as Y-End-NPRO )
I updated our lockin simulink pieces to use the newer, more streamlined lockin piece that is currently in CDS_PARTS (with new documentation block!). It means we are no longer passing clock signals through three levels of boxes.
In order to use the piece, you need to right click on it after copying from CDS_PARTS and go to Link Options->Disable Link. This forces the .mdl to save all the relevant information about the block rather than just a pointer to the library. I talked with Rolf and Alex today and we discussed setting up another model file, non-library format for putting generically useful user blocks into, rather than using the CDS_PARTS library .mdl.
The BS, ITMX, ITMY, PRM, SRM, ETMX, ETMY now have working lockins, with the input matrix to them having the 2nd input coming from LSC_IN, the 3rd from the oplev pitch, and the 4th from oplev yaw.
This necessitated a few name changes in the medm screens. I also changed the lockin clock on/off switch to a direct amplitude entry, which turns green when a non-zero value is entered.
Currently, the Mode cleaner optic suspension screens have white lockins on them. I started modifying a new set of screens just for them, and will modify the generate_master_screens. Unfortunately, this requires modifying two sets of suspension screens going forward - the main interferometer optics and the MC optics.
The reason for using Alberto's laser is that some amount of work has already gone into characterising its phase noise. Ref elog entry 2788
We use Alberto's laser for the Y end Green Locking.
Which laser are we going to use, Alberto's laser or MOPA laser ?
Just for a record. We got 4 new laser pointers (2 greens, 1 blue, and 1 green and red combination). Don't lose them.
They reside in a bucket on the SP table, where IR viewers and sensor cards also reside.
Prior to the works on the Y end setup I propose to perform the temperature scan business like Koji and Suresh did before (see this entry).
This business will allow us to easily find a beatnote at 532nm after the installation on the Y end.
I guess the right persons for this work are Bryan and Suresh.
Bryan will have a safety guidance from Steve in this after noon. So after that they can start working on it.
/* - - - coarse plan - - - */
* remove Alberto's laser from the AS table
* setup Alberto's laser on the PSL table
* put some stuff such as lenses, mirrors and etc. (Use the IR beam picked off after the doubling crystal for the main laser source)
* mode matching
Which laser are we going to use, Alberto's laser or MOPA laser ?
Steve pointed out to me today he couldn't get trends for his PEM slow channels like C1:PEM-count_full.
I experimented a bit and found for long time requests (over 20 days), it would produce minute trends up to the current time, but only if they started far enough back. So the data was being written, but something was causing a problem for dataviewer/NDS to find it.
On further investigation it looks to be some incorrect time stamps at several points in the last few months are causing the problems. Basically when Alex and I made mistakes in the GPS time stamp settings for the frame builder (daqd) code, the wrong time got written for hours to the raw minute trend data files.
So Alex is going to be running a script to go through the roughly 180 gigabytes of affected trend data to write new files with the correct time stamps. Once it done, we'll move the files over. We'll probably lose a few hours worth of recent trend data, depending on how quickly the scripts run, but after which minute trends should work as they are supposed to.