Rana pointed out that the delay may be caused by the 110B DAQ, as it integrates over 2ms (5 clock cycles at 2048Hz on the fe computer), to make low noise measurement. However, the C0DCU knows about this delay and corrects it by fudging the time stamp, before sending it to the frame builder, so that the time stamps match the actual measurement time. But, the ASS computer is not aware of such an integration time, so it does not adjust the time. We verified that it is indeed the case. This is what we did (as suggested by Rana):
We split the signal from the MODE cleaner board "OUT" port using a T-splitter to the original PENTEK board (C1:SUS-MC2-MCL-IN) and the PEM ADCU channel #2. Then measured the mutual delays between the signals that are processed by C0DCU and the ASS computer for both the MC_L signal and the corresponding output through the PEM channel. We clearly see the same delay (compare red and brown in the bottom panel) between the signals that are going through 110B and the PENTEK DAQ. This delay is a bit noisy, possibly because the PENTEK is not as low noise as the 110B is.
There is some delay (pink curve in the bottom panel) between the PENTEK DAQ and the frame builder corrected 110B output, much smaller than 2ms, could be ~200-400 u sec. Which should correspond to the 1 or 1/2 cycle delay caused by the PENTEK DAQ.
So, once we have the planned advLIGO DAQ system, there should not be any long delay. Perhaps, to solve the problem and make OAF functional soon, we will upgrade the PEM DAQ asap, rather than waiting for the rest of the upgrades...