So why is /frames full? Apparently the wiper script is either not running, or is failing to do it's job. My guess is that this is a side effect of the linux1 raid failure we had over xmas.
It actually looks like the wiper script has been running fine. There is a log from Tuesday morning:
controls@fb ~ 0$ cat /opt/rtcds/caltech/c1/target/fb/wiper.log
Tue Jan 7 06:00:02 PST 2014
Directory disk usage:
Combined 12757641076k or 12458633m or 12166Gb
/frames size 13460088620k at 94.78%
/frames is below keep value of 95.00%
Will not delete any files
df reported usage 97.72%
controls@fb ~ 0$
So now I'm wondering if something else has been filling up the frames today. Has anything changed today that might cause more data than usual to be written to frames?
I'm manually running the wiper script now to clear up some /frames. Hopefully that will solve the problem temporarily.