As of this writing, the DokuWiki appears to be working.
As you and I suspected, it looks like this was a clusterwhoops with the permissions for the NFS mount. Let's recap what happened in the past 24 hours:
- Yesterday, 8 PM: I restart the Apache server, thereby resurrecting the SVN (now conveniently located at /export/home/svn). The DokuWikis remain borked.
- Yesterday, 7 to 11 PM: Zach, Rana, and Jenne perform deep magic to get the front-end machines up and running again. This should be irrelevant for this Apache/SVN/DokuWiki witchcraft.
- Today, morning: the townsfolk happily resume their svn up and svn ci festival.
- Today, ca. 3 PM: Zach runs stopapache.sh to bring down Apache, thinking he can bring it back up momentarily with startapache.sh. But NFS is a jealous and vengeful god, and Apache now complains that it doesn't have permission to write to its logfiles, and therefore can't start httpd.
- Today, 5 PM: "How can this be?", Zach and I ask. Apache had no problem starting up yesterday night, and to our knowledge nobody futzed with chiara's NFS mount of /home/cds. This change remains mysterious to us.
Despite the aforementioned mystery, Zach and I pressed on and tried to diagnose the permissions issue. We found that even if you are logged into nodus or pianosa or rossa or whatever as the controls user, the NFS mount saw us as the user nobody (in the group nogroup). If we created a file on the NFS mount, it was owned by nobody/nogroup. If we tried to modify a file on the NFS mount that was owned by controls/controls or controls/staff, we got a "permission denied" error, even if we tried with superuser privileges.
It turns out this has to do with the vagaries of NFS (scroll down to gotcha #4). We have all_squash enabled in /etc/exports , which means that no matter your username or group on nodus, rossa, pianosa, or harpischordsa, NFS coerces your UID/GID to chiara's nobody/nogroup. Anyway, the fix was to go into chiara's /etc/exports and change this
/home/cds 192.168.113.0/255.255.255.0(rw,sync,no_subtree_check,all_squash,insecure)
to this
/home/cds 192.168.113.0/255.255.255.0(rw,sync,no_subtree_check,all_squash,anonuid=1001,anongid=1001,insecure)
where 1001/1001 are the UID/GID for chiara's controls/controls (as opposed to 65534/65534 for chiara's nobody/nogroup). That way, the NFS mount sees you as chiara's controls/controls.
In order to make chiara's NFS daemon aware of the new /etc/export settings, I ran sudo exportfs -r based on the answer to this StackOverflow question. As with all the best StackOverflow questions, the moderators closed it for being "off-topic".
[Edit, 2014-06-11, 11 AM: I've repeated the above anonuid/anongid change for the /home/cds/caltech/home/controls mount, so that nodus's /home/controls is writeable as well. I've also added a .screenrc for nodus in order to maintain sanity.] |