Realtek 889A is the hdaudio codec, right?
2. SPDIF is supported by oss_hdaudio. However, I don't have experience using it, nor using this specific codec with SPDIF. It should work, but I can't guarantee this.
3. OSSv4 works with Ubuntu (I use debian, personally). This
page might be helpful.
You can either download a .deb from this website, or build
an OSS deb from source. Then, you'll need to use 'dpkg' to install it (there's also an official debian package, but it probably doesn't exist on Ubuntu?).
After installation, you'll need to set up
software to use OSS. Many will work automatically, but you probably need to install OSS plugin packages (e.g. libsdl1.2debian-oss or -all, libesd0, libwine-oss, libcanberra-gstreamer, gstreamer0.10-plugins-good (or -bad if it's old), etc.). For testing purposes, I think you can skip most setup after install, until you're comfortable with OSS (or decide to uninstall it).
4. You don't have to reload OSS, but there's no GUI for this. The default output is /dev/dsp symlink, and the spdif output is probably /dev/oss/oss_hdaudio0/spdout0 (usually symlinked to /dev/dsp_ac3). You can either order software individually to use the spdif output and the method to do that is slightly different for most programs (e.g. 'mplayer -ao oss:/dev/dsp_ac3' etc. other programs have other switches/config files), or you can change /dev/dsp symlink back and forth (probably 'sudo ln -sf /dev/oss/oss_hdaudio0/spdout0 /dev/dsp' and 'sudo ln -sf /dev/oss/oss_hdaudio0/pcm0 /dev/dsp').
Pulseaudio using programs may be able to use its interface, but I'm not sure whether defining multiple OSS devices in its config works (the one time I tried, it copied the device description, so it was a bit clunky. Another person told me his copy couldn't use anything other than /dev/dsp device). Note that Pulse requires some manual fiddling
in order to work at all with OSSv4.
5. I don't know. You'll probably have to set module-oss to use the /dev/oss/oss_hdaudio0/spdout0 device too, but that might be buggy (see above). Btw, I suspect Pulse can be made to do AC3 with ALSA, but that probably requires some fiddling with ~/.asoundrc. That's beyond my knowledge.