I vaguely remember it being something about having a -devel package?
I'm asking because Festival is currently built for both archs, and there's really no reason. I've made it multiarch compatible (bug #228315), but really, there's no good reason for it to be, since nothing links with it.
The only program that requires it, in fact, is gnome-speech, and that communicates via a forked process. It's really meant to be used that way or as a network server.
In fact, I'm not sure it's even terribly useful to ship a devel package for this -- it's really not designed for things to be built against it in that way. You can add modules, but the intended process is to build them as part of the main package build. (Arguably the speech-tools libraries could be split off into a completely separate package and then festival built against _those_ devel packages, but the festival build process wants an incestuous sort of availability of the speech-tools source tree, so that's not really an option.) So if ditching the devel package is what it takes....