Does Rails3 work only with html5? Can I use earlier versions of html?
I think you didn't understand, not rails, but the whole concept of web development.
The server side application (no matter language, could be ruby, java, php, C, asm, whatever) just handle requests from client and sometimes (very often) send back informations to the client.
You can generate xml, html, pdf, xls, csv.... anything YOU WANT!!
You can serve xls when you client is requesting for html if you decide to. The server application "don't care" about it.
If you wanna serve xhtml instead html5 with rails 3, just write xhtml in your views and not html5. Change the doctype in your layout and it will be enough.
I'm sure you can even use frames in html5. w3cvalidator will ***** on your face, it's not "right" BUT i'm sure your browser will more or less interpret it correctly.