Yes, you can install Fedora on ReiserFS, JFS, or XFS.
None of these file systems are officially supported by the Fedora Project. (That means that you can use them, but you won’t find a lot of official help from the Fedora Project if things go wrong.)
At the installer prompt, type this for ReiserFS:
linux selinux=0 reiserfs
or this for JFS:
linux selinux=0 jfs
or this for XFS:
linux xfs
NOTE: You cannot use SELinux on ReiserFS or JFS. XFS is OK. (If you don’t know what SELinux is, you can ignore this warning.)