DISCLAIMER: None of what you'll read below is of my own work, but mostly a summary on what I read about on the Debian PPC mailing list.
I have what I believe is somewhat exciting news for G5 owners in regards to using Nvidia cards and the open source nouveau driver, but more testing needs to be done! What I'm about to explain and show should hopefully be able to allow you to at least achieve 2D acceleration.
However, before we become too excited, let us take a step back first and examine the situation with nouveau on G5 PowerMacs. As of today (7/5/2015), I'm the only other known individual who has been able to make this fix work with the other individual being the one who filled us (the Debian PowerPC mailing list) in on how to get it working. To be clear, I'm using Late 2005 PowerMac (Identifier 11,2) using a Quadro FX4500 Nvidia card.
As already known from my previous post on a fresh install of Jessie on a G5, I had to pass in an extra yaboot parameter (nouveau.noaccel=1) that would disable any acceleration with the Nvidia card using the nouveau driver. This would at least give us G5 owners with an Nvidia card a usable system, although all graphics rendering was happening on the CPU. Not the most terrible thing in the world when using a dual-core G5, but definitely disappointing knowing how much power the graphics cards have to offer.
Since the release of Jessie, there have been a handful of threads on the mailing list about issues other users were having when trying to use the nouveau driver. Many people, myself included, recommended the yaboot parameter fix due to lack of expertise and knowledge of the nouveau driver, graphics programming, and troubleshooting. Just this past week or so, another Debian user posted a thorough analysis on why the nouveau driver was having issues with the Debian 3.16.0-4 powerpc64 kernel and what could be done as a workaround to overcome the issues and at least achieve 2D acceleration. Notice that I said "workaround" and not "fix."
Basically, it came down to one or two different kernel configurations or both.
1) Right now, only a 4Kb pagesize configuration in the kernel will work with the 64-bit G5 kernel running the nouveau driver. The default pagesize of the included kernel-image (again 3.16.0-4-powerpc64 in Debian Jessie) is 64kb, which as mentioned by the poster improves performance of the driver, but again is not supported on G5 PowerMacs. To check the pagesize in use by your running kernel run the following command in a terminal:
getconf PAGESIZE
This will likely either return 4Kb or 65536Kb.
2) The kernel configuration might also be set to have the nouveau driver use MSI interrupts. As quoted by the poster regarding these interrupts:
The newer nouveau drivers in more recent kernels default to using MSI interrupts, however with the PPC G5, when using MSI interrupts, the powerpc FCODE rom on Nvidia cards does not correctly set up the MSI address (or vector).
To verify if you are using MSI interrupts or not, run the following command in a terminal:
cat /proc/interrupts
Look for the nouveau entry and confirm whether it is MSI or MPIC.
What the poster ended up doing was rebuilding a newer 3.18.16 kernel, disabling MSI interrupts as well as adjusting the pagesize in the kernel configuration to 4kb. Not only that, but he has also posted a copy of this kernel image and headers for G5 owners to download and test on their G5 machines.
So this is a call to all other G5 owners running the latest Debian Jessie release and using an Nvidia graphics card. Download and install the files linked above and post your results to the Debian PPC mailing list ASAP! If need be, drop a comment with your results, and I will plug them into the aforementioned thread on the mailing list. I would recommend downloading and installing this newer kernel image and headers on a brand new Jessie install if possible.
Remember to add in a new kernel configuration in your /etc/yaboot.conf file that links to the newly installed kernel and run sudo ybin -v to wrap it all up. From there, go ahead and reboot to the new kernel and test things out.
I can vouch that the above kernel does allow 2D acceleration now on my G5 PowerMac. Also, as I mentioned in the mailing list thread, I had to load the i2c-powermac module to stop the fans from sounding like a 747 ready for takeoff. Now that I have 2D acceleration, I can now use xrandr and arandr to set my dual monitors up properly in an extended desktop setup that spans across both of them. Now if we could only figure out how to achieve 3D hardware acceleration us G5 owners would be set for a little while longer.
Bug reports for these two problematic issues and configurations have been filed here and here. From the sounds of it, you would not be able to make these two configuration changes work on G5 with the current Debian kernel 3.16.0-4 so do not pull out your hair trying.
I did some browsing around on the nouveau website and found that this G5 PPC pagesize issue is listed as a DRM to-do on this page. My plan is to touch base with them about it again and see what, if anything, I can do to help. In the meantime, I've been reading up more and more on the nouveau driver and Linux graphics in general, so here is to hoping I'll be able to assist in some small way. The developers of nouveau have some excellent documentation over at their site so I encourage you to check that out if you would like to also assist in any way.
Also, and lastly, it appears that 4.0+ kernels present all sorts of new issues with nouveau and G5s, but if nothing else, this small bit of information outlined above is hopefully a good sized step in the right direction! We need to continue to do our duty of reporting any bugs we find regardless of whether anybody who currently is a developer wants to devote any time to them. It helps just to have that information out there and available.