I think part of it is that they simply raised prices, regardless of inflation. It's also why they went from the three hardcover, six paperback model to the nine hardcover model.
My new project is to see which Star Wars novels hit the New York Times bestseller list, although it doesn't track Internet sales which could skew it toward stuff from the 1990s. So I'll see if I can find some other bestseller lists, maybe Amazon has a historical list rather than just cumulative sales ranks?
Edit: Interesting... TFU and the Clone Wars film novelization both outsold Invincible in 2008.
Not entirely sure whether that's due to Invincible being Invincible, or the other two being multimedia releases. I'm inclined to believe the latter. The children version of the novelization outsold Traviss'. In fact, every children's TCW publication from 2008 outsold the adult novels, with the exception of one which didn't outsell TFU.
And so, the reason why continuity has gone by the wayside has been revealed. And, if I might speculate, why a reboot would actually generate more sales if the reboot resulted in a TCW-esque series starring Han, Luke, and Leia, and generated young reader books set after the films.
Edit 2: So yeah, it seems that the last time a Star Wars novel sold more than 100,000 copies in a year of its release was 2008 with TFU, TCW, and Invincible. And it didn't achieve this feat the year before. On the other hand, kids books related to TCW regularly sell over this number. Unfortunately, it seems book sales data is otherwise extremely hard to come by, and usually isn't free. And the bestselling Star Wars novel seems to have been The Phantom Menace, followed by Attack of the Clones and Heir to the Empire.
Edit 3: Oohh, I may have hit the goldmine. More to come later, I don't have time to process what I just found. Brief snippet: In 2002, both The Approaching Storm and Destiny's Way sold over 100,000. The Approaching Storm sold better than Destiny's Way by around 25,000 or so. Attack of the Clones sold a whopping 784,750.
Edit 4: Here's some numbers:
The Phantom Menace: 1,419,852
Attack of the Clones: 784,750
Vector Prime: 200,000+
Rogue Planet: 200,000+
Balance Point: 150,000+
The Approaching Storm: 125,000+
Destiny's Way: 100,000+
The Unifying Force: 107,775
The Force Unleashed: 103,232
There's also a lot of children's Star Wars books that sold in excess of 100,000 between 1999-2003 that I haven't listed, like the Jedi Apprentice books. Unfortunately, I haven't had success finding stuff from earlier than 1999, and I don't think the stuff I've found covered paperback. And my attempt to find 2006 and Betrayal's sales have been unsuccessful thus far.. but it would appear that Sacrifice didn't break 100,000. Nor did the first three FOTJ novels. So far I've found data for 1999-2003, 2007, 2008, and 2009.
I wonder if the declining sales after the initial onslaught caused by Episode I is why we saw such an increase in volume after the New Jedi Order ended. It's also notable that the first novel in that series sold in excess of 200,000 and the final novel sold just over 100,000. It's also plausible that this data is skewed based upon when a novel is published in the year; for instance, The Unifying Force was published in November, and this data only accounts for that year. OTOH, Vector Prime was published only a month earlier in its year of publication, so I think that it does represent a decline in sales through the course of the series. Plus there seems to be a general trend of sales declining through time, as the list is ordered by number of sales, but seems to be completely chronological with the exception of Attack of the Clones.
It seems plausible that Star Wars novels did not sell all that well, with the exception of Heir to the Empire and its direct sequels, until Episode I's release. If that's the case, the question is whether the license fee has risen since 1991, and whether those types of sales warrant it being renewed by Del Rey or purchased by another publisher.