Filthy Rich Clients: Developing Animated and Graphical Effects for Desktop Java Applications
There has been some advancements in the GUI front in Java.
For example, beginning from Java 6 Update 10, it is possible to make transparent and shaped windows (i.e. arbitrarily-shaped windows commonly seen in media players) in Java:
How to Create Translucent and Shaped Windows
Filthy Rich Clients: Developing Animated and Graphical Effects for Desktop Java Applications by Chet Haase and Romain Guy is a good book full of tips on making advanced GUIs in Swing.
Swing is very flexible with regards to look-and-feel and you can certainly make very good looking applications with Swing.
I recommend the book Filthy Rich Clients by Chet Haase and Romain Guy if you want to learn how to make nice Swing GUI applications.
Yes, Swing is very customizable. I recommend reading Filthy Rich Clients.
For an example, have a look at Cezanne L&F.