It seems that if you stick with the "plan, do, adapt; plan, do, adapt" idea of agile (iterations, iteration reviews) you would avoid those things by default. BDUF is just so contrary to the idea of agile estimating & planning that if you are really agile, you wont be BDUF automatically.
The purpose of release & iteration planning meetings is to make sure you are adding the most valuable features to the project for that iteration. If you keep that in mind, you'll avoid YAGNI for free.
I would very strongly recommend the Mike Cohn books on agile planning:
Update: After your clarification about avoiding YAGNI and BDUF within an iteration...
BDUF...If I felt a feature was not clearly defined before I started work on it, I would create a small "feature" or story to account for the design type portion of the work needed. So that maybe the smaller story has a story point estimate of 1 instead of the real feature's 5. That way, the design is time-boxed into the smaller story, and you will be driven to move on to the feature itself.
To avoid violating YAGNI I would work to be very clear about what the customer expects for a feature within an iteration. Only do work that maps to what the customer expects. If you think something extra should be added, create a new feature for it, and add it to the backlog of work to be done. You would then persuade the customer to see the benefit of it; just as the customer would push for a feature being done at a certain point in time.
A very good place to start as far as books are concerned is User Stories Applied and Agile Estimation and Planning both by Mike Cohn. This have excellent examples and good starting points for anyone first coming to agile methodologies.
As far as website resources they are few and far between. Probably a good place to actually start would be searching for those keywords on Google Images as many people take photos of their burndown charts and User Stories. This helped me a lot when starting. Here are some samples: Burndown Chart, and User Stories
Please note however while a burndown chart is a simple report that you run on your current story points left in an iteration, User stories are more complex than that and do require a bit of reading to wrap your head around. Start with User Stories Applied book for that.
Hope that helps!
As a compliement to akf's answer, I would like to recommend a comprehensive guide, User Stories Applied.