Open-ended questions and problems are indeed awesome. Moreover, they are an essential part of a sound education in mathematics, even at the K-12 (primary and secondary schooling) level of learning. But open-ended questions used for teaching purposes should be carefully written for sound teaching points, and teachers using them should have sufficient background in mathematics to guide student approaches to grappling with them. One of my favorite authors on mathematics education reform (Professor Hung-hsi Wu of UC Berkeley) began writing on that issue in 1994 with his article, "The Role of Open-ended Problems in Mathematics Education,"
and he followed up on that article with a wonderful article in the fall 1999 issue of American Educator, "Basic Skills versus Conceptual Understanding: A Bogus Dichotomy in Mathematics Education."
Since then, Professor Wu has written many more useful articles on mathematics education, including guides for parents, teachers, school administrators, and teacher educators on how to apply the new Common Core State Standards in mathematics better to improve mathematics education in the United States.
A good example of a beguiling textbook by a world-famous mathematician with lots of open-ended problems is Algebra, by the late Israel M. Gelfand and Alexander Shen.
Some of the problems in this book are HARD, but they are generally well posed problems of actual research interest to mathematicians, that just happen to be accessible to pupils just beginning to learn algebra.
AFTER EDIT: answering the question kindly posted below, one example I had in mind is that Gelfand asks students to figure out how many different ways there are to group terms in an expression with parentheses as the number of terms increases. This essentially asks the students to discover the Catalan number sequence.
That's why it isn't trivially easy to be a mathematics teacher. The Russian tradition of mathematics teaching, which goes all the way back to the years when Leonhard Euler researched and taught in St. Petersburg,
does an especially good job of appreciating pure math for its delightful patterns and inherent beauty and elegance while at the same time being well informed by the many applications of math to science and engineering. One of my favorite textbooks for taking a balanced approached to making mathematics teaching interesting, rigorous, engaging, and practical is the late Israel Gelfand's and Alexander Shen's textbook Algebra published by Birkhäuser.
Many of the problems are HARD--the author is not afraid to pose research-level problems to first-time learners of algebra. On the other hand, some of the problems in some sections of the book are very approachable: "How to Explain the Square of the Sum Formula to Your Younger Brother or Sister." One section, "How to Confuse Students on an Exam," is laugh-out-loud funny. I love using this book in math classes that I teach as supplementary weekend classes for third-, fourth-, and fifth-grade-age pupils who like math and who want something more challenging than what is served up by the local school systems. I have clients from seven different counties in my sprawling metropolitan area. Pure math can be fun, and applied math can be fun, and both can be more enjoyable when they are taught hand in hand.
Gelfand was a mathematician who also cared deeply about mathematics pedagogy, and his textbooks are delightful.
Gelfand poses delightful problems that give students a workout in arithmetic (and CAN'T be done with a calculator) and that build conceptual understanding and interest in higher mathematics.