For my Advanced Evaluation in Education course, we were assigned to create a mind map or presentation of some kind that communicates our understanding of evaluation and its intricacies. For my project, I chose to design an infographic that illustrates a metaphor comparing the roles that evaluators play to the roles in a traditional role playing game (be it video game or table top).

When I began the project, it was overly ambitious. I had determined that I was going to include several sub-classes for each of the RPG roles, i.e. Tank, Support, and Damage per Second (DPS). I brainstormed that Tanks would be further divided into Warrior, Knight/Paladin, and Dark Knight. For Supports I was trying to decide how I would divide the different support roles and what labels I would use. I came up with three different categories of Support characters: (1) White Mage/Healer/Cleric/Priest, (2) Troubadour/Priest/Bard/Songstress, and (3) Tactician/Scholar/Sage/Strategist. I faced a similar problem with DPS trying to choose between: (1) Archer/Hunter/Ranger, Black Mage/Wizard, and Monk/Rogue/Mercenary. As I began to detail each of the sub-classes, I realized that I was going to end up with lots of overlap (which usually exists in RPGs anyway), and that I was too focused on creating RPG classes instead of making comparisons to evaluation. Thus, I decided to only including the three basic categories of roles, and focus on my comparisons within them.

Despite concluding this project for now, there is more that I would like to do. I limited myself to an artboard that was 8.5 inches by 11 inches, so it would fit on copy paper. I would like to enlarge the artboard so I could add more content. I would like to provide more details about each evaluation method as a rationale for why I placed them where I did, e.g., explaining why I think Tank characters are formative and internal as opposed to summative and external like DPS characters. I also would have liked to list famous evaluators under each class. For example, I would have liked to add a section to the Tank category called TOP TANKS that listed evaluators like Tyler, Provus, and Weiss; a section for SUPPORTS called SUPER SUPPORTS that listed evaluators like Stufflebeam, Alkin, Guba, and Lincoln; and a section under DPS called DISTINGUISHED DPS featuring names such as Scriven, Eisner, and Accreditation “Guilds.” If I had more time, I would love to create a larger poster with more evaluation details, and more recognition for leading evaluators.

Since I was unable to share my rationales on my infographic, I decided that I would share them here. In RPGs, Tank characters initiate battles and are the first into the fray. For these reasons, I related them to internal evaluators who complete formative evaluations. Tanks also tend to focus on objectives so they know where to lead their party. This focus on goals makes tank evaluators most suited for program and goal-oriented evaluations.

Support characters tend to be the party tacticians. Supports help the rest of the party choose which actions to take based on the ways in which their support abilities can turn the tides of battle. The decision making aspect of the support’s role makes them understandably well suited for decision-oriented evaluations. As part of providing information for making decisions, support evaluators can be external or internal, but are mostly formative in nature. Support evaluators also collaborate with party members to choose the best actions, which makes them well suited for participant-oriented evaluations and capacity building.

Lastly, DPS characters are the primary damage dealers in RPGs. They usually try to stay out of the central fray, allowing tanks to take damage while they damage adversaries. Due to their fragile nature, DPS players are usually more experienced than tanks, using their knowledge to decide when to attack and when to retreat. DPS characters are also usually the characters that determine whether a battle is won or lost. Due to their experience and opportunity to influence the outcome of a battle, DPS evaluators are mostly external summative evaluators. This makes them well suited for metaevaluation and expertise-oriented evaluations. Finally, due to their expertise, DPS evaluators may sometimes approach an evaluation using a goal-free approach.

I really appreciate the opportunity we were given to express how we make connections to the field of evaluation, and how those connections help us understand the field. If you would like to read more about my design process for this project, you can do so here.

We are completing an exercise in class that reminds me just how hard writing can be. The goal of the exercise is to write an introduction to a research question in such a way that it logically leads to the research question.

The introduction should be shaped like a funnel, starting broad and then ending by arriving at the narrowly constructed research question.

I think I may struggle so much with this because my research question may still be too broad, so it’s difficult to narrow my funnel. Yet, here is where I’m at with the exercise.

Step 1 - The Question
I began by writing my research question down; however, I will not post it here in case you want to try and test my funnel yourself. You can see my research question in my previous blog post.

Step 2 - Creating Levels
I began writing the levels that would be needed to introduce my question. Here they are:

Lvl: For many U.S. citizens, education is the focus of the first 18 years of their lives. In fact, most states have Compulsory Education Laws

Lvl: We worry more about outcomes than experiences.

Lvl: But there may be a way to do both.

Lvl: The psychologist Mihály Csíkszentmihályi describes a feeling of complete and energized focus in an activity that has a high level of enjoyment and fulfillment that he calls Flow.

Lvl: Flow Theory states that there are eight mental states that can be experienced during any activity.

Lvl: Researchers have analyzed how components of Flow impact work and leisure, from factory work to surgery, and from active-leisure (e.g., playing a musical instrument, playing a sport, or carving) to watching tv.

Lvl: But little has been done to observe: “The Question”

Step 3 - The Rough Funnel
I then connected the steps into two introductory paragraphs. Here is the funnel:

Education is the primary focus of the first 18 years of most U.S. citizens’ lives. In fact, most states have Compulsory Education Laws that require school attendance for students between certain ages. In evaluating and planning for the education of these K-12 students, social and political forces tend to focus much more on the outcomes of attending school than on the emotional experiences of attending school. Focusing on outcomes is incredibly important; however, because states mandate that students spend somewhere between 8 and 13 years enrolled in an educational institution, we should focus on making these years academically enriching and enjoyable.  

One of the reasons K-12 education can be un-enjoyable is because it does not meet the personalized needs and desires of individual learners. Personalizing instruction based on students’ abilities and providing students with clear feedback concerning their progression towards educational goals can both improve educational outcomes and make education more enjoyable. Flow Theory can be used to create a framework for personalizing education in a way that makes K-12 education enriching and enjoyable. The psychologist Mihály Csíkszentmihályi describes “Flow” as the feeling of complete and energized focus, accompanied by a high level of enjoyment and fulfillment. Csíkszentmihályi claims that Flow is attainable during any activity if a task’s difficulty matches a person’s ability level and if a person can receive nearly immediate feedback about the improvement of his or her abilities. Learning in a K-12 classroom is an activity that could also benefit from Flow Theory. Researchers have analyzed how components of Flow Theory affect the outcomes and enjoyment of work and leisure, but little research has been done to observe: ____________.

Step 4 - Test the Funnel
I asked several people to read my funnel to see if they could arrive at my question. My wife had the most success at arriving at my question, but she has a pretty biased perspective and can fill in the gaps my brain makes. I also asked a couple of fraternity brothers; they got pretty close. My classmates and advisor felt like the second paragraph was a bit derailing with all of the Flow Theory stuff, so I worked to cut it down.

Step 5 - Revise the Funnel
I took the feedback from Step 4 and arrived and this new version of the funnel:

Education is a primary focus of the first 18 years of most U.S. citizens’ lives. In fact, most states have Compulsory Education Laws that require school attendance for students between certain ages. In evaluating and planning for the education of these K-12 students, social and political forces tend to focus much more on the outcomes of attending school than on the emotional experiences of attending school. Focusing on outcomes is incredibly important; however, because states mandate that students spend somewhere between 8 and 13 years enrolled in an educational institution, it is important to focus on making these years academically enriching and enjoyable.
 
Students may find the K-12 years to be unenjoyable because the experience leaves them feeling unfulfilled. Schools can likely change this if they will match individual students’ abilities to the learning tasks they are asked to complete. Csíkszentmihályi (1990) says that matching individual ability level to task difficulty creates Flow, the feeling of complete and energized focus accompanied by a high level of enjoyment and fulfillment. Researchers have shown that components of Flow increased the feelings of enjoyment and fulfillment for work and leisure activities (Csíkszentmihályi, 1994), but little research has been done to answer the question: ______.

Step 6 - Repeat Steps 4 and 5 to Satisfaction
I’m not entirely convinced my funnel is done, but I tested it with many more people this time around, ranging from K-12 and Higher Ed. educators to other graduate students, and I even added a very healthy dose of help from some former students working in education or on undergraduate degrees. There seemed to be much more consensus this time around, and I’m thankful for all the help I received.

Guesses close to the mark included:

“Whether utilizing the Flow approach to education will positively impact students’ success during and after primary and secondary education.” - Former K-12 student and Valedictorian

“If all students were impassioned by the subjects they were studying and projects they were completing, how much more would they learn?” - Former K-12 student working in education

“How beneficial Flow could be to the students of today, and how can it maximize their ability to adapt and learn as they move into the future?” - K-12 pre-sevice teacher

“How can schools increase Flow?” - K-12 teacher

“Where has Flow been show to be effective, and does this research have enough trasnferability to K-12 to show it would be worth the time of [implementation and testing]?” - Fellow graduate student

Guesses further away were:

“Is our current system of education providing enough mental stimulation for our children and teaching them not only to learn but to enjoy the process of learning, or is our primary focus on improving test scores with little to no regard for students mental aptitude and involvement; and if the latter is the case, is it possible to overhaul our archaic system and shift the focus from test scores to mental involvement?” - Rather jaded former K-12 student

“Whether the traditionally structured school day vs. something different has actually led to true achievement.” - K-12 Instructor

-AND-

“Can we play a game?” - Higher Ed. Instructor

My two most recent posts focused on my passion for engaging K-12 learners and my beliefs that Flow Theory might be a way to better engage them. While time and research may reveal that Flow Theory is a Most Valuable Player in planning K-12 instruction, the MVP in the title of this post refers to a different MVP - the Minimum Viable Product. According to Eric Ries, the minimum viable product is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort. Another way that Ries defines the minimum viable product is as the smallest thing you can build that lets you quickly make it around the build/measure/learn loop. This blog post is about the minimum viable product (MVP) for Flow Learning.

The MVP

The minimum viable product for Flow Learning is presenting the application of Flow Theory to students and teachers in the field to see if they have a need or interest in the product. I have already done this to a small sample of teachers and students, and the results are presented in my previous blog post. Presenting these ideas allowed me to test my ideas in the smallest format that could still provide me with valuable feedback for learning. What I learned from this step of the feedback loop is detailed in my previous blog post, so I want to focus on the next step in this blog post.

The next step will be to take what I learned from the results mentioned above and use them to modify my ideas to create a theory-based framework for Flow Theory that can be immediately applied to planning K-12 learning units or modules. One of the challenges to creating this MVP would be making the framework applicable to a K-12 instructional unit. In order for the application of Flow Theory to work effectively, teachers or curriculum designers would need a tool that allows them create a list of activities that will personalize learning to their students, so that students’ learning activities match their ability levels (see the image below from a previous post). The best tool for this is likely a learning management system (LMS) such as Canvas, but then students will also need to have devices that allow them to access content on the LMS. Instead of looking to create a context in which I need to add device access and LMS access for all students, it would be best to measure Flow Learning in a context that already has one-to-one devices and utilizes an LMS. I would also want to test my MVP with a small group of students and teachers so that I can get to know them well enough to understand how Flow Learning can best meet their individual learning and pedagogical needs.

Selling Flow Learning in the Bazaar

Because the idea of implementing Flow Theory depends on meeting the needs of many different users, i.e., students and teachers in different contexts, one of the methods for developing the theory-based framework is to work out its development using the bazaar model made famous by Linus Torvalds in developing Linux. In reading Raymond’s (2000) The Cathedral and the Bazaar (CatB), I discovered many useful tips in how to make the process of testing the MVP effective and beneficial. One of the most important lessons I learned from CatB is that developers “don’t really understand the problem until after the first time [they] implement a solution. The second time, maybe [they] know enough to do it right. So if [they] want to get it right, [they should] be ready to start over at least once.” This idea enforces the need to get the MVP to consumers quickly so I can learn more about the problem I am really trying to solve.

The next idea from CatB that will influence how I test and develop Flow Learning is to treat users as co-developers because doing so comes with many benefits. First, when developers treat their users as co-developers then users can assist in creating rapid improvements for the MVP. Second, incorporating more users in the development process helps in finding more ways and different ways to tease out problems with the product. Third, CatB suggests that if early testers of a product are treated as a developer’s best resource, they will become the developer’s best resource. Teachers have such little time to begin with, so I need to make sure that any time they offer to give me is valued as well as their opinions. Doing so will allow teachers and their students to provide valuable input in redesigning and updating the MVP. Lastly, in valuing beta-testers I will also benefit from their good ideas. According to Raymond (2000), “the next best thing to having good ideas is recognizing good ideas from your users [and] sometimes the latter is better.”

Conclusion

In an earlier post, I stated that in order test my theories on the usefulness of Flow Theory to K-12 instruction, I needed to test my ideas with others. Upon doing so, I have found that there is demand for a program that helps implement Flow Theory into K-12 classrooms. The next step in testing Flow Theory in the classroom is to create a minimum viable product. With such a product, I can allow users to test the product and provide me with feedback. Such feedback can be used to improve the product for re-implementation and testing. One of the keys to product development is to quickly complete the feedback loop. Choosing testers who can provide valuable feedback will help make quick improvements to the product, as well as help identify the problems that need to be solved both by and within the product. Developing the MVP is the next step towards discovering how components of Flow Theory, e.g., matching difficulty of skill and providing frequent feedback, affect a student’s learning.

The Need for a Question

In my last post, I discussed my passion and the direction I believe it is taking me. I ended that post with the idea that the next step was to get out and talk to others to see if my ideas were shared, or if I had misconceptions concerning student engagement, boredom, and anxiety, i.e., the state of flow students experience in K-12 education. To check on my ideas I sent an informal survey out to current or recent teachers as well as current or recent high school students. I received 7 responses from teachers and 13 responses from students. There were interesting discrepancies in the responses. These discrepancies can be seen in Table 1.

I found this data to be interesting because it suggests that teachers are better at recognizing boredom in students than they are anxiety (which makes sense because it is more socially acceptable to be bored than it is anxious). However, the more important suggestion from the data was that the majority or participants believe students are either bored or anxious within certain contexts. I followed the above questions by asking participants to rank the need for a program or tool that would help students experience flow in the classroom from 1-5, and whether they thought such a program would engage students on a scale from 1-5. The responses are in Table 2.

Table 2 illustrates that students feel a greater need for a program that utilizes Flow Theory than teachers do; however, both groups of participants seem to feel that such a program would help engage students in the learning process. The answers to these questions seem to reinforce the ideas from my previous post that students face boredom and anxiety in the classroom. With this information in hand, I set out to find my research question.

Clarifying The Question

In order to create a question that was focused and could be researched, I workshopped my ideas with Dr. David Wiley and other graduate students in my grant writing class. The initial version of my question was:

• To what extent would K-12 students benefit from teachers planning lessons using Flow Theory?

We took this question and identified any variables within the question, words or phrases, that could be changed to make the question more specific. This led to a question that looked like:

• (To what extent would) (K-12 students) (benefit from) (teachers planning lessons) (using) (Flow Theory)?

After listing out different words or phrases for each parenthetical phrase, the question was revised to read:

• Would K-12 students be more engaged in educational lessons that utilize Flow Theory?
  

After repeating the exercise of bracketing and specifying or rearranging different variables in the question, the question went through the following versions:

1. Would K-12 students be more engaged in educational lessons that utilize Flow Theory?
2. If teachers used Flow Theory, how (would) it affect (student’s) flow?
3. If teachers used Flow Theory, how would it affect K-12 student’s flow?
4. If teachers used Flow Theory as part of their pedagogy, how would it impact (student engagement)?
5. If teachers used Flow Theory as part of their pedagogy, how would it impact time-on-task?
6. How would (Flow Theory) affect a student’s (flow)?
7. How would components of Flow Theory, e.g., matching difficulty of task to student’s skill and providing frequent feedback, affect a student’s learning?

This final form of the question (#7) is the question that I will be focusing on for my future research. There is still more work to be done with the question, such as specifying which students I am concerned with researching. Also, I need to make sure that I am not repeating work that has already been done. Next, I must get into the literature to discover what work has been done in the realm of K-12 Flow Theory applications.