Process
PHASE I: PLANNING
Objective:
Identify three books/publications with high-contrast images suitable for augmented reality (AR) and select one for development.
Tasks:
Identify three books/publications with high-contrast images suitable for augmented reality (AR) and select one for development.
Tasks:
Choose three books with at least 7–10 high-contrast images.
Provide a rationale for selecting each book.
Select one book for AR development.
Deliverable:
Developer blog post including:
Book list and images
Rationale for selection
Chosen book
The three main books I wanted to look into doing were all based around children books and interactions to further push the imagination of the photos and content within the book. My top 3 (no particular order) are as follows:
Thrill Rides: Top 10 Roller Coasters in America
Using Adobe Aero, this project can feature interactive 3D models of the roller coasters that users can view from different angles. On-ride POV videos can provide a realistic experience of the thrill, while hotspots reveal fun facts about the coasters’ speed, height, and history. Creating an engaging and educational experience for roller coaster enthusiasts.
If You Give A Mouse A Cookie
Enhance the charm of this story with Adobe Aero by animating the mouse with a 3D model (SketchFab), allowing it to interact with users in AR. Virtual pop-up scenes from the book can appear, letting users engage with items like cookies, milk, and crayons. Each object triggers animations or narrations, bringing the storyline to life and immersing readers in an interactive storytelling experience.
Frog and Toad
Bring the world of Frog and Toad to life with AR using Adobe Aero by animating the characters and their seasonal adventures. I want to find a way to create immersive extensions to the illustrations. Users can explore interactive scenes like the gardening, baking, or riding bicycles, complete with voiceovers and animated gestures based on clicking. Seasonal settings, such as a winter snow scene or spring blossoms, create an immersive environment, while personalized interactions with Frog and Toad add to the experience.
Taking selections from each of the books, I began to lay out the ratings of what was most useful for my augmentation goals through the use of VuPhoria. From here I began to conclude the most successful augmentation would be with Frog & Toad by using Aero. Below are the results of the image analysis.
Blog Post:
Choosing "Frog and Toad" for AR Development
For my augmented reality (AR) project, I selected Frog and Toad as the focus for development due to its rich, engaging content and the unique opportunities it presents for immersive interactions. The timeless nature of this book, combined with its high-contrast illustrations and seasonal settings, made it a strong candidate for AR augmentation using Adobe Aero.Frog and Toad is a beloved children's story with diverse, visually rich scenes that lend themselves well to AR. From gardening and baking to cycling, the book offers a variety of activities that can be brought to life through interactive animations. I plan to animate the characters—Frog and Toad—as they engage in these activities, allowing users to tap or swipe to trigger actions such as planting seeds or riding bicycles. Additionally, the seasonal transitions, such as the winter snow scene or the blooming of spring flowers, provide a dynamic setting that enhances the immersive experience.
The high-contrast images in Frog and Toad also make it an ideal candidate for AR, as they stand out well in augmented reality, making it easy for users to interact with the environment. The simplicity of the illustrations allows for clear, smooth transitions between different scenes and activities, while still providing enough detail to keep users engaged. By incorporating voiceovers and animated gestures based on user interactions, the goal is to create an experience that connects users to the characters and their world, giving them the opportunity to explore the book in a more interactive and personalized way. In conclusion, Frog and Toad offers the right balance of engaging content, interactive potential, and visual appeal for my AR project. It seems like a fitting choice to develop an immersive experience that enhances the storytelling and brings the book’s adventures to life for users.
UNIT WORKSHOP SERIES: Applying to "Frog and Toad" Project
WORKSHOP 2
The second Unity workshop series taught me the basics of the Unity interface, file setup, and navigation, which felt intuitive since Unity shares similarities with Blender. This knowledge allowed me to start planning how I might augment the "Frog and Toad" scenes with interactive elements. I also learned how to use Vuforia to scan and import images into Unity, which I could apply to create AR elements, such as virtual books or objects from the story that users could engage with in their environment.
WORKSHOP 3
In this workshop, I learned how to build a simple user interface with interactive buttons. Applying this to my "Frog and Toad" project would allow me to design interactive menus where users could select different scenes or activities involving the characters. For instance, a button could trigger a short animation of Frog and Toad gardening or interacting in the seasons, enhancing the AR experience and allowing users to explore their world more interactively.
WORKSHOP 4
Workshop 4 focused on adding video and audio into Unity scenes. If I applied this to my project, it would significantly enhance the interactive experience by embedding voiceovers or background music directly into the augmented scenes. The ability to incorporate richer media in Unity, including higher video quality, would allow me to add storytelling elements such as Frog and Toad conversations, seasonal soundscapes, or narrated actions that enhance the immersion and appeal of the AR experience.
WORKSHOP 5
Workshop 5 explored creating animations based on user input, which directly relates to adding dynamic character movements in my project. Understanding node-based animation in Unity was useful, and comparing it to Adobe Aero, Unity offers more flexibility, with the ability to manage different states for characters. In the "Frog and Toad" AR experience, I could animate characters to interact in more varied ways, like responding to user gestures or tapping specific areas to trigger new actions, providing an engaging, interactive environment beyond simple animations.
WORKSHOP 6
Workshop 6 demonstrated how to integrate touch and other user interactions into Unity. This would be a major advantage for my "Frog and Toad" project, as Unity allows more sophisticated touch-based interactions, such as dragging objects or selecting different activities. In comparison, Adobe Aero offers more limited interaction types. By implementing these advanced touch interactions in Unity, users could, for example, interact with objects in Frog and Toad’s environment, like planting seeds or moving seasonal items around, creating a deeper level of engagement with the characters and their world.
PHASE II: UX DELIVERABLES
Objective:
Understand user needs and map the current (AS-IS) journey of interacting with the book before AR integration.
Understand user needs and map the current (AS-IS) journey of interacting with the book before AR integration.
Tasks:
Create a persona and user story for a specific reader.
Develop an AS-IS journey map covering pre-, during-, and post-reading activities.
Deliverable:
Updated developer blog with:
Persona
User story
AS-IS journey map
Persona:
Name: Emma
Age: 6
Occupation: Kindergarten student
Tech Abilities: Beginner – enjoys using a tablet for educational games and occasionally watches interactive videos.
Reading Level: Early readers enjoy listening to stories being read aloud.
Interests: Nature, animals, adventure stories, playing with friends, and learning new things.
Goals: Emma loves stories with cute animal characters and enjoys interactive experiences that make her feel part of the story. She wants to explore the world of Frog and Toad in a fun and engaging way while learning about friendship and nature.
Frustrations and Pain Points: Emma sometimes struggles to focus for long periods of time when a book lacks interactivity or exciting visuals.
User Story:
As a young reader (Emma), I want to experience the world of Frog and Toad in an engaging and interactive way so that I can enjoy the story more deeply, learn about the seasons, and feel like I’m part of the characters’ adventures. I want to tap, swipe, and watch animations that bring Frog and Toad to life, especially the scenes where they’re doing activities like gardening and baking, so that I can better understand and enjoy the story.
AS-IS Journey Map:
1. Pre-Reading Activity:
- Scenario: Emma sits down with her parents or teacher to read Frog and Toad. The book is opened, and the parent reads aloud to Emma, guiding her through the story.
- Emotions: Excitement and curiosity as Emma looks at the colorful illustrations and listens to the narration.
- Pain Points: Emma may lose interest if the story becomes too long or lacks interactive elements. Sometimes, Emma struggles to focus on the text without any visual stimulation beyond the pictures.
2. During Reading Activity:
- Scenario: Emma follows along as the parent reads the book. She looks at the illustrations, paying attention to the characters and scenes. At times, she asks questions or comments on the pictures.
- Emotions: Engagement when the story gets exciting, like when Frog and Toad are on an adventure.
- Pain Points: The reading experience feels passive. Emma may not fully understand or relate to some of the scenes if they aren't interactive or animated. Some pages may feel too static for her to engage with for long periods.
3. Post-Reading Activity:
- Scenario: After finishing the story, Emma discusses the events with her parents, perhaps asking questions about the characters or the plot. She might want to revisit her favorite pages.
- Emotions: Satisfaction, but also the desire to revisit the characters or the specific scenes she enjoyed the most.
- Pain Points: Without additional interactivity, Emma may forget some details of the story or not retain the experience as deeply. The lack of engagement after reading can lead to diminished excitement.
From these scenarios and thought processes we can begin introducing interactive animations, sound, and voiceovers that would create a more immersive and memorable experience, making the book even more exciting and educational for Emma.
PHASE III: DIGITIZING IMAGES
Objective:
Prepare and validate images for AR marker creation.
Prepare and validate images for AR marker creation.
Tasks:
Digitize images using a scanner or camera.
Upload images to Vuforia Target Manager to assess augmentable ratings (minimum 4 stars).
Deliverable:
Updated developer blog with:
Reflections on image digitization and AR marker creation
Screenshots of image ratings
Reflections on Image Digitization and AR Marker Creation:
For the development of Frog and Toad in augmented reality, I first focused on digitizing the images from the book, which would be super important for creating the AR experience. The process began by scanning the book's pages to capture high-quality, high-contrast images that could be effectively integrated into AR environments using Adobe Aero.
Digitizing the illustrations was key because AR relies on clear visuals to trigger interactions. Given that the book features hand-drawn illustrations, ensuring that each page scan was high-resolution was the most important. I was able to find a scanned version of the book online. I used Vuforia to rate and create files that would be integrated with the physical book pages, creating a bridge between the user and the interactive features of the book.
The challenge of creating effective AR markers was ensuring that they would be easily recognized and detected through a mobile device's camera. To solve this, I tested the markers under different lighting conditions and distances. This was easy with a few tweaks of the edits to maintain a recognizable print and rating on VuPhoria. Below is a screenshot up the updated ratings.
PHASE IV: IDEATE AND REFINE CONCEPTS
Objective:
Refine AR concepts and visualize their impact on the reading experience.
Refine AR concepts and visualize their impact on the reading experience.
Tasks:
Define AR concepts for selected markers.
Create UX deliverables reflecting improvements from AR integration.
Deliverable:
Updated developer blog with:
Concept iterations and reflections
UX deliverables
For the AR elements, I had a clear vision of the layering aspects I wanted to incorporate into the project. These layers aim to enhance the user's engagement by introducing a form of extension and 3D dimension discovery. The primary goal was to allow the user to explore and interact with the space in an immersive way while still maintaining the constraints of a tabletop format. By refining the AR concepts, I aimed to create an experience where the user feels both grounded in the physical environment and connected to the digital augmentation.
To achieve this, I focused on integrating a voice-over feature that activates during user interaction. This element provides auditory guidance and narrative depth, enhancing the storytelling aspect of the AR experience. Additionally, I explored the use of augmented reality in conjunction with artificial intelligence to dynamically expand the scene, allowing the content to evolve based on the user's engagement. For example, interacting with certain markers could trigger animations or reveal hidden layers, creating a seamless blend of physical and virtual storytelling.
PHASE V: JOURNEY MAPPING
Objective:
Address pain points in the AS-IS journey and design the journey map with AR integration.
Address pain points in the AS-IS journey and design the journey map with AR integration.
Tasks:
Identify pain points in the AS-IS journey.
Design journey map addressing these pain points.
Deliverable:
Updated developer blog with:
journey map
Scenario revisions
Journey Map for Emma’s Reading Experience
Persona : Emma, a young child exploring Frog and Toad with the guidance of a parent or teacher.
Pre-Reading Stage
Actions:
Emma sits down with her parent or teacher to begin reading. The book is opened, and the adult starts narrating while showing the illustrations.
Touchpoints:
- Book cover and illustrations.
- Verbal interaction with the adult.
Emotions:
- Positive : Curiosity and excitement about the colorful illustrations and story setup.
- Negative : Potential boredom if the session feels too long or lacks interactive elements.
Pain Points:
- Limited ways to sustain Emma’s attention.
- Static visuals may not fully engage her imagination.
During Reading Stage
Actions:
Emma listens to the story, follows along visually, and occasionally asks questions or comments on the pictures.
Touchpoints:
- Page-by-page exploration of the book.
- Verbal exchanges between Emma and the adult.
Emotions:
- Positive : Engagement during adventurous or visually exciting moments.
- Negative : Loss of interest during slower, static scenes.
Pain Points:
- The experience feels passive, relying solely on the narrative and illustrations.
- Lack of dynamic or interactive features to sustain engagement.
Post-Reading Stage
Actions:
Emma reflects on the story, asking questions and revisiting her favorite pages.
Touchpoints:
- Re-examining book content.
- Conversations about the story’s events and characters.
Emotions:
- Positive: Satisfaction from completing the story and recalling favorite moments.
- Negative : Missed opportunities for continued engagement.
Pain Points:
- Without interactive features, Emma may forget details over time.
- Limited ways to extend the experience beyond the book itself.
Opportunities for AR Integration
Pre-Reading Enhancements
- Use AR to bring the book cover or title page to life with 3D animations or character introductions.
- Add an interactive story preview to spark Emma’s interest before reading begins.
During Reading Enhancements
- Integrate AR animations that activate when Emma interacts with illustrations, such as characters moving or scenes coming to life.
- Include touch-enabled features, like guiding Frog and Toad through their adventures.
- Add voice-over narration synced with AR visuals to create a richer, more immersive experience.
Post-Reading Enhancements
- Develop AR-powered activities, such as interactive games or quizzes based on the story’s content.
- Create an “AR Memory Book” feature that allows Emma to save and explore her favorite moments with Frog and Toad.
- Offer an AR recap feature, highlighting key moments and expanding on the story’s themes.
By integrating AR at each stage of Emma’s journey, the reading experience can transform into a dynamic, interactive adventure, addressing her pain points while deepening engagement and retention.
PHASE VI: ROUGH PROTOTYPE
Objective:
Develop a preliminary AR experience using placeholder assets.
Develop a preliminary AR experience using placeholder assets.
Tasks:
Use Unity/AERO to build a rough prototype.
Test interactions and visualize the final product.
Deliverable:
Updated developer blog with:
Prototype images/screenshots
Reflections on the creative process
Prototyping
These layered assets are designed to enhance the parallax effect, providing a sense of depth and movement to the anchored imagery. At present, these files do not include animations or audio attachments, as their primary purpose is to serve as a foundational framework for testing the functionality of the anchored imagery.
By aligning these assets with the booklet on the table, users can ensure accurate proximetric interactions, verifying that the layers appropriately respond to spatial positioning and depth. The layered design is particularly effective for evaluating the dynamic interplay between the physical and augmented reality elements, helping to refine the experience for users.
In the final implementation, these assets are intended to be interactive. Users will be able to flip through layers, triggering animations and playing associated audio clips to enhance the storytelling and engagement. This interactive functionality will provide a multi-sensory experience, blending visual, auditory, and tactile elements to create a cohesive and immersive augmented reality environment.
PHASE VII: USER TESTING
Objective:
Test the prototype and gather user feedback.
Test the prototype and gather user feedback.
Tasks:
Export prototype to a mobile device for testing.
Record user interactions and collect feedback.
Deliverable:
Developer blog update with:
Walkthrough video
Reflections on user testing
User Testing Comments and Feedback
Objective
To evaluate the prototype for functionality, ease of use, and user satisfaction, focusing on the layered assets, parallax effect, proximetric interactions, and planned interactivity (e.g., flipping layers and playing audio).
User Feedback (Category Sorted)
1. Parallax Effect and Depth Perception
Comments:
"The parallax effect is noticeable but could benefit from smoother transitions between layers."
"The depth of layers feels intuitive; however, some elements seem slightly misaligned with the booklet."
Suggestions:
Refine the layer transitions to create a seamless parallax experience.
Improve the calibration between the digital and physical layers to ensure precise alignment.
2. Proximetric Interactions
Comments:
"The proximity-based interactions work well, but there’s a slight lag when moving closer to certain layers."
"It’s fun to see how the digital elements react to the movement of my phone”
Suggestions:
Optimize responsiveness for smoother and faster proximetric interactions.
3. Interactivity (Future Features)
Comments:
"I kept trying to flip the layers, even though I knew the feature wasn’t implemented yet."
"Audio cues would add so much more context since the page isn’t visible to read.”
Suggestions:
Prioritize the implementation of flipping mechanics to align with user expectations.
Develop placeholder audio to test the impact of auditory elements on the experience.
Reflections on User Testing:
This user testing session provides valuable insights into refining the prototype, ensuring it aligns with user expectations while maintaining a focus on the intended multi-purpose experience. I feel at the moment I know most of things I am looking to implement in terms of interaction. I am glad most people enjoy the product even without these interactions yet.
PHASE VIII: ASSET CURATION
Objective:
Gather or create final assets for the AR experience.
Gather or create final assets for the AR experience.
Tasks:
Curate sound, images, videos, and 3D models.
Refine style to align with user needs and design goals.
Deliverable:
Updated developer blog with:
Reflections on asset curation
Asset previews
Here, I have compiled the aligned artboards created in Photoshop, showcasing the foundational design elements for the project. These artboards were meticulously crafted using a combination of masking techniques and the Generative Fill feature in Photoshop.
The following sections present the masks applied to isolate specific areas of the imagery, enabling integration of various elements into the layered compositions. Each layer contributes to the final image, maintaining consistency and depth in the visual assets.
Below, you will find the masks, as well as the completed layered image compositions, which serve as the base assets for animations within the project.
See all assets and layers (compressed for document size): Asset Layers
PHASE IX: REFINEMENT
Objective:
Replace placeholders with final assets, address feedback, and finalize the prototype.
Replace placeholders with final assets, address feedback, and finalize the prototype.
Tasks:
Incorporate curated assets into the prototype.
Conduct a second user test and address pain points.
Deliverable:
Updated developer blog with:
Finalized walkthrough video/screenshots
Reflections on refinement process
Reflection:
The refinement process focused on replacing placeholders with finalized assets, addressing feedback from the initial user test, and polishing the prototype for a cohesive, interactive experience. Final assets significantly enhanced visual appeal and user engagement, while iterative testing ensured seamless integration and resolved minor alignment and responsiveness issues. User feedback highlighted improvements in immersion, interactivity, and proximetric interactions, though some users suggested adding subtle animations and optimizing responsiveness. Adjustments included smoother transitions, refined interactions, and uniform scaling of assets. This phase emphasized the importance of user-centric design and iterative testing, resulting in a polished prototype ready for final rollout.
The refinement process focused on replacing placeholders with finalized assets, addressing feedback from the initial user test, and polishing the prototype for a cohesive, interactive experience. Final assets significantly enhanced visual appeal and user engagement, while iterative testing ensured seamless integration and resolved minor alignment and responsiveness issues. User feedback highlighted improvements in immersion, interactivity, and proximetric interactions, though some users suggested adding subtle animations and optimizing responsiveness. Adjustments included smoother transitions, refined interactions, and uniform scaling of assets. This phase emphasized the importance of user-centric design and iterative testing, resulting in a polished prototype ready for final rollout.
Finalized walkthrough video: All Recordings
Finalized layers: All Photo Files
PHASE X: DOCUMENTATION
Objective:
Compile all deliverables into a final presentation.
Compile all deliverables into a final presentation.
Tasks:
Build and deploy the AR experience on a mobile device.
Record high-quality video of AR interactions.
Create a reflective essay and portfolio page.
Deliverable:
Developer blog update with:
Walkthrough video/screenshots
Reflective essay
Portfolio page
Refinements Made
Optimized proximetric responsiveness to reduce delays and enhance fluidity.
Incorporated subtle animations between transitions to elevate the parallax effect.
Ensured all final assets were uniformly aligned and scaled for a cohesive experience.
Lessons Learned
Iterative Testing is Key: Regular testing with users provided invaluable insights, guiding the refinement process and ensuring the prototype met expectations.
Attention to Detail: Small adjustments, like smoother transitions and optimized responsiveness, made a significant difference in the overall user experience.
User-Centric Design: Actively addressing feedback reinforced the importance of designing with the user’s perspective at the forefront.
Finalized Portfolio Links: Portfolio
Finalized walkthrough video: All Recordings
Finalized layers: All Photo Files
PHASE XI: REFLECTION
Objective:
Reflect on the project process, outcomes, and future possibilities.
Reflect on the project process, outcomes, and future possibilities.
Tasks:
Write a 300–500 word reflective essay.
Summarize findings, challenges, and implications for future work.
Deliverable:
Reflective essay covering:
Problem space and design intervention
Process documentation and analysis
Challenges and solutions
Conclusion and future implications
Essay:
The project’s journey began with the goal of creating an immersive, interactive augmented reality (AR) experience that enhances storytelling through layered visuals and dynamic user interactions. This initiative aimed to address the challenge of keeping users engaged with static content by introducing depth, interactivity, and multi-sensory elements into the narrative experience. The intervention focused on utilizing AR to bridge the gap between traditional storytelling and modern technology, allowing users to interact with and explore layered visual compositions enriched by animations and audio.
The process unfolded across several phases, beginning with ideation and concept refinement. Initial iterations emphasized developing layered assets to achieve a parallax effect and ensure alignment with physical booklets for proximetric interactions. Early prototypes allowed us to test foundational elements, including depth perception and alignment accuracy. Feedback from the first user test highlighted areas needing improvement, such as smoother transitions and enhanced responsiveness. Subsequent refinements involved replacing placeholders with finalized assets, integrating interactive features like flipping layers and triggering audio, and optimizing proximetric responsiveness for a seamless experience.
Throughout the project, challenges arose that required creative problem-solving. For example, ensuring precise alignment between digital layers and physical booklets demanded iterative calibration and rigorous testing. Additionally, maintaining a balance between visual complexity and usability was critical to avoid overwhelming users. Addressing these challenges involved refining assets, optimizing transitions, and conducting multiple user tests to validate improvements. The iterative process underscored the importance of user-centric design, as incorporating feedback was essential to achieving a polished and engaging prototype.
The outcomes of the project demonstrated the potential of AR to transform traditional storytelling into a multi-sensory, interactive experience. The final prototype successfully combined layered visuals, parallax effects, and interactive elements to create an engaging user journey. Insights from this process emphasize the value of iterative testing, detailed asset development, and prioritizing user feedback.
Looking forward, this project opens opportunities for future enhancements, such as incorporating advanced animations, adaptive audio narratives, and expanding the application to different content areas. These possibilities highlight the potential for AR to revolutionize user engagement across education, entertainment, and beyond. The lessons learned from this project provide a strong foundation for future work, emphasizing the importance of blending creativity, technology, and user input to craft impactful design solutions.
In conclusion, this project illustrates the transformative power of AR in storytelling and underscores the significance of a structured, user-centered design process. By reflecting on the challenges and achievements, the groundwork is laid for further exploration and innovation in the field of augmented reality.