This project developed an AI-powered Canva App that offers a one-stop solution for creating children's books, including story writing, storyboard drawing, animation, voice interaction, and music composition. Final outputs seamlessly integrate with Canva's design interface, enabling users to edit and enhance content before exporting. Aligned with Canva's AI education strategy, MagicBook.AI enables personalized book creation, helps families create lasting memories, makes teaching more engaging, and introduces children to the fun and convenience of AI technology.
- Frontend: React, Typescript, JavaScript
- UI Libraries: Canva UI
- Backend: Flask, Canva SDK
- AI Development: Pytorch, Jupyter Notebook
- AI Models: ChatGPT API(text2Text), Stable Diffusion(text2Img, img2Img), LoRA(fine-tuning model), AnimateDiffusion(text2Animation, img2Animation), ChatTTS(text2Speech), and MusicGen(text2Music)
- Hosting: Google Colab, Heroku
- HomePage
- Functionality: Display two subpages: "Discover" and "Generate." In "Discover," allow users to view generated stories, images, videos, audio, and text, with the option to directly import them into Canva for editing. Use "Generate" to initiate the story creation process.
- ScriptPage
- Functionality: Start with generating the story script. Require users to fill in the story description, target age range, story style, teaching content, and chapter length. After clicking "Generate," display the story background, main characters, and the content of each chapter in the outline. Provide options to regenerate or continue to the illustration stage.
- DesignPage
- Functionality: Proceed to the illustration stage. Let users select the art style and image dimensions to generate story illustrations. Allow generated images to be converted into animated videos. After completion, enable users to continue to the voiceover stage.
- VoiceoverPage
- Functionality: Choose the language, gender, age range, and voice style for the voiceover. After clicking "Generate," create the voiceover. Once completed, guide users to the music generation stage.
- MusicPage
- Functionality: Define the music style, describe the content, and choose whether it should be instrumental. After clicking "Generate," complete the music creation process.
- SummaryPage
- Functionality: Display all generated content in chapters, including images, videos, story text, character dialogues, narration, and background music, allowing users to review and apply them to Canva's design interface.
-
Comprehensive AI Integration: Utilizes 6 AI models, including ChatGPT for text generation, Stable Diffusion and LoRA for image creation, AnimateDiffusion for animations, ChatTTS for voiceovers, and MusicGen for background music, providing an all-in-one solution for creating children’s books.
-
Seamless Canva Integration: Utilizes Canva API to connect generated content directly with Canva's design interface, enabling users to easily edit and enhance their stories, images, and videos within Canva before exporting the final product.
-
User-Friendly Workflow: Offers a streamlined, step-by-step process for generating scripts, illustrations, voiceovers, and music, making it accessible for users of all levels, including beginners, to create unique, interactive children's audiobooks.
- Frontend
- Install dependencies
npm install- Start the frontend server
npm run start- Backend
- Navigate to the backend directory
cd backend- Create a virtual environment
python3 -m venv .venv- Activate the virtual environment
.venv\Scripts\activate # Windows
source .venv/bin/activate # macOS/Linux- Install dependencies
pip install -r requirements.txt- Run the Flask application
flask --app CanvaAPI run- (Devpost)
- (Demo Video)
- Xinyi Gao [Linkedin][GitHub]: Product Design, UI/UX Design, Front-end Development, AI Model Testing
- Viet Doan [Linkedin][GitHub]: UI/UX Design, Front-end Development
- Li Cui [Linkedin][GitHub]: Backend Development, AI Model Testing
- Lee ( Jiacheng ) Li [Linkedin][GitHub]: AI Development, Backend Development