Realtime Behavior Analysis

To assert a better feedback mechanism in a traditional classroom environment we propose using computer vision and machine learning techniques for real-time, objective analysis of student emotions and teacher behavior simultaneously Using a Combined model the two combined models are:

• Facial Expression
• Body Language Classifier

Facial Expression

• Video capture using python CV.

• Uses Cascade Classifier to detect faces

• CNN then classifies the detected faces to predefined categories

  • Happy
  • Sleepy
  • Angry
  • Neutral
  • Fear
  
  

  Flowchart:

  

  Results:

  • Total params: 13,111,367
  • Trainable params: 13,103,431
  • Non-trainable params: 7,936
  • Accuracy: 88.64 %

  Loss function Used: Categorical Cross-Entropy

  

Body Language Classification

• Utilizes MediaPipe Holistic to classify body poses from video data into specific categories:

  • Writing
  • Active Explaining
  • Passive Explaining
  • At Desk

• Pose Estimation model was trained on lectures videos from MIT Open Course Ware.

• Frames from the video lectures were first converted to RGB format, Mediapipe holistic model was used to extract landmarks.

• Landmarks from 10 such frames were contacted then written to csv along with class name

• This data-set was then fed to a Random Forrest model to which classifies into the given categories.

• Next step was to create a composite model that incorporates both the models, leveraging the strengths of both models and providing a holistic view.

Flowchart:

Results:

Model used: Random Forrest
Accuracy: 97.81 %

Combined Results

Both Models were run simultaneously to give combined results

Detecting teacher’s behavior:

Detecting students behaviour: