Skip to content

MaAI API Documentation

GitHub Repository (MaAI-Kyoto/MaAI)

Welcome to the official API documentation for MaAI (Real-time and Continuous Non-Linguistic Behavior Generation Software).

This documentation provides detailed specifications for each module, class, and function included in the MaAI source code. You can browse the detailed API references for each module using the top navigation bar or the left menu.

Core Modules

  • maai.encoder: Modules related to encoding audio and text features.
  • maai.encoder_components: Building blocks used by the audio encoders.
  • maai.model: Core modules for model building. Maai runs a single model; MaaiMultiple runs several of them on one shared encoder.
  • maai.modules: Transformer building blocks (GPT, GPTStereo, ALiBi attention) shared by the models.
  • maai.input: Modules for input data processing and management.
  • maai.output: Modules for output data generation and management.
  • maai.objective: Modules defining objective functions, including loss functions for optimization.
  • maai.util: A collection of utility functions.
  • maai.models: Specific model architecture definitions corresponding to various tasks and conditions (see below).

Models

Each model is selected through the mode argument of the Maai class.

  • maai.models.config: VapConfig, the configuration object shared by all models.
  • maai.models.vap (mode="vap", "vap_mc"): Voice Activity Projection, the turn-taking model that predicts who will be speaking in the near future. "vap_mc" is the noise-robust (multi-condition) variant.
  • maai.models.vap_mono (mode="vap_mono"): Single-channel turn-taking model with its own pretrained weights — one encoder and a plain causal transformer instead of the two-speaker cross-channel one. p_now, p_future and vad are single float values for the one input channel, and are not normalized between speakers.
  • maai.models.vad (mode="vad", "vad_mono"): Voice Activity Detection. Unlike the turn-taking models it does not predict the future, but detects whether each participant is speaking right now. Both channels are processed jointly with cross-channel attention, so the model can tell who is actually talking even when one speaker's voice leaks into the other's microphone (crosstalk). "vad_mono" is the single-channel variant.
  • maai.models.vap_bc (mode="bc"): Backchannel prediction.
  • maai.models.bc_det (mode="bc_det", "bc_det_mono"): Backchannel detection. Where "bc" predicts that a backchannel is about to happen, this model detects whether the utterance a speaker is producing right now is a backchannel. Both channels are processed jointly, since the interlocutor's speech is most of the evidence that a short utterance is a backchannel rather than the start of a turn. "bc_det_mono" is the single-channel variant.
  • maai.models.vap_bc_2type (mode="bc_2type"): Backchannel prediction distinguishing two backchannel types.
  • maai.models.vap_nod (mode="nod"): Head nod generation.
  • maai.models.vap_nod_para (mode="nod_para"): Head nod generation with nod parameters.
  • maai.models.vap_prompt (mode="vap_prompt"): VAP conditioned on a text prompt.

Modes whose name ends in _mono are single-channel: they are separately trained models with their own pretrained weights, take only audio_ch1, and run one encoder instead of two. They can be combined with the two-channel modes through MaaiMultiple.

Usage guides for each mode (supported languages, frame rates, and sample code) are available in the repository README.

Usage

Select the module you wish to learn more about from the navigation menu on the left. Each page is automatically generated from the Docstrings within the source code.