Echo-Team/tiny-echo-wm-flash-diffusers

This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.

Pipeline Type: EchoWMFlashBlocks

Description:

This pipeline uses a 5-block architecture that can be customized and extended.

Example Usage

[TODO]

Pipeline Architecture

This modular pipeline is composed of the following blocks:

1. text (EchoWMFlashTextConditioningStep)
2. camera (EchoWMFlashCameraConditionStep)
3. image_encoder (EchoWMVaeEncoderStep)
   • VAE encoder step that encodes the input image into normalized latents for image-to-video generation.
4. denoise (EchoWMFlashDenoiseStep)
   • Autoregressively denoise and commit each Echo-WM Flash video/audio chunk.
5. decode (EchoWMDecoderStep)

Model Components

1. text_encoder (PreTrainedModel)
2. tokenizer (PreTrainedTokenizerBase)
3. connectors (LTX2TextConnectors)
4. transformer (EchoWMTransformer3DModel)
5. vae (AutoencoderKLLTX2Video)
6. video_processor (VideoProcessor)
7. audio_vae (AutoencoderKLLTX2Audio)
8. scheduler (FlowMatchEulerDiscreteScheduler)
9. vocoder (LTX2Vocoder)

Input/Output Specification

Inputs:

• prompt (str): The prompt or prompts to guide image generation.
• max_sequence_length (int, optional, defaults to 1024): Maximum sequence length for prompt encoding.
• action (str): WASD/IJKL action program.
• height (int, optional, defaults to 704): The height in pixels of the generated image.
• width (int, optional, defaults to 1280): The width in pixels of the generated image.
• num_frames (int, optional, defaults to 241): Number of output video frames.
• frame_rate (float, optional, defaults to 24.0): Output video frame rate.
• translation_speed (float, optional, defaults to 0.05): Per-frame camera translation speed for W/A/S/D actions.
• rotation_speed_deg (float, optional, defaults to 0.4): Per-frame camera yaw speed in degrees for J/L actions.
• pitch_speed_deg (float, optional, defaults to 0.2): Per-frame camera pitch speed in degrees for I/K actions.
• pitch_limit_deg (float, optional, defaults to 40.0): Maximum absolute camera pitch in degrees.
• fov_deg (float, optional, defaults to 70.0): Horizontal camera field of view in degrees.
• num_videos_per_prompt (int, optional, defaults to 1): The number of images to generate per prompt.
• image (Image | list): Reference image(s) for denoising. Can be a single image or list of images.
• image_crf (int, optional): H.264 CRF used to re-compress the conditioning image before VAE encode, matching the compression the model was trained against. None (default) resolves from the text-encoder generation (33 through LTX-2.3, 18 for LTX-2.5). Pass 0 to skip re-compression. Requires a PIL.Image.Image when re-compression runs.
• generator (Generator, optional): Torch generator for deterministic generation.
• timesteps (list, optional, defaults to [1000, 750, 500, 250]): Distilled denoising timestep IDs for each autoregressive chunk.
• video_cache_size (int, optional, defaults to 19): Maximum number of latent video frames retained in the bounded KV cache.
• video_sink_size (int, optional, defaults to 7): Number of leading latent video frames permanently retained as the cache sink.
• output_type (str, optional, defaults to pil): Output format: 'pil', 'np', 'pt'.
• decode_timestep (None, optional, defaults to 0.0): The timestep at which the VAE decodes the final latents.
• decode_noise_scale (None, optional): Noise interpolation factor applied to the latents at the decode timestep.
• vae_tiling (bool, optional, defaults to True): Enable spatial and temporal VAE decoding tiles to reduce peak memory usage.
• vae_tile_size (int, optional, defaults to 512): Spatial tile long-side size in pixels; the short side follows the video aspect ratio.
• vae_tile_overlap (int, optional, defaults to 64): Spatial tile overlap in pixels.
• vae_temporal_tile_size (int, optional, defaults to 64): Temporal tile size in sample frames, excluding the causal boundary frame.
• vae_temporal_tile_overlap (int, optional, defaults to 24): Temporal tile overlap in sample frames.

Outputs:

• videos (list): The generated videos.
• audio (Tensor): The generated audio waveform.