Text2Sign: A Single-GPU Diffusion Baseline for Text-to-Sign Language Video Generation

Overview image generated from the compiled PDF.

Abstract

Sign language is a primary communication channel for millions of Deaf and hard-of-hearing people, yet generating signer video from text remains expensive. Text2Sign is a text-conditioned diffusion architecture for short sign-language clips that runs on a single NVIDIA L4 GPU. It combines a frozen CLIP text encoder with a 3D encoder-decoder backbone and factorized spatiotemporal attention. On a signer-disjoint How2Sign split, a longer-run checkpoint reaches validation loss 0.00999 and generates a 32-frame 64x64 clip in 12.60 seconds with 3.12 GB peak inference memory. The system remains a research baseline rather than a complete production system. Clips are low-resolution and short, and expert linguistic evaluation is still missing.

Type

Journal article

Publication

IEEE Access

Authors: Ruize Xia
Published in: IEEE Access, 2026
DOI: 10.1109/ACCESS.2026.3686260
arXiv: 2607.13164
Code: github.com/xiaruize0911/text2sign
Model: huggingface.co/xiaruize/text2sign
ORCID: 0009-0000-0501-0943

Abstract

Sign language is a primary communication channel for millions of Deaf and hard-of-hearing people, yet generating signer video directly from text remains difficult because video diffusion models are expensive to train and evaluate. This article presents Text2Sign, a text-conditioned diffusion architecture for short sign-language clips designed to run on a single NVIDIA L4 GPU rather than a multi-node cluster.

The model combines a frozen vision-language text encoder with a three-dimensional encoder-decoder backbone and factorized spatial-temporal attention. That design reduces the cost of full video attention while preserving motion coherence. On a signer-disjoint partition of How2Sign, the best short-run ablation reaches a validation loss of 0.0648, while a longer-run checkpoint reaches 0.00999. On a compact evaluation slice, that checkpoint yields SSIM 0.2403 ± 0.0238, PSNR 15.11 ± 0.42 dB, and temporal consistency 1.0000 ± 0.0000. Under 8-step DDIM sampling with guidance scale 5.0, it generates a 32-frame, 64 × 64 clip in 12.60 seconds (2.54 frames/s) with 3.12 GB peak inference memory.

Held-out audits still show only weak prompt-specific separation, and the system does not yet include expert linguistic evaluation. The contribution should therefore be read as an efficiency-oriented research baseline rather than a complete sign-language production system.

Read the paper

Last updated on Jan 1, 2026