Abstract
Traditional gesture typing systems require extensive training data and gradient-based optimization. This research explores a fundamentally different approach: freeze pre-trained VAE encoders for touch trajectories and text, then bridge their latent spaces using vector field dynamics instead of neural networks.
The key innovation is treating continuous swipe trajectories as paths through latent space, where target words act as semantic "attractors" that guide trajectory interpretation. No backpropagation, no neural networks in the coupling layer - just vector field updates that map motion patterns to meaning.
Core Architecture
Touch VAE (Frozen): Encodes multi-finger touch trajectories into continuous latent space. Pre-trained on gesture data to capture motion patterns and temporal dynamics.
Vector Field Coupling: The only trainable component. Maps from touch latent space to text latent space using gradient-free vector field updates. No neural networks - just learned vector fields that define how to translate between the two spaces.
Text VAE (Frozen): Decodes text latent vectors into words. Pre-trained to understand semantic relationships.
Continuous Trajectory Decoding
Instead of segmenting input into discrete words, the system treats the entire swipe as a continuous path through semantic space. Target words function as semantic attractors - as your finger traces a path, the system performs path integration, continuously updating which words you're moving toward based on trajectory dynamics.
One-Shot Adaptation: Magnetic Latent Interpolation
When a user demonstrates a gesture once, the system captures both the motion pattern and intended meaning in latent space, creating a "magnetic attractor" at that location. Future similar gestures are pulled toward this example's interpretation. The system learns by example placement rather than gradient descent - adapting to individual users from a handful of corrections.
Current Status
Architecture: Designed and validated theoretically
VAE Training: Touch and Text VAEs in progress
Next Steps: Implement vector field coupling, test path integration system, prototype adaptive interface
Related Links
Repository and documentation coming soon