Continuous Attention Transformer
Early approaches to CAT: rethinking attention as a continuous-time process. Research with Purdue ECE.
Continuous Attention Transformer (CAT) is early-stage research, done with Purdue ECE, that asks what attention looks like when you stop treating a sequence as a bag of discrete positions and start treating it as a signal in continuous time. The motivation is irregularly-sampled data — sensor streams, clinical time series — where the gaps between observations carry as much meaning as the observations themselves.
The formulation replaces the fixed positional grid with a continuous-time kernel, so queries and keys are evaluated as functions of real timestamps rather than integer indices. That lets a single model ingest sequences sampled at different, uneven rates without resampling or padding away the structure. The prototype is built in PyTorch and evaluated against standard irregular-time-series baselines.
A preprint is currently under review. The interesting open questions are computational: making the continuous kernel cheap enough to compete with the highly-optimised discrete attention it is trying to generalise.