Optimal time series AI

Time series problems include processing of language, audio, biosignals, RF signals, network traffic and so on – anything where the order of the data matters. Most researchers know of two main architectures used to solve time series problems: LSTMs and Transformers. We have invented a third: Legendre Memory Units (LMUs). LMUs are provably optimal at compressing time series.

Long short-term memories (LSTMs) have been called the most commercially valuable AI algorithm ever invented (Bloomberg), having been deployed in many speech, vision and text analysis systems. But LSTMs fall apart when tasked with learning temporal dependencies spanning thousands of timesteps, limiting their usefulness.

Transformers are the workhorse of NLP and the basis for the largest AI models available, including GPT-3 and Google Brain’s Switch Transformer. But Transformers are incredibly expensive in both computation and memory, O(N²) for both.

The Legendre Memory Unit (LMU) is a recurrent network that can learn temporal dependencies over millions of timesteps, unlike LSTMs. The LMU can scale to the size of transformers, but scales as O(N) in compute and O(1) in memory. In addition, LMUs require 10x less training (and data) to be as accurate as a transformer.

LMUs maintain efficient and scale-invariant representations of recent inputs and learn how to solve real world problems using those representations. LMUs can be implemented with traditional deep learning techniques on hardware you already have, or can be deployed on neuromorphic hardware or neural accelerators for massive power savings. Using LMUs, we and others have obtained the best-known results on a variety of benchmarks.

Do you have a time series problem you’re currently solving with an LSTM or Transformer?

Have a time series problem that you haven’t been able to solve yet?

Do you need to solve a time series problem in an edge or IoT device?

Take a look at our published and easily reproducible results. ABR’s patent-pending LMU is free to use for academic research and personal non-commercial uses and is available at a yearly per-processor rate commercially through our store.