Use Case

Fast keyword detection with LMUs on GPU

November 23, 2022
Trevor Bekolay

The Legendre Memory Unit (LMU) algorithm is a new alternative to LSTMs and Transformers with some key advantages, especially at the edge. NengoEdge makes it easy to build LMU models on any dataset and deploy those models anywhere. In this article, we'll show how easy it is to deploy a keyword detection model using LMUs on a general purpose Nvidia GPU with NengoEdge.

The goal of our model will be to add voice controls to a device that moves along two dimensions, like a plotter. We want to be able to say what direction to move, and to be able to stop it when it reaches the desired location by saying "stop."

Set up your dataset

The first step is to set up your dataset. In NengoEdge, you can upload your own raw audio files, or use the SpeechCommands audio files that are available when you create your account. Raw audio files are used to make Datasets, which let you specify how to use those raw audio files in your machine learning pipeline.

  1. Click on Datasets in the top bar.
  2. Under the SpeechCommands v2 heading, click New dataset.
  3. Give the dataset a name.
  4. We will use the name "Directions"
  5. Set the dataset parameters.
  6. We will select the keywords "up, down, left, right, stop"
  7. Click Create.

If you're unsure of what a parameter controls, hovering over it will display a tooltip with more information.

Find a successful test run

We recommend doing at least one shorter test run to make sure that your longer run will yield good results. If you're using your own raw audio files, you may need several test runs to find a good set of parameters.

Training time can get expensive, so shorter test runs save money in the long run. You can reuse the weights from the short run as the starting point for longer runs, so there's no downside to a test run.

  1. Click on Runs in the top bar.
  2. Click on Create new project to organize the runs for this project.
  3. We will use the name "Directions"
  4. Under the Directions heading, click New run.
  5. Set the run parameters.
  6. We will use the name "Directions test 1"
  7. We will use the LMU-1 network to leverage the power of LMUs
  8. We will set the number of training steps to 2000
  9. We will set the validation interval to 100
  10. Click the Optimize button.

These parameter will result in a run that finishes within a few hours, and will give more feedback on performance in these early training stages.

The Progress tab gives you a rough idea of whether your test run is successful.

<screenshot of progress page>

In this case, our loss is going down steadily, and the validation accuracy reaches a good level, so we will consider this a successful test run.

With the built in SpeechCommands v2 dataset, we expect performance to be very good using default run parameters. However, using your own dataset may require parameter tweaking.

If this wasn't the case, we can look at the Results tab to see where the model is underperforming to guide future test runs.

<screenshot of results page>

The confusion matrix can identify specific labels (words) that are not being classified well, which can help direct data collection when using your own dataset. General accuracy metrics can help you identify other issues. Experiment with the run parameters to see how each parameter affects performance metrics if your accuracy numbers are too low.

Set up a full training run

Once you are satisfied with your test run, create a new run to train to convergence.

  1. Click on Runs in the top bar.
  2. Under the Directions heading, click New run.
  3. TODO: name run?
  4. Click the drop-down next to Initial Weights. Select the successful test run.
  5. Set other run parameters.
  6. We will set the number of training steps to 20000
  7. We will set the validation interval to 500
  8. Click the Optimize button.

These parameter will result in a run that finishes in several hours, and will give coarse feedback compared to the test runs.

<screenshot of progress page>

You will see on the Progress page that initial performance will start around where the test run finished off. By the end of the optimization process, performance should be even better.

If you want to deploy this model to an edge device, you can stop here and go to the tutorial for your target device (TODO: link those).

Deploy and use the trained model

Your model is now ready to be deployed to your environment and used either by itself or as part of a larger system. Steps for integration will depend on your setup, so here we will show how to do a standalone deployment.

The standalone deployment contains the model and a simple test script that you can use as a guide for integrating the model from NengoEdge in your code.

<screenshot of deploy page>

  1. On the main Runs page, click the Deploy icon <TODO:insert icon> next to the fully trained run.
  2. Under the Download model heading, click the Format dropdown and select Standalone.
  3. Click the Download button.
  4. Extract the downloaded artifact.zip to your desired location.
  5. Follow the steps in README.txt to set up a Python environment.
  6. Follow the steps in test.py to test your environment and the model.
  7. Run python test.py to see the keywords classified in real time.

If everything worked, you can speak into your microphone to see your model classify keywords in real time.

<iframe width="560" height="315" src="https://www.youtube.com/embed/UooLZ5MCeE4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

Similar content from the ABR blog