Train/Infer¶
This tab is the interface for training models and running inference on new videos.
The left side-bar displays your current labeling progress, and contains a drop-down menu showing all previously trained models. The main workflows are:
“Train Networks” and “Predict on New Videos” columns are for training and inference, and detailed below.
Train networks¶
Training options¶
From the drop-down “Expand to adjust training parameters” menu, optionally change the following:
Max training epochs (default is 300)
Model seed; different seeds will lead to different model outputs that are useful for ensembling. Enter either a single integer (e.g.
0) to train one model of each type (see below), or a list of comma-separated integers (e.g.0,1,2) to train multiple models of each type.Losses used for the semi-supervised models. The PCA Multiview option will only appear if your data have more than one view; the Pose PCA option will only appear if you selected keypoints for the Pose PCA loss during project creation.
Select models to train¶
There are currently 4 options to choose from:
Supervised: fully supervised baseline
Semi-supervised: uses labeled frames plus unlabeled video data for training
Supervised context: supervised model with temporal context frames
Semi-supervised context: semi-supervised model plus temporal context frames
Note
If you uploaded a DLC project or are using the example data you will not see the context options.
Train models¶
Click “Train models” to launch sequential model training - parallel model training coming soon! A set of progress bars will appear below, one for each model.
Once training is complete for all models you will see “Training complete; see diagnostics in the following tabs” in green.
Predict on new videos¶
First, select the model you would like to use for inference from the drop-down menu.
Select videos¶
You have two options for video selection:
Upload new: upload a new video to the app. To do so, drag and drop the video file(s) using the provided interface. You will see an upload progress bar. If your video is larger than the 200MB default limit, see the FAQs.
Select previously uploaded video(s): any video previously uploaded to this tab (located in the
videos_inferdirectory) or the Extract Frames tab (located in thevideosdirectory) will be available in the drop down menu; you may select multiple videos.
Video handling options¶
You may also choose to create videos overlaid with predictions from the model.
Save labeled video (30 second clip): label the 30 second portion of the video with the highest motion energy in the model predictions. This is a good option if you want to quickly get a sense of how well the model is performing.
Save labeled video (full video): plot predictions for the duration of the entire video. This is a good option if you want to search over longer or more diverse periods of the video.
If you check one or both boxes, you will be able to view the resulting videos directly in the app in the “Video Player” tab.
Run inference¶
Click “Run inference” once the video uploads are complete, and another set of progress bars will appear.
Once inference is complete for all videos you will see the “waiting for existing inference to finish” warning disappear.
See Accessing your data for the location of inference results.
Create an ensemble of models¶
Ensembling is a classical machine learning technique that combines predictions from multiple models to provide enhanced performance. We offer the Ensemble Kalman Smoother (EKS), a Bayesian ensembling technique that combines model predictions with a latent smoothing model.
To use EKS, you must first create an ensemble of models. Then, if you run inference using the ensemble, EKS will automatically be run on the ensemble output. The steps are outlined in more detail below.
Select models for ensembling¶
Select a set of previously trained models to create the ensemble. We recommend an ensemble size of 4-5 models for a good trade-off between computational efficiency and accuracy. An ensemble can be composed in many ways; one way would be to include models of the same type (supervised, semi-supervised, etc.) using different random seeds; another way would be to include models of different types (e.g. one supervised, one semi-supervised, etc.); a combination of these approaches would work too!
Add ensemble name¶
Give your ensemble a name. This text will be appended to the date and time to form the final ensemble name (just like the other models), to prevent overwriting previous models/ensembles.
Create ensemble¶
Click the “Create ensemble” button; you will see a brief success message. The newly-created ensemble directory will contain a text file that points to the model directories of the individual ensemble members.
Running the Ensemble Kalman Smoother post-processor¶
Now that the ensemble has been created, you can run inference on videos. Navigate back to the Predict on new videos part of this tab. You should now see your ensemble in the drop-down menu of models.
Note
If your model is not in the drop-down menu, click on the three vertical dots in the top right of the tab (next to the “Deploy” button) and click “Rerun”.
You can now treat the ensemble as any other model: select one or more videos to run inference on, select any video labeling options you like, and then click “Run inference”. Upon doing so you will see multiple progress bars appear, one for each model/video combination:
Inference and labeled video creation will be skipped for any ensemble member that has already performed these tasks.
After inference and labeled video creation are completed for each ensemble member, a new progress bar will appear for the EKS model. You will see the progress of the EKS fitting process, as well as the labeled video creation if you have selected one of those options.
The outputs of EKS will be stored just like the inference outputs of a single model. This means that you may inspect the EKS traces in the Video Diagnostics tab and view the labeled video (if you have selected one of these options) in the Video Player tab.