--- title: "Retraining the classifier with validated clips" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Retraining the classifier with validated clips} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>") ``` The manual review of stage 1 produces two kinds of labelled clips: clips that contain the species (positives) and clips in which the classifier detected the species but the reviewer did not hear it (negatives). Besides measuring precision and feeding the calibration, both can be used to retrain the BirdNET custom classifier. This article shows how to turn the reviewed clips into a training data set in the layout that BirdNET-Analyzer expects, how to keep an independent set for testing the new classifier, and what has to be redone after retraining. Training itself runs in BirdNET-Analyzer (Python), not in R. The requirements quoted below come from the BirdNET-Analyzer documentation (version 2.4.0, section `birdnet_analyzer.train` of the [command line interface](https://birdnet-team.github.io/BirdNET-Analyzer/stable/usage/cli.html)); check the version you use, since the training options change between releases. ```{r} library(sheepdog) suppressPackageStartupMessages(library(dplyr)) ex <- system.file("extdata", "example", package = "sheepdog") ``` ## What BirdNET-Analyzer expects According to its documentation, the training script of BirdNET-Analyzer: * takes a folder with **one subfolder per class**, and uses the folder names as labels. The recommended label format is `_`, although other formats work; * accepts **negative samples** of a class in a folder with the same name prefixed by `-` (e.g. `-Boana raniceps_boa-ran`). Negative samples are used for training only, not for validation, and only when the folder with the positives of that class exists; * treats a folder named `Noise`, `Background`, `Other` or `Silence` as a non-event class; * resamples the audio to 48 kHz and works on 3 s segments; * recommends at least 100 audio files per species, although training also works with fewer; * accepts an independent test set (`--test_data`). The documentation also states that custom classifiers trained with BirdNET-Analyzer are licensed under CC BY-NC-SA 4.0. The clips cut by sheepdog already fit this layout: they are 3 s segments of the original recordings, and `species_code` keeps the full label of the classifier (`_`), which can be used as the folder name so that the retrained classifier reports the same labels. ## The reviewed clips We use the reviewed clips shipped with the package. In a real project the review is done in the Review tab of the BirdNET GUI, which moves each clip into a `Positive` or `Negative` subfolder; here we recreate that folder structure with empty `.wav` files from the verdicts of the example tables, and read it back with `classify_review()`, which records where each clip was found (`reviewed_path`). ```{r} validation <- read_validation_tables( list.files(file.path(ex, "validation"), full.names = TRUE)) review_dir <- file.path(tempdir(), "review") unlink(review_dir, recursive = TRUE) dest <- file.path(review_dir, validation$species_folder, ifelse(validation$verdict == "positive", "Positive", "Negative")) for (d in unique(dest)) dir.create(d, recursive = TRUE) invisible(file.create(file.path(dest, validation$clip_file))) reviewed <- validation |> select(-verdict, -reviewed_path) |> classify_review(review_dir) |> suppressMessages() count(reviewed, species_code, verdict) ``` `read_validation_tables()` stacks the `validation_by_clip.csv` tables of several sites or rounds, keeping every column and verdict. ## Keeping an independent test set A retrained classifier has to be evaluated on clips that it did not see during training. BirdNET-Analyzer holds out a validation split (`--val_split`) drawn from the training folder itself, but clips from the same site and night share background noise and calling individuals, so a random split of clips tends to overstate performance (our recommendation, not a statement of the BirdNET documentation). A stricter test keeps whole sites out of training and passes them as `--test_data`. Here we hold out one site. ## Building the training folders `build_training_set()` copies the reviewed clips into the BirdNET layout: positives in `//` and negatives in `/-/`, where `` is `train`, or `test` for the sites in `test_sites`. It reads each clip from `reviewed_path`, the place where the review left it, leaves out whole-recording clips (which have no confidence score), writes the table of copied clips to `training_set.csv`, and warns about classes with fewer positive training files than `min_files` (100 by default, the number recommended by the BirdNET documentation). ```{r} training_root <- file.path(tempdir(), "birdnet_training") unlink(training_root, recursive = TRUE) training_set <- build_training_set(reviewed, training_root, test_sites = "NHE") |> suppressMessages() list.files(file.path(training_root, "train")) count(training_set, set, verdict) ``` How many positive training files each class received: ```{r} n_pos <- training_set |> filter(set == "train", verdict == "positive") |> count(species_code) |> mutate(below_100 = n < 100) n_pos ``` With the `r nrow(validation)` reviewed clips of the example, `r sum(n_pos$below_100)` of the `r nrow(n_pos)` classes have fewer than 100 positive training files. Classes with few positives are the ones that most need clips from further review rounds before retraining. ## Which clips to add Three points about the composition of the training set follow from how the clips were chosen (our recommendations, not statements of the BirdNET documentation): * **Clips from the `"top"` design are the easy cases.** They are the highest-scoring detections, so they teach the classifier little about the calls it currently misses or scores low. Positives from the `"random_detections"` and `"random_recordings"` designs (see `vignette("sampling-designs")`) cover lower scores and calls the classifier did not detect. * **Negatives with high scores are the most informative.** They are the sounds the classifier confuses with the species. Species with many high-scoring false positives in `precision_by_bin()` (see `vignette("validation-effort")`) are the ones for which negatives matter most. * **Whole-recording clips** of `"random_recordings"` are longer than 3 s. BirdNET-Analyzer crops long files (`--crop_mode`), but the label of a whole recording says only that the species was heard somewhere in it. Cut the annotated calls into 3 s clips before using them as positives. ## Training Training runs in a Python environment with BirdNET-Analyzer installed. The commands below follow the example of the documentation (they are not run here): ```sh python -m birdnet_analyzer.train birdnet_training/train \ --test_data birdnet_training/test \ -o checkpoints/custom/Custom_Classifier.tflite python -m birdnet_analyzer.analyze recordings/ \ --classifier checkpoints/custom/Custom_Classifier.tflite ``` Use `--model_format raven` (or `both`) if the classifier will also be used in Raven. Hyperparameters such as `--hidden_units`, `--learning_rate`, `--dropout` and the upsampling options affect the result; the documentation recommends trying several settings, or `--autotune`. ## After retraining A retrained classifier produces new confidence scores, so everything that was estimated from the scores of the old one no longer applies: 1. Run the new classifier over the recordings to obtain new selection tables. 2. Draw and review a new set of clips (stage 1). Do not reuse the clips used for training: their scores are biased upwards on the data the classifier was trained on. 3. Recalibrate (stage 2, `vignette("calibration")`) and rebuild the detection histories and community matrices (stage 3). Keep `training_set.csv` with the new classifier. Before reviewing the detections of the retrained classifier, `exclude_training_clips()` drops from the new manifest the clips that overlap, for the same species and recording, a clip used in training: ```{r, eval = FALSE} manifest_v2 <- select_clips(detections_v2, ...) |> exclude_training_clips(file.path(training_root, "training_set.csv")) ``` `extract_validation_clips()` records the `classifier` label in `extraction_info.csv`, so each validation round can be traced to the classifier that produced its clips.