| Title: | Sample, Validate and Calibrate 'BirdNET' Detections |
|---|---|
| Description: | Tools to check whether automated detections from 'BirdNET' custom classifiers in passive acoustic monitoring are real. The package builds clip sets for manual validation (top-score, random-occasion or random-recording designs), summarises the reviewed clips, calibrates confidence scores into probabilities of true detection with hierarchical logistic models, and turns calibrated detections into nightly detection histories and effort-standardised community matrices. |
| Authors: | Diogo B. Provete [aut, cre] |
| Maintainer: | Diogo B. Provete <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 3.2.1.9000 |
| Built: | 2026-10-24 21:12:18 UTC |
| Source: | https://github.com/diogoprov/sheepdog |
Splits the clips of a manifest among reviewers, balanced within each
species, and gives a fraction of them to a second reviewer, so that the
agreement between reviewers can be measured with
reviewer_agreement(). The overlap is drawn within each species, so
every species gets doubly reviewed clips.
assign_reviewers(manifest, reviewers, overlap = 0.1, seed = 2026)assign_reviewers(manifest, reviewers, overlap = 0.1, seed = 2026)
manifest |
Clip manifest, as returned by |
reviewers |
Names of the reviewers (at least two). |
overlap |
Fraction of the clips of each species reviewed by two reviewers. |
seed |
Seed of the draws. The global random number generator is left as it was. |
Cut the clips with cut_clips() first, then copy each reviewer's share
to their own folder with split_by_reviewer().
The manifest with reviewer (the first reviewer of each clip)
and second_reviewer (NA for clips reviewed once).
Reads the nightly detection history written by calibrate() and writes
the matrices of community_matrices() to output_dir:
frequency_matrix_site_month_k<k>.csv,
count_matrix_site_month_k<k>.csv and effort_site_month.csv (which
also lists the dropped cells). This is the function form of the v1.2.0
script gerar_matrizes_comunidade.R.
build_community_matrices( history_file, output_dir, core_species, k_min = 2, min_nights_month = 5 )build_community_matrices( history_file, output_dir, core_species, k_min = 2, min_nights_month = 5 )
history_file |
Path to |
output_dir |
Output folder; created if needed. |
core_species |
Species codes (the |
k_min |
Repetition rule: 1, 2 or 3 reliable detections on the night. |
min_nights_month |
Site x month cells with fewer recorded nights are
dropped from the matrices (unstable frequencies) and flagged in
|
The list returned by community_matrices(), invisibly.
Copies the reviewed clips into the folder layout that the training
script of BirdNET-Analyzer expects: one folder per class, named after
the label (species_code, so the retrained classifier keeps the labels
of the current one), with the negatives of a class in a folder with the
same name prefixed by -. Clips of the sites in test_sites go to a
separate test folder, to be passed as --test_data, so that the new
classifier is evaluated on sites it was not trained on.
build_training_set( validation, output_dir, test_sites = NULL, negatives = TRUE, min_files = 100 )build_training_set( validation, output_dir, test_sites = NULL, negatives = TRUE, min_files = 100 )
validation |
Reviewed clips, as returned by |
output_dir |
Output folder; |
test_sites |
Sites held out for testing. |
negatives |
Copy the negative clips into |
min_files |
Minimum number of positive training files per class below which a warning is raised. |
The layout follows the documentation of BirdNET-Analyzer (version
2.4.0, birdnet_analyzer.train), which also recommends at least 100
files per species; classes below min_files in the training set raise
a warning. Whole-recording clips of "random_recordings" (longer than
3 s, confidence = NA) are left out: their label says only that the
species was heard somewhere in the recording.
The table of copied clips is also written to training_set.csv in
output_dir; keep it with the classifier, to leave these clips out of
later validation rounds with exclude_training_clips().
A tibble with one row per copied clip: site, species_code,
verdict, set ("train" or "test"), class_folder, clip_file,
source_audio, clip_start, clip_end, from and to, invisibly.
Runs the whole calibration stage: reads the reviewed clips, compares the
candidate models by cross-validation, refits the best one to all the
clips, optionally draws the effort curve, scores every detection of the
complete selection tables and builds the nightly detection history.
Writes to output_dir:
model_comparison.csv (compare_calibration_models());
calibration_folds.csv, when fold_group is supplied: validation
row number, site, species, grouping columns and assigned fold;
calibration_model.rds and calibration_model.txt (the refitted
best model and its summary);
calibration_curves.png (plot_calibration()), when 'ggplot2' is
installed;
effort_curve.csv (calibration_effort()), when effort_curve is
TRUE;
calibrated_detections_<site>.csv, one per site
(score_detections());
nightly_detection_history.csv (detection_history());
detection_summary.csv (summarise_history()).
calibrate( validation_files, prediction_tables, output_dir, site_map = NULL, min_confidence = 0.1, p_min = 0.95, min_detections_night = 1, dawn_hour = 12, k_folds = 5, select = c("one_se_logloss", "best_auc"), uncorrelated = FALSE, effort_curve = FALSE, effort_n = c(5, 10, 15, 20, 25, 30), effort_reps = 50, engine = c("lme4", "brms"), prior = NULL, brms_args = list(), rule = c("mean", "probability"), prob_min = 0.9, seed = 2026, fold_group = NULL )calibrate( validation_files, prediction_tables, output_dir, site_map = NULL, min_confidence = 0.1, p_min = 0.95, min_detections_night = 1, dawn_hour = 12, k_folds = 5, select = c("one_se_logloss", "best_auc"), uncorrelated = FALSE, effort_curve = FALSE, effort_n = c(5, 10, 15, 20, 25, 30), effort_reps = 50, engine = c("lme4", "brms"), prior = NULL, brms_args = list(), rule = c("mean", "probability"), prob_min = 0.9, seed = 2026, fold_group = NULL )
validation_files |
Paths to the |
prediction_tables |
Named character vector of the complete selection tables, named by site. |
output_dir |
Output folder; created if needed. The calibrated detections of a large project can take hundreds of MB. |
site_map |
Optional named character vector to fix diverging site
names, as |
min_confidence |
Detections below this confidence are not scored. |
p_min |
Threshold on the calibrated probability (not on the confidence score). |
min_detections_night |
Repetition rule: minimum number of reliable detections for a night with detection. |
dawn_hour |
Recordings that start before this hour belong to the night before. |
k_folds |
Number of folds. |
select |
Rule to choose the model; see
|
uncorrelated |
Include the m4 variant without the species
intercept-slope correlation among the candidates; see
|
effort_curve |
Draw the effort curve (slow). |
effort_n, effort_reps
|
Training sizes and repetitions of the effort curve. |
engine |
Engine of the final fit: |
prior |
|
brms_args |
|
rule |
When a detection is reliable: |
prob_min |
|
seed |
Seed of the fold assignment. The global random number generator is left as it was. |
fold_group |
Optional character vector of grouping columns. Do not
include |
The candidate family follows the observed species and sites: one
species uses the site-effect family of calibration_models(), and
a single site omits non-estimable site random effects.
The model is chosen by select_calibration_model() with the rule
select: by default, the simplest model within one standard error of
the lowest log-loss, leaving out models singular in most folds
("one_se_logloss"). Use select = "best_auc" to reproduce the 2.x
selection by highest mean AUC. When the refit is singular,
fit_calibration() warns; read calibration_model.txt and the
comparison before using the probabilities.
Converting BirdNET confidence scores into probabilities is recommended
by the BirdNET authors (Wood & Kahl 2024, doi:10.1007/s10336-024-02144-5);
calibration by logistic regression on logit(score) is applied, for
instance, by Owens et al. (2026, doi:10.64898/2026.06.01.729442).
This function extends that approach with a hierarchical multi-species
structure. This is the function form of the v1.2.0 script
calibracao.R.
A list with comparison, fit, effort (or NULL), scored
and history, invisibly.
Reports the largest R-hat, the smallest bulk and tail effective sample
sizes and the number of divergent transitions of a model fitted with
engine = "brms", and whether they pass the usual limits.
calibration_diagnostics(fit, rhat_max = 1.01, ess_min = 400)calibration_diagnostics(fit, rhat_max = 1.01, ess_min = 400)
fit |
Model returned by |
rhat_max |
Largest acceptable R-hat. |
ess_min |
Smallest acceptable bulk and tail effective sample size. |
A one-row tibble with max_rhat, min_ess_bulk, min_ess_tail,
n_divergent and ok.
How many reviewed clips per site x species are needed to calibrate well? For each training size, draws that many clips per cell (keeping at least five for testing), fits the model and computes the AUC on the remaining clips. Repetitions with fewer than 50 test clips, or whose fit fails, are dropped.
calibration_effort( validations, formula, n_train = c(5, 10, 15, 20, 25, 30), n_reps = 50, seed = 2026 )calibration_effort( validations, formula, n_train = c(5, 10, 15, 20, 25, 30), n_reps = 50, seed = 2026 )
validations |
Table returned by |
formula |
Model formula, usually the best one of
|
n_train |
Training sizes per site x species. |
n_reps |
Repetitions per training size. |
seed |
Seed of the draws. The global random number generator is left as it was. |
Cells with fewer than n_train + 5 clips contribute all but five of
their clips, so the training size actually used can be below
n_train; it is reported in n_train_mean, with a warning. The test
set shrinks as n_train grows (n_test_mean), which widens auc_sd.
A tibble with n_per_cell (the requested training size),
auc_mean, auc_sd, n_train_mean (mean clips per cell actually
used for training), n_test_mean (mean size of the test set) and
n_failed (repetitions dropped).
With fold_group = NULL, assigns clips within each site x species,
reproducing the original fold assignment. For grouped folds, give one
or more column names, e.g. "source_audio" for recordings or "night"
for nights. Site is always part of the grouping key. All species and
clips sharing that key go into the same fold; groups are shuffled and
distributed among folds within site, balancing group counts rather
than clip counts. Each site must have at least k_folds groups.
calibration_folds(validations, k_folds = 5, seed = 2026, fold_group = NULL)calibration_folds(validations, k_folds = 5, seed = 2026, fold_group = NULL)
validations |
Calibration data including |
k_folds |
Integer number of folds, at least two. |
seed |
Seed of the assignment; the caller's random state is restored. |
fold_group |
Optional character vector of grouping columns. Do not
include |
Grouped folds evaluate new recordings or nights at known sites, not transfer to new sites. Unlike clip folds, they cannot guarantee that each species appears in each training and test set. A test fold with only one verdict has no AUC but still contributes probability scores.
Integer vector of fold numbers, in the input row order.
The family of logistic calibration models compared by
compare_calibration_models(), from a single logistic regression on
the logit of the confidence score to a binomial GLMM with random slopes
by species and a site x species intercept.
calibration_models(uncorrelated = FALSE, single_species = FALSE)calibration_models(uncorrelated = FALSE, single_species = FALSE)
uncorrelated |
Add |
single_species |
Use a family for one species: global logistic
regression, a site intercept, and a site intercept and slope. With
|
The list goes from the simplest to the most complex model; that order is
the complexity used by select_calibration_model().
A named list of formulas: m0_global, m1_species_intercept,
m2_species_slope, m3_plus_site, m4_species_x_site and, with
uncorrelated = TRUE, m5_species_x_site_uncorrelated. With
single_species = TRUE: m0_global, m1_site_intercept,
m2_site_slope and optionally m3_site_slope_uncorrelated.
Builds a weakly informative prior for a calibration formula, centred on
the hypothesis that the BirdNET score is already calibrated: on the
model scale (tp ~ logit_conf), intercept 0 and slope 1 mean
P(TP) = score. The data then show how far the classifier departs from
it. Only the classes present in the formula are returned:
calibration_prior(formula, validations)calibration_prior(formula, validations)
formula |
Calibration formula (see |
validations |
Table returned by |
Intercept: normal(m, 1.5), where m is the mean of logit_conf
among the reviewed clips. 'brms' centres the predictors, so its
intercept is the logit of P(TP) at the mean logit score; with a slope
of 1, an intercept equal to that mean gives P(TP) = score;
b (slope of logit_conf): normal(1, 1) truncated at 0, centred on
a calibrated score, wide enough to allow slopes from near 0 to near 3,
and assuming that a higher score never lowers P(TP);
sd (standard deviations of the random effects): exponential(1);
cor (correlations between random effects): lkj(2), which pulls
the correlations away from +-1 (the singular fits of m4 with 'lme4').
This is an example, not a default: fit_calibration() with
engine = "brms" requires the prior to be given explicitly. Check what
it implies on the probability scale with
prior_predictive_calibration() before fitting, and its influence on
the posterior with calibration_sensitivity(). On weakly informative
priors and prior predictive checks see McElreath (2020,
doi:10.1201/9780429029608) and Gelman et al. (2026, Bayesian
Workflow, Chapman & Hall).
A brmsprior object.
Power-scaling sensitivity analysis with 'priorsense' (Kallioinen et al. 2024, doi:10.1007/s11222-023-10366-5): how much the posterior changes when the prior or the likelihood is up- or down-weighted. A prior-data conflict or a strongly informative prior shows up as sensitivity to both.
calibration_sensitivity(fit, variable = NULL, ...)calibration_sensitivity(fit, variable = NULL, ...)
fit |
Model returned by |
variable |
Parameters to analyse. |
... |
Further arguments for |
The table of priorsense::powerscale_sensitivity().
Classifies each site x species cell of the calibration by whether its detections can enter a community matrix, separating the absence of a species from a calibration that does not work:
cell_status(fit, validations, p_min = 0.95, min_n = 15)cell_status(fit, validations, p_min = 0.95, min_n = 15)
fit |
Model returned by |
validations |
Table returned by |
p_min |
Calibrated probability that makes a detection reliable, as
in |
min_n |
Minimum number of reviewed clips in a cell. |
"reliable": the calibrated curve reaches p_min within the range of
reviewed scores of the cell, and the cell has at least min_n
reviewed clips;
"probably_absent": clips were reviewed but none was a true positive.
The species can enter the matrix with zeros at that site;
"unreliable": true positives were reviewed, but the curve never
reaches p_min within the reviewed scores, or the cell has fewer than
min_n reviewed clips, or no clip at all was reviewed. A zero in such
a cell could be a false negative.
suggest_core_species() keeps the species without unreliable cells.
The final choice of core species remains with the analyst.
A tibble with one row per site x species: site, species,
n_reviewed, n_tp, max_confidence (highest reviewed score),
threshold (lowest grid score at which the calibrated probability
reaches p_min; NA if never), status and reason. The grid
contains both observed endpoints and steps of 0.01 within the
reviewed range.
Reads the result of the manual review done in the BirdNET GUI and joins
it to the clip manifest. The GUI organises the review by moving each
clip into subfolders such as <species>/Positive and
<species>/Negative, so the verdict of each clip comes from the path
where it is found. Clips still in the species folder are not reviewed.
classify_review( manifest, review_dir, positive_pattern = "positiv", negative_pattern = "negativ" )classify_review( manifest, review_dir, positive_pattern = "positiv", negative_pattern = "negativ" )
manifest |
Clip manifest: a data frame or the path to
|
review_dir |
Folder with the reviewed clips (the |
positive_pattern, negative_pattern
|
Regular expressions (case-insensitive) that identify the positive and negative folders in the path of each clip. |
The manifest with two more columns: verdict ("positive",
"negative", "not_reviewed" or "clip_missing") and
reviewed_path, the path where the clip was found. This is the table
written to validation_by_clip.csv.
Turns the calibrated nightly detection history into site x month community matrices for multivariate ordinations (PCoA, NMDS, RDA). Because those analyses do not model false positives, the entry rule is strict:
a species counts as detected on a night only when it had at least
k_min reliable detections that night, i.e. detections with a
calibrated probability >= p_min of calibrate() (column
detection_k<k_min> of the history);
only the core species enter the matrix: species whose calibration is reliable at every site. Other species would create artefactual beta diversity from differences in classifier recall among sites;
each cell is the frequency of nights with detection, standardised by effort: nights with detection / nights recorded at that site x month.
community_matrices( history, core_species, k_min = 2, min_nights_month = 5, drop_empty = FALSE )community_matrices( history, core_species, k_min = 2, min_nights_month = 5, drop_empty = FALSE )
history |
Nightly detection history, as returned by
|
core_species |
Species codes (the |
k_min |
Repetition rule: 1, 2 or 3 reliable detections on the night. |
min_nights_month |
Site x month cells with fewer recorded nights are
dropped from the matrices (unstable frequencies) and flagged in
|
drop_empty |
Drop the site x month rows in which no core species was detected. Distances such as Bray-Curtis are undefined for them. |
Effort (nights recorded) is approximated by the number of distinct nights in the history of the site, for any species. With dense detections, every night actually recorded appears in the history.
A list with three tibbles, frequency and counts (one row per
site x month, with site, month, n_nights_recorded and one column
per core species) and effort (site, month, n_nights_recorded,
included), plus trials: a numeric matrix with the dimensions of
the species columns of counts holding the nights recorded, the
Ntrials of a binomial model of the counts (e.g. gllvm::gllvm(),
which needs a matrix when effort varies among samples). The row names
of trials are <site>_<month>.
Builds the community matrices of community_matrices() for each draw
of history_draws() and summarises the frequency of each site x month
x species cell by its posterior mean and an interval, so that the
uncertainty of the calibration reaches the matrices. The per-draw
frequencies are also returned, e.g. to repeat an ordination on each
draw.
community_matrices_draws( history, core_species, k_min = 2, min_nights_month = 5, probs = c(0.05, 0.95) )community_matrices_draws( history, core_species, k_min = 2, min_nights_month = 5, probs = c(0.05, 0.95) )
history |
Table returned by |
core_species |
Species codes (the |
k_min |
Repetition rule: 1, 2 or 3 reliable detections on the night. |
min_nights_month |
Site x month cells with fewer recorded nights are
dropped from the matrices (unstable frequencies) and flagged in
|
probs |
Probabilities of the interval. |
A list with summary (site, month, species,
n_nights_recorded, frequency_mean, frequency_lower,
frequency_upper) and draws (draw, site, month, species,
frequency).
Fits each candidate model with k-fold cross-validation and scores the
predicted probabilities on the held-out folds. Folds are stratified
within site x species by default. With fold_group, recordings or
nights are kept intact across species; see calibration_folds().
Models with random effects need the 'lme4' package.
compare_calibration_models( validations, models = NULL, k_folds = 5, seed = 2026, fold_group = NULL )compare_calibration_models( validations, models = NULL, k_folds = 5, seed = 2026, fold_group = NULL )
validations |
Table returned by |
models |
Named list of model formulas. |
k_folds |
Number of folds. |
seed |
Seed of the fold assignment. The global random number generator is left as it was. |
fold_group |
Optional character vector of grouping columns. Do not
include |
The AUC measures discrimination (whether true detections get higher
probabilities than false ones). The log-loss and the Brier score are
proper scoring rules: they reward probabilities close to the observed
outcomes, which is what matters when the probabilities are compared
with p_min (Gneiting & Raftery 2007, doi:10.1198/016214506000001437).
The calibration intercept and slope come from a logistic regression of
the verdicts on the logit of the predicted probabilities in each test
fold: a well calibrated model has intercept 0 and slope 1; a slope below
1 means probabilities too extreme (overfitting).
A tibble with model, formula, auc_mean, auc_sd,
n_failed (folds whose fit failed, left out of the means),
logloss_mean, logloss_sd, brier_mean, brier_sd,
cal_intercept and cal_slope (means over the folds), prop_singular
(share of the fold fits that were singular) and complexity (position
of the model in models), ordered from the highest mean AUC. Failed
folds also raise a warning. A test fold with only one verdict has
missing AUC but contributes log-loss and Brier; it is not a failed fit.
Combines the verdicts of several reviewers into one validation table:
each clip keeps the verdict its reviewers agree on; clips with
conflicting verdicts get verdict = "disagreement" and are listed for a
consensus review. Disagreements are left out of the precision,
thresholds and calibration, which use only positive and negative
verdicts.
consensus_verdicts(reviews)consensus_verdicts(reviews)
reviews |
One table of reviewed clips (as returned by
|
A tibble with one row per clip: the columns of the reviewed
tables (from the first reviewer of each clip), verdict,
n_reviewers and reviewers (names separated by ;).
Cuts each clip of the manifest from its source recording into
<output_dir>/clips/<species_folder>/<clip_file>, one folder per species,
ready for the review tab of the BirdNET GUI. Clips that already exist are
not cut again, so an interrupted run resumes where it stopped. Missing or
unreadable recordings do not stop the run; they are flagged in status.
cut_clips(manifest, output_dir, simulate = FALSE)cut_clips(manifest, output_dir, simulate = FALSE)
manifest |
Clip manifest, as returned by |
output_dir |
Output folder; created if needed. |
simulate |
If |
Cutting uses the 'ffmpeg' library bundled with the 'av' package.
The manifest with the column status after begin_path: one of
"ok", "already_exists", "audio_not_found", "simulated" or
"error: <message>".
Aggregates the scored detections by site x species x night. A night has
a detection when at least min_detections_night detections have a
calibrated probability >= p_min; the columns detection_k1 to
detection_k3 give the same rule with k = 1, 2 and 3, as a
sensitivity analysis.
detection_history( scored, p_min = 0.95, min_detections_night = 1, rule = c("mean", "probability"), prob_min = 0.9 )detection_history( scored, p_min = 0.95, min_detections_night = 1, rule = c("mean", "probability"), prob_min = 0.9 )
scored |
Table returned by |
p_min |
Threshold on the calibrated probability (not on the confidence score). |
min_detections_night |
Repetition rule: minimum number of reliable detections for a night with detection. |
rule |
When a detection is reliable: |
prob_min |
|
A tibble with site, species, night, n_detections,
n_reliable, p_tp_max, detection_k1, detection_k2,
detection_k3 and detection.
Drops from a clip manifest the clips that overlap, for the same species
and the same recording, a clip used to train the classifier (the
training_set.csv written by build_training_set()). Use it before
reviewing the detections of a retrained classifier: scores of the
clips it was trained on are biased upwards.
exclude_training_clips(manifest, training_set)exclude_training_clips(manifest, training_set)
manifest |
Clip manifest, as returned by |
training_set |
Table returned by |
The manifest without the overlapping clips.
Writes, in one folder, everything an occupancy model with false
positives needs from a passive acoustic survey: the occasions, the
classifier scores, the reviewed clips with their sampling design and,
optionally, posterior draws of a Bayesian calibration. The folder
follows the sheepdog-occupancy schema, version 1, described in
vignette("crywolf-integration"); the crywolf package reads it, but any
software can, since every table is a plain CSV file (the calibration
draws are a Parquet file).
export_occupancy_data( output_dir, detections, recordings = NULL, validation = NULL, fit = NULL, sites = NULL, occasion_unit = c("file", "minute"), recording_duration = NULL, min_confidence = 0.1, dawn_hour = 12, species = NULL, n_draws = 100, grid_step = 0.01, seed = 2026, audio_pattern = "\\.(wav|flac|mp3)$", remap_from = NULL, remap_to = NULL, classifier = NULL, overwrite = FALSE )export_occupancy_data( output_dir, detections, recordings = NULL, validation = NULL, fit = NULL, sites = NULL, occasion_unit = c("file", "minute"), recording_duration = NULL, min_confidence = 0.1, dawn_hour = 12, species = NULL, n_draws = 100, grid_step = 0.01, seed = 2026, audio_pattern = "\\.(wav|flac|mp3)$", remap_from = NULL, remap_to = NULL, classifier = NULL, overwrite = FALSE )
output_dir |
Folder to write. Created if needed. |
detections |
Predictions, as returned by |
recordings |
The list of all recordings, as a data frame or the
path to a CSV file with the columns |
validation |
Reviewed clips: paths to |
fit |
Optional model from |
sites |
Optional data frame of site covariates, with a |
occasion_unit |
The occasion of the occupancy model: |
recording_duration |
Length of the recordings (s). Needed for minute occasions unless the 'av' package can read the files. |
min_confidence |
Predictions below this confidence are not
written; it is also the lower end of the grid of the calibration
draws. Use the |
dawn_hour |
Recordings that start before this hour belong to the night before. |
species |
Restrict to these short species codes (the last part of
|
n_draws |
Number of posterior draws of the calibration. |
grid_step |
Step of the grid of scores of the calibration draws. |
seed |
Seed of the choice of draws. The global random number generator is left as it was. |
audio_pattern |
Regular expression for the audio files listed when
|
remap_from, remap_to
|
Replace the prefix |
classifier |
Optional label of the classifier, recorded in
|
overwrite |
Replace an existing export in |
Invisibly, a list with the tables written (tibbles) and the schema (a list).
occasions.csv: one row per occasion (a recording, or a minute of a
recording), with or without predictions.
detections.csv: one row per prediction with confidence >=
min_confidence.
scores.csv: one row per occasion x species with at least one
prediction: the highest score and the number of predictions. Occasion
x species pairs absent from this table had no prediction >=
min_confidence (a censored score).
validation.csv: the reviewed clips of the "top" and
"random_detections" designs, with their verdict and inclusion
probability (only when validation is given).
recordings_verified.csv: the occasions annotated by ear in the
"random_recordings" design, with the state of each occasion (only
when validation holds such clips).
sites.csv: one row per site, with the covariates of sites.
calibration_draws.parquet: posterior draws of P(TP) on a grid of
scores for each site x species with predictions (only when fit is
given; needs the 'arrow' package).
schema.json: name and version of the schema, the settings of the
export, the lists of sites and species and the number of rows of each
table.
A clip verdict and the state of an occasion are kept apart on purpose: a positive clip proves that the species called in its occasion, but a negative clip only says that this prediction was a false positive (the species may still have called, with a lower score). A recording annotated by ear gives the state of the whole occasion.
vignette("crywolf-integration") for the columns of each table.
Runs the whole first stage: reads the selection tables, selects the
clips, cuts them, writes clip_manifest.csv and zips one package per
species. This is the function form of the v1.2.0 script
extrair_clipes_validacao.R.
extract_validation_clips( data_dir, output_dir, ..., table_pattern = "^BirdNET_SelectionTable\\.txt$", make_zip = TRUE, simulate = FALSE, classifier = NULL )extract_validation_clips( data_dir, output_dir, ..., table_pattern = "^BirdNET_SelectionTable\\.txt$", make_zip = TRUE, simulate = FALSE, classifier = NULL )
data_dir |
Root folder, searched recursively. |
output_dir |
Output folder; created if needed. |
... |
Sampling design, passed to |
table_pattern |
Regular expression for the table file names. |
make_zip |
Zip one package per species (skipped when |
simulate |
If |
classifier |
Label of the classifier that produced the selection
tables (e.g. |
Output structure:
<output_dir>/clips/<species_folder>/*.wav <output_dir>/packages/<species_folder>.zip <output_dir>/clip_manifest.csv <output_dir>/extraction_info.csv
extraction_info.csv records the provenance of the run (one key,
value row per item): the date, the versions of sheepdog and R, the
folder of the selection tables, the classifier label and every
argument passed to select_clips(). Keep it with the manifest: it tells
later rounds which classifier produced the clips (for instance, to leave
out of a validation round the clips used to retrain that classifier).
The manifest written to clip_manifest.csv, invisibly.
## Not run: # Check the selection first, then cut extract_validation_clips("N:/Arara", "N:/Arara/validation", remap_from = "N:/", remap_to = "/Volumes/Arara/", simulate = TRUE) ## End(Not run)## Not run: # Check the selection first, then cut extract_validation_clips("N:/Arara", "N:/Arara/validation", remap_from = "N:/", remap_to = "/Volumes/Arara/", simulate = TRUE) ## End(Not run)
Fits a calibration model to all the reviewed clips. The species and
site levels of validations are stored with the model, so that
score_detections() can score new detections.
fit_calibration( validations, formula, engine = c("lme4", "brms"), prior = NULL, ... )fit_calibration( validations, formula, engine = c("lme4", "brms"), prior = NULL, ... )
validations |
Table returned by |
formula |
Model formula, usually the best one of
|
engine |
|
prior |
|
... |
|
With engine = "lme4" (default), the model is fitted by maximum
likelihood ('lme4' for random effects, glm() otherwise). A model with
random effects whose fit is singular (a variance estimated at zero or a
correlation at +-1) raises a warning: the fixed effects and the
predictions can still be used, but the variance components on the
boundary should not be interpreted, and a simpler model may predict as
well. Check them with lme4::VarCorr(). The other messages of
lme4::glmer() are passed on, prefixed by lme4:, and every message of
the fit is kept in the attribute sheepdog_messages of the model
(convergence warnings of 'lme4' are warnings and are passed on as such).
With engine = "brms", the model is fitted in a Bayesian framework
with 'brms' (Bernoulli family, logit link). The prior must be given
explicitly; calibration_prior() builds an example centred on an
already calibrated score. After the fit, calibration_diagnostics()
runs and a warning is raised when R-hat, the effective sample sizes or
the divergent transitions fail the usual limits. Check the prior with
prior_predictive_calibration(), the fit with brms::pp_check() and
the sensitivity to the prior with calibration_sensitivity().
A glm, glmerMod or brmsfit model, with the attribute
sheepdog_levels (the species and site levels), used by
score_detections() and plot_calibration().
Propagates the uncertainty of a Bayesian calibration model to the
detection histories: for each of n_draws draws of the posterior, a
detection is reliable when its probability in that draw is >= p_min,
and the nightly history is rebuilt. The histories can then be turned
into community matrices with uncertainty by
community_matrices_draws().
history_draws( fit, scored, p_min = 0.95, n_draws = 200, min_detections_night = 1, min_confidence = 0.1, grid_step = 0.01, seed = 2026 )history_draws( fit, scored, p_min = 0.95, n_draws = 200, min_detections_night = 1, min_confidence = 0.1, grid_step = 0.01, seed = 2026 )
fit |
Model returned by |
scored |
Table returned by |
p_min |
Threshold on the calibrated probability (not on the confidence score). |
n_draws |
Number of posterior draws. |
min_detections_night |
Repetition rule: minimum number of reliable detections for a night with detection. |
min_confidence |
Detections below this confidence are not scored. |
grid_step |
Models fitted with With a model fitted with |
seed |
Seed of the choice of draws. The global random number generator is left as it was. |
As in score_detections(), the probabilities are computed on a grid of
scores (step grid_step) for each site x species and assigned to the
detections by their rounded score, which keeps the cost independent of
the number of detections.
A tibble with draw, site, species, night,
n_detections, n_reliable, detection_k1 to detection_k3 and
detection.
Summarises the verdicts by site, species and night. A night is
"confirmed" when at least one clip is positive,
"not_detected_in_reviewed" when clips were reviewed but none was
positive, and "not_reviewed" otherwise.
nightly_presence(validation)nightly_presence(validation)
validation |
Table returned by |
A tibble with site, species_folder, night, n_clips,
n_reviewed, n_positive, presence and max_confidence_positive.
Zips each species folder of <output_dir>/clips into
<output_dir>/packages/<species_folder>.zip: light validation packages
to send to the reviewers.
pack_clips(manifest, output_dir)pack_clips(manifest, output_dir)
manifest |
Clip manifest, as returned by |
output_dir |
The |
The paths of the zip files, invisibly.
Calibrated probability of a true detection against the confidence score, one panel per species and one line per site, with the observed precision by confidence bin as points. Needs the 'ggplot2' package.
plot_calibration( fit, validations, min_confidence = 0.1, p_min = 0.95, model_name = "" )plot_calibration( fit, validations, min_confidence = 0.1, p_min = 0.95, model_name = "" )
fit |
Model returned by |
validations |
Table returned by |
min_confidence |
Detections below this confidence are not scored. |
p_min |
Threshold on the calibrated probability (not on the confidence score). |
model_name |
Name shown in the subtitle. |
A ggplot object.
Draws the population-level calibration curve, P(TP) against the score,
for n_draws draws of a Bayesian calibration model, without the
species and site effects. With a model from
prior_predictive_calibration() it shows what the prior implies on the
probability scale (a prior predictive check); with a fitted model, the
uncertainty of the curve. Needs the 'ggplot2' package.
plot_calibration_draws(fit, n_draws = 100, p_min = 0.95, seed = 2026)plot_calibration_draws(fit, n_draws = 100, p_min = 0.95, seed = 2026)
fit |
Model returned by |
n_draws |
Number of curves. |
p_min |
Threshold drawn as a dashed line. |
seed |
Seed of the choice of draws. The global random number generator is left as it was. |
A ggplot object.
One panel per species: precision by confidence bin, with the target precision as a dashed line and bins with low support shown faded. With more than one site, each site is a line of its own colour. Needs the 'ggplot2' package.
plot_precision( precision, bins = seq(0.1, 1, by = 0.1), target_precision = 0.9, min_n = 15 )plot_precision( precision, bins = seq(0.1, 1, by = 0.1), target_precision = 0.9, min_n = 15 )
precision |
Table returned by |
bins |
Breaks of the confidence bins (left-closed). |
target_precision |
Target precision. |
min_n |
Minimum of reviewed clips for a bin to be taken seriously;
bins with fewer are flagged in |
A ggplot object.
Runs the whole post-validation step: classifies the reviewed clips and
writes, to output_dir:
validation_by_clip.csv: manifest + verdict of each clip
(classify_review());
precision_by_bin.csv: precision by species x confidence bin
(precision_by_bin());
suggested_thresholds.csv: lowest confidence reaching the target
precision, by species (suggest_thresholds());
nightly_presence.csv: confirmed presence by site x species x night
(nightly_presence());
retraining_clips.csv: positive clips for retraining
(retraining_clips());
precision_by_bin.png: precision curves (plot_precision()), when
'ggplot2' is installed.
post_validation( review_dir, manifest_path, output_dir, positive_pattern = "positiv", negative_pattern = "negativ", bins = seq(0.1, 1, by = 0.1), target_precision = 0.9, min_n = 15 )post_validation( review_dir, manifest_path, output_dir, positive_pattern = "positiv", negative_pattern = "negativ", bins = seq(0.1, 1, by = 0.1), target_precision = 0.9, min_n = 15 )
review_dir |
Folder with the reviewed clips (the |
manifest_path |
Path to the manifest of the same run as the
reviewed clips ( |
output_dir |
Output folder; created if needed. |
positive_pattern, negative_pattern
|
Regular expressions (case-insensitive) that identify the positive and negative folders in the path of each clip. |
bins |
Breaks of the confidence bins (left-closed). |
target_precision |
Target precision. |
min_n |
Minimum of reviewed clips for a bin to be taken seriously;
bins with fewer are flagged in |
This is the function form of the v1.2.0 script pos_validacao.R.
A list with the tables validation, precision, thresholds,
presence and retraining, invisibly.
Precision (share of positive verdicts) of the reviewed clips, by site,
species and confidence bin. Clips without confidence (whole recordings
of "random_recordings") are left out.
precision_by_bin(validation, bins = seq(0.1, 1, by = 0.1), min_n = 15)precision_by_bin(validation, bins = seq(0.1, 1, by = 0.1), min_n = 15)
validation |
Table returned by |
bins |
Breaks of the confidence bins (left-closed). |
min_n |
Minimum of reviewed clips for a bin to be taken seriously;
bins with fewer are flagged in |
A tibble with site, species_folder, bin, n_reviewed,
n_positive, precision and low_support.
Samples the calibration model from the prior only (no data enter the
likelihood), so that the curves it implies on the probability scale
can be checked before the model is fitted. Plot them with
plot_calibration().
prior_predictive_calibration(formula, validations, prior, ...)prior_predictive_calibration(formula, validations, prior, ...)
formula |
Calibration formula (see |
validations |
Table returned by |
prior |
Prior, e.g. from |
... |
Further arguments for |
A brmsfit sampled from the prior, usable in
plot_calibration().
Finds every BirdNET selection table (Raven format, tab-separated) under
data_dir, reads them and stacks them in one table. Each table holds the
predictions of one site, and the site is the name of the folder that
contains it. The output_ prefix of BirdNET output folders is removed, so
output_UEM/ becomes site UEM.
read_selection_tables( data_dir, table_pattern = "^BirdNET_SelectionTable\\.txt$" )read_selection_tables( data_dir, table_pattern = "^BirdNET_SelectionTable\\.txt$" )
data_dir |
Root folder, searched recursively. |
table_pattern |
Regular expression for the table file names. |
Column names are standardised to lower case without units or punctuation,
e.g. Begin Time (s) becomes begin_time and File Offset (s) becomes
file_offset.
A tibble with one row per prediction, the column site first,
followed by the standardised table columns. The columns begin_time,
end_time, species_code, confidence, begin_path and file_offset
are required.
Stacks the validation_by_clip.csv tables of several runs or sites
(output of post_validation()) and keeps every column and every
verdict, unlike read_validations(), which keeps only what the
calibration needs. Tables written by v1.2.0 (validacao_por_clipe.csv)
are also read.
read_validation_tables(files)read_validation_tables(files)
files |
Paths to the |
A tibble with the columns of the validation tables; night and
recording_date are character.
Stacks the validation_by_clip.csv tables of several sites (output of
post_validation()) and keeps the clips with a positive or negative
verdict and a confidence score. Tables written by v1.2.0
(validacao_por_clipe.csv) are also read. Whole-recording clips of
"random_recordings" have no confidence and are left out.
read_validations(files, site_map = NULL, keep = NULL)read_validations(files, site_map = NULL, keep = NULL)
files |
Paths to the |
site_map |
Optional named character vector to fix diverging site
names, as |
keep |
Optional character vector of input columns to retain, e.g.
|
A tibble with site and species (factors; species is the
species folder code), confidence, logit_conf (logit of the
confidence, bounded at 1e-4 and 1 - 1e-4) and tp (1 = true
positive, 0 = false positive), plus the columns requested in keep.
Lists the clips confirmed as positive, ready to retrain the classifier.
retraining_clips(validation)retraining_clips(validation)
validation |
Table returned by |
A tibble with site, species_code, species_folder,
clip_file, reviewed_path, confidence, night, recording_hour
and source_audio.
Measures, for each species, the agreement between reviewers on the clips reviewed more than once: the proportion of agreement and kappa, which discounts the agreement expected by chance. When a species has exactly two reviewers, this is Cohen's kappa, with the marginal frequencies of each reviewer (Cohen 1960, doi:10.1177/001316446002000104); with more reviewers, or different pairs of reviewers on different clips, Fleiss' kappa (Fleiss 1971, doi:10.1037/h0031619). Only positive and negative verdicts count.
reviewer_agreement(reviews)reviewer_agreement(reviews)
reviews |
One table of reviewed clips (as returned by
|
A tibble with species_folder, n_clips (clips with at least
two verdicts), n_reviewers, agreement, kappa and method.
Reads the complete BirdNET selection tables (not only the reviewed
clips) and predicts, for each detection, the probability that it is a
true positive. Only species present in the calibration are scored. The
night of each detection is its recording date, or the day before for
recordings that start before dawn_hour.
score_detections( tables, fit, min_confidence = 0.1, dawn_hour = 12, p_min = 0.95, lower_prob = 0.05, grid_step = 0.01 )score_detections( tables, fit, min_confidence = 0.1, dawn_hour = 12, p_min = 0.95, lower_prob = 0.05, grid_step = 0.01 )
tables |
Named character vector of selection-table paths, named by site (the site names of the calibration). |
fit |
Model returned by |
min_confidence |
Detections below this confidence are not scored. |
dawn_hour |
Recordings that start before this hour belong to the night before. |
p_min, lower_prob
|
Models fitted with |
grid_step |
Models fitted with With a model fitted with |
A tibble with site, species, date, night, hour,
confidence, p_tp, begin_path and file_offset; with a model
fitted with engine = "brms", also p_tp_lower and pr_reliable.
Picks a model from the table of compare_calibration_models().
select_calibration_model( comparison, rule = c("one_se_logloss", "best_auc"), k_folds = 5 )select_calibration_model( comparison, rule = c("one_se_logloss", "best_auc"), k_folds = 5 )
comparison |
Table returned by |
rule |
Selection rule. |
k_folds |
Number of folds of the comparison (for the standard error). |
"best_auc": the model with the highest mean AUC (the rule of
calibrate() in 2.x).
"one_se_logloss": among the models whose mean log-loss is within one
standard error (logloss_sd / sqrt(k_folds)) of the lowest, the
simplest (lowest complexity); models singular in most folds are left
out, unless all are, which raises a warning. The log-loss rewards
correct probabilities, not only their ranking, and the one-standard-
error rule prefers the simpler model when the data cannot tell the
candidates apart. This is the default of calibrate() since 3.0.0.
The name of the selected model.
Applies the sampling design to the predictions read by
read_selection_tables() and returns the clip manifest: one row per clip
to cut, with its site, species, night, source audio, start and end. No
audio is read or written; cut the clips with cut_clips().
select_clips( detections, sampling_mode = c("top", "random_detections", "random_recordings"), occasion_unit = c("file", "minute"), threshold = NULL, n_top = 2, strata = "night_hour", recording_fraction = 0.01, n_recordings_per_site = NULL, recordings = NULL, audio_pattern = "\\.(wav|flac|mp3)$", recording_duration = NULL, min_confidence = 0.1, sites = NULL, date_start = NULL, date_end = NULL, species = NULL, hour_start = NULL, hour_end = NULL, alternate_days = FALSE, target_total = NULL, balance_by_hour = TRUE, max_per_species = NULL, remap_from = NULL, remap_to = NULL, seed = 2026 )select_clips( detections, sampling_mode = c("top", "random_detections", "random_recordings"), occasion_unit = c("file", "minute"), threshold = NULL, n_top = 2, strata = "night_hour", recording_fraction = 0.01, n_recordings_per_site = NULL, recordings = NULL, audio_pattern = "\\.(wav|flac|mp3)$", recording_duration = NULL, min_confidence = 0.1, sites = NULL, date_start = NULL, date_end = NULL, species = NULL, hour_start = NULL, hour_end = NULL, alternate_days = FALSE, target_total = NULL, balance_by_hour = TRUE, max_per_species = NULL, remap_from = NULL, remap_to = NULL, seed = 2026 )
detections |
Predictions, as returned by |
sampling_mode |
|
occasion_unit |
The occasion of the occupancy model: |
threshold |
Confidence threshold of the occupancy analysis. It
defines the candidate occasions of |
n_top |
|
strata |
or a character vector of column names of With one recording per hour and |
recording_fraction |
|
n_recordings_per_site |
|
recordings |
|
audio_pattern |
Regular expression for the audio files listed when
|
recording_duration |
Length of the recordings (s), for
|
min_confidence |
Minimum confidence of a candidate prediction in
|
sites |
Restrict to these sites. |
date_start, date_end
|
Date window of the recordings ( |
species |
Restrict to these short species codes (the last part of
|
hour_start, hour_end
|
Hour window (0-23). It can cross midnight:
|
alternate_days |
Keep alternate days (or nights), anchored on the first date of each site. |
target_total |
Total clips per site, split evenly among species by
water filling: species with fewer clips than the quota enter whole and
the surplus goes to the others. In |
balance_by_hour |
Within each species, draw round-robin across the
hours of the day (covers the diel cycle) when applying |
max_per_species |
Simpler alternative to |
remap_from, remap_to
|
Replace the prefix |
seed |
Seed of every random draw. The global random number generator is left as it was. |
A tibble, the clip manifest without status, with the columns
site, species_code, species_folder, recording_date, night,
recording_hour, rank, confidence, clip_file, clip_start,
clip_end, duration, source_audio, begin_path, sampling_mode,
occasion_id, n_detections_occasion and inclusion_prob. See
vignette("crywolf-integration") for their definitions.
"top" keeps the n_top predictions with the highest confidence per
site x species x night x hour. It serves classifier retraining and the
score-to-probability calibration of calibrate(). Do not use it to
estimate false-positive rates: top-score clips hold fewer false
positives than a random sample.
"random_detections" draws, at random, n_top occasions per
stratum (by default site x species x night x hour; see strata)
among the occasions with at least one
prediction of the species with confidence >= threshold, and keeps
every prediction >= threshold of each drawn occasion. This is the
observation-confirmation design of Chambert et al. (2015,
doi:10.1890/14-1507.1).
"random_recordings" draws recordings (or minutes) among all those
recorded, with or without predictions, in proportion to the effort of
each site, for annotation by ear. This is the multiple-method design of
Miller et al. (2011, doi:10.1890/10-1396.1). The clip is the whole
recording (or minute), copied once per species to annotate, and
confidence is NA.
In both random modes, use the same threshold as the occupancy analysis.
Copies the cut clips of a manifest with reviewers
(assign_reviewers()) to <output_dir>/<reviewer>/<species_folder>/,
one folder tree per reviewer, ready for the review tab of the BirdNET
GUI. Clips with a second reviewer are copied to both folders. After the
review, classify each reviewer's folder with classify_review().
split_by_reviewer(manifest, clips_dir, output_dir)split_by_reviewer(manifest, clips_dir, output_dir)
manifest |
Manifest returned by |
clips_dir |
Folder of the cut clips (the |
output_dir |
Output folder. |
A tibble with reviewer, species_folder, clip_file and to,
invisibly.
Keeps the species with no unreliable site x species cell in the table of
cell_status(): reliable at every site, or reliable where present and
probably absent elsewhere.
suggest_core_species(status)suggest_core_species(status)
status |
Table returned by |
A character vector of species codes (the species column of
the history).
For each site and species, the lowest confidence c such that the
precision among the reviewed clips with confidence >= c reaches
target_precision, with at least min_n clips. Candidate cuts are the
observed confidences rounded to two decimals.
suggest_thresholds(validation, target_precision = 0.9, min_n = 15)suggest_thresholds(validation, target_precision = 0.9, min_n = 15)
validation |
Table returned by |
target_precision |
Target precision. |
min_n |
Minimum of reviewed clips for a bin to be taken seriously;
bins with fewer are flagged in |
A tibble with site, species_folder, threshold,
n_support and precision_at_threshold (NA when no cut reaches
the target). Without any reviewed clip with a confidence, the tibble
has no rows.
Summarise the detection history by site and species
summarise_history(history)summarise_history(history)
history |
Table returned by |
A tibble with site, species, nights (nights with any
scored detection) and nights_k1 to nights_k3 (nights with
detection under each repetition rule).