sheepdog writes two tables that
occupancy models with false positives, such as those of the crywolf
package, read: clip_manifest.csv
(extract_validation_clips()) and
validation_by_clip.csv (post_validation()).
This document fixes their columns, types and the definition of an
occasion. The two packages are independent: sheepdog does not depend on
crywolf, and crywolf reads these files without loading sheepdog.
Example files ship with the package:
library(sheepdog)
manifest <- system.file("extdata", "clip_manifest.csv", package = "sheepdog")
validation <- system.file("extdata", "validation_by_clip.csv",
package = "sheepdog")
v <- readr::read_csv(validation, show_col_types = FALSE)
v[, c("site", "occasion_id", "confidence", "n_detections_occasion",
"inclusion_prob", "verdict")]
#> # A tibble: 12 × 6
#> site occasion_id confidence n_detections_occasion inclusion_prob verdict
#> <chr> <chr> <dbl> <dbl> <dbl> <chr>
#> 1 ARA ARA01_20220110… 0.913 1 0.25 positi…
#> 2 ARA ARA01_20220110… 0.789 2 0.25 positi…
#> 3 ARA ARA01_20220110… 0.938 2 0.25 positi…
#> 4 ARA ARA01_20220111… 0.532 3 0.25 negati…
#> 5 ARA ARA01_20220111… 0.699 3 0.25 negati…
#> 6 ARA ARA01_20220111… 0.622 3 0.25 positi…
#> 7 ARA ARA01_20220111… 0.908 1 0.167 not_re…
#> 8 ARA ARA01_20220115… 0.552 4 0.167 positi…
#> 9 ARA ARA01_20220115… 0.964 4 0.167 positi…
#> 10 ARA ARA01_20220115… 0.723 4 0.167 positi…
#> 11 ARA ARA01_20220115… 0.576 4 0.167 negati…
#> 12 ARA ARA01_20220115… 0.705 1 0.333 positi…They come from a "random_detections" run (occasion =
file, threshold 0.5) on simulated data
(data-raw/make_extdata.R).
An occasion is the sampling unit of the occupancy model, set by
occasion_unit:
"file": one recording. occasion_id is the
file name without extension,
e.g. ARA01_20220110_193000."minute": one minute of a recording, from
file_offset. occasion_id is
<file>_m<minute>, with the minute zero-based
and padded to three digits, e.g. ARA01_20220110_193000_m002
for 120-180 s.The site, species and occasion (site,
species_code, occasion_id) identify the
detection history cell. With minute occasions, match them to the
occasions of the model by file and minute of
clip_start.
One row per clip. Columns, in this order:
| column | type | definition |
|---|---|---|
site |
character | Site: the folder of the selection table, without the
output_ prefix. |
species_code |
character | Classifier label, e.g. Boana raniceps_boa-ran. |
species_folder |
character | Short species code used as folder name,
e.g. boa_ran. |
recording_date |
character YYYYMMDD |
Date in the recording file name. |
night |
character YYYYMMDD |
Grouping date: equals recording_date, or the day before
for small hours when the hour window crosses midnight. |
recording_hour |
character HH |
Hour in the recording file name. |
rank |
integer | "top": rank by confidence in the site x species x night
x hour stratum. "random_detections": order of the clip in
its occasion (by file_offset). NA in
"random_recordings". |
confidence |
double | BirdNET confidence score. NA for whole-recording clips
("random_recordings"). |
clip_file |
character | File name of the clip
(<conf>_<rank>_<recording>_<start>s_<end>s.wav,
or rec_<site>_<species>_<occasion_id>.wav
for whole recordings). |
clip_start, clip_end |
double (s) | Start and end of the clip in the source recording.
clip_end is NA for whole recordings of unknown
length. |
duration |
double (s) | clip_end - clip_start. |
source_audio |
character | Path of the source recording, after remapping. |
begin_path |
character | Path as written in the selection table
(Begin Path). |
status |
character | ok, already_exists,
audio_not_found, simulated or
error: <message>. |
sampling_mode |
character | top, random_detections or
random_recordings. |
occasion_id |
character | Occasion (see above). |
n_detections_occasion |
integer | Predictions of the species with confidence >=
threshold in the occasion. NA when no
threshold was given. |
inclusion_prob |
double | Probability that the occasion (or recording) was drawn.
NA in "top", which is not a probability
sample. |
In "random_detections", every prediction >=
threshold of a drawn occasion is cut, so the number of
clips of the occasion equals n_detections_occasion. This is
the assumption of the observation-confirmation design (Chambert et
al. 2015, https://doi.org/10.1890/14-1507.1): every detection of a
checked occasion is checked.
The manifest plus two columns:
| column | type | definition |
|---|---|---|
verdict |
character | positive (true detection), negative (false
positive), not_reviewed (clip found in the review folder,
but in no subfolder matching the positive or negative pattern,
e.g. still in the species folder) or clip_missing (clip not
found in the review folder). |
reviewed_path |
character | Path where the reviewed clip was found; NA when
missing. |
Only positive and negative are reviewed
verdicts. Clips of "random_recordings"
(confidence = NA) are the second method of a
multiple-method design (Miller et al. 2011, https://doi.org/10.1890/10-1396.1), that is, manual
annotation of recordings. They are not confirmations of detections and
must not go to the validation argument of
crywolf::fp_aggregate().
export_occupancy_data() writes in one folder everything
an occupancy model with false positives needs: every occasion, the
classifier scores, the reviewed clips with their design and, optionally,
posterior draws of a Bayesian calibration. The folder follows the schema
sheepdog-occupancy, version 1, described below. The tables
are plain CSV files (missing values are empty cells); the calibration
draws are a Parquet file.
ex <- system.file("extdata", "example", package = "sheepdog")
det <- read_selection_tables(file.path(ex, "birdnet"))
out <- export_occupancy_data(
file.path(tempdir(), "occupancy"), det,
recordings = file.path(ex, "recordings.csv"),
validation = list.files(file.path(ex, "validation"), full.names = TRUE)
)
vapply(out$tables, nrow, integer(1))
#> occasions detections scores validation sites
#> 354 7553 2298 2467 4The example clips were drawn with "top", so
validation.csv serves the calibration but not the
false-positive rate of an occupancy model (see
sampling_mode below).
| field | definition |
|---|---|
schema, schema_version |
"sheepdog-occupancy" and 1. A reader
should refuse other names and versions it does not know. |
created, sheepdog_version,
classifier |
Provenance of the export. |
occasion_unit |
"file" or "minute" (see Occasion
above). |
min_confidence |
Scores below this value were not written: an occasion x species pair
absent from scores.csv had no prediction at or above
it. |
dawn_hour |
Recordings before this hour belong to the night before. |
sites, species |
Every site and every classifier label (species_code) of
the export. |
tables |
For each table, its file and number of rows. |
calibration |
With fit: engine, formula,
n_draws, grid_step and
grid_min. |
One row per occasion, with or without predictions, from the list of
recordings (recordings).
| column | type | definition |
|---|---|---|
site, occasion_id |
character | The occasion (see Occasion above). |
source_audio |
character | Path of the recording. |
recording_date, recording_hour |
character | Date (YYYYMMDD) and hour (HH) in the file
name. |
night |
character YYYYMMDD |
Night of the occasion: the day before for recordings before
dawn_hour. |
start, duration |
double (s) | Start of the occasion in the recording (0 for files) and its length
(NA when unknown). |
detections.csv has one row per prediction with
confidence >= min_confidence: site,
occasion_id, species_code,
species (the short code of species_folder, the
key of the calibration), file_offset (s) and
confidence. scores.csv summarises them by
occasion x species: max_score (the highest confidence) and
n_above_min (the number of predictions). Pairs absent from
scores.csv are censored at min_confidence.
validation.csv holds the clips of the "top"
and "random_detections" designs: site,
occasion_id, species_code,
species, clip_start, clip_end,
confidence, sampling_mode,
inclusion_prob, n_detections_occasion,
verdict (as in validation_by_clip.csv, or
disagreement after consensus_verdicts()) and
tp (1 for positive, 0 for
negative, empty otherwise), plus reviewer and
n_reviewers when present. Only clips of
"random_detections" estimate a false-positive rate;
"top" clips serve the calibration.
recordings_verified.csv holds the occasions annotated by
ear ("random_recordings"), with present (1, 0
or empty) in place of tp. The two tables are kept apart
because they say different things: a positive clip proves that the
species called in its occasion, a negative clip only that this
prediction was false (the species may have called with a lower score),
while an annotated recording gives the state of the whole occasion.
One row per site, with the covariates given in
sites.
With fit (a model of
fit_calibration(engine = "brms")), n_draws
draws of the posterior of P(TP) on a grid of scores from
min_confidence in steps of grid_step, for each
site x species with predictions: draw, site,
species, grid_index, confidence
(the grid point) and p_tp. A detection takes the value of
the grid point nearest its score, as in score_detections().
Species absent from the calibration are left out; sites absent from it
get the species effects only. Reading it needs the ‘arrow’ package (in
R) or any Parquet reader.
Up to v1.2.0 the same tables were written in Portuguese
(manifesto_clipes.csv,
validacao_por_clipe.csv). sheepdog reads them and renames
on reading. The map is:
| v1.2.0 | v2.0.0 |
|---|---|
local |
site |
pasta_especie |
species_folder |
data_gravacao |
recording_date |
noite |
night |
hora_gravacao |
recording_hour |
arquivo_clipe |
clip_file |
inicio_clipe, fim_clipe,
duracao |
clip_start, clip_end,
duration |
audio_origem |
source_audio |
begin_path_original |
begin_path |
modo_amostragem |
sampling_mode |
ocasiao_id |
occasion_id |
n_deteccoes_ocasiao |
n_detections_occasion |
prob_inclusao |
inclusion_prob |
veredito |
verdict |
caminho_wav |
reviewed_path |
Values: aleatorio_deteccoes and
aleatorio_gravacoes become random_detections
and random_recordings; positivo,
negativo, nao_validado and
clipe_ausente become positive,
negative, not_reviewed and
clip_missing; ja_existia,
audio_nao_encontrado, simulado and
erro: become already_exists,
audio_not_found, simulated and
error:. Whole-recording clips cut by v1.2.0 keep their
grav_ prefix on disk; new ones use rec_.