mhq_terr_assessments: many cases where change_location contradicts the match between grts_ranking & grts_ranking_draw

[florisvdh]

Using current master; d29cd9b.

I'm actually not using mhq_terr_assessments directly, since I understood that mhq_terr_popunits already builds on it. But as there seems to be separate information about replacement in mhq_terr_assessments, I wanted to do a quick check of that.

Code to reproduce objects

> mhq_terr_datapath <- file.path(dirname(gitroot), "n2khab-sample-admin/data/mhq_terr/rapportage2025")
> 
> mhq_terr_popunits <-
    read_vc("mhq_terr_popunits", root = mhq_terr_datapath) %>%
    as_tibble()
> 
> mhq_terr_assessments <-
    read_vc("mhq_terr_assessments", root = mhq_terr_datapath) %>%
    as_tibble()

According to METADATA.md, regarding the mhq_terr_assessments data source:

change_location: has the population unit been replaced? (TRUE/FALSE)

My assumptions:

• change_location refers to a local replacement
• local replacements in mhq_terr_popunits are reflected by grts_ranking_draw != grts_ranking, as told. This is reflected by local_replacement in below output.

While looking only at mhq_terr_popunits units that have been assessed (according to source), it can be seen that corresponding locations in mhq_terr_assessments have a number of contradictions between change_location and local_replacement.

> assessment_replacement <- 
    mhq_terr_assessments %>%
    select(assessment_date, point_code, type, is_present, change_location) %>%
    inner_join(
      mhq_terr_popunits %>%
        filter(str_detect(source, "assessment")) %>% 
        select(point_code, grts_ranking, grts_ranking_draw, type),
      join_by(point_code, type),
      relationship = "many-to-one",
      unmatched = c("drop", "error")
    ) %>%
    mutate(local_replacement = grts_ranking != grts_ranking_draw) %>% 
    relocate(local_replacement, .before = grts_ranking)
> 
> assessment_replacement %>%
    count(is_present, change_location, local_replacement) %>%
    filter(change_location != local_replacement)
# A tibble: 4 × 4
  is_present change_location local_replacement     n
  <lgl>      <lgl>           <lgl>             <int>
1 FALSE      FALSE           TRUE                  2
2 TRUE       FALSE           TRUE                249
3 TRUE       TRUE            FALSE               156
4 NA         FALSE           TRUE                  2

Given these numbers, I'm probably missing something. Maybe it has to do with revisits of the same location; below code may help to investigate all assessments of the involved locations. But as said, I'm not relying on mhq_terr_assessments (with change_location), only on mhq_terr_popunits.

> assessment_replacement %>%
    filter(change_location != local_replacement) %>% 
    semi_join(assessment_replacement, ., join_by(point_code)) %>% 
    arrange(grts_ranking, grts_ranking_draw, type, point_code) %>% 
    print(n = 25)
# A tibble: 541 × 8
   assessment_date point_code type     is_present change_location local_replacement grts_ranking grts_ranking_draw
   <date>          <chr>      <chr>    <lgl>      <lgl>           <lgl>                    <dbl>             <dbl>
 1 2015-04-20      3894_2     9120     TRUE       TRUE            FALSE                     3894              3894
 2 2017-04-21      4945_2     9130_end TRUE       TRUE            FALSE                     4945              4945
 3 2018-06-22      12710_2    9120     TRUE       TRUE            FALSE                    12710             12710
 4 2014-06-05      18546_2    6510_hu  TRUE       TRUE            FALSE                    18546             18546
 5 2020-05-20      18546_2    6510_hu  TRUE       FALSE           FALSE                    18546             18546
 6 2022-06-09      18546_2    6510_hu  TRUE       FALSE           FALSE                    18546             18546
 7 2020-07-03      23221_2    2130_hd  TRUE       TRUE            FALSE                    23221             23221
 8 2015-07-05      23257_2    9130     TRUE       TRUE            FALSE                    23257             23257
 9 2018-06-05      23385_2    9130_end TRUE       TRUE            FALSE                    23385             23385
10 2018-06-26      36774_2    9120     TRUE       TRUE            FALSE                    36774             36774
11 2015-08-11      37557_2    1330_hpr TRUE       TRUE            FALSE                    37557             37557
12 2022-09-28      37557_2    1330_hpr NA         FALSE           FALSE                    37557             37557
13 2023-10-12      37557_2    1330_hpr TRUE       FALSE           FALSE                    37557             37557
14 2014-05-30      38545_2    6510_hua TRUE       TRUE            FALSE                    38545             38545
15 2022-05-31      38545_2    6510_hua TRUE       FALSE           FALSE                    38545             38545
16 2014-06-11      41814_2    6510_hu  TRUE       TRUE            FALSE                    41814             41814
17 2020-05-13      41814_2    6510_hu  TRUE       FALSE           FALSE                    41814             41814
18 2016-06-16      46662_2    6510_hu  TRUE       TRUE            FALSE                    46662             46662
19 2014-06-13      47446_2    6510_hu  TRUE       TRUE            FALSE                    47446             47446
20 2016-09-05      58433_2    6230_hn  TRUE       TRUE            FALSE                    58433             58433
21 2023-07-05      58433_2    6230_hn  TRUE       FALSE           FALSE                    58433             58433
22 2014-06-06      63270_2    6510_hu  TRUE       TRUE            FALSE                    63270             63270
23 2020-05-13      63270_2    6510_hu  TRUE       FALSE           FALSE                    63270             63270
24 2016-06-03      78294_2    6510_hu  TRUE       TRUE            FALSE                    78294             78294
25 2024-06-05      78294_2    6510_hu  TRUE       FALSE           FALSE                    78294             78294
# ℹ 516 more rows
# ℹ Use `print(n = ...)` to see more rows

[ToonHub]

'change_location' means that the measurement was made at a distance > 1 m from the targeted reference point, so grts_ranking can be the same as grts_ranking_draw (the measurement location is in the same grts grid cell than the original reference point)

in the first version of the protocol it was allowed for certain types with low detection rate to change the location of the measurements within a distance of 100 m.

[florisvdh]

Thanks! The solution will be to clarify in the metadata.

[florisvdh]

'change_location' means that the measurement was made at a distance > 1 m from the targeted reference point, so grts_ranking can be the same as grts_ranking_draw (the measurement location is in the same grts grid cell than the original reference point)

Am I right then to expect that change_location, if TRUE, normally means that the considered point will not coincide with a cell center (is_centroid)? To check my assumption, I added below code, but since quite some combinations of TRUE x TRUE seem to exist (n = 109), my understanding may be wrong.

> mhq_terr_refpoints <-
    read_vc("mhq_terr_refpoints", root = mhq_terr_datapath) %>%
    as_tibble()
> 
> mhq_terr_assessments %>% 
    inner_join(
      mhq_terr_refpoints,
      join_by(point_code),
      relationship = "many-to-one",
      unmatched = c("error", "drop")
    ) %>% 
    count(is_centroid, change_location)
# A tibble: 5 × 3
  is_centroid change_location     n
  <lgl>       <lgl>           <int>
1 FALSE       FALSE             330
2 FALSE       TRUE              432
3 TRUE        FALSE            3209
4 TRUE        TRUE              109
5 TRUE        NA                304

The question then is also, whether is_centroid is correct in those 109 cases. It is something I make use of (it is an attribute of mhq_terr_refpoints): the MNE base sampling frame is updated for cell-center assessments for types where the cell-center is used for evaluation (NB we have 7 types that are an exception to this rule in MNE, which are evaluated at the whole cell level, not cell-center).

FWIW, change_location is missing in 304 points; is_centroid is always TRUE here. Is that a true missing (then is_centroid might be too optimistic), or is change_location actually to be considered as FALSE?

[florisvdh]

I now understand that, for MHQ:

• change_location is TRUE for all replacements, regardless of the used replacement protocol
• it can occur within or outside the cell, hence:
  • it cannot predict the presence of a shift in GRTS address
  • my above crafted variable local_replacement, defined by a shift in GRTS address, has a misleading name since it doesn't include all replacements according to the first protocol. In below code, I renamed it more specifically as grts_address_shift

So we may expect that:

1. if change_location is TRUE and the replacement location is at the cell center, then this should coincide with a shift in GRTS address, i.e. using the second protocol
2. if there is a GRTS address shift, change_location is TRUE

Checking expectation 1 in the above created object assessment_replacement (i.e. the assessments that match point code x type of mhq_terr_popunits). There seems to be just one conflict.

This is with the updated current master; 9a066b9.

> assessment_popunits_points <-
    assessment_replacement %>%
    rename(grts_address_shift = local_replacement) %>%
    filter(is_present) %>% 
    inner_join(
      mhq_terr_refpoints %>%
        select(point_code, is_centroid),
      join_by(point_code),
      relationship = "many-to-one",
      unmatched = c("error", "drop")
    )
> assessment_popunits_points %>%
    filter(change_location) %>%
    count(change_location, is_centroid, grts_address_shift)
# A tibble: 4 × 4
  change_location is_centroid grts_address_shift     n
  <lgl>           <lgl>       <lgl>              <int>
1 TRUE            FALSE       FALSE                155
2 TRUE            FALSE       TRUE                 238
3 TRUE            TRUE        FALSE                  1
4 TRUE            TRUE        TRUE                 107
> assessment_popunits_points %>%
    filter(change_location, is_centroid, !grts_address_shift)
# A tibble: 1 × 9
  assessment_date point_code type    is_present change_location grts_address_shift grts_ranking
  <date>          <chr>      <chr>   <lgl>      <lgl>           <lgl>                     <dbl>
1 2023-08-16      101169_1   2130_hd TRUE       TRUE            FALSE                    101169
# ℹ 2 more variables: grts_ranking_draw <dbl>, is_centroid <lgl>
> mhq_terr_popunits %>%
    filter(point_code == "101169_1")
# A tibble: 1 × 9
  point_code grts_ranking grts_ranking_draw sac   legacy_site type    polygon_id    phab source             
  <chr>             <dbl>             <dbl> <lgl> <lgl>       <chr>   <fct>        <int> <chr>              
1 101169_1         101169            101169 TRUE  FALSE       2130_hd 524269_v2023    80 assessment/habitat…

Should we assume here that change_location must be FALSE, or is it is_centroid that is mistaken?

Checking expectation 2. The first row of the tibble below seems to be a problem, i.e. many points without change_location but where the GRTS address shifted:

> assessment_popunits_points %>%
    filter(grts_address_shift) %>%
    count(grts_address_shift, change_location)
# A tibble: 2 × 3
  grts_address_shift change_location     n
  <lgl>              <lgl>           <int>
1 TRUE               FALSE             249
2 TRUE               TRUE              345
> assessment_popunits_points %>%
    filter(grts_address_shift, !change_location) %>%
    count(grts_address_shift, change_location, is_centroid)
# A tibble: 2 × 4
  grts_address_shift change_location is_centroid     n
  <lgl>              <lgl>           <lgl>       <int>
1 TRUE               FALSE           FALSE         207
2 TRUE               FALSE           TRUE           42

Probably change_location should also be TRUE in these situations?

[florisvdh]

Checking expectation 2. The first row of the tibble below seems to be a problem, i.e. many points without change_location but where the GRTS address shifted:

> assessment_popunits_points %>%
    filter(grts_address_shift) %>%
    count(grts_address_shift, change_location)
# A tibble: 2 × 3
  grts_address_shift change_location     n
  <lgl>              <lgl>           <int>
1 TRUE               FALSE             249
2 TRUE               TRUE              345
> assessment_popunits_points %>%
    filter(grts_address_shift, !change_location) %>%
    count(grts_address_shift, change_location, is_centroid)
# A tibble: 2 × 4
  grts_address_shift change_location is_centroid     n
  <lgl>              <lgl>           <lgl>       <int>
1 TRUE               FALSE           FALSE         207
2 TRUE               FALSE           TRUE           42

Probably change_location should also be TRUE in these situations?

Maybe this is still within expectation for legacy sites, i.e. these might not have change_location as TRUE even while they were selected using a different grts_address_drawn? Then the problem can be narrowed down to the non-legacy sites:

> # recreate object to also contain the legacy_site attribute
> assessment_popunits_points_legacystatus <- 
    mhq_terr_assessments %>%
    select(assessment_date, point_code, type, is_present, change_location) %>%
    inner_join(
      mhq_terr_popunits %>%
        filter(str_detect(source, "assessment")) %>%
        select(point_code, grts_ranking, grts_ranking_draw, type, legacy_site),
      join_by(point_code, type),
      relationship = "many-to-one",
      unmatched = c("drop", "error")
    ) %>%
    filter(is_present) %>% 
    mutate(grts_address_shift = grts_ranking != grts_ranking_draw) %>%
    relocate(grts_address_shift, .before = grts_ranking) %>%
    inner_join(
      mhq_terr_refpoints %>%
        select(point_code, is_centroid),
      join_by(point_code),
      relationship = "many-to-one",
      unmatched = c("error", "drop")
    )
> assessment_popunits_points_legacystatus %>%
    filter(grts_address_shift, !change_location) %>%
    count(grts_address_shift, change_location, legacy_site, is_centroid)
# A tibble: 3 × 5
  grts_address_shift change_location legacy_site is_centroid     n
  <lgl>              <lgl>           <lgl>       <lgl>       <int>
1 TRUE               FALSE           FALSE       FALSE          73
2 TRUE               FALSE           FALSE       TRUE           42
3 TRUE               FALSE           TRUE        FALSE         134

This would mean that the third line is still OK; only the first two would be problematic.

Checking which types are involved:

> assessment_popunits_points_legacystatus %>%
    filter(grts_address_shift, !change_location, !legacy_site) %>% 
    count(type, sort = TRUE)
# A tibble: 19 × 2
   type            n
   <chr>       <int>
 1 2120           30
 2 6410_mo        16
 3 2130_hd        14
 4 2170            9
 5 6230_hn         8
 6 1330_hpr        7
 7 7140_meso       6
 8 2190_mp         5
 9 6230_ha         3
10 6510_hu         3
11 7140_oli        3
12 2130_had        2
13 2190_overig     2
14 6230_hmo        2
15 2330_bu         1
16 4030            1
17 6230_hnk        1
18 6510_hua        1
19 6510_huk        1