#764 - Support cursor-based pagination from NetBox 4.6

[arthanson]

Fixes: #764

Adds opt-in cursor-based pagination for .all() and .filter(). NetBox 4.6 introduced cursor pagination as an alternative to offset pagination: instead of skipping a growing number of rows with offset, the server pages forward using the primary key as a cursor, which performs significantly better on very large result sets.

Usage

nb = pynetbox.api(url, token=token, pagination="cursor")
devices = nb.dcim.devices.all()  # pages via the server's `start` cursor, following `next` links

pagination defaults to "offset", so existing behavior is unchanged.

Below is a script that shows the new pagination, will need to update the token.

Manual verification script

#!/usr/bin/env python3
"""Manual test for cursor-based pagination (NetBox 4.6+, PR #787).

Exercises the cursor pagination path of ``.all()`` / ``.filter()`` against a
live NetBox instance and verifies the review-comment fixes:

1. ``pagination="cursor"`` walks every page by following the server's
   ``next`` links and returns the same objects as offset pagination.
2. ``len()`` on a cursor RecordSet refetches the count NetBox omits (null)
   in cursor mode.
3. ``threading=True`` + ``pagination="cursor"`` emits a UserWarning because
   cursor pages are fetched sequentially (the threading config is a no-op).
4. A transport-level failure in the version probe falls back to offset
   instead of propagating (api.py exception handling).

Requires a NetBox 4.6+ instance whose ``dcim.devices`` endpoint has more than
one page of results (the script forces a small page size to guarantee this).

Usage: python manual_tests/test_cursor_pagination.py
"""

import os
import sys
import warnings
from unittest.mock import patch

import requests

sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

import pynetbox
from packaging import version

NETBOX_URL = os.environ.get("NETBOX_URL", "http://127.0.0.1:8000/")
NETBOX_TOKEN = "nbt_cUOdzNJvWx3q.v1YQC2Q1smP0PyN4soKGucD3BrbYZcu1qyztXu2c"

# Small page size so the device list spans multiple cursor pages and the
# next-link following is actually exercised.
PAGE_SIZE = 2


def test_version_and_effective_pagination():
  print("1. server version + effective pagination resolution")
  api = pynetbox.api(NETBOX_URL, token=NETBOX_TOKEN, pagination="cursor")
  server_version = api.version
  print(f"   NetBox {server_version} at {NETBOX_URL}")
  if version.parse(server_version) < version.parse("4.6"):
      raise SystemExit(
          f"   NetBox {server_version} does not support cursor pagination "
          "(need 4.6+); aborting."
      )
  effective = api._effective_pagination()
  assert effective == "cursor", f"expected 'cursor', got {effective!r}"
  print("   _effective_pagination() == 'cursor'")
  return api


def test_cursor_matches_offset(cursor_api):
  print("2. cursor pagination returns the same devices as offset")
  offset_api = pynetbox.api(NETBOX_URL, token=NETBOX_TOKEN, pagination="offset")

  offset_devices = list(offset_api.dcim.devices.all())
  cursor_devices = list(cursor_api.dcim.devices.filter(limit=PAGE_SIZE))

  print(f"   offset returned {len(offset_devices)} device(s)")
  print(f"   cursor returned {len(cursor_devices)} device(s) (page size {PAGE_SIZE})")

  assert len(cursor_devices) > PAGE_SIZE, (
      f"need more than one page of devices to test pagination; got "
      f"{len(cursor_devices)} with page size {PAGE_SIZE}"
  )

  offset_ids = sorted(d.id for d in offset_devices)
  cursor_ids = sorted(d.id for d in cursor_devices)
  assert offset_ids == cursor_ids, (
      "cursor and offset returned different device id sets:\n"
      f"   only in offset: {set(offset_ids) - set(cursor_ids)}\n"
      f"   only in cursor: {set(cursor_ids) - set(offset_ids)}"
  )
  print(f"   both strategies returned the same {len(cursor_ids)} device id(s)")


def test_cursor_len_refetches_count(cursor_api):
  print("3. len() on a cursor RecordSet refetches the omitted count")
  record_set = cursor_api.dcim.devices.filter(limit=PAGE_SIZE)
  count = len(record_set)
  print(f"   len(filter(limit={PAGE_SIZE})) == {count}")
  assert count > PAGE_SIZE, f"expected count > {PAGE_SIZE}, got {count}"


def test_threading_cursor_warns():
  print("4. threading=True + cursor emits a no-op warning")
  api = pynetbox.api(
      NETBOX_URL, token=NETBOX_TOKEN, pagination="cursor", threading=True
  )
  with warnings.catch_warnings(record=True) as caught:
      warnings.simplefilter("always")
      effective = api._effective_pagination()
  assert effective == "cursor", f"expected 'cursor', got {effective!r}"
  messages = [str(w.message) for w in caught if issubclass(w.category, UserWarning)]
  assert any("threading" in m for m in messages), (
      f"expected a threading no-op UserWarning, got: {messages}"
  )
  print(f"   warned: {messages[0]}")


def test_version_probe_network_error_falls_back():
  print("5. transport error in version probe falls back to offset")
  api = pynetbox.api(NETBOX_URL, token=NETBOX_TOKEN, pagination="cursor")
  with patch(
      "requests.sessions.Session.get",
      side_effect=requests.exceptions.ConnectionError("boom"),
  ):
      effective = api._effective_pagination()
  assert effective == "offset", f"expected 'offset', got {effective!r}"
  print("   ConnectionError during probe -> _effective_pagination() == 'offset'")


def main():
  api = test_version_and_effective_pagination()
  test_cursor_matches_offset(api)
  test_cursor_len_refetches_count(api)
  test_threading_cursor_warns()
  test_version_probe_network_error_falls_back()
  print("PASS")


if __name__ == "__main__":
  sys.exit(main())

[arthanson]

@claude /review Only report on issues of potential concern; we don't need commentary on anything that doesn't require further consideration.

[github-actions]

Claude finished @arthanson's task in 2m 23s —— View job

---

Code Review

• Read changed files
• Analyze for potential issues
• Post findings

---

Three concerns worth addressing:

1. Network exceptions not caught in version probe (api.py:219-224)

_effective_pagination() only catches RequestError and InvalidVersion, but self.version calls requests.Session.get(), which raises requests.exceptions.ConnectionError, requests.exceptions.Timeout, etc. on network failure — not RequestError. If the version probe fails at the transport level, the exception propagates instead of falling back to offset as the docstring promises.

# api.py ~line 219
try:
    self._cursor_supported = version.parse(self.version) >= version.parse("4.6")
except (RequestError, InvalidVersion):   # ← misses ConnectionError, Timeout, etc.
    self._cursor_supported = False

Suggested fix: catch requests.exceptions.RequestException (the base class for all requests transport errors) alongside the existing exceptions. Fix this →

---

2. Silent misconfiguration: threading=True + pagination="cursor" is not warned (api.py, query.py)

The docs correctly note cursor pagination can't be parallelised, but passing both threading=True and pagination="cursor" is silently accepted. At runtime the cursor path in get() simply ignores self.threading, so the user gets sequential results with no indication their threading configuration is a no-op.

A warnings.warn() at __init__ time (or inside _effective_pagination() when cursor is confirmed and threading is True) would surface the misconfiguration immediately instead of leaving it as a hard-to-debug performance surprise. Fix this →

---

3. No integration tests for cursor pagination

The feature is entirely tested against mock responses. An integration test against a live NetBox 4.6 instance would catch if the actual start parameter semantics, null count field, or next link format differ from what was mocked. This is especially worth noting because cursor pagination is a new NetBox 4.6 feature and the implementation makes non-obvious assumptions (start=0 for the first page, count being null not absent, next always being the raw URL to follow).

Even a single integration test in tests/integration/ gated behind a version check (skip unless NetBox >= 4.6) would provide a meaningful regression target once 4.6 lands in the CI matrix.

---

No concerns with the overall design, the offset fallback logic, the get_count() lazy-fetch approach, or the test coverage for mocked scenarios.

[arthanson]

claude review issues fixed.

[jeremystretch]

@claude /review Only report on issues of potential concern; we don't need commentary on anything that doesn't require further consideration.

[github-actions]

Claude finished @jeremystretch's task in 3m 4s —— View job

---

Code Review

• Read changed files
• Analyze for potential issues
• Post findings

---

Two remaining concerns:

1. warnings.warn stacklevel points into library internals (api.py:258)

The call chain when the warning fires is: user code → Endpoint.all() → _effective_pagination() → warnings.warn(..., stacklevel=2). With stacklevel=2, Python attributes the warning to the line inside endpoint.py where _effective_pagination() is called — not to the user's code. The user's traceback will show a pynetbox internal, which makes it harder to find the misconfigured call site.

stacklevel=3 would point to the user's .all() or .filter() call, which is the actionable location.

# api.py:258 — should be stacklevel=3
warnings.warn(
    "threading=True has no effect with cursor pagination; ...",
    stacklevel=2,   # ← points to endpoint.py internals
)

Fix this →

---

2. No runtime enforcement of ordering + cursor incompatibility

The docs explicitly state: "an explicit ordering filter cannot be combined with cursor pagination." However there is no runtime warning or error when ordering is passed to .filter() with cursor mode active. The request is silently forwarded to NetBox with both start=0 and ordering=<value> set — behavior that is server-defined and may produce duplicate pages, missing records, or an API error that surfaces as a RequestError rather than a clear misconfiguration message.

Since pynetbox already validates parameters in strict-filter mode, a targeted warnings.warn (or even just a check and raise in filter()) when ordering is present and cursor pagination is in effect would make this footgun explicit before the network round-trip.

Fix this →

---

The three issues flagged in the previous review (transport exceptions in version probe, threading no-op warning, and integration tests) have all been addressed.

[jeremystretch]

Approved with two minor suggestions.

[arthanson]

Added the two changes, so needs re-review.