fix(geometry): validate rotation matrix in Rot3 constructor

[jashshah999]

Summary

Fixes #2300

Passing a non-rotation matrix to Rot3 (e.g., all zeros, non-orthogonal, or det != 1) causes undefined behavior and segfaults in the Python bindings.

Added a validity check in the matrix constructor that throws std::invalid_argument when the input is not a valid rotation matrix (checks orthogonality and determinant with tolerance 1e-9).

Also adds a public Rot3::IsValid(const Matrix3&, double tol) static method that users can call to check validity before constructing.

Test plan

• Invalid matrix input now raises RuntimeError in Python instead of segfaulting
• Valid rotation matrices still construct without error
• Rot3::IsValid can be used independently to check matrices

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds explicit rotation-matrix validation to Rot3 construction to prevent undefined behavior/segfaults (notably in Python bindings) when given invalid 3×3 matrices.

Changes:

• Validate input matrices in Rot3 matrix constructors and throw std::invalid_argument on invalid rotations.
• Introduce Rot3::IsValid(const Matrix3&, double tol) as a public static helper.
• Implement orthogonality + determinant checks in Rot3.cpp.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
gtsam/geometry/Rot3.h	Adds input validation + new IsValid API and documents it.
gtsam/geometry/Rot3.cpp	Implements Rot3::IsValid with orthogonality and determinant checks.

[dellaert]

Please check co-pilot comments. I'll run CI if there is another commit.

[jashshah999]

Addressed all 4 Copilot comments (replied to each):

1. Quaternion init order fixed (identity first, validate, then assign)
2. #include <cmath> already present
3. Extracted shared throwIfInvalid() helper
4. Constructor uses relaxed 1e-6 tolerance via kConstructorTol

All in commit 7c8623b.

[dellaert]

Pls check CI issues

[ProfFan]

It does not make sense to check for validity in the constructor, this adds tons of overhead

[dellaert]

It does not make sense to check for validity in the constructor, this adds tons of overhead

I tend to agree but I'm also a supporter of having more stringent error checking from Python. How could we have our cake and eat it too? In C++ we don't want this overhead.

[jashshah999]

Good point on overhead. A few options:

1. Wrap in #ifndef NDEBUG so it only runs in debug builds
2. Add a static Rot3::Checked(const Matrix3&) factory that validates — keeps the constructor fast but gives a safe path for users who want it
3. Only validate in the pybind wrapper layer (since the original issue Invalid input to gtsam.Rot3 causes segfault in python wrapper. #2300 was a segfault from Python)

Option 3 is probably the best tradeoff — zero overhead in C++, safe from Python. Happy to rework the PR in whichever direction you prefer.

[dellaert]

Option 3 seems the best, but I have no idea how to do it :-)