-
Notifications
You must be signed in to change notification settings - Fork 140
feat(otel): set ate.* actor telemetry identity on ateapi spans #412
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 2 commits
5ba9da5
5230263
8067f63
692d7ac
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
|
zoez7 marked this conversation as resolved.
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,6 +30,7 @@ func (s *Service) PauseActor(ctx context.Context, req *ateapipb.PauseActorReques | |
| if err := validatePauseActorRequest(req); err != nil { | ||
| return nil, err | ||
| } | ||
| setSpanActorRefIdentity(ctx, req.GetActor().GetAtespace(), req.GetActor().GetName()) | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. out of curiosity, what will be happened if instead of setSpanActorRefIdentity, the "setSpanActorIdentity" will be called.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We can't call the full one early. At line 33 we only have the Attributes overwrite by key, so the later wins and we get the latest version on success. On error we return early with only the ref, so no version. I guess we could also add the version on mid-workflow failures if we move it inside the workflow. Maybe that's better as a follow-up if it's worth it. What do you think?
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd keep it as Happy to add a line to the attr doc spelling out "resulting version" or similar if that makes it better? |
||
|
|
||
| actor, err := s.actorWorkflow.PauseActor(ctx, req.GetActor().GetAtespace(), req.GetActor().GetName()) | ||
| if err != nil { | ||
|
|
@@ -42,6 +43,7 @@ func (s *Service) PauseActor(ctx context.Context, req *ateapipb.PauseActorReques | |
| return nil, err | ||
| } | ||
|
|
||
| setSpanActorIdentity(ctx, actor) | ||
| return &ateapipb.PauseActorResponse{Actor: actor}, nil | ||
| } | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| // Copyright 2026 Google LLC | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package controlapi | ||
|
|
||
| import ( | ||
| "context" | ||
|
|
||
| "go.opentelemetry.io/otel/trace" | ||
|
|
||
| "github.com/agent-substrate/substrate/internal/ateattr" | ||
| "github.com/agent-substrate/substrate/pkg/proto/ateapipb" | ||
| ) | ||
|
|
||
| // setSpanActorIdentity annotates the RPC's server span (from ctx) with the | ||
| // actor's full identity. A no-op when ctx carries no recording span. | ||
| func setSpanActorIdentity(ctx context.Context, a *ateapipb.Actor) { | ||
| trace.SpanFromContext(ctx).SetAttributes(ateattr.ActorIdentity(a)...) | ||
| } | ||
|
|
||
| // setSpanActorRefIdentity is setSpanActorIdentity for the identity subset known | ||
| // before the Actor record resolves, so a failed lookup still carries who/where. | ||
| func setSpanActorRefIdentity(ctx context.Context, atespace, actorID string) { | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. the parameter name is
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fair, but this turned out to be a bit deeper when started looking into it. |
||
| trace.SpanFromContext(ctx).SetAttributes(ateattr.ActorRefIdentity(atespace, actorID)...) | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Similarly here, can we rename
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I went with |
||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,149 @@ | ||
| // Copyright 2026 Google LLC | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package controlapi | ||
|
|
||
| import ( | ||
| "context" | ||
| "testing" | ||
|
|
||
| "go.opentelemetry.io/otel" | ||
| "go.opentelemetry.io/otel/attribute" | ||
| sdktrace "go.opentelemetry.io/otel/sdk/trace" | ||
| "go.opentelemetry.io/otel/sdk/trace/tracetest" | ||
|
|
||
| "github.com/agent-substrate/substrate/internal/ateattr" | ||
| "github.com/agent-substrate/substrate/pkg/proto/ateapipb" | ||
| ) | ||
|
|
||
| // The functional harness registers no otelgrpc StatsHandler, so there is no | ||
| // server span on the client path. These tests instead call the Service methods | ||
| // in-process under a self-provided recording root span, which stands in for the | ||
| // span the otelgrpc handler injects in production and exercises the same | ||
| // trace.SpanFromContext(ctx).SetAttributes path. | ||
| func installSpanRecorder(t *testing.T) *tracetest.SpanRecorder { | ||
| t.Helper() | ||
| sr := tracetest.NewSpanRecorder() | ||
| tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr)) | ||
| prev := otel.GetTracerProvider() | ||
| otel.SetTracerProvider(tp) | ||
| t.Cleanup(func() { otel.SetTracerProvider(prev) }) | ||
| return sr | ||
| } | ||
|
|
||
| func rootSpanAttrs(t *testing.T, sr *tracetest.SpanRecorder, fn func(ctx context.Context)) map[attribute.Key]attribute.Value { | ||
| t.Helper() | ||
| ctx, root := otel.Tracer("test").Start(context.Background(), "root") | ||
| fn(ctx) | ||
| root.End() | ||
| for _, s := range sr.Ended() { | ||
| if s.Name() == "root" { | ||
| m := make(map[attribute.Key]attribute.Value, len(s.Attributes())) | ||
| for _, kv := range s.Attributes() { | ||
| m[kv.Key] = kv.Value | ||
| } | ||
| return m | ||
| } | ||
| } | ||
| t.Fatal("root span not recorded") | ||
| return nil | ||
| } | ||
|
|
||
| func assertSpanStr(t *testing.T, attrs map[attribute.Key]attribute.Value, key attribute.Key, want string) { | ||
| t.Helper() | ||
| v, ok := attrs[key] | ||
| if !ok { | ||
| t.Errorf("missing %s", key) | ||
| return | ||
| } | ||
| if v.AsString() != want { | ||
| t.Errorf("%s = %q, want %q", key, v.AsString(), want) | ||
| } | ||
| } | ||
|
|
||
| func TestCreateActor_StampsFullSpanIdentity(t *testing.T) { | ||
|
krisztianfekete marked this conversation as resolved.
Outdated
|
||
| ns := namespaceForTest("ns-span-create") | ||
| tc := setupTest(t, ns) | ||
| defer tc.cleanup() | ||
| createTemplate(t, tc, ns) | ||
|
|
||
| sr := installSpanRecorder(t) | ||
| attrs := rootSpanAttrs(t, sr, func(ctx context.Context) { | ||
| if _, err := tc.service.CreateActor(ctx, &ateapipb.CreateActorRequest{ | ||
| Actor: &ateapipb.Actor{ | ||
| Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "id1"}, | ||
| ActorTemplateNamespace: ns, | ||
| ActorTemplateName: "tmpl1", | ||
| }, | ||
| }); err != nil { | ||
| t.Fatalf("CreateActor: %v", err) | ||
| } | ||
| }) | ||
|
|
||
| assertSpanStr(t, attrs, ateattr.AtespaceKey, testAtespace) | ||
| assertSpanStr(t, attrs, ateattr.ActorIDKey, "id1") | ||
| assertSpanStr(t, attrs, ateattr.ActorTemplateNameKey, "tmpl1") | ||
| assertSpanStr(t, attrs, ateattr.ActorTemplateNamespaceKey, ns) | ||
| if v, ok := attrs[ateattr.ActorVersionKey]; !ok || v.Type() != attribute.INT64 || v.AsInt64() != 1 { | ||
| t.Errorf("%s = %v, want int64 1", ateattr.ActorVersionKey, v.Emit()) | ||
| } | ||
| } | ||
|
|
||
| func TestDeleteActor_StampsRefSpanIdentity(t *testing.T) { | ||
| ns := namespaceForTest("ns-span-delete") | ||
| tc := setupTest(t, ns) | ||
| defer tc.cleanup() | ||
| createTemplate(t, tc, ns) | ||
| if _, err := tc.service.CreateActor(context.Background(), &ateapipb.CreateActorRequest{ | ||
| Actor: &ateapipb.Actor{ | ||
| Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "id1"}, | ||
| ActorTemplateNamespace: ns, | ||
| ActorTemplateName: "tmpl1", | ||
| }, | ||
| }); err != nil { | ||
| t.Fatalf("seed CreateActor: %v", err) | ||
| } | ||
|
|
||
| sr := installSpanRecorder(t) | ||
| attrs := rootSpanAttrs(t, sr, func(ctx context.Context) { | ||
| if _, err := tc.service.DeleteActor(ctx, &ateapipb.DeleteActorRequest{ | ||
| Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "id1"}, | ||
| }); err != nil { | ||
| t.Fatalf("DeleteActor: %v", err) | ||
| } | ||
| }) | ||
|
|
||
| assertSpanStr(t, attrs, ateattr.AtespaceKey, testAtespace) | ||
| assertSpanStr(t, attrs, ateattr.ActorIDKey, "id1") | ||
| } | ||
|
|
||
| // The early ref stamp must land on the span even when the operation fails, so a | ||
| // failed resume is still attributable to who/where. | ||
| func TestResumeActor_ErrorStillStampsRefSpanIdentity(t *testing.T) { | ||
| ns := namespaceForTest("ns-span-resume-err") | ||
| tc := setupTest(t, ns) | ||
| defer tc.cleanup() | ||
|
|
||
| sr := installSpanRecorder(t) | ||
| attrs := rootSpanAttrs(t, sr, func(ctx context.Context) { | ||
| if _, err := tc.service.ResumeActor(ctx, &ateapipb.ResumeActorRequest{ | ||
| Actor: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "missing"}, | ||
| }); err == nil { | ||
| t.Fatal("expected error resuming missing actor") | ||
| } | ||
| }) | ||
|
|
||
| assertSpanStr(t, attrs, ateattr.AtespaceKey, testAtespace) | ||
| assertSpanStr(t, attrs, ateattr.ActorIDKey, "missing") | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| // Copyright 2026 Google LLC | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| // Package ateattr projects an Actor onto substrate's ate.* identity attributes. | ||
| // Identity is a span-level subject attribute (the producer is the substrate | ||
| // component, the actor is the subject), so it belongs on spans rather than the | ||
| // resource, and uses substrate's own ate.* namespace rather than service.*. | ||
| package ateattr | ||
|
|
||
| import ( | ||
| "go.opentelemetry.io/otel/attribute" | ||
|
|
||
| "github.com/agent-substrate/substrate/pkg/proto/ateapipb" | ||
| ) | ||
|
|
||
| // Dotted ate.* matches the metric-instrument naming (atenet.*, atelet.*), not the | ||
| // ate.dev/ slash form used for k8s labels and stdout log fields. | ||
| const ( | ||
| AtespaceKey = attribute.Key("ate.atespace") | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ate.actor.atespace?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think I prefer
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. the ActorTemplateNameKey on line 37 should be |
||
| ActorIDKey = attribute.Key("ate.actor.id") | ||
| ActorTemplateNameKey = attribute.Key("ate.actor.template.name") | ||
| ActorTemplateNamespaceKey = attribute.Key("ate.actor.template.namespace") | ||
| ActorVersionKey = attribute.Key("ate.actor.version") | ||
| ) | ||
|
|
||
| // ActorRefIdentity returns the subset knowable before the Actor record resolves. | ||
| func ActorRefIdentity(atespace, actorID string) []attribute.KeyValue { | ||
| return []attribute.KeyValue{ | ||
| AtespaceKey.String(atespace), | ||
| ActorIDKey.String(actorID), | ||
| } | ||
| } | ||
|
|
||
| // ActorIdentity is nil-safe; a nil Actor yields zero-valued attributes. | ||
| func ActorIdentity(a *ateapipb.Actor) []attribute.KeyValue { | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: I think this should be
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is now done. |
||
| return []attribute.KeyValue{ | ||
| AtespaceKey.String(a.GetMetadata().GetAtespace()), | ||
| ActorIDKey.String(a.GetMetadata().GetName()), | ||
| ActorTemplateNameKey.String(a.GetActorTemplateName()), | ||
| ActorTemplateNamespaceKey.String(a.GetActorTemplateNamespace()), | ||
| ActorVersionKey.Int64(a.GetMetadata().GetVersion()), | ||
| } | ||
| } | ||

There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: rename this to
setSpanActorAttributesto distinguish fromsetSpanActorRefIdentity?