Feature Request: Log Pattern Detection & Drilldown UX for VictoriaLogs Grafana Datasource

[JeelDasvani]

Current Limitation in LogsQL

VictoriaLogs currently supports:

• stats by (_msg) for grouping identical logs
• pattern_match() for filtering known patterns

However, there is no native LogsQL function to automatically generate a structural template/fingerprint from log messages by removing high-cardinality values (IDs, IPs, UUIDs, timestamps, etc.).

Because of this:

• Each dynamic log line becomes its own group
• True pattern clustering is not possible
• Datasource UI cannot implement pattern exploration reliably

Similar functionality exists in tools like Loki and Datadog, but currently requires manual workarounds or external preprocessing when using VictoriaLogs.

Backend Issue: See VictoriaMetrics/VictoriaLogs#1082 for the pattern() function request in the VictoriaLogs core.

---

Problem Context

VictoriaLogs backend already provides powerful log aggregation via LogsQL. However, it currently lacks a native way to automatically group logs by structural pattern.

This makes pattern discovery difficult for most users and limits observability use cases like:

1. Detecting common log patterns automatically
2. Grouping similar logs by structure
3. Click-to-drilldown into matching logs
4. Building dashboards for error patterns without manual LogsQL

Comparison: Grafana Loki provides a "Log Patterns" tab with one-click grouping and drilldown. VictoriaLogs is capable of similar functionality once the core primitive exists.

---

Proposed Solution: pattern() Aggregation Function

Would adding a LogsQL pattern() aggregation function be a reasonable direction?

Proposed Syntax

stats count() by pattern(_msg)

Where pattern(_msg) returns a normalized template by replacing high-cardinality tokens with placeholders.

The goal is not perfect NLP-style templating, but a fast heuristic that replaces high-cardinality tokens (numbers, UUIDs, IPs, hashes) with placeholders so similar logs can be grouped efficiently.

Example

Input log:

User 4829 failed login from 192.168.1.1

Output pattern:

User <N> failed login from <IP>

This would enable queries like:

_time:1h | stats count() by pattern(_msg) | sort by count desc

Returning a table of discovered patterns with counts.

---

Datasource UX Improvements (Once Backend Primitive Exists)

If pattern() becomes available, the datasource plugin could provide:

1. "Show Patterns" Button in Query Editor

Auto-transforms the current query into a pattern aggregation query.

2. Click Pattern → Drilldown Logs

When a user clicks a pattern row, auto-generate a filtered query to show matching logs.

3. Pattern Tab in Explore View

Add an optional tab next to "Logs": Logs | Patterns

Showing:

• Top patterns
• Error patterns
• Pattern counts
• First/last seen

4. Dashboard Support

Allow pattern queries to be used in dashboard panels for:

• Top error patterns
• Most frequent log structures
• New patterns (last 1h)

---

Real-World Use Cases

1. Identify top production errors by pattern
2. Detect new/unseen error patterns after deployment
3. Build dashboards showing most frequent failures
4. Reduce log noise by grouping similar messages
5. Enable click-to-drilldown debugging workflow

This capability would significantly improve VictoriaLogs adoption for teams evaluating alternatives to Loki or Datadog for high-volume production observability.

---

Technical Notes

• Backend: Would require new pattern() function in LogsQL (see VictoriaLogs#1082)
• Datasource: Query transformation logic, pattern table rendering, drilldown query generation
• No breaking changes: This would be additive functionality
• Performance: The function could use lightweight heuristics similar to existing log clustering approaches (tokenization + placeholder substitution) while preserving VictoriaLogs performance characteristics

---

Questions for Discussion

1. Is a pattern() aggregation function something that fits the LogsQL design philosophy?
2. Are there alternative approaches being considered for log pattern clustering?
3. Would this be better suited for the VictoriaLogs core repo instead?

---

Happy to help test UX or contribute to the datasource side if the backend primitive becomes available.