Skip to main content

Admin Chat and Message Analysis

Carlos Camacho
Updated

Admin Chat and Message Analysis give administrators deep visibility into how the AI processes each message. Admin Chat works the same way as the regular Chat Interface but with additional diagnostic tools. Message Analysis lets you drill into any individual message to see the full execution timeline, token breakdown, cache performance, and model reasoning.

Tenant admin access required. Admin Chat requires tenant admin permissions. Message Analysis also requires tenant admin access.

Admin Chat

Admin Chat is identical to the regular Chat Interface with one key addition: a View Analysis button on each AI response. This button opens Message Analysis for that specific message, showing you exactly what the AI saw and did.

Admin Chat also provides:

  • Access to both My Documents and Document Library libraries
  • Full conversation search across all tenant conversations
  • All the same tools, Extended Thinking, and preferences as regular chat

Message Analysis

Message Analysis is an LLM observability dashboard that shows everything the AI model saw and did during message processing.

How to Access

  • From Admin Chat — click View Analysis on any AI response
  • Direct URL — navigate to Message Analysis and enter the partition key (pk), sort key (sk), or message ID

Query Form

If accessing directly, enter at least one of:

  • Partition Key (pk) — format: USER#...
  • Sort Key (sk) — format: THREAD#...#MESSAGE#...
  • Message ID — format: msg-...

Click Load Analysis to fetch the data.

Summary Statistics

Five KPI cards appear at the top of the analysis:

Card What It Shows
Model Which AI model processed the message (Sonnet 4.6, Opus 4.6, or Haiku 4.5)
Duration Total processing time in seconds
Total Tokens Total token count (clickable — expands the Token Breakdown section)
Cache Status Green "Hit" (tokens read from cache — 90% cost reduction), Blue "Write" (tokens written to cache), or Gray "No cache"
Steps Number of execution steps the model took

Token Breakdown

Click the Total Tokens card to expand the breakdown. It shows a two-column comparison:

Cached Content (left column, green)

Content that stays the same across messages and can be cached for cost savings:

  • Base Template — the system prompt template
  • Tenant Policies — your organization's AI policies
  • Profile Memory — user context and preferences
  • Roadmap — conversation planning data
  • Tool Definitions — definitions of available tools

If a cache hit occurred, a note shows: "Anthropic cache hit: X tokens (90% cost reduction)".

Dynamic Content (right column, orange)

Content that changes each turn and is billed at full rate:

  • Session Info — current session metadata
  • Recent Messages — last few messages in the conversation
  • Older Messages — earlier conversation history
  • Current Message — the user's latest message
  • Tool Results — data returned by tools (shown with per-tool token counts)

Execution Timeline

Below the token breakdown, the execution timeline shows a chronological list of every step the model took. Each step is expandable and shows:

  • Turn number and action type (tool_call or response)
  • Duration in milliseconds (and Time to First Token if available)
  • Tools called (if applicable)
  • Token usage per turn: input, output, cache read, cache creation
  • Stop reason
  • Response preview (truncated)

Trace ID

A Trace ID is shown for each analysis. Click the copy button to copy it — this ID can be used in CloudWatch Logs Insights for deeper backend debugging:

filter @message like 'trace-xxx'

PDF Jobs Panel

If the conversation involved PDF generation, a panel shows all PDF jobs with:

  • Status badge — color-coded (pending, processing, complete, failed)
  • Mode — Generated or Refined
  • Document title and version
  • Page count and progress
  • Processing timeline — Created, Build, Design, Render, Validation phases with duration for each
  • Metadata — Job ID, Document Type, File Size, S3 location

Troubleshooting

Load Analysis returns no data

Solutions:

  • Make sure at least one field (pk, sk, or message_id) is filled in
  • Verify the IDs are correct — you can get them from the Admin Chat View Analysis button
  • Very old messages may have expired from the analysis store

Cache Status shows "No cache"

Solutions:

  • Cache hits depend on conversation context staying consistent
  • The first message in a new conversation always shows "Write" (creating cache)
  • Subsequent messages in the same conversation should show "Hit"

Need help? Contact our support team at support@nxtconstruction.ai

Comments

0 comments

Please sign in to leave a comment.