Skip to main content

Observation Testing

Carlos Camacho

Observation Testing lets BECI administrators test the NSPEC observation generation AI model. You can load real observations from the NSPEC database, generate AI-powered equivalents, and compare the output against the original human-written text — with detailed evaluation metrics and full pipeline visibility.

BECI administrators only. This feature is only visible to users with BECI admin access. If you don't see the Observation Testing tile on the Home screen, contact your administrator.

Accessing Observation Testing

  1. From the Home screen, click the Observation Testing tile
  2. The page opens with three tabs: Select, Compare, and History

Select Tab — Loading Test Data

Use the Select tab to load observations from the NSPEC database for testing.

Filter Options

Filter Options Default
Office All Offices, or select a specific office All Offices
Inspector All Inspectors, or select by name/email All Inspectors
Report Type All Report Types, or select a specific type All Report Types
Sample Size 1–100 10
Start Date Date picker 30 days ago
End Date Date picker Today
Only observations with images Checkbox Enabled

Loading Observations

  1. Set your desired filters
  2. Click Fetch Test Data
  3. A table of matching observations appears, showing a text preview, timestamps, and image caption count
  4. Click any observation row to select it — you'll be taken to the Compare tab automatically

Use Clear Filters to reset all filters to their defaults.

Compare Tab — Generating and Evaluating

The Compare tab is where the AI generates an observation and you can evaluate the quality against the original.

Generating an Observation

  1. The selected observation's details are shown: original text, inspection ID, app user, office, and report type
  2. If the observation has image captions, they are displayed with position numbers
  3. Select the NSPEC Environment:
    • Dev — uses dev.nspec.nxtconstruction.ai for context lookup
    • Prod (default) — uses nspec.nxtconstruction.ai
  4. Click Generate Observation
  5. The AI generates the observation text and displays it alongside the original

Pipeline Debug Info

After generation, click the pipeline debug toggle to see the full execution breakdown. The pipeline shows 9 steps, each with a status icon (success/skipped/error), name, and duration:

  1. Parse Request
  2. NSPEC Context
  3. Fetch Images
  4. RAG Retrieval
  5. Condition Classification
  6. Style Examples
  7. Build System Prompt
  8. Build User Message
  9. Bedrock Generation

Each step is expandable for detailed diagnostic information.

Evaluation Metrics

After generation, the system automatically evaluates the generated text against the original. Metrics are shown in card layout:

Metric What It Measures
Similarity Cosine similarity between original and generated text, displayed as a percentage
Length Word and sentence counts for original vs generated, with a ratio comparison
Format Standards references found, sentence compliance, passive voice detection, pattern analysis
AI Quality Scores (0–100%) for Technical Accuracy, Professional Tone, Completeness, Clarity, and Standards References

History Tab — Viewing Past Executions

The History tab shows all past observation generation executions.

Filters

  • Office, Inspector, Report Type — same dropdowns as the Select tab
  • Source — filter by "Testing Page" or "API" to distinguish manual tests from automated runs
  • Search — search by message ID or inspection ID

Execution Records

The table shows: Execution ID, Status (badge), Office, Inspector, Report Type, Duration (ms), and Timestamp. Status values include:

  • queued — waiting to execute
  • training — in progress
  • evaluating — running quality evaluation
  • review_needed — completed but needs human review
  • deployed — approved and deployed
  • failed — execution error

Click any row to expand it and see the full prompts (system prompt, user message), arguments (inspection ID, input text, RAG proposal count), results (output text, sentence count), and execution time breakdown.

Troubleshooting

I don't see the Observation Testing tile

Solutions:

  • This feature requires BECI admin access. Contact your administrator if you believe you should have access.

Fetch Test Data returns no results

Solutions:

  • Widen your date range (try the last 90 days instead of 30)
  • Remove the Office/Inspector/Report Type filters to broaden the search
  • Uncheck "Only observations with images" if image-based observations are rare in your dataset

Generation fails or times out

Solutions:

  • Try switching the NSPEC Environment (Dev vs Prod) — the other environment may have the context data available
  • Check the pipeline debug info to identify which step failed
  • If RAG Retrieval fails, the document corpus may not be indexed for that inspection's report type

Need help? Contact our support team at support@nxtconstruction.ai

Comments

0 comments

Please sign in to leave a comment.