Analytics Dashboard
Location: apps/analytics_dashboard/
Tech Stack: Streamlit, Plotly, Pandas
Purpose: Monitor RAG performance, costs, and quality metrics in real-time
🎯 Overview
The Analytics Dashboard is a Streamlit-based web application for monitoring your RecoAgent deployment. It provides real-time visualizations of performance metrics, cost tracking, and quality scorecards.
Use this when:
- Monitoring production systems
- Tracking costs and usage
- Analyzing quality trends
- Debugging performance issues
- Reporting to stakeholders
⚡ Quick Start
# Navigate to dashboard directory
cd apps/analytics_dashboard
# Install dependencies
pip install -r requirements.txt
# Set database connection
export DATABASE_URL="postgresql://..."
# Run dashboard
streamlit run app.py
# Open browser: http://localhost:8501
📊 Dashboard Features
1. Real-Time Metrics
What You See:
- Requests per minute/hour/day
- Average latency trends
- Error rates
- Active sessions
Filters:
- Time range (last hour, day, week, month)
- User segments
- Query types
- Endpoints
2. Cost Tracking
Metrics Tracked:
- Total spend by day/week/month
- Cost per query
- LLM token usage
- Embedding costs
Visualizations:
- Cost trends over time
- Cost breakdown by model
- Budget alerts
- ROI calculations
3. Quality Scorecards
RAGAS Metrics:
- Context Precision
- Context Recall
- Faithfulness
- Answer Relevancy
Trends:
- Quality over time
- Performance by query type
- User satisfaction scores
4. Usage Analytics
User Behavior:
- Most common queries
- Peak usage times
- Session duration
- User retention
System Health:
- Cache hit rates
- Database query times
- API response times
📈 Screenshot Tour
┌─────────────────────────────────────────────────────────┐
│ RecoAgent Analytics Dashboard │
├─────────────────────────────────────────────────────────┤
│ │
│ ⏰ Time Range: Last 7 Days [Refresh] [Export] │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Requests │ │ Latency │ │ Cost │ │
│ │ 12,345 │ │ 850ms │ │ $123.45 │ │
│ │ ↑ 15% │ │ ↓ 5% │ │ ↑ 10% │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Requests Over Time │ │
│ │ [Interactive Line Chart] │ │
│ └────────────────────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Quality Metrics (RAGAS) │ │
│ │ Context Precision: 0.82 ✅ │ │
│ │ Context Recall: 0.75 ✅ │ │
│ │ Faithfulness: 0.88 ✅ │ │
│ │ Answer Relevancy: 0.85 ✅ │ │
│ └────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
⚙️ Configuration
Create .env file:
# Database Connection (metrics source)
DATABASE_URL=postgresql://user:pass@localhost/recoagent
# LangSmith (for tracing data)
LANGSMITH_API_KEY=your-key
LANGSMITH_PROJECT=recoagent-prod
# Dashboard Settings
DASHBOARD_PORT=8501
REFRESH_INTERVAL_SECONDS=60
MAX_DATA_POINTS=10000
# Alert Thresholds
COST_ALERT_THRESHOLD=100
LATENCY_ALERT_THRESHOLD=2000
ERROR_RATE_ALERT_THRESHOLD=0.05
🚀 Features in Detail
Cost Analysis
Daily Cost Breakdown:
Date | Requests | LLM Cost | Embed Cost | Total |
-------------|----------|----------|------------|--------|
2024-10-09 | 1,234 | $12.34 | $1.23 | $13.57 |
2024-10-08 | 1,456 | $14.56 | $1.46 | $16.02 |
2024-10-07 | 987 | $9.87 | $0.99 | $10.86 |
Cost Optimization Recommendations:
- "Switch to GPT-3.5 for simple queries to save 70%"
- "Enable caching to reduce costs by 40%"
- "Reduce chunk overlap to save on embedding costs"
Performance Monitoring
Latency Distribution:
- P50: 650ms
- P90: 1,200ms
- P99: 2,500ms
Slowest Queries:
- Query: "Explain hybrid search..."
- Latency: 3.2s
- Reason: Large document set
Quality Tracking
Quality Trends:
Week | Precision | Recall | Faithfulness | Overall |
----------|-----------|--------|--------------|---------|
Week 40 | 0.82 | 0.75 | 0.88 | 0.82 |
Week 39 | 0.79 | 0.73 | 0.85 | 0.79 |
Week 38 | 0.77 | 0.71 | 0.83 | 0.77 |
Improvement: +6.5% over 3 weeks
📤 Exporting Data
CSV Export
# Export metrics to CSV
st.download_button(
"Download CSV",
data=df.to_csv(),
file_name="recoagent_metrics.csv"
)
PDF Reports
# Generate PDF report
generate_pdf_report(
time_range="last_7_days",
metrics=["cost", "quality", "performance"]
)
🔔 Alerting
Slack Integration
# Send alerts to Slack
if daily_cost > threshold:
send_slack_alert(
channel="#recoagent-alerts",
message=f"Daily cost ${daily_cost} exceeds threshold ${threshold}"
)
Email Reports
# Send daily email summaries
schedule.every().day.at("09:00").do(
send_email_report,
recipients=["team@company.com"]
)
🎨 Customization
Add Custom Metrics
# custom_metrics.py
def calculate_custom_metric(df):
"""Calculate your custom metric"""
return df.groupby('user_id')['cost'].sum()
# Display in dashboard
st.metric(
"Cost per User",
f"${calculate_custom_metric(df):.2f}"
)
Custom Visualizations
import plotly.express as px
# Create custom chart
fig = px.scatter(
df,
x='latency',
y='cost',
color='quality_score',
title='Cost vs Latency'
)
st.plotly_chart(fig)
📱 Mobile Access
The dashboard is mobile-responsive! Access on phone/tablet for on-the-go monitoring.
🔗 Integration
With Production API
Dashboard automatically pulls data from:
- PostgreSQL metrics database
- LangSmith traces
- Prometheus metrics
With Other Tools
Export data to:
- Grafana
- Datadog
- Custom BI tools
🆘 Troubleshooting
| Issue | Solution |
|---|---|
| Dashboard not loading | Check Streamlit port 8501 is free |
| No data showing | Verify DATABASE_URL connection |
| Slow refresh | Reduce MAX_DATA_POINTS or time range |
| Missing metrics | Check LangSmith integration |
📚 Related Docs
Start monitoring now: Run streamlit run app.py and open http://localhost:8501! 📊