# Symph Agency QA Protocol
**Version:** 0.1 (Draft — Pending Rona Review)
**Author:** ARIA
**Requested by:** Rona Maceda (@itsronami)
**Status:** 🟡 Draft — Do Not Implement

---

## 1. Purpose

This protocol defines the quality assurance standards, gates, artifacts, and responsibilities for all products moving through the Symph Agency pipeline. It applies to every phase from Discovery through Operate and is enforced by ARIA (automated) with human sign-off at critical gates.

**Core principle:** No phase advances without a QA gate passing. No artifact delivered without a QA review on record.

---

## 2. Roles & Responsibilities

| Role | Responsibility |
|------|---------------|
| **ARIA (Agent)** | Automated test execution, artifact generation, issue logging, gate enforcement |
| **QA Lead (Human)** | Protocol ownership, gate sign-off on P0/P1 issues, protocol updates |
| **Product Owner** | Accepts/rejects Delivery Acceptance Report |
| **Builder Agent** | Fixes issues flagged by QA before re-submitting |

---

## 3. QA Gates — Phase Overview

Each phase has a mandatory gate. Work cannot advance until the gate passes.

| Phase | Gate Name | Who Signs Off | Blocking? |
|-------|-----------|--------------|-----------|
| Discovery | PRD Quality Gate | QA Lead | ✅ Yes |
| Design | Design Testability Gate | QA Lead | ✅ Yes |
| Build | Build QA Gate | ARIA (auto) + QA Lead for P1+ issues | ✅ Yes |
| Market | Marketing Claim Gate | QA Lead | ✅ Yes |
| Sell | Sales Readiness Gate | QA Lead | ✅ Yes |
| Deliver | Delivery Acceptance Gate | Product Owner + QA Lead | ✅ Yes |
| Operate | Health Threshold Gate | ARIA (auto) | ⚠️ Alert only |

---

## 4. Phase-by-Phase QA Standards

---

### 4.1 Discovery Phase

**Trigger:** PRD submitted for build

**QA Actions:**
- Review PRD for completeness against standard checklist
- Identify requirement gaps (missing user stories, undefined edge cases)
- Flag ambiguous acceptance criteria
- Identify technical and product risks
- Verify user stories follow: "As a [role], I want [action] so that [outcome]" format

**Gate Criteria (all must pass):**
- [ ] All user stories have acceptance criteria
- [ ] No undefined technical dependencies
- [ ] Risk register completed
- [ ] Scope clearly bounded (what's IN vs OUT)
- [ ] Success metrics defined

**Required Artifact:** PRD Audit Report

---

### 4.2 Design Phase

**Trigger:** Design mockups / specs submitted

**QA Actions:**
- Review UI flows for testability (can each state be reached and verified?)
- Check component specs against brand guidelines
- Identify untestable states (e.g., loading states with no fallback)
- Verify mobile responsiveness is designed in (not an afterthought)
- Flag accessibility concerns (color contrast, touch targets)

**Gate Criteria:**
- [ ] All user story flows have corresponding UI screens
- [ ] Empty states designed for all data lists
- [ ] Error states designed for all forms
- [ ] Mobile breakpoints specified
- [ ] Brand guidelines applied (correct colors, fonts, logo)

**Required Artifact:** Design Testability Review (section in Test Specification Doc)

---

### 4.3 Build Phase

**Trigger:** Builder agent signals build complete

**QA Actions (Ralph Wiggum Loop — repeat until all pass):**

**Round 1 — Static Checks**
- `npm run build` passes with zero errors
- No mock/hardcoded data in lists or dashboards
- Firestore collection names in code match Firestore rules
- No `.env` values exposed in client-side code
- No console.log statements with sensitive data

**Round 2 — Auth Flow**
- Sign up with new email → redirect to dashboard
- Log in with existing credentials → redirect to dashboard
- Log out → redirect to login page
- Access protected route while logged out → redirect to login
- Session persists after page refresh

**Round 3 — Core CRUD**
- Create a record → appears in list
- Update a record → changes reflected immediately
- Delete a record → removed from list
- Data persists after browser refresh
- Data persists after logout → login

**Round 4 — Edge Cases & Error States**
- Submit empty required form → validation errors shown
- Submit invalid data → appropriate error shown
- Network error handling (graceful, not blank screen)
- Empty list states show (not blank screen)
- All buttons clickable and functional

**Round 5 — Final Sweep**
- Mobile view (375px) — no overflow, all elements accessible
- No console errors
- All navigation links work
- Page titles and labels are correct (no "undefined", "null", "Lorem ipsum")
- OWASP security check (no exposed secrets, CORS correct)

**Gate Criteria:**
- [ ] All 5 rounds pass
- [ ] Zero P0/P1 issues open
- [ ] `npm run build` clean
- [ ] No mock data
- [ ] Auth end-to-end works
- [ ] Data persistence verified

**Required Artifacts:**
- Test Specification Document
- Test Execution Report
- Requirements Traceability Matrix (RTM)
- Updated Defect Log

---

### 4.4 Market Phase

**Trigger:** Marketing assets or copy submitted for publish

**QA Actions:**

**Asset QA (5-Layer Pipeline):**
1. Hard Failure Detection — corrupt file, unreadable text, placeholder content
2. Brand Compliance — correct logo, correct colors, correct typography
3. Aesthetic Quality — composition, readability, no visual artifacts
4. Content Relevance — claim matches product, no factual errors
5. Platform Fit — correct dimensions, format, aspect ratio for target platform

**Claim Verification:**
- Every marketing claim must be traceable to a passing test or verified product behavior
- "Feature X does Y" → verify Feature X does Y in the live build
- Pricing claims → verify against current pricing config
- Performance claims → verify against Lighthouse or benchmark results

**Gate Criteria:**
- [ ] All assets pass 5-layer QA
- [ ] All claims traced to verified product behavior
- [ ] No banned language or patterns (no AI tells, no em dashes in copy)
- [ ] Social handles are correct

**Required Artifacts:**
- Marketing Asset QA Report (per asset)
- Marketing Claim Verification Log

---

### 4.5 Sell Phase

**Trigger:** Demo script or proposal submitted

**QA Actions:**
- Validate demo script flows against actual live product
- Verify pricing in proposal matches current pricing config
- Check feature claims in proposal against build test results
- Flag any promised features not yet built

**Gate Criteria:**
- [ ] Demo script executable end-to-end on live product
- [ ] All pricing accurate
- [ ] No unbuilt features promised
- [ ] Proposal claims backed by verified test results

**Required Artifact:** Sales Readiness Report

---

### 4.6 Deliver Phase

**Trigger:** Product deployment to production

**QA Actions:**
- Post-deploy smoke test (HTTP 200 on all critical routes)
- Auth flow works on production environment
- Environment configuration verified (correct Firebase project, correct API keys)
- No staging/dev data or references in production
- Onboarding flow walkthrough (first-time user experience)
- Rollback procedure documented and tested

**Gate Criteria:**
- [ ] All critical routes return 200
- [ ] Auth works on production
- [ ] No dev/staging config leaked to prod
- [ ] Onboarding flow completes without errors
- [ ] Rollback procedure documented

**Required Artifacts:**
- Delivery Acceptance Report
- Environment Configuration Summary
- User Manual / Onboarding Guide (if first deploy)

---

### 4.7 Operate Phase

**Trigger:** Continuous (every 2nd heartbeat + weekly report)

**QA Actions:**
- HTTP health check on all live apps
- Process health check (app running, port responding)
- Synthetic user flow check (weekly — auth + core feature)
- Incident-triggered regression (on any reported bug, re-run full test suite)
- Performance baseline monitoring (Lighthouse weekly)

**Thresholds:**
| Metric | Warning | Critical |
|--------|---------|----------|
| HTTP status | Non-200 once | Non-200 3x in a row |
| Response time | > 3s | > 10s |
| Lighthouse score | < 70 | < 50 |
| Error rate | > 1% | > 5% |

**Required Artifacts:**
- Weekly Operational Health Report
- Incident Regression Report (on each incident)

---

## 5. Required Artifacts — Templates

---

### 5.1 PRD Audit Report

```
# PRD Audit Report
Product: [name]
PRD Version: [version]
Auditor: ARIA
Date: [date]
Status: PASS / FAIL / NEEDS REVISION

## Completeness Check
- [ ] All user stories present
- [ ] Acceptance criteria defined
- [ ] Success metrics defined
- [ ] Scope bounded (in/out)
- [ ] Technical dependencies listed

## Gaps Found
| # | Gap | Severity | Recommendation |
|---|-----|----------|---------------|
| 1 | ... | P1/P2/P3 | ... |

## Risks Identified
| # | Risk | Likelihood | Impact | Mitigation |
|---|------|-----------|--------|-----------|
| 1 | ... | H/M/L | H/M/L | ... |

## Verdict
[PASS / FAIL with summary]
```

---

### 5.2 Test Specification Document

```
# Test Specification
Product: [name]
Version: [build version or commit]
Author: ARIA
Date: [date]

## Test Scope
[What is and is not being tested]

## Test Cases
| TC-ID | User Story | Test Description | Steps | Expected Result | Priority |
|-------|-----------|-----------------|-------|-----------------|---------|
| TC-001 | US-001 | User can sign up | 1. Go to /signup... | Redirected to dashboard | P1 |

## Out of Scope
[What is explicitly not tested and why]
```

---

### 5.3 Test Execution Report

```
# Test Execution Report
Product: [name]
Build: [commit hash or build ID]
Environment: dev / staging / prod
Executed by: ARIA
Date: [date]
Duration: [time]

## Summary
| Metric | Count |
|--------|-------|
| Total Test Cases | 0 |
| Passed | 0 |
| Failed | 0 |
| Skipped | 0 |
| Pass Rate | 0% |

## Failed Tests
| TC-ID | Description | Actual Result | Severity | Issue ID |
|-------|-------------|---------------|----------|---------|
| TC-003 | ... | ... | P1 | AGY-XXX |

## Build QA Gate
Status: PASS / FAIL
Blocking Issues: [count]
Recommendation: [Proceed / Block — fix issues first]
```

---

### 5.4 Requirements Traceability Matrix (RTM)

```
# Requirements Traceability Matrix
Product: [name]
Date: [date]

| Requirement ID | User Story | Test Case ID | Test Status | Issue ID (if failed) |
|----------------|-----------|--------------|-------------|---------------------|
| REQ-001 | As a user, I can sign up | TC-001 | ✅ Pass | — |
| REQ-002 | As a user, I can log in | TC-002 | ❌ Fail | AGY-045 |
```

---

### 5.5 Defect Log

```
# Defect Log
Product: [name]
Last Updated: [date]

| Defect ID | Phase | Title | Severity | Status | Reporter | Assignee | Created | Resolved |
|-----------|-------|-------|----------|--------|---------|---------|---------|---------|
| DEF-001 | Build | Login form not redirecting | P1 | Open | ARIA | Builder | date | — |
```

---

### 5.6 Marketing Claim Verification Log

```
# Marketing Claim Verification Log
Product: [name]
Asset/Copy: [file or campaign name]
Date: [date]

| Claim | Source (asset/copy) | Verified Against | Status | Notes |
|-------|-------------------|-----------------|--------|-------|
| "Sync in real-time" | Homepage hero | Build TC-012 | ✅ Verified | — |
| "Free forever plan" | Pricing page | pricing.md | ✅ Verified | — |
| "50% faster reports" | LinkedIn post | No test exists | ❌ Unverified | Needs test or remove claim |
```

---

### 5.7 Sales Readiness Report

```
# Sales Readiness Report
Product: [name]
Demo Version: [URL or build]
Date: [date]
Prepared by: ARIA

## Demo Script Validation
| Step | Script Says | Actual Behavior | Status |
|------|------------|-----------------|--------|
| 1 | Click "Create Report" | Modal opens | ✅ Pass |

## Pricing Accuracy
| Plan | Quoted Price | Config Price | Match? |
|------|-------------|-------------|--------|
| Starter | $29/mo | $29/mo | ✅ Yes |

## Feature Completeness
| Feature Promised | Built? | Verified? |
|-----------------|--------|----------|
| Export to PDF | ✅ Yes | ✅ Yes |
| Team collaboration | ❌ No | ❌ No — REMOVE FROM PITCH |

## Verdict
READY / NOT READY
[Blocking items if not ready]
```

---

### 5.8 Delivery Acceptance Report

```
# Delivery Acceptance Report
Product: [name]
Version: [commit hash]
Production URL: [url]
Deploy Date: [date]
Deployed by: ARIA

## Smoke Test Results
| Check | Status |
|-------|--------|
| Homepage loads (200) | ✅ Pass |
| Auth works (signup + login) | ✅ Pass |
| Core feature functional | ✅ Pass |
| No dev/staging config in prod | ✅ Pass |
| SSL valid | ✅ Pass |

## Environment Configuration
| Key | Value |
|-----|-------|
| Firebase Project | symph-[slug] |
| Environment | Production |
| Domain | [domain] |
| Deploy Method | Cloud Run / Caddy |

## Known Issues at Launch
| Issue ID | Description | Severity | ETA |
|---------|-------------|----------|-----|
| — | None | — | — |

## Rollback Procedure
[Step-by-step rollback instructions]

## Acceptance
- [ ] Product Owner sign-off
- [ ] QA Lead sign-off

Status: ACCEPTED / REJECTED
```

---

### 5.9 Operational Health Report

```
# Operational Health Report
Week of: [date range]
Generated by: ARIA

## Apps Status
| App | URL | Uptime | Avg Response | Last Checked |
|-----|-----|--------|-------------|-------------|
| [name] | [url] | 99.9% | 240ms | [date] |

## Incidents This Week
| Date | App | Issue | Duration | Resolution |
|------|-----|-------|----------|-----------|
| — | — | — | — | — |

## Open Defects (All Products)
| Defect ID | Product | Severity | Age (days) | Status |
|----------|---------|----------|-----------|--------|

## Health Score
| Product | Score | Trend |
|---------|-------|-------|
| [name] | 95/100 | ↑ |

## Recommendations
[Any actions needed]
```

---

## 6. Defect Severity Definitions

| Severity | Definition | Response Time |
|---------|------------|--------------|
| **P0 — Critical** | App unusable, auth broken, data loss possible | Immediate — blocks everything |
| **P1 — High** | Core feature broken, significant user impact | Same session fix |
| **P2 — Medium** | Feature degraded but workaround exists | Next build cycle |
| **P3 — Low** | Minor UI issue, cosmetic, low user impact | Backlog |

---

## 7. Escalation Paths

| Situation | Action |
|-----------|--------|
| P0 issue found | Stop build. Post to #aria. Tag QA Lead immediately. |
| P1 issue found in production | Post to #aria-status. Log incident. Begin fix immediately. |
| QA gate fails 3x (same issue) | Escalate to human — tag QA Lead + product owner |
| Marketing claim cannot be verified | Remove claim or flag for human decision — never publish unverified |
| Deploy smoke test fails | Auto-rollback. Post to #aria. Do not retry without fix. |

---

## 8. Flaky Test Registry

Tests that fail intermittently (not consistently) are tracked separately to avoid false gate blocks.

```
# Flaky Test Registry
| Test ID | Description | Failure Rate | Root Cause (if known) | Action |
|---------|-------------|-------------|----------------------|--------|
| TC-012 | Realtime sync check | 30% | Network timeout on CI | Increase timeout |
```

**Policy:** A test with >20% flake rate is quarantined (excluded from gate) until root cause is fixed.

---

## 9. Implementation Notes (For Review)

The following requires implementation work after this protocol is approved:

1. **Embed QA checklist in all build spawn tasks** — closes subagent blindness
2. **Auto-generate Test Execution Report** after every build QA run
3. **Auto-generate Delivery Acceptance Report** on every deploy
4. **Weekly Operational Health Report** as a scheduled heartbeat task
5. **RTM generation** from PRD user stories → test cases
6. **Defect Log** as structured YAML files in `agency/issues/` (upgrade from current flat INDEX.md)

---

## 10. What's Out of Scope (v0.1)

- Load/stress testing
- Penetration testing (covered separately by OWASP agent)
- A/B test result analysis
- Accessibility audits (WCAG full compliance)
- Third-party API reliability testing

These may be added in v0.2 based on team feedback.

---

*Draft v0.1 — Pending QA Lead review by Rona Maceda*
*Do not implement until approved*
