# UI Incident Resolution - 2026-02-09

**Reported by:** Paul Gonia  
**Time:** 17:15 UTC  
**Complaint:** "UI is currently broken, why does this keep happening"  
**Status:** ✅ RESOLVED

## Investigation Summary

### Issue #1: Code Bug (Immediate Cause)
**Location:** `/src/app/api/discovery/opportunities/route.ts` line 133-135  
**Error:** `TypeError: Cannot read properties of undefined (reading 'toLowerCase')`  

**Root Cause:** Scan YAML files had inconsistent field names:
- Some used `title` (expected by code)
- Some used `name` or `id`
- Some had no name field at all

Code tried to call `.toLowerCase()` on undefined values → crash.

**Fix:** Added null safety checks:
```typescript
if (!scanOpp.title) continue;  // Skip invalid entries
if (!existing.name || !opp.name) return false;  // Null-safe comparison
```

### Issue #2: Process Management (Recurring Cause)
**Problem:** PM2 was in a crash loop with **523 restarts** due to port conflicts.

**Pattern:**
1. PM2 starts `npm start -- -p 4000`
2. Process binds to port successfully
3. npm wrapper exits (normal behavior)
4. PM2 detects exit and tries to restart
5. Port 4000 still occupied by previous Next.js instance
6. `EADDRINUSE` error
7. PM2 restarts again → infinite loop

**Why it kept happening:** Every deploy or restart would trigger this loop. Paul would see whichever instance happened to be serving at that moment - sometimes old buggy code, sometimes mid-crash.

## Actions Taken

1. ✅ Fixed code bug (null safety)
2. ✅ Killed all zombie next-server processes
3. ✅ Rebuilt with fix (`npm run build`)
4. ✅ Removed PM2 temporarily (source of crash loop)
5. ✅ Started clean process: PID 1094612
6. ✅ Created `/home/aria/projects/symph-agency/start.sh` with port conflict prevention
7. ✅ Created PID file: `/home/aria/projects/symph-agency.pid`
8. ✅ Verified API working (81 opportunities, no errors)
9. ✅ Documented root cause and prevention measures
10. ✅ Updated Paul via dashboard chat

## Current Status

- **Process:** Running (PID 1094612)
- **Port:** 4000 (clean, no conflicts)
- **API Health:** ✅ Working
- **Errors:** None in recent logs
- **Management:** Manual (no PM2)

## Prevention Measures

See `agency/docs/ui-breakage-prevention.md` for details.

**Immediate (DONE):**
- Null safety in API code
- Startup script with health checks

**Next 48h:**
- Monitor stability
- No restarts = success

**Week of Feb 10:**
- Add scan file validation
- Implement health monitoring
- Add auto-alerting to Discord
- Improve deploy script

## Files Modified

- `/home/aria/projects/symph-agency/src/app/api/discovery/opportunities/route.ts` (bug fix)
- `/home/aria/projects/symph-agency/start.sh` (new startup script)
- `agency/docs/ui-breakage-root-cause.md` (documentation)
- `agency/docs/ui-breakage-prevention.md` (prevention plan)
- `agency/chats/dashboard/chat.yaml` (updated Paul)

## Monitoring Commands

```bash
# Check process status
ps aux | grep $(cat /home/aria/projects/symph-agency.pid)

# Check logs
tail -f /home/aria/projects/.aria-logs/symph-agency.log

# Verify API health
curl http://localhost:4000/api/discovery/opportunities

# Check for zombies
ps aux | grep "next-server" | grep "4000"

# Restart if needed (use this!)
/home/aria/projects/symph-agency/start.sh
```

## Lessons Learned

1. **PM2 + npm wrappers = trouble** - PM2 interprets npm exit as app crash
2. **Defensive coding matters** - One undefined value crashed entire feature
3. **Zombie processes accumulate** - Need proper cleanup on restart
4. **Manual process management is OK** - PM2 adds complexity, not always needed
5. **Data validation should happen at source** - Scan scripts should validate before writing YAML

## Success Criteria

- [x] Site responding (HTTP 200)
- [x] API returning data without errors
- [x] No crash loops
- [x] Paul notified
- [ ] 48h uptime with zero restarts (monitoring)
- [ ] Prevention measures implemented (by Feb 12)
