How we built a software factory to handle 6x more product feedback
TL;DR
As our two-engineer Cosmos Advisor team shipped more automations to more customers, weekly product feedback grew rapidly - reaching 30+ feedback-threads per week. Investigating questions, reproducing bugs, finding owners, filing tickets, and putting up fixes began consuming an estimated 90% of our time. Our roadmap nearly stalled.
Instead of immediately growing the team, we built a Feedback Triager expert (i.e. agent) with Cosmos. It follows each Slack report through its full lifecycle: gathering evidence, performing root-cause analysis (RCA), answering questions, routing feedback to other channels, creating tickets, and—when the fix is clear—handing the issue to a PR Author expert (i.e. agent). Humans retain product judgment and prioritization; agents handle the repetitive investigation and execution around those decisions.
The result is a small team that can stay responsive to customers without turning its roadmap into a support queue.
Success created a new bottleneck
The two-engineer Cosmos Advisor team builds out-of-the-box automations and the Advisor experience across code hosts, ticket trackers, and collaboration platforms. That includes workflows for code review, incident response, feedback triage, and large engineering projects across GitHub, GitLab, Slack, Microsoft Teams, Jira, and other surfaces.
As our feature set and customer base expanded, feedback volume surged. Success had created a new bottleneck: the faster we shipped, the more time we spent supporting what we had already built.
Reports arrive in a Slack feedback channel dedicated to our team. They come from:
Over a recent two-week period, we handled 60 product feedback threads—an average of 30 per week, up from about five per week only a few weeks earlier:
| Type | Typical example | Share of threads |
|---|---|---|
| Feature request | A customer needs a new workflow or configuration option. | 8.33% |
| Platform support | A workflow behaves differently on GitLab, Azure DevOps, Teams, or Jira—platforms we cannot dogfood as deeply as our primary stack. | 8.33% |
| Production bug | A released workflow fails or behaves unexpectedly in a customer environment. | 6.67% |
| Pre-release feedback | Internal dogfooding finds a problem in a new automation before customers encounter it. | 68.34% |
| Other product feedback | Internal post-release bugs, positive feedback, or broader product concerns. | 8.33% |
| Total | 100.00% |
Of those 60 threads, 50% were fixed or had a concrete fix underway shortly after they were reported.
The problem was not simply the number of Slack messages. Each report could require reading a thread, reproducing behavior, searching code and documentation, checking logs, finding related tickets, deciding ownership, answering follow-up questions, and sometimes writing a fix.
At the peak, we estimated that this consumed roughly 90% of the team's time. We were staying responsive, but execution on longer-term work had begun to stall.
Hiring was one option, but much of the workload was repetitive context reconstruction rather than product judgment. We wanted agents to absorb investigation and routine execution while engineers retained prioritization and product decisions.
How our feedback loop works—from report to resolution
Every new root message in our team's feedback channel starts a long-running Feedback Triager session. That session owns the thread for its lifetime, so it can incorporate replies, corrections, edits, and new evidence without reconstructing the conversation on every turn.
1. Intake
The Triager reads the report and surrounding context, acknowledges it, and determines what information is needed. It asks at most one focused clarifying question when a critical fact is missing.
2. Investigation
For bugs, regressions, and unexpected behavior, it performs RCA across code, configuration, tests, documentation, logs, metrics, deployed state, existing tickets, and related Slack threads. Runtime evidence establishes what happened; static evidence explains why. The Triager labels conclusions as confirmed, tentative, or still missing evidence rather than presenting a plausible guess as fact.
3. Action
The next step depends on the evidence:
| Finding | Action |
|---|---|
| A question, one-off issue, expected behavior, or unsupported request | Answer or explain the decision in Slack; no ticket required. |
| The issue belongs to another team | Route it to the right team with the RCA and evidence already attached. |
| A matching issue already exists | Link or enrich the existing ticket instead of creating a duplicate. |
| The RCA and localized fix are clear | Launch a PR Author to implement the fix immediately, then send it through review and verification. |
| The problem is open-ended or the RCA remains unclear | Create a Linear backlog issue with the evidence gathered so far; prioritize and investigate it with the rest of the roadmap. |
Clear, well-scoped fixes should not wait for another planning cycle; ambiguous problems and feature requests belong in Linear for prioritization and deeper work.
How to make feedback triage effective
Feedback triage needs a software factory
A Feedback Triager can produce actionable work much faster than a manual queue. If the downstream engineering system cannot absorb that work, code review and verification become the new bottlenecks.
Our feedback loop therefore connects several specialized Cosmos experts:
The objective is not to optimize one step. It is to shorten the complete loop from product feedback to a verified outcome. The same software factory also supports large engineering projects and incident response.
The end state: feedback scales without scaling the team
We are happy with the operating model today. Our two-person team now spends roughly 30% of its time on feedback, down from an estimated 90%, and our primary focus is back on the long-term roadmap.
We no longer need to grow the team solely to keep pace with feedback. As we create more features, the Feedback Triager expands our capacity to investigate and route the resulting feedback, while PR Author, code review, and verification experts absorb downstream remediation.
Other early lessons:
How to adopt this workflow
Ask the Cosmos Advisor to set up a Feedback Triager for your team. With a standard collaboration and ticketing stack—Slack or Microsoft Teams paired with Jira, Linear, GitHub Issues, or GitLab Issues—Advisor can autonomously configure the expert, integrations, triggers, and routing workflow.
Start with one team and one feedback channel. Advisor will connect the codebase, tracker, and evidence sources; configure answer, routing, deduplication, and filing rules; choose the level of human approval; and connect downstream review and verification. Measure the baseline first, then expand authority as the workflow proves reliable.
The goal is not to create more tickets or PRs. It is to make every product feedback thread reach the right outcome with less repetitive human work—so a small team can support a growing product and keep building what comes next.
Build your own software factory for product feedback
Cosmos gives engineering teams the shared context, runtime controls, integrations, and human checkpoints to triage feedback, investigate root cause, and route it to the right outcome.
Originally published on @augmentcode blog. Written by @AkshayUtture001.