Tests grounded in the product. Releases gated by people.
AI-Driven QA Automation
- Company
- Tyroo Technologies
- Role
- SDE II
- Employment period
- Feb 2023 — Present
I’m architecting a QA process that follows a Bitbucket pull request from its versioned Confluence PRD through test design, execution, human UAT and production regression. Six specialist agents handle repeatable work; people approve tests, releases and production responses. The proposal defines the gates, evidence and failure paths for a one-team pilot.
- Multi-agent orchestration
- RAG
- Bitbucket
- Confluence
- Jira
- Human approval gates
Separate test design, execution, triage, regression, sanity and reporting responsibilities.
A named owner and a pass condition at each step, with defined responses to failures.
Writing, execution, bug reporting, retesting and regression / sanity move to agents. UAT stays manual; this is a scope count, not an effort-saving result.
Test sets, UAT, release and production sanity responses. People also rule on bug disputes.
Scope from the proposal dated 28 September 2026. Architecture and pilot planning; these are proposed responsibilities, not measured production outcomes.
Follow the system.
Explore the constraint, the decision, how it was checked, and the result. The full narrative follows below.
Test the requirement behind the change.
Manual test cycles delay PR feedback and require repeated execution after fixes. The proposed flow ties every change to its Jira story and versioned Confluence PRD, while keeping UAT and release judgment with people.
- Jira key in branch name and PR title
- Confluence PRD version recorded per run
- Missing requirements block the PR build
- Bitbucket PR
- Jira story
- Versioned Confluence PRD
From manual execution to human review.
Architecture and process prepared for approval. The one-team pilot is proposed; impact will be measured against the current manual flow.A requirement behind every run.
The Jira issue key connects code, product intent and test evidence. Versioned requirements keep the test history explainable.
- 01
Bitbucket pull request
The branch name and PR title carry the Jira issue key. PR events trigger the orchestrator; the Bitbucket–Jira integration exposes the PR on the issue.
- 02
Linked Jira story
The issue key resolves the story and its Confluence PRD. Jira holds approved test sets, run results, evidence, bugs and the ticket’s current state.
- 03
Versioned Confluence PRD
The product requirements ground generated tests. Every run records the Confluence page version, so its requirements remain traceable.
A missing Jira key or PRD link fails the PR build with a reason and blocks merging. Every generated or rewritten test set needs human approval, including after a PRD change.
Clear jobs. Clear accountability.
One orchestrator and six specialists handle repeatable work. No agent approves its own output.
Orchestrator
Listens for Bitbucket PR, update, merge and deployment events; retrieves the linked story and PRD through Atlassian APIs; assigns agent work, moves Jira tickets and sends notifications.
- Agent / 01
Test design
Writes PRD-grounded test cases under the anti-flake policy. Rewrites rejected or changed sets for human approval.
- Agent / 02
Execution
Runs functional, API and UI tests on the QA environment, captures evidence and reruns every story test after a fix.
- Agent / 03
Bug triage
Separates product defects from test and environment problems. Files confirmed bugs only with the required evidence.
- Agent / 04
Regression
Maintains the regression suite, adds newly approved tests, retires old ones and runs the full suite on the production test brand.
- Agent / 05
Sanity
Checks critical paths on the production test brand after deployment. Escalates failures for a QA engineer’s decision.
- Agent / 06
Reporting
Writes results to Jira and sends the release report to stakeholders at completion.
People hold the gates.
- QA lead
- Approves every test case set and rewrite; rules on bug disputes.
- Manual tester
- Runs UAT, signs off the build and chooses the response to UAT failures.
- Tech lead
- Approves release after UAT and green Bitbucket merge checks.
- QA engineer
- Assesses production sanity failures and decides the response.
From PR to production evidence.
Each stage has an owner and a condition for moving forward. Failed checks return to a defined response before the change continues.
PR raised
Resolve the Bitbucket PR to its Jira story and versioned product requirements before testing starts.
- Owner
- Orchestrator
- Pass condition
- Jira key present; Confluence PRD linked.
Response & continuityA missing key or PRD link fails the PR build with a reason and blocks merging.
Test design
Generate cases from the recorded PRD version. The QA lead reviews coverage, correctness and the anti-flake policy.
- Owner
- Test design agent + QA lead
- Pass condition
- A person approves the complete test set.
Response & continuityRejections return with comments for a rewrite and another approval. PRD changes restart design and reapproval.
Automated testing
Run approved tests on a healthy QA environment. Retry a failed test once, then triage; log lower-severity bugs in Jira.
- Owner
- Execution + bug triage agents
- Pass condition
- No open blocker or critical bugs.
Response & continuityPause for environment issues. Confirmed defects return to the developer; a fix linked to its bug ID reruns every test for that story.
Manual UAT
A person exercises the build that passed automated testing. UAT remains fully manual.
- Owner
- Manual tester
- Pass condition
- The tester signs off the tested build.
Response & continuityThe tester chooses the failure response. Missing coverage adds a test; flagged backlog items stay linked to the story and may accompany approval with a note.
Merge and release
Use the test history in Jira and enforced Bitbucket checks to authorize the merge and deployment.
- Owner
- Tech lead
- Pass condition
- UAT approved; merge checks green; release approved.
Response & continuityA changed PR resets approvals. Every new commit is retested before it can move forward.
Production sanity
Immediately after deployment, check critical paths using only isolated production test data.
- Owner
- Sanity agent + QA engineer
- Pass condition
- All critical-path checks pass on the test brand.
Response & continuityNotify stakeholders immediately. The QA engineer assesses product, test or environment failures and decides the response; rollback is never automatic.
Production regression
Add the story’s newly approved tests to the suite and run full regression after sanity.
- Owner
- Regression agent
- Pass condition
- The full regression suite passes on the test brand.
Response & continuityApply the proposal’s evidence policy to failures: triage first, then file confirmed product bugs with evidence.
Completed
Record results in Jira and attach the release report for stakeholders.
- Owner
- Reporting agent
- Pass condition
- Full regression passed; ticket closed.
Response & continuityCompletion depends on the production regression gate; the report makes the release history available to the team.
The failure paths are part of the design.
Reliability comes from what happens when a test is flaky, a bug is disputed or a release needs an urgent fix. Open a field note for its policy.
Tests that do not fail at random.
The test design agent follows an explicit anti-flake policy, and the QA lead checks it during approval. A test that flakes twice within 30 days is quarantined: it cannot block a release or file a bug until it has been rewritten and approved again.
- Use dedicated test IDs, never screen position, styling or generated class names.
- Wait for a specific UI or API condition, never a fixed delay.
- Give each test independent data setup and cleanup; avoid run-order dependencies.
- Use fixed, known data without relying on the current date, time zone, random values or live third-party services.
- Give each test one clear goal so a failure points to a cause.
An environment failure is not a product bug.
Before each run, the execution flow checks QA environment health. An unhealthy environment pauses testing and is reported as an environment issue. Failed tests are retried once before triage separates product defects from test and environment problems.
Confirmed bugs require steps to reproduce, expected versus actual results, screenshots or video, logs and severity. Jira rejects agent-filed bugs missing any of that evidence. A developer fix references the bug ID and triggers every test case for the story again.
- Health check before execution.
- One retry, then triage.
- Complete evidence before a Jira bug.
- Full-story retesting after each fix.
A disputed bug should improve the suite.
Developers can dispute an agent-filed bug with a valid written reason. The QA lead rules; stakeholders see the dispute and its outcome. If accepted, the bug closes, the test cases are corrected and the story returns to In QA. If the bug stands, the developer fixes it.
Every accepted dispute becomes a correction to the test cases so the same false report does not recur. The monthly review tracks overruled agent bugs as a measure of report accuracy.
Manual UAT keeps the product judgment.
The manual tester receives a build that has passed automated testing and decides the UAT outcome. A UAT failure caused by a missing test adds that case to the suite.
Flagged items go to the Jira backlog linked to the original story. The tester may still approve the release with that item noted; this remains a human decision.
Production checks need their own boundary.
Sanity runs first after deployment, checking critical paths in the first minutes. Full regression follows and incorporates the story’s approved tests. Both run inside a dedicated production test brand with test data isolated from customer data, billing and analytics.
A sanity failure immediately notifies stakeholders. The agent gathers evidence; a QA engineer decides whether it is a real product issue, a test problem or an environment problem. Rollback never happens automatically, and the tech lead and developer are notified so they can respond if the decision requires a rollback or hotfix.
- Critical-path sanity before full regression.
- Test data isolated from customer activity.
- Human decision on every sanity failure.
Urgency does not remove the gates.
Only P0 and P1 Jira tickets enter the hotfix process. Testing targets the fix and approvals are fast-tracked, while every gate still applies. The proposal specifies an eight-step hotfix flow with three human decisions.
A QA engineer and tech lead are named on call each week. The back-merge at step 7 is mandatory: the ticket cannot close before it is complete. Every hotfix is reviewed monthly to understand why the issue reached production.
- P0 / P1 only; other changes use the standard eight stages.
- Targeted tests and fast-tracked human approvals.
- Mandatory back-merge before closure.
- Every hotfix enters the monthly quality review.
Keep changed and competing PRs traceable.
QA and UAT share an environment, so overlapping PRs are deployed there one at a time. The QA lead sets the order when parallel PRs have equal priority. Every new PR commit is retested, and Bitbucket resets approvals when the PR changes after approval.
A changed PRD notifies stakeholders and sends test cases through update and reapproval. An agent or pipeline outage notifies the team for DevOps to restore service. The proposal describes a 12-state Jira lifecycle: the orchestrator tracks transitions, while people retain test approvals, UAT decisions, release, dispute rulings and sanity decisions.
Notifications carry a next action.
The Jira issue and team chat channel carry notifications to the responsible people. Stakeholders include the developer, QA lead, manual tester, tech lead and product owner.
- Test approval waiting → QA lead approves or rejects with comments.
- Evidence-backed bug → developer fixes or disputes; QA lead is notified.
- Dispute and ruling → stakeholders follow the QA lead’s decision.
- UAT failure → developer, QA lead and tech lead follow the tester’s chosen response.
- Sanity failure or hotfix → stakeholders are notified immediately; the named decision owners respond.
- Pipeline outage → stakeholders are notified and DevOps restores service.
- Equal-priority PRs → QA lead and both developers resolve test order.
- PRD change → stakeholders update and reapprove tests.
- Completion → stakeholders receive the attached release report.
Prove it with one team first.
The proposal defines four rollout phases and three gates. It requests the first three phases with one team, then a pilot review before wider rollout.
Keep the current manual flow available during phases 2 and 3. Compare pilot measures with the manual baseline at review, and seek approval before rolling out to all teams.
Baseline first. Measure monthly.
These are evaluation criteria. The proposal reports no baseline values, numerical targets or pilot results.
- PR → ready for UAT
- Feedback and delivery speed
- Manual QA hours / release
- Effort shifted to review and judgment
- Bugs found in production
- Release quality
- Agent bugs overruled in disputes
- Bug-report accuracy
- Flaky test rate
- Suite reliability
- Time to retest a fix
- Fix-and-retest speed
What the pilot needs.
- Adopt the eight-stage process, sub-processes and quality controls for the pilot team.
- Provision a production test brand containing only test data, isolated from customers, billing and analytics.
- Configure Jira key rules, required bug-evidence fields, enforced Bitbucket merge checks and Confluence as the PRD source. The proposal identifies Bitbucket Premium as a merge-check dependency.
- Name a QA lead for test approval and disputes, plus a weekly on-call QA engineer and tech lead.
- Run phases 1–3 with one team, then return for pilot review and approval before expanding to all teams.
Review production bugs, overruled agent bugs, flaky tests and every hotfix. Turn each finding into a new test case or rule, then use the next review to assess the process again.
Have a similar systems challenge?
Let’s talk