All engineering studies
Owned study 02 / AI systems architecture

Tests grounded in the product. Releases gated by people.

AI-Driven QA Automation

Proposal for approval · one-team pilot planned
Company
Tyroo Technologies
Role
SDE II
Employment period
Feb 2023 — Present

I’m architecting a QA process that follows a Bitbucket pull request from its versioned Confluence PRD through test design, execution, human UAT and production regression. Six specialist agents handle repeatable work; people approve tests, releases and production responses. The proposal defines the gates, evidence and failure paths for a one-team pilot.

In this study
  • Multi-agent orchestration
  • RAG
  • Bitbucket
  • Confluence
  • Jira
  • Human approval gates
6 + 1specialists + orchestrator

Separate test design, execution, triage, regression, sanity and reporting responsibilities.

8 stagesPR to production completion

A named owner and a pass condition at each step, with defined responses to failures.

5 of 6QA activities proposed for agents

Writing, execution, bug reporting, retesting and regression / sanity move to agents. UAT stays manual; this is a scope count, not an effort-saving result.

4 decisionsprincipal human approval points

Test sets, UAT, release and production sanity responses. People also rule on bug disputes.

Scope from the proposal dated 28 September 2026. Architecture and pilot planning; these are proposed responsibilities, not measured production outcomes.

The engineering, step by step

Follow the system.

Explore the constraint, the decision, how it was checked, and the result. The full narrative follows below.

Follow the systemOwned study 02 / AI systems architecture
Constraint / 01

Test the requirement behind the change.

Manual test cycles delay PR feedback and require repeated execution after fixes. The proposed flow ties every change to its Jira story and versioned Confluence PRD, while keeping UAT and release judgment with people.

  • Jira key in branch name and PR title
  • Confluence PRD version recorded per run
  • Missing requirements block the PR build
  1. Bitbucket PR
  2. Jira story
  3. Versioned Confluence PRD
Conceptual sequence · based on the project narrative
The constraint ledgerHuman-gated QA architecture · proposal for approval
Proposal /

From manual execution to human review.

Architecture and process prepared for approval. The one-team pilot is proposed; impact will be measured against the current manual flow.
Operating blueprint / 01

A requirement behind every run.

The Jira issue key connects code, product intent and test evidence. Versioned requirements keep the test history explainable.

  1. 01

    Bitbucket pull request

    The branch name and PR title carry the Jira issue key. PR events trigger the orchestrator; the Bitbucket–Jira integration exposes the PR on the issue.

  2. 02

    Linked Jira story

    The issue key resolves the story and its Confluence PRD. Jira holds approved test sets, run results, evidence, bugs and the ticket’s current state.

  3. 03

    Versioned Confluence PRD

    The product requirements ground generated tests. Every run records the Confluence page version, so its requirements remain traceable.

Before the first test

A missing Jira key or PRD link fails the PR build with a reason and blocks merging. Every generated or rewritten test set needs human approval, including after a PRD change.

Operating blueprint / 02

Clear jobs. Clear accountability.

One orchestrator and six specialists handle repeatable work. No agent approves its own output.

Coordination layer

Orchestrator

Listens for Bitbucket PR, update, merge and deployment events; retrieves the linked story and PRD through Atlassian APIs; assigns agent work, moves Jira tickets and sends notifications.

  1. Agent / 01

    Test design

    Writes PRD-grounded test cases under the anti-flake policy. Rewrites rejected or changed sets for human approval.

  2. Agent / 02

    Execution

    Runs functional, API and UI tests on the QA environment, captures evidence and reruns every story test after a fix.

  3. Agent / 03

    Bug triage

    Separates product defects from test and environment problems. Files confirmed bugs only with the required evidence.

  4. Agent / 04

    Regression

    Maintains the regression suite, adds newly approved tests, retires old ones and runs the full suite on the production test brand.

  5. Agent / 05

    Sanity

    Checks critical paths on the production test brand after deployment. Escalates failures for a QA engineer’s decision.

  6. Agent / 06

    Reporting

    Writes results to Jira and sends the release report to stakeholders at completion.

Four principal human decisions

People hold the gates.

QA lead
Approves every test case set and rewrite; rules on bug disputes.
Manual tester
Runs UAT, signs off the build and chooses the response to UAT failures.
Tech lead
Approves release after UAT and green Bitbucket merge checks.
QA engineer
Assesses production sanity failures and decides the response.
Operating blueprint / 03

From PR to production evidence.

Each stage has an owner and a condition for moving forward. Failed checks return to a defined response before the change continues.

  1. PR raised

    Resolve the Bitbucket PR to its Jira story and versioned product requirements before testing starts.

    Owner
    Orchestrator
    Pass condition
    Jira key present; Confluence PRD linked.
    Response & continuity

    A missing key or PRD link fails the PR build with a reason and blocks merging.

  2. Test design

    Generate cases from the recorded PRD version. The QA lead reviews coverage, correctness and the anti-flake policy.

    Owner
    Test design agent + QA lead
    Pass condition
    A person approves the complete test set.
    Response & continuity

    Rejections return with comments for a rewrite and another approval. PRD changes restart design and reapproval.

  3. Automated testing

    Run approved tests on a healthy QA environment. Retry a failed test once, then triage; log lower-severity bugs in Jira.

    Owner
    Execution + bug triage agents
    Pass condition
    No open blocker or critical bugs.
    Response & continuity

    Pause for environment issues. Confirmed defects return to the developer; a fix linked to its bug ID reruns every test for that story.

  4. Manual UAT

    A person exercises the build that passed automated testing. UAT remains fully manual.

    Owner
    Manual tester
    Pass condition
    The tester signs off the tested build.
    Response & continuity

    The tester chooses the failure response. Missing coverage adds a test; flagged backlog items stay linked to the story and may accompany approval with a note.

  5. Merge and release

    Use the test history in Jira and enforced Bitbucket checks to authorize the merge and deployment.

    Owner
    Tech lead
    Pass condition
    UAT approved; merge checks green; release approved.
    Response & continuity

    A changed PR resets approvals. Every new commit is retested before it can move forward.

  6. Production sanity

    Immediately after deployment, check critical paths using only isolated production test data.

    Owner
    Sanity agent + QA engineer
    Pass condition
    All critical-path checks pass on the test brand.
    Response & continuity

    Notify stakeholders immediately. The QA engineer assesses product, test or environment failures and decides the response; rollback is never automatic.

  7. Production regression

    Add the story’s newly approved tests to the suite and run full regression after sanity.

    Owner
    Regression agent
    Pass condition
    The full regression suite passes on the test brand.
    Response & continuity

    Apply the proposal’s evidence policy to failures: triage first, then file confirmed product bugs with evidence.

  8. Completed

    Record results in Jira and attach the release report for stakeholders.

    Owner
    Reporting agent
    Pass condition
    Full regression passed; ticket closed.
    Response & continuity

    Completion depends on the production regression gate; the report makes the release history available to the team.

Operating blueprint / 04

The failure paths are part of the design.

Reliability comes from what happens when a test is flaky, a bug is disputed or a release needs an urgent fix. Open a field note for its policy.

Tests that do not fail at random.

The test design agent follows an explicit anti-flake policy, and the QA lead checks it during approval. A test that flakes twice within 30 days is quarantined: it cannot block a release or file a bug until it has been rewritten and approved again.

  • Use dedicated test IDs, never screen position, styling or generated class names.
  • Wait for a specific UI or API condition, never a fixed delay.
  • Give each test independent data setup and cleanup; avoid run-order dependencies.
  • Use fixed, known data without relying on the current date, time zone, random values or live third-party services.
  • Give each test one clear goal so a failure points to a cause.

An environment failure is not a product bug.

Before each run, the execution flow checks QA environment health. An unhealthy environment pauses testing and is reported as an environment issue. Failed tests are retried once before triage separates product defects from test and environment problems.

Confirmed bugs require steps to reproduce, expected versus actual results, screenshots or video, logs and severity. Jira rejects agent-filed bugs missing any of that evidence. A developer fix references the bug ID and triggers every test case for the story again.

  • Health check before execution.
  • One retry, then triage.
  • Complete evidence before a Jira bug.
  • Full-story retesting after each fix.

A disputed bug should improve the suite.

Developers can dispute an agent-filed bug with a valid written reason. The QA lead rules; stakeholders see the dispute and its outcome. If accepted, the bug closes, the test cases are corrected and the story returns to In QA. If the bug stands, the developer fixes it.

Every accepted dispute becomes a correction to the test cases so the same false report does not recur. The monthly review tracks overruled agent bugs as a measure of report accuracy.

Manual UAT keeps the product judgment.

The manual tester receives a build that has passed automated testing and decides the UAT outcome. A UAT failure caused by a missing test adds that case to the suite.

Flagged items go to the Jira backlog linked to the original story. The tester may still approve the release with that item noted; this remains a human decision.

Production checks need their own boundary.

Sanity runs first after deployment, checking critical paths in the first minutes. Full regression follows and incorporates the story’s approved tests. Both run inside a dedicated production test brand with test data isolated from customer data, billing and analytics.

A sanity failure immediately notifies stakeholders. The agent gathers evidence; a QA engineer decides whether it is a real product issue, a test problem or an environment problem. Rollback never happens automatically, and the tech lead and developer are notified so they can respond if the decision requires a rollback or hotfix.

  • Critical-path sanity before full regression.
  • Test data isolated from customer activity.
  • Human decision on every sanity failure.

Urgency does not remove the gates.

Only P0 and P1 Jira tickets enter the hotfix process. Testing targets the fix and approvals are fast-tracked, while every gate still applies. The proposal specifies an eight-step hotfix flow with three human decisions.

A QA engineer and tech lead are named on call each week. The back-merge at step 7 is mandatory: the ticket cannot close before it is complete. Every hotfix is reviewed monthly to understand why the issue reached production.

  • P0 / P1 only; other changes use the standard eight stages.
  • Targeted tests and fast-tracked human approvals.
  • Mandatory back-merge before closure.
  • Every hotfix enters the monthly quality review.

Keep changed and competing PRs traceable.

QA and UAT share an environment, so overlapping PRs are deployed there one at a time. The QA lead sets the order when parallel PRs have equal priority. Every new PR commit is retested, and Bitbucket resets approvals when the PR changes after approval.

A changed PRD notifies stakeholders and sends test cases through update and reapproval. An agent or pipeline outage notifies the team for DevOps to restore service. The proposal describes a 12-state Jira lifecycle: the orchestrator tracks transitions, while people retain test approvals, UAT decisions, release, dispute rulings and sanity decisions.

Notifications carry a next action.

The Jira issue and team chat channel carry notifications to the responsible people. Stakeholders include the developer, QA lead, manual tester, tech lead and product owner.

  • Test approval waiting → QA lead approves or rejects with comments.
  • Evidence-backed bug → developer fixes or disputes; QA lead is notified.
  • Dispute and ruling → stakeholders follow the QA lead’s decision.
  • UAT failure → developer, QA lead and tech lead follow the tester’s chosen response.
  • Sanity failure or hotfix → stakeholders are notified immediately; the named decision owners respond.
  • Pipeline outage → stakeholders are notified and DevOps restores service.
  • Equal-priority PRs → QA lead and both developers resolve test order.
  • PRD change → stakeholders update and reapprove tests.
  • Completion → stakeholders receive the attached release report.
Operating blueprint / 05

Prove it with one team first.

The proposal defines four rollout phases and three gates. It requests the first three phases with one team, then a pilot review before wider rollout.

Pilot boundary

Keep the current manual flow available during phases 2 and 3. Compare pilot measures with the manual baseline at review, and seek approval before rolling out to all teams.

Baseline first. Measure monthly.

These are evaluation criteria. The proposal reports no baseline values, numerical targets or pilot results.

PR → ready for UAT
Feedback and delivery speed
Manual QA hours / release
Effort shifted to review and judgment
Bugs found in production
Release quality
Agent bugs overruled in disputes
Bug-report accuracy
Flaky test rate
Suite reliability
Time to retest a fix
Fix-and-retest speed
Requested approval

What the pilot needs.

  1. Adopt the eight-stage process, sub-processes and quality controls for the pilot team.
  2. Provision a production test brand containing only test data, isolated from customers, billing and analytics.
  3. Configure Jira key rules, required bug-evidence fields, enforced Bitbucket merge checks and Confluence as the PRD source. The proposal identifies Bitbucket Premium as a merge-check dependency.
  4. Name a QA lead for test approval and disputes, plus a weekly on-call QA engineer and tech lead.
  5. Run phases 1–3 with one team, then return for pilot review and approval before expanding to all teams.
Monthly quality review

Review production bugs, overruled agent bugs, flaky tests and every hotfix. Turn each finding into a new test case or rule, then use the next review to assess the process again.

Have a similar systems challenge?

Let’s talk