AI transition stages and dimensions
The pathway from conceptualisation to AI system implementation is typically a three-stage process involving a proof of concept (PoC), pilot and then release into production or the operational environment. In the Australian Government context, this may involve (for example):
- PoC: An agency builds a generative AI tool using historical case files to test whether AI can effectively summarise long policy documents, identify relevant points and provide decision support for staff. Depending on the objectives, the PoC boundaries must be set appropriately – whether in a constrained business environment or a constraint-free setting.
- Pilot: The AI tool is deployed to a small group of policy officers in one division, using redacted real documents, with feedback collected on accuracy and usability.
- Production: The AI tool is rolled out agency-wide, fully integrated with secure document repositories, supported by IT and governed under AI Assurance and the Digital Service Standard.
In practice, multiple PoCs may be conducted to experiment with distinct components of a broader initiative, forming a composite foundation for implementation. The components may include technical evaluation and business process fit or validation. Several PoCs may also be conducted in parallel to test various aspects of the evaluation area of focus i.e. open source versus AI proprietary technology type.
All activities that form part of the pathway to production need to comply with all relevant Australian laws and legislation.
Stages: PoC vs pilot vs production
The table below provides a comparative definition of each stage whilst also identifying the various aspects that could be considered in implementation design and ongoing solution sustainment.
Definition
- PoC: Early, experimental build to test concepts, feasibility, or specific technical components. Short time frame, budget constrained.
- Pilot (transit to scale): Limited-scale, real-world implementation to validate value, usability and readiness. Budget constrained and timeframe specified.
- Production (scale): Fully deployed, enterprise-scale system integrated into business-as-usual operations.
Purpose
- PoC: Explore ideas, prove feasibility and learn quickly.
- Pilot (transit to scale): Validate solution in controlled conditions before scaling.
- Production (scale): Deliver consistent, reliable, compliant service at scale.
Users
- PoC: Internal teams, stakeholders or test group.
- Pilot (transit to scale): Real users (staff, citizens) in a limited or controlled setting.
- Production (scale): All intended end-users (public, agency-wide, cross-government).
Metrics
- PoC: Technical feasibility, functional performance, scientific and empirical test metrics.
- Pilot (transit to scale): User satisfaction and feedback, business operation impact.
- Production (scale): Operation performance, business operation measures, ongoing monitoring.
Data
- PoC: Use of synthetic data due to privacy and data governance risk.
- Pilot (transit to scale): Live or near-live real data with governance and safeguards.
- Production (scale): Full live production data under governance frameworks.
Security
- PoC: Foundational, meets data, infrastructure and IP requirements.
- Pilot (transit to scale): Controlled and increased oversight.
- Production (scale): Full compliance, robust and ongoing monitoring.
Infrastructure
- PoC: Light-weight or sandbox environment.
- Pilot (transit to scale): Core processing and integration fully tested under high volumes.
- Production (scale): Full enterprise integration.
Integration
- PoC: Minimal or stand-alone; often mocked or simulated.
- Pilot (transit to scale): Partial/controlled integration with legacy systems or cloud services.
- Production (scale): Full enterprise integration with other government systems and services.
Investment
- PoC: Low (time, cost, compliance).
- Pilot (transit to scale): Medium (infrastructure, governance, partial training).
- Production (scale): High (enterprise infrastructure, full compliance, long-term sustainability).
Governance
- PoC: Light-touch; risk-based exemptions allowed; compliance with PSPF, ISM and AI Assurance. Data governance and data-type considerations must be addressed to meet security and privacy requirements.
- Pilot (transit to scale): Increased oversight, ethics and assurance processes; compliance with PSPF, ISM and AI Assurance. Strengthened data governance controls for increased scale of sensitive data and information exposure.
- Production (scale): Full compliance with PSPF, ISM, AI Assurance, procurement and audit standards. Robust governance and incident management for operation-wide exposure to data and information.
Risk tolerance
- PoC: High – failure is acceptable and expected. Expectation that PoC clearly states assumptions.
- Pilot (transit to scale): Moderate – failure is still possible but must be controlled.
- Production (scale): Low – failures must be rare and managed through continuity planning.
Outcome
- PoC: Proof of feasibility – decide if idea has merit.
- Pilot (transit to scale): Proof of viability – decide whether to scale to production.
- Production (scale): Operational service – provides measurable, sustainable outcomes.
Lifecycle
- PoC: Disposable, often short-lived.
- Pilot (transit to scale): Transitional – can be iterated or scaled.
- Production (scale): Long-term – maintained, supported and continuously improved.
Procurement (if applicable)
- PoC: Short-term contract, vendor checks, performance evaluation, IP ownership, auditable purchase orders.
- Pilot (transit to scale): Partial compliance meeting all mandatory requirements for the controlled setting, including formal purchase orders.
- Production (scale): Full compliance, long-term contract, open standards, formal exit planning.
Dimension considerations
A critical part of ensuring PoC to Scale success includes thorough and detailed planning. The framework laid out below is intended to assist agencies with a pro-active approach to planning. It considers the key dimensions of transition from system inception to ongoing sustainment.