Case study
CSIRO
Moving national infrastructure dashboards to serverless AWS
CSIRO’s COAG and BITRE dashboards give policymakers, analysts and the public access to national infrastructure and transport data.
As their underlying technology aged, CSIRO needed a partner to modernise the applications, improve accessibility and take responsibility for their ongoing development, hosting and support.
Airteam upgraded both dashboards and redesigned their AWS infrastructure around a simple characteristic of the data: people request it far more often than it changes.
Quick facts
CSIRO’s COAG and BITRE dashboards were upgraded, made more accessible and migrated from server-based AWS infrastructure to a serverless architecture.
The new architecture uses CloudFront, S3, API Gateway and Lambda to cache frequently requested data at the edge and run compute only when it’s needed.
Airteam continues to develop, host, test and maintain both dashboards.
- Client: CSIRO
- Industry: Government and science
- Services: Technical modernisation, accessibility, AWS serverless migration, hosting, ongoing development and support
Modernising critical public data infrastructure
CSIRO’s dashboards provide infrastructure and transport statistics used by federal policymakers, industry analysts and the public.
The applications were still doing their job, but the technology underneath them needed attention.
CSIRO didn’t have the internal capacity available to take on that work, so it looked for a partner that could assume responsibility for both the immediate modernisation and the platforms’ ongoing operation.
There were four priorities:
- Upgrade the underlying programming language, dependencies and frameworks
- Improve accessibility and introduce new functionality
- Move the applications onto more efficient AWS infrastructure
- Take responsibility for ongoing development, hosting, support and maintenance
The infrastructure presented an interesting opportunity.
CSIRO's previous EC2 server infrastructure
Rethinking an always-on architecture
The original architecture used a conventional server-based AWS setup.
A user request would arrive at an Application Load Balancer, which forwarded it to one of at least two running EC2 instances. The application then queried the RDS database, returned the result through the load balancer and logged the request to CloudWatch.
Additional EC2 instances could be added or removed as traffic changed.
It was a reliable architecture, but it meant maintaining compute capacity around the clock.
The way people used the dashboards suggested another approach: the underlying statistics change relatively infrequently. The same data can be requested many times before it needs to be refreshed.
Airteam redesigned the architecture around that pattern.
The new serverless infrastructure that Airteam implemented
Serving the data from the edge
In the new architecture, requests first reach an Amazon CloudFront distribution.
CloudFront uses AWS edge locations to serve cached responses closer to the user. If the requested content is already cached and still valid, it can be returned without invoking the application or querying the database again.
When something isn’t cached, the request follows one of two paths.
Static content
- Images and other static assets are retrieved from Amazon S3.
Statistical and chart data
- CloudFront sends the request through API Gateway, which invokes an AWS Lambda function. Lambda retrieves the required configuration from AWS Secrets Manager, queries the RDS database and returns the result.
The response is then cached by CloudFront for subsequent requests, while Lambda logs requests through CloudWatch.
The result is an architecture where many requests can be satisfied without reaching application compute or the database at all.
And when compute is required, Lambda runs for the request rather than relying on application servers running continuously.
For a public data platform, that’s a useful architectural question to ask: how often does the underlying data change compared with how often people request it?
When reads significantly outnumber updates, caching and on-demand compute can change the infrastructure you actually need.
Modernising more than the infrastructure
Moving to serverless AWS was only one part of the work.
Airteam also completed major version upgrades across the dashboards’ programming language, dependencies and frameworks, reducing the technical debt associated with ageing software.
Separate VPCs segregate different environments, supporting data integrity and privacy requirements.
New functionality also expanded what users could do with the dashboards, including additional graphs, tables and data exports.
Making data visualisations easier to read
Accessibility improvements focused on how statistical information was communicated.
Some charts previously relied on colour alone to distinguish between data series. That creates problems for users who can’t reliably perceive those colour differences.
Airteam introduced symbols, shading and other visual differentiators so that colour was no longer the only way to interpret the data.
This brought the visualisations into better alignment with WCAG requirements while preserving the amount of information the dashboards could communicate.
It’s an important consideration for any data-heavy interface: accessibility applies to the information inside a chart, not just the controls around it.
Ongoing support
Modernising ageing software doesn’t end when the new infrastructure goes live.
Airteam continues to host, test, maintain and develop both dashboards, managing the AWS environment and responding as technical requirements, accessibility expectations and CSIRO’s needs change.
The result is a modernised platform backed by an ongoing development and support team, allowing CSIRO to continue making national infrastructure and transport data available without rebuilding the internal capacity to maintain the applications itself.