A national system has a lot happening at once, across states, regions, industry partners, and the applications they use to exchange information. The team supporting it needs to understand how those parts are working together, from the availability of a service to the submissions arriving from a state and the data being retrieved by a partner.
That is how I think about mission control for an application: a common place where the team can follow the operation as a whole and get to the detail when something needs attention. It should show the current state of the system, enough history to understand what has changed, and the upcoming work that we need to prepare for.
For the national system we support, that includes application health and deployments, but it also includes state data freshness, review queues, API usage, and support requests. We have been bringing these together in an internal application called Mission Control, with the aim of making the day-to-day work of supporting the system easier as its use grows.
Why We Needed a Common View
We already had much of this information, but getting to it involved several different tools. Alarms and infrastructure details were in monitoring tools, usage statistics came from reports or database queries, and deployment information and release schedules were maintained elsewhere. The same group of people needed all of it, so gathering information became a recurring part of supporting the application.
Correlating what we found was more difficult because the scope of a problem was not always apparent from any one source. We needed to compare operational records and activity across the system to establish who was affected and decide where to investigate or follow up.
That state-level detail is especially important for the national system we support. Facilities and state systems submit information, state and regional staff review it, and other users retrieve it for their own work. A national total can show steady activity even when one state’s information has stopped changing, so we need to be able to move between the overall picture and the individual programs and states behind it.
I wanted that investigation to be part of our normal operational view, with less time spent logging into different systems and assembling reports. Having the information together should also help us reach out with something specific to discuss, whether that is a period with missing updates, a growing review queue, or an API operation that keeps failing.
Building Mission Control Around the Operation
We organized Mission Control so the team can begin with the overall status and move into a particular application, state, or type of activity. Development, preproduction, and production have separate views, which lets us check a release before it goes out as well as follow what is happening in the live system. The overview brings together application health and deployed versions, with more detailed views for events, statistics, states, support requests, and upcoming work.
Deployments and operational events appear on a shared timeline, which lets us compare a change in behavior with releases, alarm transitions, database events, and data loads. From a deployment, we can follow the record back to the code change and continue the investigation in the tools that hold the underlying detail. We still need those tools, but we can approach them with a better idea of what we are looking for.
State Activity and Data Freshness
For the state view, we brought review queues and data freshness together with program statistics so the team can see where activity needs a closer look. Support information is available alongside those views, including open requests, their age, and the pattern of requests being created and resolved.
We track both when our reporting information was refreshed and when the underlying state data changed, so the team can assess freshness at both points in the process.
The interpretation also depends on the program and its reporting schedule. Some programs receive frequent updates, while others have long periods with little expected activity; a pending submission may also be waiting for a valid reason. We use the view to identify where to investigate, then look at the program and reporting period before deciding whether to contact someone.
API and Website Usage
We kept the differences between our APIs visible because industry, regulatory, and public users interact with the system in different ways. For the public API, we look at endpoint usage, response times, failures, and activity by API key; regulatory integrations also need detail about reads, writes, program modules, and state attribution.
Even assigning activity to a state requires a clear definition, since the caller’s state may be different from the state whose records they requested. Public API keys may have no state association at all, so a common screen still needs to preserve the meaning of each measure.
We included website searches, page views, and downloads as well, since those are other ways people use the information the system collects. Taken together, these views help us understand incoming submissions and outgoing use, including where partners are having trouble and which services people rely on.
Preparing for Upcoming Work
The Upcoming view brings in scheduled deployments, the next production release and its notes, certificate expirations, access-key age, and pending database maintenance. The deployment calendar remains the source for the schedule, and Mission Control reads it so the team can see planned work without maintaining the same information in two places.
Keeping the Collection Process Manageable
We collect application statistics on a regular schedule and make the summaries available to Mission Control. This keeps routine reporting work separate from someone opening a page and helps keep the collection process manageable as usage grows.
The interface shows when information was refreshed so the team can judge whether it is current enough for the investigation. Scheduled statistics help us follow changes in activity over time, while live monitoring and alerts handle problems that need an immediate response. If a source is unavailable, that needs to be clear to the person using the view.
Supporting a Growing National System
As more daily work depends on electronic exchanges, uptime and response time become a larger part of supporting the system. Our ability to identify a problem and coordinate with the affected partners needs to keep up with that use.
I would like us to support that growth without a corresponding increase in the manual work of checking systems, running queries, and preparing reports. Bringing routine operational information together is one way to reduce that effort and help control the cost of supporting a larger system.
Mission Control is still evolving as we add sources and more statistics become available through application releases. My view is that it earns its place when the team can use it to understand which part of the national operation needs attention and work through the issue with the people involved. As more of the work becomes electronic, I want us spending our time helping states and partners use the system successfully, with less effort required to piece together what is happening.
