A few months ago, we launched a public website that makes hazardous-waste site information easier to search and understand. From the beginning, one of our ideas for the Hazardous Waste Information Platform, or HWIP, was to make the same data available through an API. Researchers, industry partners, and software developers should be able to use public data without manually working through a website one page at a time.
The endpoints already existed because the web application used them. It would have been easy to document those endpoints, call them public, and move on.
We decided to wait.
Publishing an API creates a different kind of commitment. We needed to think through how people would find and understand the services, how much traffic the system could reasonably handle, what would happen when someone made an expensive request, and how we would communicate with consumers when something changed. The data was already public, but that did not mean the service could be unmanaged.
We recently finished that work and released the API. The most useful lessons came from the parts that initially looked like small details.
Public Does Not Mean Unlimited
While we were still thinking through the design, I had a casual conversation at an Exchange Network working meeting with someone familiar with EPA’s Air Quality System API, which is part of AirData. They mentioned that the API had seen a lot of abuse. Some of it may have been intentional, but much of it was probably ordinary users making more requests than they realized. A loop runs too often. A client retries without backing off. Someone asks for far more data than the application was designed to return in one request.
The intent does not matter much to the server. A well-meaning script can consume the same compute as a bad actor.
That conversation reinforced something we already suspected: request volume was only part of the problem. One very broad geospatial search or one enormous page of results could be more expensive than many small requests. We needed limits on page sizes and search radii as well as limits on how frequently someone could call the service.
AWS Web Application Firewall protects the application at the edge, but the application understands the request. It knows which endpoint is being called, which API key is involved, and how much work the request may create. We added controls inside the application so it could apply those rules more precisely and return a useful response when a caller reached a limit.
That last part matters. A throttled request should not look like a mysterious server failure. The
API returns the proper 429 status, a consistent error body, and a Retry-After header telling the
caller when to try again. For a developer—or an automated client—that difference turns a failure
into something the client can handle.
The API Key Was Not a Security Boundary
We resisted requiring an API key for a while. The data is public, and our goal was to reduce friction. We did not want someone to create an account, manage a password, request roles, or work through a complicated authorization process just to retrieve information that was already available on the website.
At the same time, completely anonymous traffic left us with no practical way to operate the service. If one caller created a problem, we needed to identify and disable that key without affecting everyone else. If an API changed, we needed a way to communicate with the people using it.
We settled on a small registration process: first name, last name, email address, and an optional organization. The user verifies the email and receives a key. That is enough to associate traffic with a real contact without turning access to public data into an account-management project.
The key is an operational control, not a confidentiality boundary. It gives us a contact handle, a throttle bucket, and an off switch.
We still treat it like a credential. The full key is shown only once, and the application stores a protected hash rather than the key itself. Logs use a short prefix instead of printing the key. If someone loses a key, registering the same email again starts a verified rotation process rather than requiring a separate recovery system.
Minimal should mean small, not careless.
An Internal Endpoint Is Not Yet a Public Contract
The first version of the website already had useful endpoints. The browser called them every day. That did not make them ready for outside consumers.
A web application controls how it calls its own services. It knows which fields to send, which defaults to assume, and how many records to request. An outside developer does not share those assumptions. Neither does an automated agent trying to construct a request from the published documentation.
Pagination was a good example. Once external consumers began testing the API, we had to be much more precise about the starting position, page size, total count, and sort order. A paginated database query without a stable order can return duplicate records on one page and skip records on another. The API now adds a unique Site ID as a final sort value so the sequence remains stable across pages.
We also learned that documenting a default value does not necessarily make the application apply it. The OpenAPI description could say that a missing page size defaults to a certain value, while the JSON request still reached the application as null. The runtime behavior had to be enforced in the code, not merely described in the schema.
The same applied to request limits. Programmatic searches now have bounded page sizes and geospatial radii. When a caller asks for more than the supported amount, the application reduces the request to the permitted maximum. The public website can continue using the endpoints as it did before, while external callers receive the additional controls designed for automated use.
These were not large architectural changes. They were the work required to turn application plumbing into a predictable public interface.
Documentation Became Part of the Product
We used OpenAPI and Swagger to build the reference documentation directly into HWIP. It is styled to match the rest of the website, and users can read about a service, supply their key, and try a request without leaving the RCRAInfo HWIP Public APIs page.
The interactive console follows the same rules as a real external caller. Even though it runs inside our website, it still requires an API key and receives the same limits and errors. Otherwise, the documentation would demonstrate behavior that someone writing a script could not reproduce.
Generating the reference was only the beginning. A tool can list endpoints and fields, but it cannot decide whether the language makes sense to the intended audience.
Inside RCRAInfo, we have historically used terms such as handler ID. Industry users are more likely
to know that value as a Site ID, so the public API uses siteId and explains the relationship. A
boolean flag such as operatingTsdf also needs more than a data type. A consumer needs to know that
TSDF means treatment, storage, and disposal facility and what a true value represents. Reference
codes need descriptions and a clear way to find valid values.
That translation took more time than expected. It was also some of the most important work.
Documentation now has another audience as well. Developers increasingly describe what they want to an AI assistant or agent and ask it to construct the request. We have already heard from a consumer using the API that way. Clear property names, realistic examples, predictable errors, and documented relationships between lookup values and search fields make that possible.
An agent can read an OpenAPI document very quickly. It can also misunderstand an ambiguous field very quickly. The growing use of these tools makes precise documentation more important, not less.
The API Should Not Solve Every Data-Access Problem
HWIP now provides three different ways to work with the data.
For researchers who need the complete dataset for deep analysis, a full CSV export is the right tool. For users who want a smaller dataset but still prefer a file, the website provides filtered downloads. The API is intended for targeted, automated questions: search for matching sites, return lightweight summaries, and retrieve more detailed sections for a particular site when needed.
That separation influenced the API design. Search results do not return every available fact about every matching site. They return a smaller object with the Site ID, name, location, and important status flags. A consumer can then call the more granular site-detail services for permitting, waste codes, activities, ownership, manifests, or other information.
Trying to make the API replace the bulk exports would have produced a worse API and a less efficient way to download the same data. Different access patterns deserve different tools.
A Public API Is an Operational Promise
The endpoints were the easy part because they already existed. Releasing them responsibly meant deciding what behavior we were willing to support as a public contract.
We needed enough identity to operate the service without creating unnecessary barriers. We needed limits that protected shared compute without making normal use frustrating. We needed errors that helped automated clients recover. We needed documentation that translated the system’s internal language into terms people—and increasingly their tools—could understand.
My view is that this is the right balance for public government data. Do not add a heavyweight security model when the data does not require one. But do not confuse public access with unlimited, anonymous, and unexplained access either.
Start with the actual risk. Add the smallest controls that let the service remain reliable. Then treat the documentation and behavior with the same care as the code.
That is enough for a useful first release, and it gives us room to learn from how people actually use it.
