According to President Biden, it has handled 8 million applications so far.
The launch marks yet another high-profile, high-stakes launch of a government digital service, coming months after the debut of covidtests.gov, the site for ordering COVID-19 test kits through the mail. In each case, a cloud of uncertainty hung over the site prior to launch, owing to continued skepticism that government is capable of building and operating such services, and that they can meet the expectations of the public.
As we did with covidtests.gov, let’s take a quick look and see what we can learn about the implementation of this application from a systems design perspective. We’re just looking at what we can see publicly, with no special access or insight into what’s going on under the hood.
Many people have noted that the entirety of the application is a single web page with a brief form. The simplicity of the application has become one of the most commented on features of this program. Indeed, loading the page, we see a relatively simple page, with a few eligibility criteria up top, leading into the form itself.
Taking a look at the HTTP response headers for the application page, we note “server: AkamaiNetStorage”.
Akamai, a major CDN and network service provider, offers a NetStorage product, which serves static web content, such as HTML pages like the student aid application, from a set of computers distributed geographically over the world, which allows for fast delivery to end users’ browsers and resilience against spikes in demand. Serving a web page directly from an object store like NetStorage is a common tactic: allow a big, seasoned infrastructure provider like Akamai to handle traffic and absorb threats like DoS attacks, keeping requests off an origin server, or avoiding hosting an origin server altogether (in the case of static content like this).
Can DNS tell us more about this page and the overall studentaid.gov site?
It looks as if the entire studentaid.gov site is being served from Akamai. This tells us that HTTP requests to studentaid.gov are handled by web servers at Akamai, which then look up content by URL path like /debt-relief/application in NetStorage.
Now that we know how the application is served to us, let’s see if we can learn anything about how the application is submitted. What server is handling the form submission, and how is it doing that?
Typically a web form is implemented in HTML with the <form> element. The debt relief form is not contained within such an element, which means the Federal Student Aid team is handling the submission manually via JavaScript, as opposed to using the browser’s built-in <form> facilities. This makes it a little more work to figure out what URL the form’s data is being submitted to, as we can not merely look at the action attribute on the <form> element.
To see where and how the form is posted, I’ll need to fill the form out. Since I am myself not eligible for the debt relief program, I didn’t want to fill it out with my personal information. I decided to fill out the form with obviously fake dummy data to pass the validation checks, click on the submit button, and intercept the form submission before it goes out over the network, preventing it from actually reaching the FSA servers. This means I can’t personally inspect the subsequent result page (i.e., “Thank you for your submission”), if any, since the request is never actually sent to the server, but since a result page tends not to be as interesting from an infrastructure perspective, I’m okay with that tradeoff.
In order to intercept the form submission, I set a global breakpoint on XHR events in the developer tools. Clicking the “Submit” button, a breakpoint was reached before the request was sent out on the network, allowing me to inspect the call stack and any local variables, including the destination URL and any form data.
This tells us that the server URL handling the form submission is https://user-api.studentaid.gov/debt-relief-api/submit-attest-form. That domain name is new to us, let’s look it up.
edgekey.net is a domain name owned by Akamai. The apig- bit indicates that the server that answers this name is probably part of Akamai’s API Gateway product. This is what we might expect from a part of the submission that handles form submissions from user requests.
Notice the submit-attest-form bit of the URL. Those words comport with our understanding of the program, that the borrower is attesting they are who they say they are. Where previous applications for benefits would have identity verification up front — by, for example, requiring the user to create an account and submit proofs of their identity against a provider of verification services — the administrators of the debt relief program have chosen to prioritize simplicity of application, at the expense of a likely higher percentage of erroneous (typos, etc.) and fraudulent submissions.
Presumably, they’re accounting for this likelihood on the backend processing side, since ultimately the program boils down to consenting for Federal Student Aid to access income and relevant information about the borrower from other agencies. They are likely going to determine eligibility by matching the fields of the submitted record, like name and Social Security number. Either a person meets the criteria or they don’t, and that information is entirely on the government’s side. A person could fraudulently submit an application for debt relief masquerading as another person if they knew their Social Security number, but the worst thing that would happen in that case would be the applied-for person receiving a benefit they were ordinarily eligible for. The risk of abuse, from the program side, is therefore likely to be low.
The risk of abuse from a bad-network-actor perspective, though, is high. Lacking a login or other means of requiring a user be authenticated prior to accessing the site, there is nothing stopping someone or a bot from spamming the form submission endpoint in a DoS attack, attempting to either overwhelm the system from a network and compute resources perspective, or to bog down FSA employees with many spurious entries in their backend database to process.
Presumably, this is the kind of thing the Akamai API Gateway is in place to help mitigate, and why Akamai in general was chosen. They are experienced in detecting and blocking, or at least reducing the harm, of DoS attacks. Exposing an application like this directly to the internet presents these and other serious risks, and it is prudent to front it with a service provider experienced in dealing with them.
Back to the submission. The local variables in the call stack frame also carry the payload for the form data.
{
"consentDtmEpochMillis":1666129852851,
"consentFlag":true,
"formStartDtmEpochMillis":1666129759626,
"locale":"en-us",
"source":"E",
"submissionDtmEpochMillis":1666130115605,
"timezoneOffsetMin":300,
"submittedDob":"1965-01-01",
"submittedEmail":"fake@example.com",
"submittedFirstName":"Fake",
"submittedFormerLastName":null,
"submittedLastName":"Fake",
"submittedMiddleInitial":null,
"submittedSsn":"123456789",
"submittedTelephone":"8885551212",
"uuid":"92ca6197-86f5-4e12-a6cb-c25bd6b0f8ce+1666130115606"
}
It’s a standard JSON object, nothing more or less than what was in the form itself, a simple serialization. From here, the data is likely persisted to a database at FSA somewhere (could be a cloud service provider, or it could be an on-premise system — at this point we don’t know any more about the internals). What we know about the program from public reporting is that records will be processed gradually over coming weeks, in some backend workflow at FSA. The key point here is that the user is not waiting online for a response, like they might be for a different program.
This choice radically simplifies the architecture and implementation of the debt relief program website, as none of the machinery needed to build a system that responds with low-latency to the user is required, such as linking disparate government databases and all the caching and redundancy — read, high-level, sophisticated digital service operations — necessary therein.
Like covidtests.gov, the FSA debt relief website boils down to a simple form, served quickly from robust, battle-hardened infrastructure that the government does not own or operate itself. The form submission targets a backend handler designed for heavy demand, which whisks the data away to be processed later, while the user moves on and expects to receive the benefit at some point in the future.
Compared with a system like, for example, HealthCare.gov, in which eligibility determination and enrollment, and all the intermediate steps required, are conducted synchronously while the user is at their browser waiting for a response, this is a dramatically less complex approach. The major tradeoff is that the user does not know, at the point of submission, whether they are eligible for the benefit. But it affords the agency the flexibility to reach out asynchronously, via email most likely, if they need to follow up with additional information — think of it as a manual version of form validation.
For a program that needed to be stood up very quickly, in the span of a few months, there was likely not the time required to build such an automated system, so distilling the problem down to a simple form submission allowed them to launch a site and have it work for users, pushing the processing question down by deferring it. This is arguably a sensible approach for government digital services in general, as it starts simple, proves out the core of the program, and gives the agency more information and time to expand if desired, while de-stressing the effort of launching a new service under intense scrutiny.
