An internal essay about our engineering culture once said:
We should make it easy for engineers to build internal Ad Hoc applications — anything from Slack bots to assist with managing the homework queue to fun badges for employee profiles. Democratizing this process will open the doors for internal apps that could catch on, whether for productive purposes or fun ones.
That’s where our “Pizza Platform” comes in: a streamlined platform that enables developers to more easily build and deploy internal-facing apps. Internal authentication, cloud billing, IT compliance and security concerns, and a host of other details are simply taken care of by the platform. By leveraging various Google Cloud Platform (GCP)’s features and building a few key pieces of developer-supporting tooling, we provide a simple platform that gives developers freedom to focus on their app’s functionality. This all manifests our concepts for platform design of reuse of services and separation of responsibility.
This platform strategy for the Pizza Platform has had three positive effects:
- Developers are more willing to build solutions to company problems they see, whether that’s a toy program for the company Slack or a useful application to help manage infrastructure for a billable project.
- Engineers get hands-on experience with Google Cloud Platform (GCP), which comes up less frequently in our government client work than other cloud infrastructure providers.
- Pizza Platform builds in good governance practices, and gives us better controls for easier security and controlled cost.
I’d previously experienced Google’s internal, App Engine-based platform provided to its engineers for a similar purpose. The Google internal platform enabled employees to create applications ranging from niche debugging tools to broad, fun parts of company culture. Their approach was a good comparison for ours, and while our system had similar goals of ease of use and apps being securely locked down to employees, two constraints were different:
- We needed the project to be budget-efficient, allowing for a relatively consistent cloud spend even as the number of applications increased.
- We had to make do with publicly available cloud features. Since Google owned the entire platform, their version of this solution used internal networking features that we didn’t have available as public customers.
We considered a variety of tools across different cloud providers, but decided to build our Pizza developer platform on Google Cloud Platform (GCP) because it most closely matched our constraints. Specifically:
- GCP’s Identity-Aware Proxy (IAP) makes it easy to secure applications to be employee-viewable only, since Ad Hoc uses Google Workspace for our primary corporate identity for most of our day-to-day work.
- App Engine Standard makes it easy to deploy applications that won’t be racking up runtime costs while not in use, allowing us to develop this platform on a more consistent budget.
As a bonus, choosing GCP for this platform gives us an easy way to build some hands-on experience with it among our engineers, who are typically more experienced with AWS.
In addition to IAP and App Engine, our solution uses a mix of GCP’s resource hierarchy, some DNS trickery, and a small custom admin application that ties the pieces together. We’ll dive into the technical weeds in a moment. First, we’ll discuss the governance approach to the platform.
Since the goal of the internal employee application platform is to be easy to use and low in red tape, it doesn’t have the governance requirements that other, more involved platforms typically include. We were able to leave off rules about technical stack, visual design consistency, and accessibility checks. Instead, we focused on managing cost and security.
Here, it was especially important to use a carrots and sticks approach to platform governance.
The carrot: built in starter kits
We encouraged developers to use the platform securely and cheaply by providing simple starter kits in a variety of standard stacks. Whether the person wants to work in Python + Django or Node + Express, there’s a ready-made starter app available, complete with deployment scripts that create a GCP project, then deploy and configure the application as expected.
For example, in order to keep costs low, our starter kits default to using the App Engine standard environment, limiting spending on infrequently-used applications to pennies per day. (We discuss this feature of the platform more in the technical explanation section, later.) And for locking the applications down to employees, the starter kits immediately turn on the Identity-Aware Proxy (IAP) after deploying the first application.
We love the kits because they set up applications in the low-cost, secure Cloud configuration that we’d prefer. This is how we “make the right path the easy path”, helping ensure that engineers are using the platform in the ways we prefer. It’s not a “preventing malicious actors” kind of firm guardrail, but rather a well-paved path that makes it easy for our engineers to get from 0 to 1 quickly and with the infrastructure configuration we expect to see.
The stick: automated notifications
It’s difficult to grant engineers full access to GCP projects and prevent them from spending too much or misconfiguring their application. But we can build ways to detect when that has happened, and alert those involved to quickly course-correct.
For cost controls, we added billing alerts on the main billing account to identify when costs start to balloon quicker than expected. While the starter kits default to low-cost runtimes, engineers sometimes spin up other services outside what we have budgeted for this platform. Alerts help us quickly identify issues, such as users starting up a more expensive deployment and forgetting to tear it down.
Similarly for security, we can’t enforce that IAP remains on. But our management application is able to periodically scan applications in the GCP resource folder containing all Pizza Platform GCP projects, ensuring that IAP is enabled for any web applications, and notifying us on Slack if there’s a problem.
Overall, the detailed “getting started” guide and application starter kits help with making the right thing the easy thing, and automated notifications help alert us when something isn’t right.
The technical details
Let’s dig into the specifics of how this is all wired up! We’ll talk about each component and how it fits into the bigger picture.
App Engine’s standard environment acts as a simple place to deploy applications with low-cost and low-infrastructure complexity. Even though it’s been around for over a decade, App Engine’s recent “second-generation runtimes” made it much more useful, by removing the dependency on Google’s custom framework and adding support for many more language runtimes.
App Engine has several great features that make it ideal for hosting employee-built applications:
- The “scale-to-zero” model, where applications that aren’t in active use don’t remain running, and therefore don’t cost anything most of the time. The tradeoff is having to experience the application’s cold-boot time, but given the experimental nature of an employee application playground, that’s an acceptable tradeoff for this use case.
- Support for custom domains, so that applications can be easily served via subdomains of a company domain. This includes an API for easy provisioning.
- SSL certificate auto-provisioning for those custom domains, letting us enforce that employee applications run over HTTPS.
Altogether, these features made App Engine an easy, low-cost place to host employee applications.
Identity-Aware Proxy (IAP)
Restricting this platform’s applications to be only accessible by employees is performed with GCP’s Identity-Aware Proxy. This has to be set up per application, and can’t be enforced by GCP policy.
Once enabled, permission can be granted to the organization through an IAM policy addition. The easiest way to grant to everyone in the organization is to create an “everyone” group alias using Google Workspace — see Google’s instructions for details. That account can be granted the
roles/iap.httpsResourceAccessor role on the project (or a containing folder) to allow everyone in the organization to view the application.
We used our
adhoc.pizza domain for this platform, with applications hosted at subdomains like
my-app.adhoc.pizza. Employees had long ago created a silly website at this domain based on a picture of our company logo turned into a pizza. Using this domain emphasized the playfulness of the platform and non-official nature of these applications.
Getting DNS working for this subdomain setup came in two parts: first, pointing our domain records at App Engine; second, having a mechanism to wire up custom domains to specific App Engine applications.
First, we delegated all subdomains to point at GCP, with a wildcard
CNAME record for
*.adhoc.pizza pointing to
ghs.googlehosted.com. Now, when a lookup is performed for
some-application.adhoc.pizza, the resolved IP address will point generically to GCP. This simplifies the infrastructure so that we don’t have to set up new DNS records for each application — though we could accomplish that with Cloud DNS if needed.
Second, we need GCP to know which subdomain goes with which application, so that GCP can route requests correctly. App Engine applications can be served from custom domains, which can be configured via API. Critically, the user requesting the custom domain be attached to an App Engine application must meet two criteria:
- They must be a verified owner of the domain in the Google Search Console.
- They must be an App Engine Admin on the GCP project containing the application.
The most secure way to meet these criteria is to create a new service account that can be granted these permissions, rather than granting domain ownership to many employees. We’ll call this account the “Domain Manager.”
To meet the first criterion, we added the Domain Manager service account to the Google Search Console as an owner of the
adhoc.pizza domain. This allows that service account to use any
adhoc.pizza subdomains when attaching custom domains to App Engine applications.
For the second App Engine Admin criterion, keep reading on.
IAM and Folder Structure
In order for the Domain Manager service account to be able to set custom domains on applications, it needs to have permissions to update all of the App Engine applications in this platform. This is where GCP’s folder structure comes in handy: we can grant it permissions on the Pizza Projects folder that contains the employee projects.
Following the principle of least privilege, we made a custom IAM role named “Pizza Domain Manager”, granting the following permissions:
Granting the domain manager service account this role on the Pizza Projects folder allows it to set custom domains on applications, as well as support scanning for security concerns as described earlier.
But how do we actually use this service account? Keep reading…
To successfully set custom domains on App Engine applications using the authority of the Pizza Domain Manager, we need a privileged application that can act as that account. So we built a small admin application that houses the Pizza Domain Manager service account, that lets employees do privileged configuration on their applications.
Specifically, there are two privileged actions that this admin application helps users do as requirements to launch:
- Setting custom domains on App Engine applications
- Attaching the company billing account to the project
Since this admin application has control over all employee Pizza projects, as well as the company billing account, it performs a few verifications before making any project changes:
- Verifying that the GCP project being modified is in the Pizza Projects folder
- Verifying that the user visiting this application (through IAP) is an owner on the project
If those criteria can be met, the admin application will happily proceed.
Getting Started scripts
Finally, as described above, we have starter kits with getting-started scripts and documentation that tie all this together from the point of the user. The script does the following steps, in order:
- Creates a GCP project in the Pizza Projects folder
- Enables the relevant GCP APIs (
- Points users to the admin application to connect the billing account for their project
- Deploys the starter application in the language of their choice to App Engine
- Sets up IAP on the application
- Points users to the admin application to pick a custom subdomain
And just like that, we’ve walked the user through an easily-deployed, company-billed, secured-to-the-organization internal application!
Our initial launches with engineers proved out our hypotheses, after some early iteration on usability: that this made it easy for developers to build internal-facing projects, that this could run applications at low cloud costs, and that it would encourage new projects by giving engineers somewhere supported to host them.
Since launching this, our engineers have used this platform for over a dozen projects, including a technical blogging site for engineers, a “tilde-style” hosting environment for static content, a search engine for our Google Drive-based content library, and a daily slackbot that posts new emoji to users. Empowering our engineers to be able to build and launch tools that solve problems makes us more effective as an organization.
Altogether, this project proves out our platform strategy: that solving common problems once, making them reusable, and focusing on developer experience can accelerate developers — and in this case, enables them to build and launch tools that otherwise would have never existed.