Creating a platform for employee-internal apps with GCP

An internal essay about our engineering culture once said:

We should make it easy for engineers to build internal Ad Hoc applications — anything from Slack bots to assist with managing the homework queue to fun badges for employee profiles. Democratizing this process will open the doors for internal apps that could catch on, whether for productive purposes or fun ones.

That’s where our “Pizza Platform” comes in: a streamlined platform that enables developers to more easily build and deploy internal-facing apps. Internal authentication, cloud billing, IT compliance and security concerns, and a host of other details are simply taken care of by the platform. By leveraging various Google Cloud Platform (GCP)’s features and building a few key pieces of developer-supporting tooling, we provide a simple platform that gives developers freedom to focus on their app’s functionality. This all manifests our concepts for platform design of reuse of services and separation of responsibility.

This platform strategy for the Pizza Platform has had three positive effects:

1. Developers are more willing to build solutions to company problems they see, whether that’s a toy program for the company Slack or a useful application to help manage infrastructure for a billable project.
2. Engineers get hands-on experience with Google Cloud Platform (GCP), which comes up less frequently in our government client work than other cloud infrastructure providers.
3. Pizza Platform builds in good governance practices, and gives us better controls for easier security and controlled cost.

Our constraints

I’d previously experienced Google’s internal, App Engine-based platform provided to its engineers for a similar purpose. The Google internal platform enabled employees to create applications ranging from niche debugging tools to broad, fun parts of company culture. Their approach was a good comparison for ours, and while our system had similar goals of ease of use and apps being securely locked down to employees, two constraints were different:

1. We needed the project to be budget-efficient, allowing for a relatively consistent cloud spend even as the number of applications increased.
2. We had to make do with publicly available cloud features. Since Google owned the entire platform, their version of this solution used internal networking features that we didn’t have available as public customers.

Our solution

We considered a variety of tools across different cloud providers, but decided to build our Pizza developer platform on Google Cloud Platform (GCP) because it most closely matched our constraints. Specifically:

1. GCP’s Identity-Aware Proxy (IAP) makes it easy to secure applications to be employee-viewable only, since Ad Hoc uses Google Workspace for our primary corporate identity for most of our day-to-day work.
2. App Engine Standard makes it easy to deploy applications that won’t be racking up runtime costs while not in use, allowing us to develop this platform on a more consistent budget.

As a bonus, choosing GCP for this platform gives us an easy way to build some hands-on experience with it among our engineers, who are typically more experienced with AWS.

In addition to IAP and App Engine, our solution uses a mix of GCP’s resource hierarchy, some DNS trickery, and a small custom admin application that ties the pieces together. We’ll dive into the technical weeds in a moment. First, we’ll discuss the governance approach to the platform.

Streamlined governance

Since the goal of the internal employee application platform is to be easy to use and low in red tape, it doesn’t have the governance requirements that other, more involved platforms typically include. We were able to leave off rules about technical stack, visual design consistency, and accessibility checks. Instead, we focused on managing cost and security.

Here, it was especially important to use a carrots and sticks approach to platform governance.

The carrot: built in starter kits

We encouraged developers to use the platform securely and cheaply by providing simple starter kits in a variety of standard stacks. Whether the person wants to work in Python + Django or Node + Express, there’s a ready-made starter app available, complete with deployment scripts that create a GCP project, then deploy and configure the application as expected.

For example, in order to keep costs low, our starter kits default to using the App Engine standard environment, limiting spending on infrequently-used applications to pennies per day. (We discuss this feature of the platform more in the technical explanation section, later.) And for locking the applications down to employees, the starter kits immediately turn on the Identity-Aware Proxy (IAP) after deploying the first application.

We love the kits because they set up applications in the low-cost, secure Cloud configuration that we’d prefer. This is how we “make the right path the easy path”, helping ensure that engineers are using the platform in the ways we prefer. It’s not a “preventing malicious actors” kind of firm guardrail, but rather a well-paved path that makes it easy for our engineers to get from 0 to 1 quickly and with the infrastructure configuration we expect to see.

The stick: automated notifications

It’s difficult to grant engineers full access to GCP projects and prevent them from spending too much or misconfiguring their application. But we can build ways to detect when that has happened, and alert those involved to quickly course-correct.

For cost controls, we added billing alerts on the main billing account to identify when costs start to balloon quicker than expected. While the starter kits default to low-cost runtimes, engineers sometimes spin up other services outside what we have budgeted for this platform. Alerts help us quickly identify issues, such as users starting up a more expensive deployment and forgetting to tear it down.

Similarly for security, we can’t enforce that IAP remains on. But our management application is able to periodically scan applications in the GCP resource folder containing all Pizza Platform GCP projects, ensuring that IAP is enabled for any web applications, and notifying us on Slack if there’s a problem.

Overall, the detailed “getting started” guide and application starter kits help with making the right thing the easy thing, and automated notifications help alert us when something isn’t right.

The technical details

Let’s dig into the specifics of how this is all wired up! We’ll talk about each component and how it fits into the bigger picture.

App Engine

App Engine’s standard environment acts as a simple place to deploy applications with low-cost and low-infrastructure complexity. Even though it’s been around for over a decade, App Engine’s recent “second-generation runtimes” made it much more useful, by removing the dependency on Google’s custom framework and adding support for many more language runtimes.

App Engine has several great features that make it ideal for hosting employee-built applications:

• The “scale-to-zero” model, where applications that aren’t in active use don’t remain running, and therefore don’t cost anything most of the time. The tradeoff is having to experience the application’s cold-boot time, but given the experimental nature of an employee application playground, that’s an acceptable tradeoff for this use case.
• Support for custom domains, so that applications can be easily served via subdomains of a company domain. This includes an API for easy provisioning.
• SSL certificate auto-provisioning for those custom domains, letting us enforce that employee applications run over HTTPS.

Altogether, these features made App Engine an easy, low-cost place to host employee applications.

Identity-Aware Proxy (IAP)

Restricting this platform’s applications to be only accessible by employees is performed with GCP’s Identity-Aware Proxy. This has to be set up per application, and can’t be enforced by GCP policy.

Once enabled, permission can be granted to the organization through an IAM policy addition. The easiest way to grant to everyone in the organization is to create an “everyone” group alias using Google Workspace — see Google’s instructions for details. That account can be granted the roles/iap.httpsResourceAccessor role on the project (or a containing folder) to allow everyone in the organization to view the application.

We can use the gcloud CLI to do all of this in our starter kit scripts, first creating an OAuth brand and OAuth client, then enabling IAP web, and finally granting the IAM role for access.

DNS

We used our adhoc.pizza domain for this platform, with applications hosted at subdomains like my-app.adhoc.pizza. Employees had long ago created a silly website at this domain based on a picture of our company logo turned into a pizza. Using this domain emphasized the playfulness of the platform and non-official nature of these applications.

Getting DNS working for this subdomain setup came in two parts: first, pointing our domain records at App Engine; second, having a mechanism to wire up custom domains to specific App Engine applications.

First, we delegated all subdomains to point at GCP, with a wildcard CNAME record for *.adhoc.pizza pointing to ghs.googlehosted.com. Now, when a lookup is performed for some-application.adhoc.pizza, the resolved IP address will point generically to GCP. This simplifies the infrastructure so that we don’t have to set up new DNS records for each application — though we could accomplish that with Cloud DNS if needed.

Second, we need GCP to know which subdomain goes with which application, so that GCP can route requests correctly. App Engine applications can be served from custom domains, which can be configured via API. Critically, the user requesting the custom domain be attached to an App Engine application must meet two criteria:

1. They must be a verified owner of the domain in the Google Search Console.
2. They must be an App Engine Admin on the GCP project containing the application.

The most secure way to meet these criteria is to create a new service account that can be granted these permissions, rather than granting domain ownership to many employees. We’ll call this account the “Domain Manager.”

To meet the first criterion, we added the Domain Manager service account to the Google Search Console as an owner of the adhoc.pizza domain. This allows that service account to use any adhoc.pizza subdomains when attaching custom domains to App Engine applications.

For the second App Engine Admin criterion, keep reading on.

IAM and Folder Structure

In order for the Domain Manager service account to be able to set custom domains on applications, it needs to have permissions to update all of the App Engine applications in this platform. This is where GCP’s folder structure comes in handy: we can grant it permissions on the Pizza Projects folder that contains the employee projects.

Following the principle of least privilege, we made a custom IAM role named “Pizza Domain Manager”, granting the following permissions:

• appengine.applications.get
• appengine.applications.update
• appengine.operations.get
• resourcemanager.folders.get
• resourcemanager.folders.list
• resourcemanager.organizations.get
• resourcemanager.projects.createBillingAssignment
• resourcemanager.projects.deleteBillingAssignment
• resourcemanager.projects.get
• resourcemanager.projects.getIamPolicy
• resourcemanager.projects.list

Granting the domain manager service account this role on the Pizza Projects folder allows it to set custom domains on applications, as well as support scanning for security concerns as described earlier.

But how do we actually use this service account? Keep reading…

Admin Application

To successfully set custom domains on App Engine applications using the authority of the Pizza Domain Manager, we need a privileged application that can act as that account. So we built a small admin application that houses the Pizza Domain Manager service account, that lets employees do privileged configuration on their applications.

Specifically, there are two privileged actions that this admin application helps users do as requirements to launch:

1. Setting custom domains on App Engine applications
2. Attaching the company billing account to the project

Since this admin application has control over all employee Pizza projects, as well as the company billing account, it performs a few verifications before making any project changes:

• Verifying that the GCP project being modified is in the Pizza Projects folder
• Verifying that the user visiting this application (through IAP) is an owner on the project

If those criteria can be met, the admin application will happily proceed.

Getting Started scripts

Finally, as described above, we have starter kits with getting-started scripts and documentation that tie all this together from the point of the user. The script does the following steps, in order:

1. Creates a GCP project in the Pizza Projects folder
2. Enables the relevant GCP APIs (appengine.googleapis.com, cloudbuild.googleapis.com, cloudresourcemanager.googleapis.com, and iap.googleapis.com)
3. Points users to the admin application to connect the billing account for their project
4. Deploys the starter application in the language of their choice to App Engine
5. Sets up IAP on the application
6. Points users to the admin application to pick a custom subdomain

And just like that, we’ve walked the user through an easily-deployed, company-billed, secured-to-the-organization internal application!

Outcomes

Our initial launches with engineers proved out our hypotheses, after some early iteration on usability: that this made it easy for developers to build internal-facing projects, that this could run applications at low cloud costs, and that it would encourage new projects by giving engineers somewhere supported to host them.

Since launching this, our engineers have used this platform for over a dozen projects, including a technical blogging site for engineers, a “tilde-style” hosting environment for static content, a search engine for our Google Drive-based content library, and a daily slackbot that posts new emoji to users. Empowering our engineers to be able to build and launch tools that solve problems makes us more effective as an organization.

Altogether, this project proves out our platform strategy: that solving common problems once, making them reusable, and focusing on developer experience can accelerate developers — and in this case, enables them to build and launch tools that otherwise would have never existed.