Replacing Sawzall — a case study in domain-specific language migration



In a earlier put up, we described how knowledge scientists at Google used Sawzall to carry out highly effective, scalable evaluation. Nevertheless, during the last three years we’ve eradicated virtually all our Sawzall code, and now the area of interest that Sawzall occupied in our software program ecosystem is usually crammed by Go. On this put up, we’ll describe Sawzall’s position in Google’s evaluation ecosystem, clarify a number of the issues we encountered as Sawzall use elevated which motivated our migration, and element the methods we utilized to realize language-agnostic evaluation whereas sustaining robust entry controls and the flexibility to write down quick, scalable analyses.

Any profitable programming language has its personal evolutionary area of interest, a set of issues that it solves unusually properly. Typically this area of interest is created by language options. For instance, Erlang has robust instruments for setting up distributed programs constructed into the language. In different circumstances, options equivalent to normal libraries and a language’s group of customers are extra essential — the primary motive that R is a good language for statistics is that it’s extensively utilized by statisticians and has an enormous number of helpful statistics libraries. To be able to perceive the explanation for Sawzall’s decline, we have now to first perceive the area of interest that it occupied in Google’s software program ecosystem.

Our previous discussion of Sawzall centered on one in every of Sawzall’s largest strengths — it makes it simple to write down highly effective evaluation scripts shortly for duties like computing statistical aggregates or computing a Poisson bootstrap. As such, it’s nice for writing fast one-off evaluation code and iterating on it as we come to a greater understanding of the information. The identify of the language is suggestive — the precise bodily Sawzall® (trademark Milwaukee Device) that the language is called after is a flexible hand software that may make fast work of logs.

Determine 1: A bodily Sawzall sawing bodily logs.

Sawzall additionally has essential strengths in one other crucial space — entry management and auditing. The enter to evaluation jobs typically contains personally identifiable info like IP addresses, and there are strict guidelines that restrict what analysts can do with this knowledge. We want to have the ability to reply a number of questions on any evaluation earlier than it runs:

  • Ought to this analyst have entry to this knowledge in any respect? 
  • If they need to have entry, which fields ought to they have the ability to learn? Our enter data are protocol buffers, and we’ve annotated the fields of our logged protos to point which of them might comprise delicate knowledge (e.g. a consumer’s IP tackle) and which of them are innocuous (e.g. the period of time it took to course of a request). Studying delicate fields requires a powerful justification.
  • In the event that they’re studying delicate fields, what code are they really operating? We wish to have the ability to audit the precise code that’s getting used to do any delicate evaluation.

Briefly, we wish fine-grained management over who has entry to knowledge, and visibility into what they’re doing with it. Sawzall supplied an excellent answer to all these points. We ran a centralized service known as Sawmill that managed all Sawzall evaluation on our logs.

Determine 2: Within the Sawmill execution setting, customers ship their Sawzall evaluation scripts to Sawmill Server, which performs authorization, applies entry filters, and launches a MapReduce job on the consumer’s behalf in a restricted execution zone the place the consumer isn’t allowed to run arbitrary binaries.

You possibly can ship your Sawzall code to Sawmill, and it could just be sure you have entry to the information that you just wish to analyze. In the event you do, it could add some code to the start of your script to filter out any fields that you just don’t have entry to and file your script for auditing functions. Then it could begin a MapReduce which runs your Sawzall code on every employee. Since your Sawzall code runs inside a sandbox, it can’t get entry to the uncooked, unfiltered logs knowledge. It solely sees filtered enter.

Issues with Sawzall

This setup is nice for entry management and auditing, nevertheless it additionally creates some issues. Since we’re counting on the Sawzall sandbox to implement our entry insurance policies, we have now to be sure that un-sandboxed code doesn’t run alongside our Sawzall evaluation. If the evaluation may name unsafe code (e.g. user-controlled C++ capabilities), it may bypass our sandbox and browse delicate fields earlier than they’re filtered. Sawzall does present a approach of calling capabilities written in different languages as if they had been Sawzall capabilities. These capabilities are known as intrinsics, they usually present a bridge between Sawzall and the remainder of the world.

At Google, intrinsics had been generally used to supply an interface to giant, complicated C++ libraries and to work together with exterior companies through RPC. Nevertheless, since intrinsics present a technique to get away of the Sawzall sandbox, every one wanted to be fastidiously vetted for security earlier than it could possibly be whitelisted to be used. As increasingly folks began utilizing Sawzall, the demand for brand new intrinsics grew shortly and have become a standard level of friction for interoperability with companies or libraries from different groups inside Google.

The necessity to stop arbitrary un-sandboxed code from interacting with Sawzall evaluation additionally put robust constraints on the execution setting the place evaluation runs. For instance, if a consumer may run arbitrary applications alongside their sandboxed evaluation, they might have the ability to examine the reminiscence of their Sawzall program and extract unfiltered knowledge that they shouldn’t have entry to. To keep away from this state of affairs, we needed to reserve compute assets for logs evaluation with restrictions on what sorts of applications will be run and who can launch them, making our evaluation infrastructure a lot much less versatile.

These issues had been manageable when Sawzall occupied a small, well-contained area of interest. However because the group utilizing Sawzall turned bigger and extra numerous, the issues turned extra acute and the constraints of a domain-specific language turned extra essential.

Sawzall could also be a wonderful hand software, however many groups at Google got here to want one thing extra akin to heavy industrial equipment. Sawzall is at its greatest for small, centered analyses. Whereas Sawmill itself is giant, refined infrastructure that enables Sawzall evaluation to scale up and course of huge quantities of knowledge effectively, Sawzall just isn’t well-suited for constructing giant built-in pipelines with refined testing and launch administration. Groups constructed their core enterprise logic in Sawzall, however with out an object system or any help for user-defined interfaces it turned very onerous to handle a big codebase. These issues aren’t distinctive to Google — different firms which have adopted Sawzall for his or her analytics wants have reported similar difficulties.

Sawzall possible may have continued as a small, area of interest language, nevertheless it was sufficiently helpful that folks needed rather more out of it, and people wants grew past what the language and its related entry management and execution mannequin may present.

Language-Agnostic Evaluation

Step one towards fixing these issues was eradicating the tight hyperlink between entry controls on logs knowledge and the Sawzall execution mannequin. By putting these controls exterior of the Sawzall sandbox, we will open the door for evaluation written in any language with out weakening our means to regulate entry to delicate knowledge.

If we permit customers to run arbitrary un-sandboxed code on the information, we have now to alter the mannequin for a way we filter out delicate fields. As soon as the information will get to the consumer’s binary, it’s too late for filtering. We due to this fact want a separate service that proxies entry to the uncooked knowledge and enforces our entry management insurance policies earlier than the information ever makes its technique to analysts.

We’ve constructed simply such a system, known as the logs proxy. It supplies a language-agnostic interface for studying logs knowledge, and it applies all the required filtering logic earlier than sending the information alongside to shoppers. There are a number of fascinating wrinkles to this course of (for instance, what if I wish to do a be part of that’s keyed by a discipline that might be filtered out?), and we’ve needed to clear up some robust efficiency optimization issues to deal with the size of research at Google, however the elementary thought could be very easy.

Determine 3: Within the logs proxy execution setting, consumer evaluation code by no means has direct entry to logs knowledge. No restricted zone is important, as a result of the logs proxy filters out delicate fields earlier than they’re accessible to evaluation code.
Because the logs proxy decouples our knowledge entry coverage from the programming language used for evaluation, particular person groups now have extra freedom to decide on the language that most closely fits their wants. Nevertheless, since evaluation libraries can typically get very difficult, and a number of groups typically share frequent knowledge sources, there’s an financial system of scale in selecting a standard language for many evaluation.

At Google, most Sawzall evaluation has been changed by Go. Go has the benefit of being a comparatively small language which is straightforward to study and integrates properly with Google’s manufacturing infrastructure. Quick compile instances and rubbish assortment make Go a pure match for iterative growth. To ease the method of migrating from Sawzall, we’ve developed a set of Go libraries that we name Lingo (for Logs in Go). Lingo features a desk aggregation library that brings the highly effective options of Sawzall aggregation tables to Go, utilizing reflection to help user-defined varieties for desk keys and values. It additionally supplies default conduct for establishing and operating a MapReduce that reads knowledge from the logs proxy. The result’s that Lingo evaluation code is commonly as concise and easy as (and typically less complicated than) the Sawzall equal.

For example, think about the spam classification activity from an earlier post on Sawzall on this website, the place the aim is to measure the impression of two variations of a spam classifier on completely different web sites. Right here’s how that code appears in Lingo:
package deal spamcount

import (


// For every website, observe whether or not or not it’s spam in accordance with

// the outdated and new spam scores.

kind SpamCount struct {

  Previous int
  New int
  URLs int

func spamCount(rating float) int {
  // A file with a spam rating above 0.5 counts as spam.

  if rating > 0.5 {
    return 1
  return 0

// stats is a sum desk with string keys (website identify), and

// SpamCount values (the outdated and new spam counts and whole

// rely of URLs).

var stats = desk.Sum(“my_stats”, “website”, SpamCount{})

func Mapper(w *webpage.WebPage) {
  // Every file is a protocol buffer of kind WebPage, which
  // has a url discipline which the spam package deal can classify.


  stats.Emit(websites.SiteFromURL(w.GetUrl()), SpamCount{

    Previous: spamCount(spam.SpamScore(w.GetUrl())),
    New: spamCount(spam.NewSpamScore(w.GetUrl())),
    URLs: 1,

The construction of this Lingo program is similar to its Sawzall equivalent, due to the desk library. It outputs a desk of summed spam scores, keyed by website names. The desk library makes use of the identical output encoding as Sawzall, so the output of this program is byte-for-byte similar to its Sawzall equal. This vastly simplifies the method of migrating away from Sawzall for groups.

The good thing about this work is that logs evaluation is now rather more versatile and higher built-in into Google’s broader software program ecosystem. The logs proxy has decoupled the selection of language from the execution and entry management mannequin for evaluation, which provides groups the liberty to make their very own willpower about what language most closely fits their wants.


Shifting away from Sawzall has been an enormous job. Partly that’s as a result of Sawzall was fairly profitable at its authentic aim — make it simple for analysts to write down fast, highly effective evaluation applications. Consequently there was a number of Sawzall code to be migrated. Nevertheless, Sawzall was in some methods a sufferer of its personal success. There’s a pure stress for any domain-specific language between staying extremely centered on its drawback area and rising to accommodate the wants of customers who wish to stretch the language in new instructions. Sawzall’s growth was formed by this stress from the very starting — early designs didn’t even embrace the flexibility to outline capabilities, however capabilities had been shortly added when it turned obvious that the language couldn’t meet customers’ wants with out them. Over time, many extra options have been added. However because the language grows, the rationale for utilizing a website particular language moderately than a common goal language turns into increasingly diluted.

Thankfully, we’ve discovered that with fastidiously designed libraries we will get a lot of the advantages of Sawzall in Go whereas gaining the benefits of a strong general-purpose language. The general response of analysts to those adjustments has been extraordinarily constructive. At this time, logs evaluation is without doubt one of the most intensive customers of Go at Google, and Go is the most-used language for studying logs by means of the logs proxy. And for customers preferring a special language, the logs proxy supplies a language-agnostic technique to learn logs knowledge whereas complying with our entry insurance policies. Wanting ahead, we will’t predict precisely what route logs evaluation at Google will go subsequent, however we do know that its path gained’t be constrained by our selection of programming language.


Source link

Write a comment