I take great inspiration from “The Joel Test”. When it was written in 2000, most of the practices included were considered unnecessary or time-consuming. But good ideas presented plainly have a way of spreading, and nearly everything in the original article is now commonplace.

Joel Spolsky wrote the test as a helpful guide for software engineers when they go out and interview. It’s very difficult to know the quality of a software engineering organization through the standard back and forth of an informal interview. These questions can quickly be answered ‘yes’ or ‘no’. I’ve been in the job market myself recently and found it very difficult to assess the quality of analytics in an organization.

Remember, as with the original Joel Test, answering yes to all these questions doesn’t guarantee success, but it gives you much better odds. Lots of no’s is a sure sign that this will be a difficult workplace.

  1. Are all the data transformations used in the analytics system in source control? I’m amazed at how many organizations will write stored procedures without any requirement that they go through source control. You must guarantee the system’s state to guarantee how anything will function. This is entry-level sloppiness that will cause late nights and hard-to-find bugs. Maybe you’d have an excuse a few years ago, but modern transformation frameworks give you all the tools you need.
  2. Does this source control system run automated tests when code is checked in? There’s a lot to be said about the kinds of tests that are useful in a data system, but it’s sufficient to say if you aren’t running any tests at all, you are shipping low-quality code.
  3. Can changes to the analytics system be deployed via CI/CD? Manual deployments are still typical in data. Having developers make changes manually to the production system means you have no guarantees about what is in production.
  4. Can analysts write queries in their own isolated sandbox environment? With modern cloud services, writing and executing transformations in an individual sandbox environment should be trivially easy. A shared ‘Staging’ area is insufficient as shared resources will be inconsistent, and there is no guarantee that another developer won’t make changes that produce side effects.
  5. Is there a staging area of the analytics system where end users can see the results of transformations before they are pushed to production? Similarly, if there isn’t a staging area to demo changes to the data before pushing to production, the production area becomes a test environment.
  6. Are data quality tests run on production assets? Many analytics teams will wait for end users to tell them their data is wrong. A higher-quality data team will continually add regression tests and check for uniqueness or known values in the production system. End users may point out business logic changes, but SQL errors or join condition bugs should be caught by data quality tests.
  7. Is there a written on-call process for production incident resolution? This is another basic best practice often ignored on data teams. Not having an on-call procedure ensures that whoever answers messages first becomes permanently on-call. Production incident resolution should be a shared responsibility in any team, and workflows need to be standardized.
  8. Is there a process for deleting and removing analytics assets no longer in use? Any system that only grows and never removes data will eventually become bloated, expensive, and unmaintainable. Demanding that data engineers ask everyone at the company if they are using something is also a non-starter. Having an agreed-upon process for deprecating old pipelines is the only way to maintain the system.
  9. Can an end user look up information about assets in the analytics system without asking a data analyst? A catalog or some type of metadata maintenance is necessary if the system is to be truly ‘self-serve.’ Anything less and analysts will be constantly chasing down information that could have existed in documentation. It doesn’t have to be complicated, but it does have to answer the question, “What data do we have about ‘X’?”
  10. Can an end user, in the visualization tool, view a metric total, then split it subgroups (region, user, etc), and then look at the individual rows that comprise that total? Going from high-level totals down to individual rows of data is the most basic business intelligence task there is. Without it, business users can’t answer any of their data questions. They will then go back to data analysts to seek out the problematic data points that their BI tool of choice should have been able to surface.

Analytics engineering is improving every year, but many organizations still operate at a 2 or lower. Getting your score to a 4 is trivially easy with most transformation frameworks. Getting to an 8 or higher requires serious design of how the system will work. If you score a 10, let me know so I can send you my resume.