When you’re creating brand new software, life is simple. Your focus is exclusively on the functionality of the software. It’s a care-free, liberating world. Coding is an act of pure creation. If you can think it, you can build it, and typically very quickly. You could see a change in production just a minute or two.

There’s so much you don’t have to think about or acknowledge, at all.

Your focus is exclusively on the behavior of the thing you are trying to build. It’s like running through a green field with your eyes closed. You can run full sprint ahead and absolutely fly without worrying about hitting anything.

If you want to add a new page, go ahead! If you want to add a textbox somewhere, you can! Just do it.

In this green field, a textbox is just a textbox. It only takes a minute to add it.

Then, you start seeing success.

People start using your software. People start charging money for your software. Your software starts to become a product. A business forms around it. Customers start requesting things, and more importantly: they start depending on its continued functioning and existence.

People start asking for more. More capabilities. More features. More connections with other things - things that were never meant to be connected to. Other people start making decisions for thing you’re building. You can’t even call them people anymore, now they’re called stakeholders. The green field becomes a tangled bramble of needs and requests. It becomes muddy.

The brown field

In this brown field, the textbox is no longer “just a textbox”. There’s so much more to it. There’s business rules, often lost in nuance.

Rules around:

• How it’s displayed - masking, borders, colors, shapes, sizes, positions, margins

• How it’s tracked - analytics, auditing, tracking who, when, and where

• How it’s accessible - screenreaders, mobile, translations, color-blind support

• How it’s secured - character limits, XSS prevention, rate limiting, authorization

• How it’s validated - input rules, how errors display,

• How it’s expected - stakeholder training, user habits, data migrations

• How it’s operated - cost control for usage-based vendors, feature flags for visbility

• How it’s maintained - rules in one usage vs. another, reusability of its functionality

• How it’s observed - error handling and reporting, notifications if it breaks

• How it’s communicated - loading indicators, animations, user delight, labeling

• How it’s scaled - paint and render performance, server optimizations

• How it’s effective - A/B testing of impact, pre-filling of data, ease of completion

• How it’s used - if users understand it, if they make mistakes, if they’re faster

• How it’s valued - why customers want it, why the business needs it, if it matters

• …and so much more.

The textbox isn’t just a textbox anymore. It’s an entire ecosystem with a long history of past decisions, some forced by the context, and others made by dozens of people all without awareness of the other decisions. It’s connected to other elements with just as much complexity, all likewise interconnected in mysterious, likely undocumented ways. Some of those decisions may be written down or remembered, others not.

Hidden complexity

All of that complexity isn’t visible to any single person.

If you aren’t a marketer, you won’t notice that it can be targeted by the tag manager. If you aren’t visually impaired, you won’t notice it has screen reader support. If you aren’t using an lower-end machine you won’t notice it’s been made faster. If you don’t speak another language, you won’t notice it has translation. If you aren’t coming from another site, you won’t notice it pre-populates. If you aren’t the finance leader, you won’t notice the revenue impact. If you aren’t the auditor, you won’t notice the DOM has a label that’s technically a violation of a long forgotten rule.

Yet, you as the engineer still have to consider it all. You have to reckon with and factor in these decisions as you define and implement change after change after change - all under what always seems to be a tighter and tighter deadline.

In the green field, hours was enough to build an entire product. In the brown field, it’s sometimes barely enough to understand what a request is even asking for.

You have to make changes within the constraints of the past and future, known and unknown. Changes will eventually break something, and the affected party will fill a bug report which you have to then complete and fix.

That’s the price of success - people use the software and notice when it doesn’t behave according to how they think it should.

How should it behave?

It’s the hardest question you might be able to ask someone at a company that’s been around for a while.

The stakeholders in the company won’t be able to define it for you. Nobody actually knows the intricacies - not in full. You have to discover it.

So, you talk to users to find our their needs. You analyze how people are using the product. Where they get stuck. Where they click and don’t click. Where they enter. Where they leave. How all that translates into the how the user gets their job done and the company accomplishing their goals.

But, that’s still not enough.

Knowing what you intend the software to do is just part of the problem. You also have to know what it was intended to do and how it works today.

There’s no easy map - just code and history. To find it, you have to do good old fashioned archaeology. You search commit histories, Slack discussions, old documentation, random meeting notes, talk to co-workers, view logs. Bit by bit, you’re able to build context that makes your follow-on decisions implementing more accurate.

Speed vs. Accuracy

The challenge is - context building takes time. It’s time away from developing and coding. It affects the speed at which you can deliver the change. It might be too slow for the tastes of the company or your manager. You might even feel you aren’t really an engineer anymore.

When faced with the prospects of having to have “conversations about your throughput”, you might try to reverse course: moving faster, deciding faster, acting faster.

Rookie mistake - all that does is just cause you to miss some hidden interconnection nobody knew about and cause an incident. Now, you just traded the throughput conversation for one about quality.

At this point it’s easy to get frustrated. The natural tendency of an engineer is to push back. You might start to reject tickets due to “incomplete requirements”, or start approaching every change request like a contract negotiation, requiring volumes of forms to be defined and filled out.

That approach won’t work. Asking others to define it won’t help because they don’t know. They don’t have the knowledge, nor do they have the time. As the engineer, you at least have the advantage of being able to look under the hood and live in the product. For some of your co-workers, they only think about the product for 30 seconds a day - they have other day jobs they have to do.

It’s up to you to figure it out. The more you dive into it, the more you can grow and improve. Embrace the breadth. It’s not just about the technology.

Becoming a product engineer

When you’re a product engineer, you have to “wear a lot of hats”. The product hat. The finance hat. The user hat. The engineering hat. The investor hat. The customer hat.

None of them will seem like they fit.

You might find it hard to think about things from each of the perspectives. To wear each hat and truly adopt that persona as your own, even just for a minute. To think through a problem as if you were the person the hat was made for.

Well, nobody said Product Engineering was easy.

It’s difficult. It’s high pressure. You’ll constantly hear questions from stakeholders asking “Why can’t we go faster?” and “Why can’t we stop breaking things?” They might even start comparing you to other teams and other companies, not realizing those teams are on newer products and in greener fields.

It’s easy to feel stuck - to feel that any move you make is going to be wrong.

Grind away at it anyways. In situations like this, the only winning move is to lose and learn.

The mistakes will happen. Focus on improving, learning, and sharing knowledge. Focus on getting the job done.

Little by little, it’ll get easier. The patterns will get clearer. The cause and effect will start becoming automatic. The chaos will turn into complexity which will turn into simplicity.

You’ll start thinking deeply. Thinking about how the users will interact with the change. How it’ll impact the business. What they’ll expect. What they’ll want. What they’ll need. It’ll just become automatic.

You start being able to avoid problems entirely, and to find insights that have a huge impact. It’ll become second nature to think about things like accessibility and scalability while thinking about revenue and compliance and usability. You’ll be able to see the thread of how the click will translate across 20 hops to impact the bottom line.

It’ll all makes sense. It’ll all connect. You see it in your head, with every decision you make.

Being an expert

The next time you need to add a textbox, you’ll be more confident. You’ll consider the impact - to everyone: users, customers, the business, your peers, your system, your product, yourself. The non-functional requirements that many don’t even know exist, you’ll just consider through your minute-by-minute actions. You feel like you’ve got this. You’ll know you’ve got this. You’ll be wiser, faster, better.

It’ll take just a minute to add the textbox: a long journey to arrive at the same destination.

But not quite. This time, you’ll have considered the “-ilities”. You’ll have considered the things not mentioned. You’ll have done the things not thought about. You’ll have considered things that aren’t relevant now, but will be soon. You’ll have factored in the past, present, and future.

You’ll complete your ticket - not what was defined, not even what was intended, but what was needed. Whether the field is green or brown won’t matter anymore.

You’ll be happy with the result. You’ll have made it look easy. It actually will be easy.

After all, it’s just a textbox.

A problem I’ve seen a lot is the back-end controller accidentally serializing and returning confidential data to the front-end.

Sensitive things like password hashes, API access credentials, etc. may accidentally be serialized by magic framework defaults.

Promiscuous serialization

It’s quite easy to accidentally do, especially on a large team with multiple projects. A lot of frameworks put convenience over security, serializing all fields by default in the absence of a whitelist.

As a result, common developer errors may contribute to hidden security vulnerabilities.

In Ruby on Rails, for example, not specifying a serializer when returning an object from a controller will simply return all of the attributes for that object.

class UsersController < ActionController:Base
  
  # If there is no UsersSerializer, this method leaks sensitive data  def show
    user = User.find(params[:id])
    render user
  endend

This might happen in a lot of situations. A developer may forget to add the serializer. You may be working with a name-spaced model and the auto-magic resolution doesn’t kick in just right.

Even if you do specify a serializer, you’re not entirely safe. If that serializer uses relationships in the serializer, those relationships will have all of their attributes serialized as well:

class UsersController
  
  def show
    User.find(params[:id])
  endendclass UsersSerializer
  
  attributes :id, :name
  has_many :secret_notes # leak!end

You can see there’s a lot of potential for data to be leaked — on larger teams it is inevitable for something to happen eventually.

How do you have secure defaults in an environment that fights against you?

Simple — use a tripwire.

A tripwire?

Tripwires are things that have an effect upon a certain trigger.

In this basic implementation, the tripwire is a field that, when serialized, will throw an error. You can do this in in the following way (example using Ruby on Rails):

• Add an automatically serialized column to the table you want to protect

• In the model, create a getter that throws an error

class User < ActiveRecord::Base   def tripwire
       raise 'default serialization is not allowed'
   endend

When the framework attempts to serialize this model using the default promiscuous method, it will encounter this field and throw an error, alerting the developer that something went wrong.

This tripwire optimizes the system in favor of failing early and preventing the leaking of sensitive information. It may result in an increase in errors if it ever makes it to production, but will prevent the greater issue of leaking sensitive data. Fail-open can lead to all sorts of confidentiality breaches, whereas the fail-closed technique of the tripwire can prevent these from happening at the cost of potentially preventing data access in legitimate situations.

Software development is full of mantras that are chanted by developers of all levels as prima facie evidence that justifies and proves the rationality of their decisions.

Who amongst us hasn’t heard people or told people to YAGNI, DRY, or KISS? They’re popular mantras because, when followed properly, they work to guide and nudge thinking towards more effective development and programming.

However, they have a dark side.

Because these terms are intentionally vague and broad, they can be misapplied or over-applied in many situations. These memes are then used as justification to continue applying the wrong approach to a problem.

They’re hard to argue against because philosophically these generalized statements are correct. We want to believe them. We have to believe them. To not believe them makes one feel as if they are not a developer. This makes them difficult to argue against, combined with the fact that they contain only the broadest of truths and are not applicable in every situation.

KISS — Keep it Simple, Stupid

Despite hurting feels across the world, KISS, or “keep it simple, stupid” is often used to promote simplicity and prevent the over-engineering tendencies many engineers have.

There might be a hundred ways to implement any one requirement. KISS implies that the simplest method is often the best in terms of complexity, speed, and cost.

For example — an engineer may need to implement a way to change the program configuration. They may be evaluating two different approaches: read from the database, or read from a file.

Developers apply KISS in this situation: avoid complexity and dependencies caused by the additional database tables by choosing the simpler method of reading from a file.

What’s wrong with it?

The dark side of KISS is that is can lead you down the path of the minimum, which if applied over time eventually results in subpar outcomes.

Let’s suppose a second requirement follows that the configuration needs to be changeable during runtime by internal employees.

A KISS-focused iteration after this could be to give internal administrators access to the server to change the configuration file. However, this goes down a path that results in more ineffective results than you would have had you taken the slightly more complex option of putting it in a database table.

KISS is also often used to defend under-developed solutions and gloss over implicit or unspoken requirements. In the example above, one of the unspoken requirements is that we don’t want the server to go down if we changed the configuration. However, because we’ve gone down the route offered by KISS, this constraint is not even mentioned or addressed.

Sometimes situations involve real complexity, and in these situations, KISS can dangerously scope out important considerations.

YAGNI — You Aren’t Gonna Need It

YAGNI is often used to prevent engineers from over-engineering their solutions in an effort to address situations or requirements that don’t currently need to be addressed.

Let’s face it — engineers have a tendency to overbuild for use cases they will never see, or inappropriately delay value delivery for minimal gains in architectural and code cleanliness — code-golf is a thing, as are ivory towers.

YAGNI is the mantra that acts as the counter-balance to the path of infinite “what if?” questions that engineers can travel down. It prevents scope creep that comes from attempting to address the problems of an uncertain future that are better left to be solved for when they actually occur (if they ever).

What’s wrong with it?

YAGNI has a dark side. It is often used by engineers as a reason for under-engineering in a misguided attempt to justify sloppy work with nebulous promises of unverifiable gains in development speed.

How do you tell if it is being misapplied? When YAGNI is used as the reason for NOT doing something, it’s time to ask yourself some questions:

• Is it being used to avoid solving a problem that is inherent to the domain?

• Is it being used to avoid developing something inherently foundational to the product or software (eg. authorization for web applications)?

• Are speed benefits from avoiding the additional work being tracked to ensure it actually leads to faster development?

• Is the cost of doing it or implementing it actually as costly as believed?

YAGNI can be a powerful guiding principle when applied judiciously. However, it can also be used as a crutch to excuse sloppy development and low quality work in the name of faster delivery speeds that never materialize.

A tale of a highly detailed project

I once worked on a project that had almost no abstractions. Instead, every line of code was at the lowest possible level of detail. Functions were thousands of lines long, forcing you to read every line to understand what it was doing. There were few classes or reusable libraries.

The team was hesitant to create classes or even move code into functions. They believed it was over-engineering and yelled “YAGNI”: why move code into a function when it wouldn’t be called by anything else? Why create abstractions we didn’t need when copy-pasting was so much faster?

I had to remind them that engineering was about developing primitives and abstractions that hide the details so that you can focus on more important concepts and ideas.

Software is founded on abstractions: from on/offs to bits to registers to memory management to instructions to code to functions to classes to libraries to programs to services. Without these and other abstractions, our minds are forced to contend with details that aren’t relevant for what we’re trying to do, greatly distracting from solving the real problem at hand.

More often than not, these details don’t matter.

An alternative — You Are Gonna Need It

Spend a bit of extra time thinking about and modeling the technical and business domain. If it is some common function such as logins, file downloads, or auditing, think about how you would implement it, and then implement something minimal that will move you towards the ideal solution.

Remember — the value of the plan is in the planning. Forcing yourself to think about all of the “What Ifs?” will help ensure you don’t paint yourself into a corner that will prevent the code from evolving to support changing requirements. It doesn’t mean you have to everything at once — you can still ensuring you move in small, incremental steps.

This will save a tremendous amount of resources in the long-run, especially if it is something foundational to future development efforts, such as code related to UI components, design systems, authorization, endpoints, or data layers. These are things that are common and highly likely to be needed as the product evolves.

If you solve a problem just once and write the solution in a way where you can use it anywhere else you encounter the problem, you have effectively saved an exponential amount of time from future development.

DRY — Don’t Repeat Yourself

DRY is all about not doing the same thing over and over and over and over and over and over and over. It encourages developers to think about commonalities and not just robotically copy-paste functionality.

By emphasizing centralization of shared functionality, teams can more easily keep code volume in check, preventing a cascading propagation of changes should a commonly copied piece of code require modification.

What’s wrong with it?

DRY casts a terrible spell on developers who take it too far and centralize code that has the same form but not the same purpose. They take code that might be structurally similar and tie use cases together, causing a change in one to lead to an unintended or undesired change in the other. I see this often in React/Redux developers who use Redux to avoid having to think about state management.

This unintended globalization of previously localized data can lead to a lot of negative consequences.

A tale of a User

In one project I worked on, we had a a list of users on the front-end. The developers were judiciously following DRY, and placed these values in a centralized cache for use by all of the other pages, including two in particular: a user list page and a user lookup search.

This proved to be a mistake. Even though both components happened to use a list of users, they had completely different requirements and sources of change for that data:

• The user list page wanted many additional details in the payload, whereas the user lookup wanted just the name and ID.

• The user list page contained PII such as full names and email addresses, whereas the user lookup only contained minimal public information.

• The user list page was manually editable and sortable, whereas the lookup was not.

Because these two shared the same centralized data store, it wreaked havoc on the front-end. Sometimes data was missing. Other times items came in out of order. Sometimes using the user lookup caused the user list to disappear. It was a mess.

The sources of truth for these pieces of data should have been completely different, but they had been DRY’d up and combined.

An alternative — Do Repeat Yourself

Whenever you store or use a piece of data or write some code, always ask yourself the following questions:

• What are the reasons this code or data may change, and do those reasons apply to this situation?

• What use cases are leveraging this code or data, and is this shared piece generic enough to suit both cases without impacting either one?

• What stakeholders would request changes to this piece of functionality, and could they simultaneously both be right and disagree?

If you identify that the form may be similar but the purposes or reasons different, give yourself permission to repeat yourself.

The Lesson

Be judicious in your application of mantras. While they can help guide thinking and are useful rules of thumb, they were never intended to be dogmas.

Don’t be a lazy thinker. Always be cognizant of the “Why?” behind what you are doing, and know when to break the rules.

Explore the conceptual architecture of monitoring systems and learn how to build your own system observation engine.

Have you ever seen products with status pages? Ever wonder how it all works and how to make your own? Most companies don’t do it themselves — they use a 3rd-party vendor like StatusPage combined with automated reporting tools like Pingdom. If you’re using this in a production environment, chances are you’ll want to use a vendor.

However, if you’re curious about design approach and possible ways to build something like this on the quick, then this is the article for you!

Firstly — a conceptual exploration of “Monitoring”

The purpose of a monitoring system is simple: determine whether something is working or not.

That statement has a bit more nuance than meets the eye, so let’s take a closer look.

What are you monitoring?

It’s important to recognize that the “something” that gets monitored can be anything. Oftentimes, it is a system within your control, such as an application server. Other times, it is a vendor system. Perhaps it is a mix of both, such as a technical process that needs to contact a vendor system.

Approaching it from the perspective that the monitored target is arbitrary keeps your thinking from going down the path of a specific implementation.

What does it mean to “work”?

A system that is working can mean a lot of different things. The most binary definition is whether the system is up (working) or down (not working).

That’s an incomplete answer, though.

• If the system is up, but not doing what it was intended to, does it mean it is working?

• How about if it is doing what it is intended to do, but only for half our users?What about 1% of our users?

• Does a system have to perform flawlessly to be considered “working”?

• If the system is doing what it is supposed to, but delayed, is it “working”?

Based on this ambiguity here, it is clear that from a design perspective, the meaning of what it means to be “working” relies on us to define. The acceptable limits of which errors are tolerated, the threshold, is also determined by us as the operators of the system.

Following this train thought, it means “working” is an arbitrary definition.

What are we building?

If we refine the answers to our conceptual questions above, we are left with the essence of monitoring: we are building something that can tell whether another system is functioning based on the parameters we define.

In essence — observing. There are, of course, other elements of monitoring such as alerting, recovering, etc. — we’ll get to that later.

The basic parts of a monitoring system

The Monitor

The Monitor is the component that contains the general logic for monitoring. Note that it doesn’t contain the logic for monitoring a specific system, but rather the algorithm for monitoring:

• Step 1: Check a system and get a result

• Step 2a: If the result is “working”, do something

• Step 2b: If the result is “not working”, do something

• Step 3: Repeat step 1

If we break that down into a single section of code, we’ll get the following:

loop
  result = check_system

  if result.working?
    puts 'system up'
  else
    puts 'system down'
  end
end

Obviously, some details here are missing — we’ll get to this.

The Configuration

Remember — one of our key constraints is the fact that the system being monitored is arbitrary. This means that the Monitor doesn’t actually know how to talk to or check the system.

The configuration defines what the Monitor actually does. This is a piece of data, articulated as code or stored in a database.

configuration = {
  frequency: "2.days",
  target: "PaymentVendor"
}

One could easily write code to interpret this:

def is_time_to_check_system?(frequency)
  interval_number = frequency.split('.').first
  interval_unit = frequency.split('.').second

  interval = if interval_unit == 'days'
    interval_number * (60 * 24)  
  elsif interval_unit == 'minutes'
    interval_number * 60
  end

  return last_observed_at + interval < Time.now
end

The resulting algorithm:

loop
  next unless is_time_to_check_system?(configuration[:frequency])

  result = check_system

  if result.working?
    puts 'system up'
  else
    puts 'system down'
  end
end

Where the configuration lives is not particularly relevant to the monitoring system — it could be in the code, in a file, or even in the database if you want users to be able to set up these configurations themselves or want them defined easily during runtime!

The Integration

This is the part of the system that actually knows how to contact the thing being monitored. You’ll want to make sure it has a consistent calling interface for the monitor to use, to allow for multiple kinds of integrations to be monitored.

Let’s suppose you’re monitoring a 3rd-party payment system that has an API that always returns a response with a status code:

• SUCCESS if the request succeeded

• FAILED if the request failed

You can create a library that calls this API endpoint and interprets the response into a standard format your monitoring system can understand.

class PaymentVendorMonitorIntegration < MonitorIntegration
  def call(params = {})
    vendor_response = VendorLib.http_call(params)
    monitor_result = to_monitoring_system_result(vendor_response)
    
    return monitor_result
  end

  def to_monitoring_system_result(vendor_response)
    return {
      status: vendor_response.status == 'success' ? 'UP' : 'DOWN',
      message: vendor_response.body['message']
    }
  end
end

Here, the standard interface for a MonitorIntegration is #call and the result which contains a status and a message . While an actual implementation will contain many more things, this fundamentally is the core of determining a system’s status.

def check_system(target)
  monitor_class = "#{target}MonitorIntegration".constantize
  monitor_class.new.call
end

You could the integrate with the above code:

loop
  next unless is_time_to_check_system?(configuration[:frequency])

  result = check_system(configuration[:target])

  if result.working?
    puts 'system up'
  else
    puts 'system down'
  end
end

is_time_to_check_system?(frequency) # ...
check_system(target) # ...

The Standardized Response

The important part of monitoring multiple systems is you want to standardize the response regardless of what that system gives you.

You could map the to_monitoring_system_result into a class:

class MonitorResult

  def initialize(params = {})
    @params = params
  end

  def working?
    @params[:status] == 'up'
  end

end

Now, when you receive a result from a monitor, you can expect to be able to call working? and have it return a consistent result.

That’s the basics of observing!

Obviously, your code will have to change to support different integration targets using some sort of system identifier.

There’s multiple iterations available from here:

• Supporting more advanced system checks, such as running more complex scripts

• Supporting specificity of endpoints, or adding parameters to checks

• Moving configurations into the database to allow for user-configured checks

These I leave as an exercise to the reader.

What do you want to do after you monitor a system?

If a tree falls in the forest and no living thing is around to hear it, does it make it sound? Does it matter if it did?

There’s no point in monitoring something if you don’t have an action item arising out of it. What we do after we determine whether something is working or not working matters just as much as determining it in the first place.

Our monitor above just logs to the console — that’s not useful. You’ll want to add additional behaviors beyond just observing. These may include:

• Alerting — Letting people know when the system is operating outside of defined norms.

• Tracking — Recording statuses over time so that trends and long-term empirical baselines can be established

• Recovering — Triggering automatic actions that can help the system recover.

We’ll explore an approach for each of these.

Alerting

Suppose you want to send a Slack message if a critical system is down. You an do something like:

def send_incident_notification
  SlackNotification.send_to('#outages', 'System is down!')
end

Which could go here:

loop
  next unless is_time_to_check_system?(configuration[:frequency])

  result = check_system(configuration[:target])

  if result.working?
    puts 'system up'
  else
    send_incident_notification
  end
end

Tracking

You’ll likely want to have a history of whether a system is down or not so you can report and track uptime over a larger period of time.

The naive approach

The first approach is simple: log the results of all in your favorite data store.

def log_monitor_result(result)
  MonitorResult.create!(status: result.working?, 
                        occurred_at: Time.now)
end

You can incorporate it easily after getting the result:

loop
  next unless is_time_to_check_system?(configuration[:frequency])

  result = check_system(configuration[:target])

  log_monitor_result(result)

  if result.working?
    puts 'system up'
  else
    send_incident_notification
  end
end

Once you have a history of statuses, you can easily create a chart that shows uptime and availability at whatever frequency you desire.

A smarter approach

This naive approach has a key downsides — you’ll store a lot of records.

• If you’re monitoring just one system every minute, that’s 1,440 records / day, or 525,600 per year.

• If you’re monitoring multiple things, such as 50 different systems, that’s 72,000 records / day or 26,280,000 per year.

A better approach isn’t to log every attempt, but to log only if the status has changed.

This would only log the things we actually care about —outages or recoveries, and allow us to assume the last known status for any time period afterwards.

This has a best-case storage requirement of 1 database record if your system never goes down.

def is_different_than_last_status?(result)
  previous_result = MonitorResult.all.order(:occurred_at).desc.first
  previous_result.try(:status) != result[:status]
end

Our new algorithm looks like below:

loop
  next unless is_time_to_check_system?(configuration[:frequency])

  result = check_system(configuration[:target])

  if is_different_than_last_status?(result)
    log_monitor_result(result)
  end

  if result.working?
    puts 'system up'
  else
    send_incident_notification
  end
end

Future iterations

Tracking additional things can help you monitor and manage even more — you could easily track:

• Additional data like HTTP status codes

• Response payloads to inform outage resolution

• Response times for performance management

and more!

Recovering

If you have a reliable way to make the system function properly, you can execute this in the event of downtime fairly easily, creating self-recovering systems.

def attempt_system_recovery
  send_restart_command # I leave this to your imagination
end

Incorporating it when your system is seen as down can help automate recoveries.

loop
  next unless is_time_to_check_system?(configuration[:frequency])

result = check_system(configuration[:target])

if is_different_than_last_status?(result)
    log_monitor_result(result)
  end

if result.working?
    puts 'system up'
  else
    send_incident_notification
    attempt_system_recovery
  end
end

This is, of course, a proof of concept, and not something you should use in production — unconstrained automated system recoveries can easily cause your system to never recover or do things that makes the incident even worse. Be judicious!

A final generalization

What do Alerting, Tracking, and Recovering all have in common? That’s right — their lifecycle is tied to the monitoring system.

When a monitor is triggered, and a result is received, these are ultimately actions that occur. One could take the system a step further and generalize it into a MonitorAction.

class MonitorAction
  def do(params = {})
  end
  
  def should_do?(config, result)
  end
end

Implementations of Alerting, Tracking, and Recovering could look like:

class AlertMonitorAction < MonitorAction
  def do(params = {})
    SlackNotification.send_to('#outages', 'System is down!')
  end

  def should_do?(config, result)
    result.working?
  end
end

class TrackMonitorAction < MonitorAction
  def do(params = {})
    MonitorResult.create!(status: result.working?, 
                          occurred_at: Time.now)
  end

  def should_do?(config, result)
    previous_result = MonitorResult.order(:occurred_at).desc.first
    previous_result.try(:status) != result[:status]
  end
end

class RecoverMonitorAction < MonitorAction
  def do(params = {})
    send_restart_command
  end
end

The algorithm could call a perform_actions function that checks whether something should happen:

def perform_actions(configuration, result)
  configuration[:actions].each{ |action_name| 
    action = "#{action_name}Action".constantize.new
    if action.should_do?
      action.do(configuration, result)
    end
  }
end

The algorithm would be changed to:

configuration = {
  frequency: "2.days",
  target: "PaymentVendor",
  actions: ['Monitor', 'Track'] # whatever you want to run
}

loop
  next unless is_time_to_check_system?(configuration[:frequency])

  result = check_system(configuration[:target])
  
  perform_actions(configuration, result)
end

You can see how this could evolve even further to support different actions, parameterized actions per configuration, or even actions before the system is checked.

You could even move the configuration into the database so that it could be defined by the user.

That’s it!

Monitoring systems are conceptually fairly simple.

However, a real production system is so much more: production monitors are backgrounded, off of the main application, and have some sort of user-defined interface to set up. There could be even more specificity like monitoring endpoints, status interpretation beyond a binary “up” / “down”, etc.

Real-life complexity comes from ensuring it functions reliably and can perform reliable recovery if needed, and can track to the granularity needed.

Did you like this article? Let me know in the comments, or connect with me on LinkedIn!

Learn how to create a reusable export generation system using the template design pattern.

Reports. It’s often one of the first features developers are asked to build by the business. Once someone has data, what do they want to do with it?

That’s right — view it.

Engineers have problems with reports

Many engineers approach reports in a use-case particular way.

They’ll get a requirement such as “create a .csv report that displays all purchased items from the store” and they’ll go and create that particular feature.

The next time they may get a request to “create a .csv report of all users belonging to the account” and they’ll go and create that feature as well.

As time goes on, the business asks for different formats, different fields, etc. These requests add up, and after a few iterations, the system ends up with more than a dozen report implementations, all with their own bugs and quirks.

Worse yet, if any major iterations are needed, such as customizability or some new performance improvement, the change will have to be made over and over (and likely slightly differently each time), leading to significant overhead implementation and maintenance costs just to generate a comma-separated list! But, it doesn’t have to be this way. There are some simple techniques you can use to avoid this pain altogether.

What Is a report?

First, let’s understand — what is a report, exactly? Every report has these four basic components:

• data records

• data values

• labels

• format

Data records

The data is the actual data that you are using to populate your report. Maybe it is a bunch of events in your system. Perhaps it is transaction data. It’s likely data pulled from your database.

There’s likely a specific scope to the data to limit the data based on some parameter, such as date range.

Data values

It’s not enough to have just the data as a whole. People are often interested in specific data points — specific fields of the data. A financial person may only be interested in dollar amounts of a settlement, whereas a security monitor may be interested in an address mismatch status.

Labels

Labels provide meaning to the data. It’s what distinguishes this…:

5 | 3 | 230 | 40

…from this:

id | event_id | amount_paid_cents | fee_paid_cents
5  | 3        | 230               | 40

Labels turn meaningless values into data and provide human-understandable context to otherwise arbitrary values.

Format

Finally, there’s the format of the report. The format could be arbitrary and what format is needed ultimately depends on the consumer. If the consumer is an API, the format might be JSON.

If the consumer is a FTP drop-off or a person intending to import it into another system, it may be a CSV. If the consumer is a data person interested in slicing and dicing, it may end up as an Excel spreadsheet or even a set of database import commands.

What Are the Steps of Generating a Report?

Now that we know the pieces, we can look at the algorithm: what is the sequence of steps needed to generate any kind of report?

Turns out, it’s not a complicated algorithm! It can be encapsulated into the following:

• Setup — Get parameters for the report

• Fetching records

• Map — Get a particular set of fields from each record

• Convert each set of fields into an entry in the report

• Send the report back to the user

Setup

The first step is setup — sending the parameters into the report to adjust how it behaviors. This is often data used for scoping — some collection of records needs to be filtered and reduced into a subset (eg. by an account, by date, etc.)

Fetch

Once you have the parameters for your report, you’ll want to start retrieving those records. You can apply the filters as appropriate:

Map

You’re likely interested in a particular set of attributes and fields in the record. You can turn your records into that particular set, and optionally enrich each entry with additional data.

Convert

Once you have your collection — in this case, a set of hashes, you’ll want to convert it into a specific format, such as CSV.

Want to support other formats? No problem:

Putting It All Together

Each of the steps above may have its own details, but you can generally encapsulate it into something like below:

A use-case specific implementation might look like:

Note how easy it is to tie into the existing functionality of a report — you can quickly create many different kinds of reports without worrying about how to generate or deliver them.

If it looks familiar — it’s because it might just be!

This is the template pattern — a design pattern that helps encapsulate the sequence of an algorithm and the implementation of the invariant steps while leaving the variant steps available to be “filled in” by a subclass.

Additional Iterations

There’s a lot of other concerns and possible iterations a report might have, such as:

• Performance — record batching, record size management

• Customization — specifying headers, changing ordering

• Delivery — SFTP, download, email

This functionality can be added to the base class or subclasses as appropriate without changing the overall structure of the system. Standardization of inputs and outputs allows for tie-ins into other parts of your system.

For example, if you already have a module that delivers files via email, you can easily plug in the output of your report generator into that subsystem, or call it from a function like deliver_to_email(email).

That’s it! Creating reusable software that separates the mechanism from the use case, allowing it to be used in many other use cases, is as simple as breaking it down into small parts and thinking carefully about inputs and outputs.

Did you like this article? Let me know in the comments, or connect with me on LinkedIn!

When I was younger, I had a top-of-the-line gaming computer. I spent thousands of hours playing games like Team Fortress 2, Minecraft, Guild Wars, and Age of Empires.

One of my favorite activities after I had thoroughly explored a game was the practice of modding. Modding allowed you to create or download packages of software that changed or added to how the software behaved — new levels, textures, or even game mechanics! The possibilities were limitless.

Now that I’m a software developer, I have a better understanding of how many of these programs supported such massive extensibility.

They used a concept called plugins.

Plugin systems

Plugins allow you to write subprograms that then hook into or are attached to a larger program. These subprograms then run, modifying or adding to the behavior of the running program.

In order to write a plugin, the program itself has to be written (or hacked) to support the plugin. Once this capability exists, you can “plug and play” tremendous amounts of functionality.

The Concepts

Plugin systems come in many shapes and forms — to illustrate the design, the following basic concepts can help frame your thinking:

The program

The first piece is the program itself, the thing whose behavior you are actually trying to enhance or modify. This might be a game like Skyrim or it might be a business application like Sketch. It might even be your own program!

Whatever it is, it offers some set of behaviors and capabilities that you want to change. More importantly, it does not know about the changes you want to make.

The hook

Something in the program you’re trying to change has to run the code it doesn’t know about. Otherwise, it would be very difficult to change the behavior.

This point at which the code is run is called the hook. Rails’ model callbacks or Vue’s component lifecycle hooks are examples of hooks used out in the wild.

A simple example of a hook is below:

See how the onSaveHook has no behavior. In this example, it is expected that this behavior will be filled in by a class extending RecordSaver and implementing new behavior in the subclass.

The plugins

The plugins are the code that other people write and “plug in” to enhance the capabilities of the program.

You could potentially extend the RecordSaver class from the example above with functionality such as below:

class LoggedRecordSaver < RecordSaver
  def onSaveHook()
    puts 'Saving record.'
  end
end

Now when LoggedRecordSaver#save() is called, it will print Saving record. after calling #writeToDisk() to the console. This behavioral enhancement was done by extending RecordSaver — a basic way to enhance a program’s behavior.

Note also that RecordSaver doesn’t know anything about LoggedRecordSaver: the coupling is unidirectional.

The loader

Having a plugin is useless if it never actually manages to run. Something needs to load the Plugin to start executing its code.

There are a lot of techniques available to actually make this happen — two methods are:

• Plugin-driven, where the plugin knows how to access the program and self-registers.

• Program-driven, where the program knows how to find plugins and loads any it finds.

In plugin-driven loading approaches, the program provides a method by which the plugin can register itself with the program. In program-driven loading approaches, the program finds a plugin, such as by loading all files in a folder with the name *_plugin or by loading a manifest.

A Chatbot Written Using Plugins

A while ago, I spent a weekend writing a chatbot, OneLine, to tell me puns and send me reminders using natural language.

The approach was simple: receive a message, do something arbitrary with it, and send back a response. To build all this, I build a basic plugin system that implemented the concepts mentioned here.

The loader

OneLine had a part of the program, a subset of static methods on the Core::Plugin module, define an implementation to:

• Load a plugin (#load)

• Track all loaded plugins (@@plugins)

• Call all loaded plugins (#call_all)

Note that the loader and the hook are both combined in this example.

The loader is the load function that Plugin implementations can call to make the program aware of it.

The hook is very explicitly a call_all function that is called when the program receives a message.

The plugin interface

It also defined an expectation for plugin implementations — behaviors all plugins should support. In this case:

• A check to see whether the plugin should run or not (#process?)

• A method that will be called to run the plugin (#process)

• A standardized response the plugin was expected to return (#to_response)

The #process function is called by the hook (upon receipt of the message).

The plugin implementations

Finally, there are the plugin implementations that parse and interpret the message and do something with it. I made plugins to:

• Tell me a joke (lib/oneline/jokes)

• Keep track of my to-do list (lib/oneline/scheduler)

• …and more!

The simplest example, a plugin that tells you the current time, is below.

That’s It!

Pretty straightforward! Approaching things in this manner left room for extensibility by others, kept different parts of the code segmented and isolated, and allowed me to build and support extensions, such as supporting a web server or conversations via SMS.

While plugin systems aren’t appropriate for all scenarios, they can be highly useful in some cases.

Full Source Code Repository

Feel free to check out the full source code for the chatbot (specifically in the lib/oneline folder):

JGefroh/oneline
OneLine is a personal assistant chatbot intended to simplify my life. You’re a busy person with many important things…github.com

Did you like this article? Let me know in the comments, or connect with me on LinkedIn!

Learn how to create a dynamic heatmap of the United States by yourself, without having to buy a license from a company that sells charts.

A strangely large number of companies I’ve worked with somehow inevitably get to the point where they want to build or integrate a heatmap of some kind.

In the past, I would reach for the nearest charting library like HighCharts. Why reinvent the wheel, and besides, their charting is cool!

Over time, as I felt the itch to incorporate heatmaps into my weekend projects, I kept bumping it off due to the price. I kept asking myself “is this too expensive for a personal project?”

I don’t know about you, but for me the answer is yes. Some charting libraries charge a licensing fee of $500 per developer. This prices out almost every individual looking to use it for any “for-fun” usage. I lose far too much money buying SPY puts in the stock market to be able to afford spending $500 for a weekend project.

So, like all engineers, I spent half an hour and made it myself.

What is a heatmap?

A heatmap essentially shows a map and colors in various aspects of it based on some sort of scale.

The “map” doesn’t necessarily have to be a map — it can be a periodic table or a bunch of circles. What really matters is that there are distinct areas that will be colored differently depending on what bucket that distinct area falls in to.

As I later learned, these are common called chloropeths.

The components

There’s three aspects of a heatmap as far as web applications are concerned.

• the map

• the code

• the data

The map

First off, you need an SVG of a map that you can color. I took this one from the Wikimedia Commons:

Breaking down an SVG

If you ever interacted with an SVG file, you may have noticed it behaves differently than other image files. Some sites don’t let you upload them, they maintain their quality when their size changes, and your browser seems to be the default file opener in a lot of cases.

Well, that’s because they are different. SVGs actually consist of markup. If you inspect an SVG file, you’ll see all of the markup that makes up the SVG image.

All of that incomprehensible garbage is actually the data that then makes up the individual lines and shapes. This means that you can do a lot of things like interact with it using Javascript and CSS. It behaves like HTML, so you can add styles, ids, attributes, etc.

That’s where the true power is.

Prepare your SVG

So, let’s assume you found or created your SVG. If you create one, I don’t recommend coding it — use a drawing tool that will output an SVG.

You need to prepare your SVG by adding identifiers to all of the various parts you want to independently color.

The map above was particularly convenient — it was already categorized into states. For others, you may need to do it yourself.

Each of the individual <path> elements that make up a state had an ID attached to it with the ISO 3166 Alpha-2 code.

Hawaii, for example, has an id of HI. If your map is missing it, simply add the id="HI" to the <path> that makes up Hawaii.

The Code

Now, let’s write the coloring function. It’ll be a function that takes an identifier and sets a color. We’ll flesh this out later once we structure the data appropriately — for now let’s stub it out and return a randomly selected color.

We don’t care about the color specifically or how it is chosen. What we are caring about is tying it together.

// map.js
function setColorFor(state) {
  let color = Math.random() > 0.5 ? 'red' : 'blue';

  let element = document.getElementById(state)
  if (element) {
    element.style.fill = color;
  }
}

setColorFor('HI');

Put them both in an HTML file

Let’s go and put them together.

<!-- map.html -->
<html>
  <body>
    <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 930 585">
      <!-- Put the rest of the SVG here... -->
    </svg>

    <script type="text/javascript" src="map.js"></script>
  </body>
</html>

You should see something like below:

The Data

Now, we get to the final piece of the puzzle: the data.

Structure of data

We need to structure the data in a way we can push it to the map while maintaining domain independence. All this means is that we shouldn’t have to change the code to make it work with another use case.

Suppose we are using this map to visualize the extent of coronavirus cases in the United States. Let’s make up some numbers and see what a subset of this data would look like:

let positiveTests = {
  HI: 500,
  WA: 1000,
  MD: 10,
  FL: 9001
}

Great! We clearly have a key-value pair of state code to value. Now, how do we bucket these into groups along a scale? I’m not a data algorithm expert, so we’ll do a bit of a janky approach.

First calculate how we want to distribute it

The first decision to make — how many buckets do we want?

Let’s suppose we want to have the following breakdown:

• High (≥ 5000) [Red]

• Moderate (≥ 1000 and < 5000) [Orange]

• Low (≥ 500 and < 1000) [Yellow]

• Minimal (< 500) [Green]

• Unknown [Grey]

We effectively have 5 buckets that we will divide the values into. This is what I will call the distribution scale (not the actual name since I’m not a data guy).

This distribution scale can be hard-coded as below:

function getColorFor(value) {
   if (!value) {
     return 'grey';
   }
   else if (value >= 5000) {
     return 'red';
   }
   else if (value >= 1000) {
     return 'orange';
   }
   else if (value >= 500) {
     return 'yellow'
   }
   else {
     return 'green';
   }
}

Now, let’s incorporate this getColorFor(value) function into the setColorFor(state) function we created above in our map.js script:

// map.js
let data = {
  HI: 500,
  WA: 1000,
  MD: 10,
  FL: 9001
}

function setColorFor(state) {
  let color = getColorFor(data[state]);
  let element = document.getElementById(state)
  if (element) {
    element.style.fill = color;
  }
}

function getColorFor(value) {
  // ...
}

setColorFor('HI');

At this point, Hawaii should turn yellow — this makes sense because our data says that Hawaii has ≥ 500 cases. However, all the other states in the dataset are still grey.

Coloring the other states

The other states are still grey because the coloring of states is hard-coded.

The setColorFor('HI') line is the line of code that is triggering the color change. We could go and write out every single state, but that is boring and just poor engineering.

Let’s go and make it so that we don’t have to list every state individually. Let’s remove setColorFor('HI') and replace it with the below code.

//map.js
// ...all other code

function applyData(data) {
  Object.keys(data).forEach(function(key) {
    setColorFor(key);
  });
}

applyData(data);

Now we should see all the states we included in our dataset light up like a rainbow:

You can now see how you would start adding new colors, etc. It’s all very straightforward to do.

Addressing other concerns

There’s some other areas of concern you may encounter while attempting to do this in a production environment.

Making it work with frameworks like Vue.js, React, or Angular

I used vanilla JS and HTML in the examples to simply illustrate the concepts, but many modern web applications are written using actual frameworks and libraries.

An application of the approach in a production environment would likely involve many other aspects such as componentization, theming, etc. and leverage actual frameworks or libraries.

The great news is that you can easily follow the same exact technique and accomplish the goal regardless of the front-end technology. Vue, for example, allows you to use directive bindings directly on the SVG <path> elements.

Linked below is a Github repository I created illustrating the full approach describe above in vanilla Javascript, with an included sample of what it would look like as a Vue.js component:

JGefroh/example-heatmap
This repository illustrates an example of how to create your own heatmap. Read my accompanying blog post for a…github.com

Automatically scaling the heatmap

You may notice that, when resizing the browser window, your map doesn’t automatically adjust in size. You’ll need to do some extra configuration to make this work.

For the map above, adding a viewbox attribute with a value of 0 0 930 585 to the SVG element made it scale appropriately for me (designers may shriek in terror at the eyeballing I did). Different contexts and SVGs will require different scaling approaches.

If you want to learn all of the in-and-outs of SVG resizing and scaling, Amelia Bellamy-Royds wrote a fantastic article about in on CSS-Tricks:

How to Scale SVG | CSS-Tricks
The following is a guest post by Amelia Bellamy-Royds. Amelia has lots of experience with SVG, as the co-author of SVG…css-tricks.com

Creating a linear distribution

Perhaps you don’t want to pre-define the buckets, but rather divide the groups into a known number based on an interval that is dependent on the data.

You can do this by finding the maximum value in the dataset and then dividing it by the number of bucket you would like to have. This will give you the interval, which you could then color the sections by:

function getInterval(numbers) {
  const NUMBER_OF_BUCKETS = 3;

  let maximum = Math.max.apply(Math, numbers);

  return maximum / NUMBER_OF_BUCKETS;
}

function getStyleFor(stateCode, valuesByStateCode) {
  let interval = getInterval(Object.values(valuesByStateCode));
  let value = valuesByStateCode[stateCode]

  if (value <= interval) {
    return 'fill: #01C7E5';
  }
  else if (value <= interval * 1) {
    return 'fill: #03C2DF'
  }
  else if { // ...and so on
  }

}

Dynamic color and data scales

You can dynamically set the color or data scale by accepting an ordered array of colors, or a function that determines it.

If you truly wanted, you could make it fully data-driven. Something like below can be done:

let valuesByStateCode = {
  HI: 30,
  WA: 50,
  FL: 2
};

let buckets = [
  { color: 'green' },
  { color: 'blue' },
  { color: 'red' },
];

function getInterval(numbers) {
  let maximum = Math.max.apply(Math, numbers);
  return maximum / buckets.length;
}

function getStyleFor(stateCode) {
  let interval = getInterval(Object.values(valuesByStateCode));
  let value = valuesByStateCode[stateCode]

  let bucketIndex = Math.floor(value / interval);
  let bucket = null;

  if (isNaN(value)) {
    bucket = buckets[0];
  }  
  else if (bucketIndex >= buckets.length) {
    bucket = buckets[buckets.length - 1];
  }
  else {
    bucket = buckets[bucketIndex]
  }

  return `fill: ${bucket.color}`;
}

This code accepts an ordered array of colors and automatically figures out the number of buckets to divide the values into. Once that is done, it buckets the values received into those colors automatically.

You could then code further and create a gradient off a base color instead of having a pre-defined color scale if you wanted to, but I leave that as an exercise to the reader.

And that’s about it!

That’s the basics of creating a heatmap. Turns out you can do it in about half an hour and save yourself or your company thousands of dollars if cost is a concern and all you need are some basics.

Note that you aren’t limited to maps of the United States. The technique illustrated above will work for pretty much any heatmap-style coloring. As long as you have a properly constructed SVG, you’re good to go.

Did you like this article? Let me know in the comments, or connect with me on LinkedIn!

Take an in-depth look at how I designed, developed, and migrated a legacy system to a new feature flag system

A previous company had a problem: our deploys were thousands of lines in size, took nearly an hour, and were massively risky. Engineers hated deploying, which led to a backed-up queue of dozens of pull requests awaiting deployment. Product managers reasonably wanted to wait to release things until it was all done, which meant our releases could contain thousands of lines of code, any one of which could cause a catastrophic crash. The result? Incomplete work was piling up, value wasting away without delivery.

Something had to change.

Releasing Deploys

To help resolve these issues, one of the things I implemented was technology improvements that allowed us to separate the concept of a deploy from the concept of a release.

Previously, the deploy was the release, as soon as the code made it to production, it was being used by millions of users. This made deployments take on incredible risk.

This directly put engineers at odds with the product manager. Engineers wanted to minimize risk through smaller deploys, but product managers wanted to release working, comprehensive solutions. Instead of working together, our lack of technology capabilities created an environment with a combative dynamic.

We wanted to ensure that our technology could support releasing functionality at a later time than when it was deployed. Put another way, we wanted to be able to send code to production without users being able to see it, and then allow the business and product managers to own the release of features on their schedule.

To accomplish this separation, I pursued the development and integration of a centralized feature flagging system as a ninja project in-between my scheduled work.

What Are Feature Flags?

A feature flag system, also known as a feature toggler, boils down to storing whether a feature is enabled or not and then checking it when you need to. If the feature is enabled, allow whatever it is you are toggling. If not, hide it from users.

Code-wise, it theoretically resolves to boolean check and the execution (or not) of an accompanying code path. An example:

run_code_path if flag

The flag

The flag itself is a value that the code checks for in making its determination to run or not run a specific piece of code. This is the thing that the code would check for when making a runtime determination.

The code path

The code path is the code that is being toggled or disabled/enabled.

Perhaps it checks whether a page component can be displayed or not. Maybe it adds an extra fee to a transaction or not. Maybe it isn’t even completed, and you are just putting the skeleton in place.

These various reasons for hiding the code will determine the longevity of the flag, how it is used, and where it is placed, but the end mechanics are the same.

The check

The check is the code that determines whether the code should run or not. It’s the if-statement that will look at the flag and run the code path.

For us, we divided the system conceptually into two different kinds of checks: system-level checks and context-level checks.

• System-level checks perform checks system-wide. If the check passed for one case, it would pass for all cases.

• Context-level checks perform checks within a specific context, such as within the scope of a User record. If the check passed for one context, it did not determine whether it would pass for another.

Finally, checks could be layered. By combining flags for system-level and context-level, we could toggle functionality for features on specific records, or records belonging to a specific user, or records belonging to users within a specific group, the possibilities are endless.

What Feature Flags Are Not

One critical thing to point out is what feature flags specifically are not intended for.

Because it is a boolean check, you can be tempted to leverage it in any situation where you would have a boolean check. However, just because you can doesn’t mean you should. Developers can fall into the trap of using it for account-specific authorization related logic just because it exists. It is important to not mix this area of concern.

You should never use a feature flag system as an account permission check, such as checking if a user is an administrator or not. Even if the base concept of checking a flag is the same, everything else is different enough to warrant separate consideration. Security is important, and concepts like authorization should be addressed as a primary concern within your system, not relegated to an unrelated feature flag infrastructure.

Dealing With Legacy Work

The product already had a few false starts in this area scattered throughout the codebase.

The most common feature flag implementations I saw in this product, as well as many others throughout my career, fell into these three camps:

• A boolean on a record

• A string array on a record

• An environment variable check

The boolean column

The boolean on a record is simple, add a boolean representing whether the feature is enabled, then check it:

process_surcharge if @cause.surcharge_enabled?

The downside of this approach is that if you have dozens of features, or flags that can be shared across multiple kinds of records, you’d end up with just as many feature flag columns all over your system, and various checks for them.

It would also clutter database tables with things specific to mechanics of how the system works, and not the domain. This can get unwieldy quickly.

The string array

The second approach was clearly designed to solve the first downside of supporting multiple features.

process_surcharge if @cause.features.contains?('surcharge')

It solves one problem but still has the other downsides. It also then introduced a problem of being inconsistent with the already established feature flag pattern, which hadn’t been ported over to the new approach.

Add in a couple of years of proliferation and we ended up with dozens of places where one or the other were being used.

The environment variable

Finally, system-wide features were flagged with environment variables.

process_surcharge if ENV['IS_SURCHARGE_ENABLED']

Solving for legacy constraints

Greenfield would allow us to do anything we wanted, but we live in a legacy world. Any new solution we built had to corral all of these various partial solutions to ensure there was only one true way of toggling a feature.

Otherwise, developer psychology would lead to developers continuing to create their own or copying one of the other feature toggling mechanisms due to the inconsistency.

First Steps

We started by addressing the legacy constraints. We solved the problem of all of these disparate methods of feature checking by adding a layer of indirection and consolidating all of the differences within that layer.

The implementation

We created a service class called FeatureToggler with a method called enabled? that accepted a flag and a record:

class FeatureToggler
  def enabled?(flag, record)
  end
end

Because different kinds of records in our system had a different way to check whether a feature was enabled or not, we enumerated all of those ways within the enabled? function:

def enabled?(flag, record)
 return record.features.contains?(flag) if record.instance_of?(User)
 return record.send("#{flag}?") if record.instance_of?(Cause)
 # ...and so on...
end

We then replaced all of the various if x.<y>_enabled?, if z.features.contains?(a) checks with calls to the new function.

The code movement screamed of violations of every kind from “don’t repeat yourself” to “single responsibility principle” to “breaking the rules of basic inheritance.”

It was all for the greater good, though. Sometimes you have to make the code a bit messier before you can clean it up. Like a sliding puzzle, you may need to do the “3 steps forward, 2 steps back” dance until you complete the refactor.

Minor updates to our existing test suite ensured it continued to pass. The new consolidated interface actually made it possible to split out the testing of feature toggling functionality from other tests and creating a specific suite just for that, which made testing flagging much easier and more comprehensive.

A New Data Model

Once the FeatureToggler abstraction layer was in place, we wanted to transition to a new data model, one that was divorced from all of the other data models within our system.

The new data model

value
value was the flag itself, the value we would check in our code to determine if a flag was running or not.

status
status was a string that represented whether the flag was enabled or disabled.

owner_id and owner_type
owner_id and owner_type were two fields that tracked the owning record in a polymorphic manner. If the owner was a Contract record with ID 2, owner_id would be 2 and owner_type would be Contract.

Having no owner meant that the flag was intended to be a system flag, not a context flag.

Transitioning to the New Data Model

Transitioning was a bit more complex than we desired. Our transition plan required us to transition two aspects of flags: reads and writes.

Reads were easy

Reads were straightforward, we could insert a check in our new FeatureToggler layer to look at the new data model:

def enabled?(flag, record)
 FeatureFlag.exists?(owner: record, enabled: :true, value: flag)
end

Writes were a bit more challenging

Writes were made harder due to legacy constraints.

In order to allow operations to toggle features on our old system, we had a CRUD interface that was data-model aware and generated via DSL. This meant that it was hard-coded and specifically knew whether it was inserting into an array, setting a boolean on a column on a specific record.

There was no way to abstract that behavior, we were severely limited in our ability to change it out. If we added a flag using the new data model, it would not reflect in the old one. If we added a flag in the old data model, it would not reflect in the new one. This meant flags could go out of sync. It was a potential source of significant confusion.

Legacy constraints were forcing us to head down the path of an all-or-nothing release, which was explicitly an anti-goal of our initiative.

We couldn’t have both systems being written-to in parallel, which meant we had to do a complete transition if we wanted to migrate writes.

The solution: Only transition reads

We decided to not migrate writes just yet and only migrate the reads. We kept the old data model as the source of truth for changes by propagating any changes made to them down to the new data model.

We wrote a migration to take all of the feature flags we had stored in the other various approaches and copied them to the new data model.

To keep the data between the old and the new data model in sync, we added an extra step to the existing writes through callbacks, ensuring we were also writing the same data to the new data model in the various locations flags were being written using the approaches.

Finally, we changed the reads of feature flags to point to the new system, but to also verify results via a quorum system that logged when the checks disagreed:

def enabled?(flag, record)
 if new_enabled?(flag, record) == old_enabled?(flag, record)
   return new_enabled?(flag, record)
 else
   log_quorum_failure(flag, record)
   return old_enabled?(flag, record)
  end
end

We also decided to eat our own dog food and feature flag the feature flag check to determine whether to use the old system or the new system as a just-in-case. Talk about extra safe.

By removing the requirement of transitioning writes, we greatly simplified the project and it allowed us to deliver a small piece of value rapidly. Sometimes the simplest thing is to do nothing.

Flawless victory

We ran the system in production for a while, and it ran perfectly, we didn’t see any disagreements being logged, except for those we intentionally created as a test.

At that point, we switched over to using the new data model for all of our reads and started looking at transitioning writes.

Transitioning Writes

Because we had full confidence in the stability of the reads and the usefulness of the new system as a whole, we approached migrating writes with a whole lot more confidence.

We had two data models in play, with the old data model being the source of truth for changes. Since changes to the old data model got synced to the new data model, this meant we were still reliant on all of the limitations of the old approach.

We needed to make our new data model the source of changes. Because our old system was reliant on an inflexible DSL for feature flag modification, this meant that we had to create a new user interface to provide the same functionality that was expected before we could actually transition writes over.

Turns out it was a piece of cake with the new data model. Operationally supporting the new flagging data model was as complex as inserting and reading records in a table. There were no special rules or gotchas. The interface literally boiled down to a table with some buttons, it was CRUD functionality in its purest form.

Releasing

The actual release was pretty straightforward.

We provided a brief training to the main group of people who modified feature flags, wrote and shared a one-sheet to document it extensively, and then provided links to the new interface where the old interfaces were.

Iterations and Improvements

Our new system was fantastic.

It allowed us to rapidly apply different kinds of feature flags or apply multiple flags within different contexts without the need for database migrations.

Cascading flag checks were now easy, we could toggle functionality based on a specific record or for a specific user. We didn’t stop there. Now that we weren’t limited by legacy constraints, we could quickly add more functionality as time went on.

Descriptive enumerations

One problem with the old feature flag approach was that nobody knew what the flags actually did or meant.

Because many of them were boolean fields or string values without descriptions, it was difficult to tell exactly what the consequences of toggling them were. Unless you were already familiar with the system, you would never be able to find out.

When we transitioned them over, we preserved their unfortunate naming.

For non-engineers, flags with values like bpfee and gp were indecipherable. It provided a terrible user experience and led to a lot of operational errors that ultimately made their way back to engineering in the form of bug reports and investigations, which we wanted to prevent.

We wanted to clearly describe what flags did and what they meant, so we introduced a data model to list what the available options were:

We changed our modification interface to decorate the list of feature flags with these details, providing significantly more clarity to anyone modifying the feature flags.

They no longer had to see just:

bpfee

They also saw what it meant:

Receipt Basis Points Fee (bpfee)

Display fee percentages charged as basis points on the receipt.

The increased clarity led to significantly fewer errors and questions reaching us, resolving a large source of upstream issues.

Defaults

Sometimes when a record gets created we want certain flags to be automatically enabled. By adding a column to the FeatureFlagOption table, default_status, we can achieve this:

Now, whenever a record was created, we could pull up all of the flags where default_status was enabled and automatically enable the flags on that record.

Categorization

Over time, you get a lot of different feature flags, all for different purposes.

Some are flags intended for short-term use, typically to hide code until it is done or to mitigate risk by performing canary releases. Once the code is deployed, the flag should get removed. Some are flags intended for mid-term use, such as those used to perform experiments. Some flags are intended for long-term use. Features might be toggled based on the contract or plan the customer is on.

Each of these flags have different owners and usages, so it is important to be able to separate them. By adding a category to FeatureFlagOption, we are able to scope the enumeration and behavior even further:

Experimentation

Sometimes, we want flags that split groups into different cohorts for A/B testing of features. We could store the various parameters of experiments in the FeatureFlagOption, to be sent to the A/B subsystem:

While an A/B system is outside the scope of this post, I recommend checking out the split gem if you are using ruby.

Performance

As usage of the feature flag system grows, you don’t want to be performing that many database reads. The abstraction layer provides a perfect place to put in a caching layer as well to ensure rapid lookups of feature flags.

As always, remember the words of the famed Donald Knuth:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

Ensure you have an actual need to improve performance and that the usage patterns are appropriate for your solution.

As another tip: don’t randomly add caching logic ad-hoc. It quickly gets out of hand when you try and figure out whether you are dealing with cached data or not. It is made monumentally more complex when caching layers on the controller, data, and database all start interacting with each other.

Add caching thoughtfully and with centralized controls so you know exactly how stale the data is and how to force a refresh, if necessary.

Client-side checks

Our new technology platform greatly leveraged a lot of the feature flag infrastructure. We created a front-end service and component that showed/hid nested components based on the feature flag’s status.

By exposing flag checks in an API and then consolidating all of its logic within the service and component, we were able to avoid having feature flag logic all over the front-end.

The Effects

The transition went incredibly smoothly, and it was shocking how much separating releases from deploys improved team collaboration and delivery effectiveness.

Deploy sizes went down and frequency went up because developers could deploy code without causing production issues. This in turn decreased the change failure rate since deploys were smaller and bugs were easier to catch before they made it to production.

Releases went more smoothly and adapted better to the business’ timetable because it didn’t require engineers to keep track of different branches of work and deal with integration challenges for tens of thousands of lines of code.

It wasn’t a silver bullet to all of our problems, but it helped streamline processes and remove a lot of collaboration friction, paying off dividends.

Frequently Asked Questions

Why didn’t we go with a vendor?

There’s a lot of vendors that provide flag functionality like Optimizely or LaunchDarkly. Why didn’t we go with them, why reinvent the wheel? In a word: budget.

At the time, we were under significant budget restrictions, and it was difficult to get any additional spending approved. As engineering didn’t control our budget, we had to make do with what we had.

Even getting resources to work on technical debt like this was impossible, we had to ensure we made progress on engineering initiatives during the time we found in the “in-between:” the small slices of time between tickets, projects, and meetings.

We had to work within the context we had.

Why didn’t we go with a library or gem?

We examined a lot of gems and realized that none of them suited our particular desires for future use cases, legacy constraints, or migration requirements. Several gems in particular would have required a big-bang approach to deployment, which was a level of risk we didn’t want to accept.

A few would have required a complete overhaul of how teams external to our department understood flagging, which expanded the scope of the changes we wanted to make and increased the delivery burden significantly by adding more stakeholders. We were already working on this as an unofficial ninja-project. We didn’t want to increase the odds of it never being delivered.

Most of the gems we looked at would have required us to go the route we did anyways. We decided to kick the selection down the road and roll our own quickly, but hide all-access behind an interface so we could swap out the implementation in the future if resources freed up.

We were already working on this as an unofficial ninja-project. We didn’t want to increase the odds of it never being delivered.

Why didn’t you just transition everything over at once?

An all-or-nothing approach was not desired for a few reasons.

Recall that this was a ninja project we were doing in-between actual work. This meant I had a few minutes here, an hour or two there to actually complete this migration. An all-or-nothing approach would have required significantly more focus than we could afford to allot.

We did introduce extra complexity in the migration by doing it in smaller pieces. However, we de-risked almost the entire initiative by doing so. Feature flags were used extensively within the system for a variety of reasons including contractual obligations, and an error in it would have been highly consequential.

Doing it in smaller pieces was absolutely worth the safety, even if it ultimately wasn’t needed in the end.

Where Do You Go From Here?

If you’re interested in learning more about feature toggling, Martin Fowler’s site has an in-depth article written by Pete Hodgson on feature toggling which I highly recommend.

Feature Toggles (aka Feature Flags)
Pete Hodgson Pete Hodgson is an independent software delivery consultant based in the San Francisco Bay Area. He…martinfowler.com

Walk through the components of a minimal rules engine and understand some of the thought processes behind approaching its design.

Imagine this scenario

You’re a busy engineer working in a company with too much to build and too few resources to do it. Departments constantly ask for changes only an engineer can make. Marketing continuously asks for email on-boarding changes. Operations wants more and more batch tooling. Sales wants to test new pricing structures every other week. You and your team are overwhelmed with requests.

What are your options?

Stop progress on planned work to make these requested changes? A constant stream of interruptions is terrible way to get any real work done, and who knows if these asks are even worth interrupting the real work for.

Ignore it? You’ll make progress on your planned work, but you’ll also have a bunch of external stakeholders now suddenly very angry at your department. Besides, isn’t technology supposed to help the business? Is it really helpful to have the business miss out on potentially significant opportunities just because you couldn’t spare an hour or two?

Triage the asks around a planned schedule? Planned interruptions are better than unplanned ones, but you’re still potentially limiting the progress of other departments and forcing them to work around your schedule — not quite as collaborative as one could be.

Imagine if there was a way to prevent these requirements from every reaching you, and to give the other departments tools to pursue these opportunities without your involvement. They’d be able to make internal pivots without having to rely on engineering to perform the work.

If such a solution existed, you’d save a lot of time and be able to focus on more important problems.

Enter the rules engine.

What is a rules engine?

A rules engine is a system that performs a set of actions based on specific conditions which can be configured during runtime. This means that an effectively set up rules engine would not require engineers to change a system’s business logic.

Engineers build rules-based systems all the time, albeit unconsciously. Every time you code an if-else statement, you are effectively creating a hardcoded rule that is followed by the system.

You can go a step further and make these systems configurable dynamically. When I say dynamic, I mean that the behavior of the system is not determined by flows coded by engineers, but by configuration through data.

These so-called “codeless” systems are not new. Software like Zapier, Hubspot, and IFTTT operate off of the same concepts, as do more technical implementations like Boomi or Drools.

A rules engine is a system that performs a set of actions based on specific conditions which can be configured during runtime.

The fragility of concrete

Why are rules engines even needed?

By default, most engineers concern themselves on the details of what the rules are of the code they are writing. They were concrete implementations specific to the business use case they are encountering.

This makes sense — code does exactly what it says it does, so engineers need to know what needs to be done to write the code properly.

However, this coupling to the details of the use case makes the code they write churn significantly when the details of the use case changes. While this approach works in many cases, sometimes the rate of change can become overwhelming, especially on smaller teams or more dynamic environments like post-investment startups.

It’s easy to see how these things can change. Business logic is implemented to fulfill a requirement in what behavior the software should exhibit. These requirements can come from many sources — business demands, improvements in UX, regulatory needs, technical drivers, etc.

Examples include:

• A “welcome” email must be sent 2 hours after a user signs up

• If a payment account holds more than $100,000, over 3 consecutive days, the entire amount must be disbursed immediately

• Employees that are a member of this union should accrue vacation time at double the standard rate for 2019, but not 2021.

All of these can also vary in levels of stability, or how often the details change.

For example, a requirement based on data obtained through user analytics may change weekly, whereas a requirement based on a financial regulation may change once every couple of years. A requirement based on a union contract may only change when the contract is renegotiated.

When it does change, the implementation must change alongside of it.

The evolution of an implementation

Let’s examine a hypothetical implementation of the “welcome” email requirement as it evolves over time.

A “welcome” email must be sent 2 hours after a user signs up

The first stage

Developers can be relied on to do the easiest thing. Most engineers boil business logic down to an if-this-do-that: if this happens, perform this action.

The first (and unfortunately often last) implementation most developers take is the direct approach.

It’s so easy — a single line of code can fulfill this requirement:

def on_user_create
   WelcomeEmail.send(in: 2.hours)
end

The second stage

As time passes, the business often demands changes to details that seemed concrete and set in stone in the past.

Change is a constant, and software is SOFTware for a reason — it has to be malleable to fit the situation and circumstances.

What if we discover that 2 hours is too long, and we want to change it to 15 minutes? If we do, an engineer needs to go and change it.

def after_user_create
   WelcomeEmail.send(in: 15.minutes)
end

Change is a constant, and software is SOFTware for a reason — it has to be malleable to fit the situation and circumstances.

The third stage

If the business is performing experiments to see what kind of on-boarding leads to the best retention and conversion rates, it’s likely this value will change a lot.

If this happens enough times, the developer will hopefully get smart and move this detail out of the code so it can be configured during runtime.

def after_user_create
   WelcomeEmail.send(in: Configuration.get('welcome_email_wait'))
end

Configuration.get could access the database, read a configuration file, look at an environment variable, or perhaps even randomly decide. The important factor is that it is now determined at runtime instead of hard-coded.

The fourth stage

Each individual requirement taken in isolation often ends at the third stage.

However, if there is a collection of related requirements, patterns can start to emerge which the intuitive engineer can pick up on.

Let’s say the business also wants to send a tips and tricks email. An engineer is likely to code the following implementation, being consistent with prior work:

def after_user_create
   WelcomeEmail.send(in: Configuration.get('welcome_email_wait'))
   TipsEmail.send(in: Configuration.get('tips_email_wait'))
end

Once again, however, it requires a engineer to add support for the new kind of email.

Examining the evolution

As we’ve seen above — each change is easy and small, but it adds up over time, quite insidiously. It’s easy to see how it can quickly grow out of control in a system where hundreds or thousands of requirements may be implemented in parallel, often under high delivery pressure.

Even in this simple circumstance, time has caused unrelated emails to be tightly coupled to record lifecycle, which makes systems more complicated.

In the above scenario, the developer made the judgement to couple the mechanism to the use case. This means the developer tied together what was being done to how something should be done to why something was being done. They coupled the business domain to the technical domain — the mechanism became tied to the use case.

They coupled the business domain to the technical domain — the mechanism became tied to the use case.

When we introduced the tips and tricks email, we essentially had to repeat the implementation. The implementation relied on specific details — namely when something was being done, how long to wait, and what was actually being sent. These details then changed, forcing changes to the implementation (aka. extra work for engineers).

The details don’t matter

The truth is that these details don’t matter.

From a technical perspective, it shouldn’t matter whether a welcome email is sent 2 hours after the user signs up or 15 minutes after. Nor should it matter that it was a welcome email or a tips and trick email.

That’s the realm of the business — these are merely use cases for the base technology.

The implementation itself should remain the same — as far as the system should be concerned, it sent a Foobar notification upon creation of a Whatsittoya record, and can use (mostly) the same exact code to send a Fizzbuzz notifcation upon the deletion of a Whatchamacallit record.

When it is sent or what is being sent are details that just aren’t as concrete. These are the details that should be abstracted away because these are the details that are the most likely to change over time.

You don’t want to have to engineer something every time a requirement changes. You want to give that power to the business, barring some contractual, security[1], or regulatory requirement.

[1] Security also includes job security.

Building a rules engine

So, now that we’ve examined the context in which a rules engine would be useful, how do you actually build one?

There’s a lot of ways, but conceptually a minimal rules engine includes the following components:

Rule

A rule is a business policy. Within the technical domain of the rules engine, it is a collection of a set of triggers, conditions, and effects that are applied by the rules engine.

Trigger

The trigger is the thing that determines whether the engine should attempt to run through a rule or not. In most cases, it is contextual.

In simple systems, it could be a simple string check or even hard-coded, such as a string on a Rule with the value on_create and PaymentAccount to indicate that the rule should only be executed when a record of type PaymentAccount is created.

In more complex systems, it could be a full context check that looks at things like whether the user is logged in or the kind of record being worked on.

Condition

The Condition determines if the Rule should be applied in that particular circumstance and on that particular record.

It has some minor overlap conceptually with a trigger, but also enough differences to warrant a separate discussion.

A trigger is more of a generalized determination of whether a rule’s conditions should even be checked, whereas condition is more of an instance-specific check. Smaller systems can potentially fold the trigger in to a a condition, but more complex systems will likely benefit from separating the two.

The Condition itself will likely have some sort of reference to actual code that performs the check itself, with some parameters to pass in to the function.

For example, you could store a Condition record in the database that stores the name of the condition class as well as some accompanying parameters. The code could then dynamically initialize the condition class it identifies, passing in the parameters and the record to check against.

Effect

The Effect is what happens once a Rule is triggered and its Conditions pass. It is typically a function or execution of a function. It may be something as simple as setting a field or something as complex as kicking off an entire workflow.

Like the Condition, it will likely have some sort of reference to actual code that applies the Effect itself.

Engine

Finally, you have the engine itself. This is the thing that will actually perform the bulk of the work. It’ll accept records, load a list of rules, check whether those rules should be applied based on specific triggers and conditions, and then apply the effects of the rules.

Some other areas of concern

In addition to the above components, you’ll also likely encounter secondary areas of concern when building the rules engine. Things like auditing changes to rules, tracking the history of rules being applied, enabling the configuration of rules to specific people, event-based effects, rule DSLs, etc. are all related subsystems, but also outside the scope of the core of the rules engine.

The flow of a rules engine

How would this rules engine actually work?

• Step 1: Trigger the engine

• Step 2: Get the Rules

• Step 3: Check the Conditions

• Step 4: Apply the Effects

Step 1: Trigger the Engine

The first step that occurs is the engine gets triggered. The entry point could be a function Engine#run. This function is passed a record and its associated context (eg. if the record is newly created, or the engine is being called in an update, etc.)

Step 2: Get the Rules

The engine will get the list of rules that apply for that specific context. Perhaps it will load it in real-time from the database. Maybe it pre-loaded it from a configuration file when the system booted.

Either way, these Rules will have associations to Conditions and Effects, which is what the important part is for the next couple of steps.

Step 3: Check the Conditions

The Condition would be a boolean check that determines whether the Rule’s Effects should be applied or not. If a Rule has multiple Conditions, further boolean logic should be performed to determine how the Conditions are evaluated to determine whether the Rule applies or not (eg. all or nothing, any, quorum).

Step 4: Apply the Effects

If the Rule’s Conditions pass, it is time to apply the Effects. The Effect itself is arbitrary. You‘ll want to determine how to handle cases in which the application of an Effect failed.

Once this happens, the Rule has been applied. You’ve done it!

A concrete example

Suppose you had to set up a user account to be marked as a “Popular” member once a certain number of views for the profile has been reached. You could theoretically hard-code this, but it would require an engineer to change in the future. Good engineering minimizes the cost and impact of likely future changes.

If you had a rule engine that you could use to configure this instead, engineers would not be required.

A Rule could be created that:

• had the Trigger of view

• had a Condition of view_count being greater than 100

• had an Effect of setting a field of the user to popular

The record would then run automatically through the rules engine, and could also be changed in the future.

Sample code

Having a hard time conceptualizing this? Not to fear — I’ve created a small Github repository to illustrate some of the concepts here.

JGefroh/rules-engine
This small library demonstrates a system capable of operating as a dynamic rules engine. You can read more in my…github.com

Note that this isn’t intended to be a production-grade system, but rather a simplified illustration of the above concepts. Actual rules engines have many more moving parts and take into consideration a whole host of other cases.

As a final disclaimer: improperly implemented rules engines or rules engines applied outside of the appropriate context can also make your system incredibly complicated and lead to a loss of organizational agility, so be judicious in their usage.

Rules engines are highly useful tools — in the right context. If implemented properly, they can significantly ease development burden and improve the agility of other departments within your organization.

In many startups I’ve worked with, image uploading was a part of their web application’s workflow. From user avatars to uploadable inventory pictures, it was a common-enough feature to be present in almost every system.

Rather unsurprisingly, many of those startups’ solutions to their uploading suffered from the same issues. Logic to handle image uploads was ad-hoc. File processing happened at the same time as the request, causing the application server’s request queue to back up.

In short, it wasn’t so much a system as a bunch of disparate workflows cobbled together (typical of most startups). The result? A lot of bugs in image handling, mysterious crashes that can’t be traced, and random timeouts.

I’m here to show you a better way.

Understanding the technical concerns

Image uploading can be complex, and use-cases vary depending on the system.

There’s a lot of concerns with images in general that most people don’t think of when they are diving into building it out.

Images in a web application can easily touch on the following technical concerns:

• Displaying the image on the front-end

• Authorizing users to download the image

• Authorizing users to upload the image

• Uploading the image to your server in a scalable way

• Validating the image data

• Processing the image, performing cropping, optimization, and other tasks

• Creating image variants, such as banners and thumbnails

• Storing the images

• Associating the images to whatever records you’re uploading them for (such as user avatars or campaign banners).

While not all of these would be present, a solid system will be able to flex and adapt to support these use cases with minimal changes.

The Tradeoffs

When thinking of a solution that can address these concerns, it quickly becomes evident that the final solution will be more complex than adding a column to your user model and an endpoint to upload the image, and calling it a day.

It’ll be helpful to dive into the tradeoffs to consider in a solution.

Scaling

Image uploads are notorious for crashing servers or causing timeouts. If a user attempts to upload a 10 megabyte image, that’s a lot of resource usage:

• 10 megabytes of your server‘s memory tied up

• A request handler being tied up for the entire amount of time it takes to upload 10 megabytes

• CPU usage to deal with the image upload

If you’re dealing with 1 or 10 users uploading images, it’s not a big deal. However, if your system is actually used by users, it’ll quickly go out of control.

As a result, the architecture is required to be focused on reducing the amount of time your server is actually handling an image uploading request to almost nothing, and offloading the actual upload to another service (such as S3 or an in-house service dedicated to uploads). You don’t want image uploading taking up all of your web server’s capacity.

Security

Image uploads and processing are a massive source of security holes. Any endpoint that lets you tie up a massive amount of resources is vulnerable to denial-of-service attacks (intentional or not).

On an even more worrisome note, image processing is a poorly understood aspect of engineering that has led to some tragic security flaws, providing random users the ability to execute arbitrary commands as a root-level user.

Our system architecture has to handle images in a secure way that promotes availability but also provides integrity of user data.

Authorization

On the security front, there’s also business logic specific to our system. Not all images should be publicly available. Perhaps there are situations where users should not be able to download images uploaded by other users. Perhaps there might be rules surrounding who can upload an image.

Our system has to be able to handle these domain-specific authorization cases easily.

Consistency

I’ve written about the value of consistency in the past. We don’t want a different way to upload an image for every kind of image we want to upload. Image upload use-cases like user avatars and campaign banners should all flow through the same image uploading workflow and should require minimal or no code change to support.

Variants

When an image is uploaded, there might be multiple places and ways it is used. Places like thumbnails, backgrounds, and profile images might use the same image in different ways. We might serve a lower resolution image to mobile users to save on bandwidth.

Whatever the case, we have to support the creation and usage of variants in our upload system. Creating variants can take a lot of processing power, and it’s not something we want our primary web server to do. The architecture has to offload that to a separate, asynchronous service.

Growth

All subsystems should be able to grow independently of the rest of the system. You never know when you’ll see an increased demand in usage. By keeping well defined boundaries in your system , you can easily convert them into micro-services or a separate deployment that scales horizontally.

The Architecture

With these trade-offs in mind, let’s now take a look at an architecture for uploading images that fulfills the criteria.

The Components

What are each of the parts of the architecture intended for?

The various little nuances and decision points in the architecture provide significant advantages to scalability, security, and flexibility.

But, what are those nuances? What is gained or lost by each decision point? Why are we choosing to use signed URLs or other elements? Let’s dive into the details.

Image Upload API

The image upload API handles the lifecycle of an image upload request. It’ll perform authentication, authorization, auditing, rate limiting, and other items, but it’ll leave the actual management of the file data to other services.

It never touches file data. I leave it as an exercise to the reader to find ways to make it RESTful.

Image Upload Service

The Image Upload Service encapsulates the logic of calling the endpoints in the Image Upload API with the right data in the right order, hiding it behind a single method interface. This temporal coupling is not something you want spread out through your entire system, so having a single source of truth for this algorithm is incredibly important.

Signed URLs

A signed URL is a URL that has authorization parameters in it. We can use signed URLs for uploading and downloaded to provide protection and ensure that only authorized users perform these actions.

Signed Upload URLs
In a simple case,/images/get_upload_url could return an unsigned URL: /uploads.

However, this means that anyone could just call /uploads and fill our data store with random files or use it as their personal file server. This is clearly not desirable.

/images/get_upload_url could return a signed URL:
/uploads?write_token=a99Xioajksf23.

This means that we have an opportunity to authenticate the user, authorize them, rate limit, or audit who generated the signed URL. If someone abuses our upload service, we have the means to stop that user.

Signed Download URLs
If we didn’t have signed download URLs, anyone could download the image if they knew the URL: /images/joseph-gefroh.png

This may be desirable in many cases, but in some cases, such as private files, it may be highly undesirable. In these cases, we wouldn’t want the URL to be publicly available.

We can gate access behind our own endpoint. 
images/get_download_url, which returns a temporary token such as:
/downloads?read_token=a99Xioajksf23.

Just like uploads, this provides us an opportunity to authenticate, authorize, limit, and audit user access to the images.

Cloud File Storage

We use a separate file storage service (such as S3) to reduce the burden on our web servers.

Upload endpoint
File uploads can take a long time and can be resource-intensive. By offloading such operations to a service dedicated to this, we can ensure the rest of our system continues operating smoothly.

Download endpoint
We also never serve the files directly from our web server for the same reasons listed above — we instead deliver it from the external file store. Any request to our server instead returns another URL or redirects to the appropriate service.

Image Processing Job

Image processing can be highly memory and CPU intensive — it’s not something you want to perform on your application or web server, which is busy handling other requests.

We turn this into an asynchronous call that is offloaded to another service so that we can keep our primary web server unblocked and performant.

Image Metadata Record

We store the metadata of the image record in our database because we have to track it. There’s no sense uploading an image without a way to retrieve or manage it later. What use is uploading a photo intended to be used as a user avatar if we have no way to associate with the user record in question?

It’s important to note that we do not store the actual image itself in our database, we merely store the metadata, which we then use later to construct the various URLs to it.

Why don’t we store the image URL? Storing the image URL is fragile, and the link is susceptible to breaking if the URL ever changes for any reason. Storing the metadata needed to construct the URL is a lot more robust — we can easily change things like the domain name, and build the appropriate URL without having any downtime.

The Algorithm

Let’s examine how we would use the components in our system to actually upload the image:

• Step 1: Client request an upload URL from the server (REQUEST)

• Step 2: Client uploads the image data to the upload URL (UPLOAD)

• Step 3: Client tells the server the upload is completed (CONFIRM)

• Step 4: Server processes image in background (PROCESS)

• Step 5: Client checks image processing status (CHECK)

• Step 6: Server is done processing image, notifies client (FINALIZE)

Step 1: Client request an upload URL from the server (REQUEST)

Why is the first step not an image upload? Remember — we don’t want the image data to ever even touch our servers. It is far too much of a resource hog and denial-of-service vulnerability.

Instead, we ask our server to give use the URL we should upload to. This URL could point to a 3rd-party cloud storage, such as S3, or another system specifically built to handle the load of image uploading.

During this step, the server can generate a random URL that is:

• time restricted

• audited

• authorized

These upload URLs are pre-signed URLs. That is, the URL the server returns has query parameters that indicate all of the authorization in it required to upload to the 3rd-party service.

After performing authorization checks, the server also creates a record in the database to track this individual image upload, with data on:

• the name of the file

• the type of the file

• the URL of the file

• the status (eg. requested, uploaded, processed)

• the associations of the image (eg. user, campaign)

• the kind of association (eg. banner, avatar)

• write token (a generated token that must be provided to modify the image)

• read token (a generated token that must be provided to read the image)

• any other audit data

The server returns this data to the client.

Step 2: Client uploads the image data to the upload URL (UPLOAD)

This is a fairly straightforward step.

The client, now armed with the Upload URL from the server, simply performs a POST request to that URL. That service then accepts that data and stores it.

Step 3: Client tells the server the upload is completed (CONFIRM)

The client makes a request to the server with the token the server returned earlier, telling the server that the upload was completed.

Step 4: Server processes image in background (PROCESS)

The server verifies the token, and then checks for that upload request.

It kicks off a job that will process the image — verifying file integrity, creating variants, and performing optimizations without blocking requests to the web server.

Step 5: Client checks image processing status (CHECK)

Processing images takes some time, and you don’t want the client blocking a request. The client should check back occasionally to see if the processing is done.

Step 6: Server is done processing image, notifies client (FINALIZE)

Eventually, the check will pass and server is going to return the image URL. Now, the client is free to use the image.

The security here is provided by the image URL. If the image is protected, the URL to access the image would point to our server, and any request by the client would have to provide a read token, which the server could use to perform authorization as needed and generate a temporary signed URL for that image.

If the image is meant to be public, the image URL could be a direct reference to the image, bypassing our server altogether.

That’s it.

Implemented, this image uploading system can scale significantly and handle a lot of potential use cases. Adding new images would be as simple as adding a few lines of code.

The complexity of the system would be hidden behind clean interfaces that have implementations that, once built, would rarely change.

Built correctly, it could also support general file uploading as well!

Did you like this article? Let me know in the comments, or connect with me on LinkedIn!

A feature you often need in web applications is the ability to order, filter, group, or organize records based on some arbitrary properties. In many situations, it’s simply called a “tag”, but the purpose and behavior is the same whether it’s called categories, metadata, groups, or descriptors.

Take a look at the storefront for the popular PC gaming platform STEAM. STEAM allows users to add text tags to games. Users can add any tag to a game, marking it as a “strategy” game, “painful” to play, or even just “goat”. Users can use these tags to filter their search to only show similarly tagged games, making it much easier to find the games they want.

Let’s take a look at several ways to approach tagging.

The Naive Approach

The naive approach you can do is to store the tags in as a comma separated string within a single database column, as shown below:

PRO: It’s dead simple.

Let’s face it. Adding a single column is really the easiest thing you can do.

CON: Querying is a pain.

How do you find potatoes that are delicious and warm, but not buttery or just delicious or just warm? Writing the query to do that would be difficult and cumbersome, especially as the number of possible tags increased.

CON: Prone to formatting errors

If a user adds a comma to their tag, you suddenly have a problem — that tag gets split into two tags.

CON: Limited by field length

Database columns have a maximum length and you might hit it.

Separate Columns

An evolution of the first approach is to store each tag of a record in its own column, conveniently labeled tag1, tag2, tag3, and so on.

PRO: It looks right.

Having multiple columns for multiple values seems to be following sound database practices…right? (not in this case).

CON: Querying might actually be harder.

It might look like a good idea on the surface, but querying is actually harder. Now you have to include every single column in the query. Additionally, your code would have to check for whether a tag existed in every possible tag column since there’s no guarantee that a tag would be in any specific column for any record. tag1 might have “buttery” in Record 1, but Record 2 might have “warm” in tag1 and “buttery” in tag2.

CON: Adding more tags is difficult.

Every time you wanted to add more tags, you would need to add a new database column and change all your queries in the process. Just like the naive approach, you’d reach the limit pretty quickly.

The Separate Model

A third approach is to create a separate model for the tag. If you’re tagging a Potato, for example, you would create a model called PotatoTag that had a reference to the original Potato via a potato_id foreign key.

PRO: Querying is easier than the naive approach

In order to find all potatoes with the tag “buttery”, you would simply query for any potato with a PotatoTag that said “buttery”.

PRO: No practical limit

Unlike the naive approach, you don’t have a practical limit to the number of tags any one record could have.

CON: Separate model for every kind of tag

PotatoTag, GameTag, SnailTag — every kind of model you have would require its own table, which would get out of hand very quickly.

The Polymorphic Model

Instead of having a separate model for every tag, you have a generic model that is polymorphic.

Polymorphic database models essentially means that it can reference tables it doesn’t know about with the help of an extra field. In a typical relationship you would store the primary key of a reference (like the numeric id). However, with a polymorphic association, you would also store the type of the object associated with the id (eg. taggable_id and taggable_type). This allows you to associate records with multiple kinds of objects, even if they share the same id but are different types.

PRO: Benefits of Separate Model Approach

This approach shares the benefits of the separate model approach but without the maintenance burden of actually having separate models.

PRO: Tag anything and use the same exact code

Polymorphic models let you stick to the Don’t Repeat Yourself (DRY) principle. The code to interact with tags is exactly the same no matter what kind of record you’re tagging.

The Polymorphic Model with Constraints

Let’s say you don’t want users to be able to create arbitrary tags, but rather select from a limited set of tags. You could easily enforce this rule in the backend code. Let’s add a twist and say that you don’t know ahead of time what tags are in that set of allowed tags. This could happen in situations where administrators could choose what tags are available.

You’d need to allow administrators to store the allowed tags, and then draw from that list of allowed tags when normal users want to tag a record. A model where the tag information lies in a TagOption instead of in the tag itself can support this situation. The Tag itself would simply be an associative entity between the Taggable record and the TagOption, with the TagOption itself having the actual information of the tag.

PRO: Limit tags to pre-approved set.

This approach lets you limit tags to an arbitrary set of tags.

PRO: Globally change tags

Because the actual tag information lives in the TagOption, you can change the meaning of a tag without changing the models associated with it. For example, if you had a tag option called “orange” and you wanted to change it to say “Orange”, you could by changing a single record (the TagOption)instead of every Tag in the database.

These are a few of the ways you could design a tag feature in your system. Remember that a tag can be anything, not just a word or name! It could be a color, a priority, or really any discrete value you can use to categorize or group records.