A few weeks after external engineers join a team, someone usually asks whether the arrangement is working. The honest answer tends to rest on impressions: the standups feel productive, the backlog seems to move, or a manager has a vague worry. Impressions are a poor basis for a decision that affects budget and roadmap.
Delivery data is a better basis, provided it is chosen carefully. A few signals, read together and compared with the team’s own earlier numbers, say more about augmentation than a long status report does.
Take a baseline before anyone joins
The most common measurement mistake is starting too late. Without numbers from before the engagement, any later figure has nothing to be compared with, and a good or bad month cannot be told from normal variation.
Record the current state of a few signals for the product or service the new engineers will work on: how long changes take to reach production, how often the team deploys, how long pull requests wait for review, and how many deployments cause incidents. A few months of history is usually enough to see the normal range. In the staff augmentation model that JetRuby describes, senior engineers join the client’s team and work in its process under the client’s direction, which also means the baseline belongs to the client: nobody else holds the history of the team’s delivery, so the starting position has to be recorded on the client side before the comparison can mean anything.
The DORA metrics as a common vocabulary
The most widely used set of software delivery metrics comes from the DORA research program. Its current definitions split into two groups (DORA, software delivery performance metrics). Three describe throughput: change lead time, meaning how long a change takes from commit to running in production; deployment frequency; and failed deployment recovery time. Two describe instability: change fail rate, the share of deployments that need immediate intervention such as a rollback or hotfix, and deployment rework rate, the share of deployments that are unplanned and made in response to a production incident.
The two groups matter because they check each other. Throughput going up while instability also goes up is not an improvement; it is speed bought with quality. A healthy augmentation shows throughput stable or rising with instability flat or falling.
Two cautions from the research itself
The DORA guidance is explicit about how the metrics should not be used, and the warnings apply directly to augmentation.
First, the metrics are meant to describe a single application or service. Comparing very different applications is misleading. Do not compare the team that received external engineers with another team that works on a different system and call the difference an effect.
Second, the guidance warns against setting the metrics as targets. Mandating a number invites people to game it. Used as a guide, the metrics point to where to look. Used as a quota, they stop measuring anything. The goal, in the guidance’s framing, is a team’s improvement against its own past, not competition with others.
Add the signals that DORA does not cover
The DORA metrics are measured at the point of deployment. Augmentation shows its first effects earlier in the flow, so a few upstream signals are worth adding.
- Cycle time. The time from starting work on an item to finishing it. If it lengthens after external engineers join, look at handoffs and unclear ownership before looking at individuals.
- Review latency. The time between a pull request being opened and receiving its first substantive review. When it grows, in-house reviewers are probably absorbing the extra load, which cancels part of the capacity that was added.
- Pull request size and rework. Large pull requests and repeated rounds of changes suggest that expectations are not shared.
- Work in progress. The number of items open at once. A rising number often signals that the team is starting more than it finishes.
- Escaped defects. Bugs found after release in areas where the new engineers worked, compared with the same measure for earlier periods.
Every item on this list is available from the repository and the ticket system, without asking anyone to fill in a form.
Read the signals as a pattern
A single metric rarely tells the story. Patterns are more reliable.
If throughput rises and review latency rises with it, the team has more output than its reviewers can absorb, and the fix is on the review side. If lead time stays flat but escaped defects rise, the new work may be passing through review without enough scrutiny. If cycle time falls and change fail rate is stable, the added capacity is being used well. If every number is unchanged after a quarter, either the engineers are working on something other than the measured product or the constraint was never capacity.
That last case matters because adding people does not help when the bottleneck lies elsewhere, for example in approvals, slow environments or unclear priorities. Measurement can show this early and cheaply, which is among its best uses.
Add a short qualitative check
Numbers cannot answer some questions: whether in-house engineers feel supported or burdened, whether external engineers understand the product’s reasoning, and whether decisions are being made where the context lives. A brief monthly conversation with both groups, using the same three or four questions each time, fills the gap. Useful questions are what slowed you down this month, what was unclear, and what would you change in how we work together.
Treat answers as hypotheses to compare with the metrics. A complaint about slow reviews that matches a rising review latency is a finding. One that does not match anything may be a communication problem worth solving separately.
Decide in advance what a good result looks like
Agree on the review points at the start: a first look after the opening weeks, a fuller assessment after a quarter. Decide which signals will be read, who reads them, and which patterns would lead to a change such as adjusting the review process, narrowing the scope of the external engineers’ work, or ending the arrangement. This is different from setting targets. The decision criteria describe what the team will do in response to a pattern, and do not demand a fixed number.
Team augmentation works when it adds capacity without lowering stability, and the way to see that is to measure the team against its own history, with a small set of signals, from the first week. Tracking started later can still be useful, yet it will never be as informative as a baseline taken before the first new engineer joined.
