diff --git a/src/pages/blog.astro b/src/pages/blog.astro index 9831296..e85ae96 100644 --- a/src/pages/blog.astro +++ b/src/pages/blog.astro @@ -82,106 +82,6 @@ import NewsletterSignup from '../components/NewsletterSignup.astro';
-
-

Dark Factory Series

-
- - - - - - - - - - - - - - - -
-
- -
-
diff --git a/src/pages/blog/human-in-the-loop-seam.astro b/src/pages/blog/human-in-the-loop-seam.astro index 9e570d2..b09fd8f 100644 --- a/src/pages/blog/human-in-the-loop-seam.astro +++ b/src/pages/blog/human-in-the-loop-seam.astro @@ -4,7 +4,7 @@ import BaseLayout from '../../layouts/BaseLayout.astro'; Human-in-the-loop is a phrase everyone uses. Almost nobody uses it correctly.

-

I have been in four meetings this quarter where someone said, "the human is in the loop," and meant, "a person reviews the diff before the merge." I have been in three more where someone said the same thing and meant, "a person approves the deploy to production." I have been in two where the speaker meant, "we are researching this." All seven were talking about different systems, and none of them were using the phrase the way I would.

- -

Here is how I would use it. A human is in the loop at a seam. The seam is a place where the system is underdetermined. The seam is the place where the spec does not say what the right answer is, and the factory has to either ask or guess. The human's job is to be the answer to the questions the spec cannot answer. The factory's job is to surface those questions, at the right time, in the right form, to a human who can answer them.

+

A human is in the loop at a seam. The seam is a place where the system is underdetermined. The seam is the place where the spec does not say what the right answer is, and the factory has to either ask or guess. The human's job is to be the answer to the questions the spec cannot answer. The factory's job is to surface those questions, at the right time, in the right form, to a human who can answer them.

The wrong place to put the human

-

The wrong place to put the human is in the implementation. If the spec is complete enough to be compiled, the factory should compile it. Adding a human to "review the code" before the merge is putting the human in a place where the human is duplicating the factory's work. The factory just generated the artifact from the spec. The human reviewing the artifact is asking "did the factory do what the spec said." The factory can answer that question with a provenance graph. The human is not adding information; the human is adding latency.

+

The wrong place to put the human is in the implementation. If the spec is complete enough to be compiled, the factory should compile it. Adding a human to "review the code" before the merge is putting the human in a place where the human is duplicating the factory's work. The factory just generated the artifact from the spec. The human reviewing the artifact is asking whether the factory did what the spec said. The factory can answer that question with a provenance graph. The human is not adding information. The human is adding latency.

-

I have watched teams do this. I have watched the human review take two days. I have watched the human approve what the factory produced. I have watched the human introduce small changes that break the spec. I have watched the spec and the artifact drift apart because the human was the seam, and the seam was in the wrong place. The seam should have been in the spec, not in the review.

+

Teams that put the human in the review get a familiar pattern. The review takes two days. The human approves what the factory produced. The human introduces small changes that break the spec. The spec and the artifact drift apart because the human was the seam, and the seam was in the wrong place. The seam should have been in the spec, not in the review.

The right place to put the human

The right place to put the human is on the seam where the spec is underdetermined. There are three of these seams in a dark factory, and they are different from each other.

-

The first seam is the policy seam. The policy is the human's signature on the system. The policy says, "this kind of artifact, in this kind of environment, is allowed to promote." The policy is underdetermined because the policy is a human value judgment, not a derived fact. The factory cannot derive what is allowed; the factory can only enforce what has been declared. The human is the one who declares. The factory is the one who enforces.

+

The policy seam. The policy is the human's signature on the system. The policy says: this kind of artifact, in this kind of environment, is allowed to promote. The policy is underdetermined because the policy is a human value judgment, not a derived fact. The factory cannot derive what is allowed. The factory can only enforce what has been declared. The human declares. The factory enforces.

-

The second seam is the spec seam. The spec is underdetermined in places. The spec describes intent. The intent does not always fully constrain the artifact. The factory will produce a correct artifact from an underdetermined spec; the artifact will be one of many correct artifacts, and the choice between them is a taste decision. The human is the one who has the taste. The factory surfaces the choice. The human makes the call. The spec is updated with the call. The factory's next artifact is constrained by the updated spec.

+

The spec seam. The spec is underdetermined in places. The spec describes intent. The intent does not always fully constrain the artifact. The factory will produce a correct artifact from an underdetermined spec. The artifact will be one of many correct artifacts, and the choice between them is a taste decision. The human has the taste. The factory surfaces the choice. The human makes the call. The spec is updated with the call. The factory's next artifact is constrained by the updated spec.

-

The third seam is the audit seam. The audit is the place where the human checks that the factory did what the spec said, and the spec was the right call. The audit is the only seam where the human gets to ask the second-order question. The audit is the only seam where the human can change the spec. The audit is the only seam that closes the loop. Without the audit, the factory is the only one learning. With the audit, the human is learning too.

+

The audit seam. The audit is the place where the human checks that the factory did what the spec said, and the spec was the right call. The audit is the only seam where the human gets to ask the second-order question. The audit is the only seam where the human can change the spec. The audit is the only seam that closes the loop. Without the audit, the factory is the only one learning. With the audit, the human is learning too.

What the human looks like at each seam

-

At the policy seam, the human is a writer. The human writes the policy. The policy is a small artifact, versioned, signed, reviewed. The policy is not code; the policy is intent. The factory is the executor. The human does not need to know the factory's internals. The human needs to know what the factory is for.

+

At the policy seam, the human is a writer. The human writes the policy. The policy is a small artifact, versioned, signed, reviewed. The policy is not code. The policy is intent. The factory is the executor.

-

At the spec seam, the human is an editor. The human updates the spec to capture a decision the factory surfaced. The spec is a living document. The spec is not a wiki; the spec is a file the factory reads. The human does not need to be a programmer. The human needs to be a domain expert who can describe what they want in the domain's vocabulary.

+

At the spec seam, the human is an editor. The human updates the spec to capture a decision the factory surfaced. The spec is a living document. The spec is not a wiki. The spec is a file the factory reads. The human does not need to be a programmer. The human needs to be a domain expert who can describe what they want in the domain's vocabulary.

-

At the audit seam, the human is a reader. The human reads the factory's output. The human reads the spec the factory consumed. The human reads the policy the factory enforced. The human asks, "did the right thing happen?" The human's answer goes into the next spec, the next policy, the next audit. The human's answer is the loop closing.

+

At the audit seam, the human is a reader. The human reads the factory's output, the spec the factory consumed, the policy the factory enforced. The human asks: did the right thing happen? The human's answer goes into the next spec, the next policy, the next audit.

Why this is harder than it looks

-

The hard part is that the seams are not labeled in the artifact. The factory produces an artifact. The artifact has a hash, a signature, a provenance graph. None of those say "this decision was made by a human at the spec seam." The factory has to be designed to make the seam visible. The factory has to be designed to ask the question at the right time. The factory has to be designed to wait for the answer, or to ship a default, depending on the policy.

+

The seams are not labeled in the artifact. The artifact has a hash, a signature, a provenance graph. None of those say "this decision was made by a human at the spec seam." The factory has to be designed to make the seam visible. The factory has to be designed to ask the question at the right time. The factory has to be designed to wait for the answer, or to ship a default, depending on the policy.

-

This is the work of designing a dark factory. The work is not the build pipeline. The work is the seam management. The work is the question of when to ask, when to wait, and when to ship.

+

This is the work of designing a dark factory. The work is not the build pipeline. The work is the seam management.

-

Most teams skip this work. Most teams put the human in the implementation review, because that is the obvious place. The obvious place is the wrong place. The wrong place is the place where the human is duplicating the factory's work, and the spec is drifting from the artifact, and the audit cannot find the seam because the seam is in the wrong layer.

+

The single rule

-

The single sentence

- -

If you remember one thing from this post, remember this. The human belongs on the seam where the spec is underdetermined, not on the seam where the implementation is risky.

+

The human belongs on the seam where the spec is underdetermined, not on the seam where the implementation is risky.

Most teams have it backwards. Most teams put the human where the implementation is risky. Most teams have the human review the code, the test, the deploy. Most teams are duplicating the factory's work, and they do not know it. The factory can review the code. The factory can run the test. The factory can ship the deploy. The factory cannot decide what the code should be. The factory cannot decide what the test should check. The factory cannot decide what the deploy should do.

diff --git a/src/pages/blog/light-vs-dark-factories.astro b/src/pages/blog/light-vs-dark-factories.astro index a93f8b2..3ff697c 100644 --- a/src/pages/blog/light-vs-dark-factories.astro +++ b/src/pages/blog/light-vs-dark-factories.astro @@ -4,7 +4,7 @@ import BaseLayout from '../../layouts/BaseLayout.astro'; -

Pure dark factories are for vendors. Pure light factories are for teams that do not trust their pipeline. The real work is in between.

+

Pure dark factories are for vendors. Pure light factories are for teams that do not trust their pipeline. The interesting engineering is in between.

-

I have watched three engineering organizations try to go fully dark. Two of them rolled back inside six months. The third one shipped, and the on-call rotation got three times worse, because every failure that used to be caught at a human review was now a 3 AM page.

+

The ends are easy to describe. The middle is where the work is.

-

I have watched four organizations try to stay fully light. One of them is still running. The other three quietly automated the gate anyway and stopped calling it "human review."

- -

The lesson is not "dark is bad" or "light is bad." The lesson is that the ends are easy to describe and the middle is where the engineering happens.

- -

The light factory, named

+

The light factory

A light factory is what most teams have when they say "we have a CI/CD pipeline." Code is written. Tests run. A human reviews the diff. A human approves the merge. A human triggers the deploy. A human watches the deploy. A human confirms the deploy is healthy.

@@ -37,41 +33,39 @@ import BaseLayout from '../../layouts/BaseLayout.astro';

This is the team that hits a wall at fifteen engineers. Not because they cannot write code. Because they cannot move code through the human gates fast enough to keep up with the demand.

-

The dark factory, named

+

The dark factory

-

A dark factory replaces the human handoffs with policy. The spec is the input. The policy is the rules. The artifact is the output. Nothing in the middle requires a person to act.

+

A dark factory replaces the human handoffs with policy. Spec in, artifact out. The policy is the rules. Nothing in the middle requires a person to act.

-

This is the team that ships 200 deploys a day. The team is small. The team is bored. The product is moving fast, except when the policy is wrong, at which point the product is moving very fast in a bad direction and no one notices until the customer does.

+

This is the team that ships 200 deploys a day. The team is small. The product is moving fast, except when the policy is wrong. When the policy is wrong, the product moves fast in a bad direction and no one notices until the customer does.

-

What the middle looks like

+

The middle is a position, not a transition

-

The middle is not a transition. The middle is a position. It has three properties:

+

The middle has three properties.

    -
  1. The pipeline is the gate. The merge to main is not gated by a reviewer; it is gated by a policy expression. The policy is code. The code is reviewed.
    2. -
  2. The human is the auditor. The human's job is to inspect what the pipeline produced, not to be the pipeline. The audit is sampling-based, not exhaustive. The audit answers "did the policy work this week," not "is this merge correct."
    3. -
  3. The spec is the source of truth. The code is generated. The test suite is generated. The deploy manifest is generated. The thing the human writes is the spec, and the spec is what gets reviewed.
    4. +
  4. The pipeline is the gate. The merge to main is not gated by a reviewer. It is gated by a policy expression. The policy is code. The code is reviewed.
    5. +
  5. The human is the auditor. The human inspects what the pipeline produced. The audit is sampling-based, not exhaustive. The audit answers "did the policy work this week," not "is this merge correct."
    6. +
  6. The spec is the source of truth. The code is generated. The test suite is generated. The deploy manifest is generated. The thing the human writes is the spec, and the spec is what gets reviewed.
-

None of this is new. What is new is that the spec is now something a non-engineer can write in a domain-specific language, the compilation is fast enough to run on every commit, and the audit surface is small enough that one person can look at all of it in an afternoon.

+

What is new is the surface area. The spec is something a non-engineer can write in a domain-specific language. The compilation is fast enough to run on every commit. The audit surface is small enough that one person can look at all of it in an afternoon.

Why the middle is harder than either end

-

At the dark end, you only have to maintain the policy. The policy is the system.

+

At the dark end, you maintain the policy. The policy is the system.

-

At the light end, you only have to maintain the people. The people are the system.

+

At the light end, you maintain the people. The people are the system.

-

In the middle, you maintain the policy, you maintain the people, and you maintain the seam between them. The seam is the part that nobody talks about in conference talks. The seam is the part that fails. The seam is the part that determines whether your factory runs at all.

+

In the middle, you maintain the policy, you maintain the people, and you maintain the seam between them. The seam is what fails. The seam is what determines whether the factory runs at all.

A seam is a place where the human's authority and the pipeline's authority have to be reconciled. Promotion to production is a seam. Incident response is a seam. The decision to roll back is a seam. The decision to skip a test is a seam. The decision to add a new test is a seam.

-

Each seam has a default. The default is what the pipeline does when no human acts. The default has to be defensible. If the default is "ship," and the human does not act, the system has shipped. If the default is "do not ship," the system has stalled.

- -

You will be tempted to make every default "do not ship." That is the light factory wearing a different uniform. You will be tempted to make every default "ship." That is the dark factory pretending it does not have humans. The middle is the work of choosing the default for each seam, and defending the choice.

+

Each seam has a default. The default is what the pipeline does when no human acts. If the default is "ship" and the human does not act, the system has shipped. If the default is "do not ship," the system has stalled. The default has to be defensible. The default has to be chosen.

How to know which position you are in

-

Ask one question: when the human does nothing, does the system ship or stall?

+

One question. When the human does nothing, does the system ship or stall?

If the answer is ship, you are dark. If the answer is stall, you are light. If the answer is "it depends on which stage," you are in the middle. The middle is where you want to be, and the middle is the part you have to design on purpose.

diff --git a/src/pages/blog/operator-patterns-for-software-factories.astro b/src/pages/blog/operator-patterns-for-software-factories.astro index 1d246c5..2fd42cd 100644 --- a/src/pages/blog/operator-patterns-for-software-factories.astro +++ b/src/pages/blog/operator-patterns-for-software-factories.astro @@ -4,7 +4,7 @@ import BaseLayout from '../../layouts/BaseLayout.astro'; An operator is a control loop with a domain in it. That is the whole definition. Everything else is implementation.

-

I have written thirteen production operators. Six of them are still running. The other seven were replaced by better versions of themselves, which is the correct outcome. Here is what I learned.

+

A factory is composed of operators, each one owning a seam. The seams are the spec, the build, the test, the deploy, the promote, the rollback. Each seam gets one operator. The operators do not call each other. The operators watch the cluster and react.

-

The operator is the unit of automation in a software factory. The factory does not have one operator; the factory is composed of operators, each one owning a seam. The seams are: the spec, the build, the test, the deploy, the promote, the rollback. Each seam gets one operator. The operators do not call each other. The operators watch the cluster and react.

+

The spec operator

-

That is the design pattern. The rest of this post is the implementation choices inside it.

+

The spec operator owns a SoftwareSpec custom resource. A human writes the spec. The operator validates the spec. The operator emits a normalized form into the cluster, signed, with provenance attached.

-

Pattern 1: the spec operator

+

The spec operator is the only operator that takes human input. Every other operator in the factory watches a spec-shaped resource. The spec operator is the auditor's first stop: what specs exist, who wrote them, what versions are live.

-

The spec operator owns a SoftwareSpec custom resource. A human writes the spec. The operator validates the spec. The operator emits a normalized form of the spec into the cluster, signed, with provenance metadata attached.

+

Keep it thin. The spec operator's job is to validate and emit. It does not run tests. It does not trigger builds. It does not know what a build is.

-

The spec operator is the only operator that takes human input. Every other operator in the factory watches a spec-shaped resource. The spec operator is the auditor's first stop: "what specs exist, who wrote them, what versions are live."

- -

I write these in Go with controller-runtime. I keep them thin. The spec operator's job is to validate and emit. It does not run tests. It does not trigger builds. It does not know what a build is.

- -

Pattern 2: the build operator

+

The build operator

The build operator watches a normalized spec. The build operator produces a BuildArtifact custom resource containing the hash, the SBOM, the provenance graph, and the signed manifest. The build operator does not know about clusters. The build operator does not know about deployment. The build operator's job is to take a spec and produce an artifact that other operators can verify.

-

This is the operator I have rewritten the most. The first three versions did too much. They knew about tests, about deployment, about promotion. Every time they knew too much, they became unmaintainable. The current version is 400 lines of Go. It does one thing. It is boring. Boring is what I want from a build operator.

+

The build operator is the operator that wants to know too much. Every time the build operator learns about tests, about deployment, about promotion, it becomes unmaintainable. The build operator that survives is the one that does one thing. Boring is what you want from a build operator.

-

Pattern 3: the deploy operator

+

The deploy operator

The deploy operator watches a signed BuildArtifact and a target environment. The deploy operator produces a Deployment custom resource. The deploy operator's job is to take a verified artifact and put it in a verified place.

-

The deploy operator is where most teams make their first mistake. They have the deploy operator trigger a kubectl apply, or call a Helm release, or invoke a serverless deploy. All of these are wrong. The deploy operator should produce a declarative resource, and a separate, generic Kubernetes primitive (Deployment, StatefulSet, whatever) should reconcile to that resource. The deploy operator is the translator. The cluster is the executor.

+

The deploy operator is where most teams make their first mistake. They have the deploy operator trigger a kubectl apply, or call a Helm release, or invoke a serverless deploy. All of these are wrong. The deploy operator should produce a declarative resource, and a separate, generic Kubernetes primitive should reconcile to that resource. The deploy operator is the translator. The cluster is the executor.

-

Why this matters: when the deploy operator produces a resource, the resource can be inspected, audited, rolled back, frozen. When the deploy operator pushes to a remote API, none of those things are true. The remote API is the system. The remote API is what the auditor cannot see.

+

When the deploy operator produces a resource, the resource can be inspected, audited, rolled back, frozen. When the deploy operator pushes to a remote API, none of those things are true. The remote API is the system. The remote API is what the auditor cannot see.

-

Pattern 4: the policy operator

+

The policy operator

The policy operator is the seam. The policy operator watches deployments and consults a policy bundle. The policy bundle is a versioned, signed set of rules. The policy operator's job is to decide which deployments are allowed to promote to which environment.

-

This is the operator that is hardest to write. The first version I wrote used OPA. The second version I wrote used Cedar. The third version I wrote used a custom DSL because none of the policy engines could express what I needed without a three-page Rego file that nobody on the team could read. The fourth version is the one in production. It is 800 lines. It is the most important operator in the factory.

+

The policy operator is the one that is hardest to write. The off-the-shelf policy engines (OPA, Cedar, Rego, the rest) cannot express what most factories need without a policy file that nobody on the team can read. The version that survives is a custom DSL, small enough to fit on a page, expressive enough to encode the rules the business actually needs.

-

The policy operator is where the human's authority lives in a dark factory. The policy is the human's signature on the system. The policy operator enforces that signature. Everything else in the factory can be fully dark; the policy operator is the part that cannot be.

+

The policy operator is where the human's authority lives in a dark factory. The policy is the human's signature on the system. The policy operator enforces that signature. Everything else in the factory can be fully dark. The policy operator is the part that cannot be.

-

The pattern I will not use again

+

The anti-pattern

-

Early on, I tried to write one operator per application. One operator per microservice. The factory had 30 operators, and every one of them was a slightly different shape. The team spent more time learning the operators than they spent shipping product.

+

One operator per application. One operator per microservice. The factory has 30 operators, and every one of them is a slightly different shape. The team spends more time learning the operators than they spend shipping product.

-

The mistake was treating the operator as a wrapper around the application. The operator is not a wrapper. The operator is a wrapper around a type of change. The factory has one operator per type of change, regardless of how many applications use that type. The application is a parameter, not a peer.

+

The mistake is treating the operator as a wrapper around the application. The operator is not a wrapper around the application. The operator is a wrapper around a type of change. The factory has one operator per type of change, regardless of how many applications use that type. The application is a parameter, not a peer.

The composition rule

@@ -71,12 +67,6 @@ import BaseLayout from '../../layouts/BaseLayout.astro';

This rule sounds like bureaucracy. It is not. It is what makes the factory auditable. When operator A triggers operator B, the trigger is a side effect of state, and the state is in etcd, and the etcd is queryable. When operator A calls operator B's API, the trigger is hidden in operator A's code, and the audit is impossible. Choose the auditable version. The factory you build next year will thank you.

-

What this gets you

- -

A factory built from these patterns has a single engineer operating it. The single engineer is the spec writer and the policy writer. The single engineer does not write operators day to day; the operators are stable. The single engineer writes the policy when the business changes. The single engineer writes the spec when the product changes.

- -

That is the leverage. A single engineer, writing two kinds of artifact, with a factory that ships 200 times a day. The factory is the platform team's product. The single engineer is the platform team's customer. The factory is dark. The customer is not.

-
diff --git a/src/pages/blog/the-taste-problem.astro b/src/pages/blog/the-taste-problem.astro index 0aafaf7..a112a92 100644 --- a/src/pages/blog/the-taste-problem.astro +++ b/src/pages/blog/the-taste-problem.astro @@ -21,13 +21,11 @@ import BaseLayout from '../../layouts/BaseLayout.astro';
-

A dark factory can ship code. It cannot ship the judgment that knew the code was right. That is the part nobody talks about, and it is the part that determines whether the factory works.

+

A dark factory can ship code. It cannot ship the judgment that knew the code was right.

-

There is a thing that a senior engineer knows that a junior engineer does not. It is not the language. It is not the framework. It is not the syntax of the patterns. It is the shape of the right answer before the answer exists.

+

There is a thing a senior engineer knows that a junior engineer does not. It is not the language. It is not the framework. It is the syntax of the patterns. It is the shape of the right answer before the answer exists. The senior engineer looks at a 200-line function and says, in 30 seconds, "this is the wrong shape; it should be three things, not one." The junior engineer will spend three days on the same function, write the same wrong answer, run the tests, and ship it. The tests pass. The code is fine. The shape is wrong. The shape is what is going to cost the team a quarter.

-

I have watched a senior engineer look at a 200-line function and say, in 30 seconds, "this is the wrong shape; it should be three things, not one." The junior engineer will spend three days on the same function, write the same wrong answer, run the tests, and ship it. The tests pass. The code is fine. The shape is wrong. The shape is what is going to cost the team a quarter.

- -

That is taste. Taste is the human judgment that knows the right shape of a system before it exists. Taste is the part that does not transfer through a code review, because the code review happens after the shape is fixed. Taste does not transfer through a style guide, because the style guide is the part of the taste that can be made explicit. The part that is not explicit is the part that matters.

+

That is taste. Taste is the human judgment that knows the right shape of a system before it exists.

Why the dark factory has a taste problem

@@ -37,13 +35,13 @@ import BaseLayout from '../../layouts/BaseLayout.astro';

Most teams discover this six months in. The factory is shipping. The customers are complaining. The metrics are flat. Someone looks at the artifacts and realizes the artifacts are exactly what the specs said to build. The specs are exactly what the engineers wrote. The engineers wrote what the market asked for. The market asked for the wrong thing. The factory shipped the wrong thing at scale.

-

That is the taste problem. The factory did not break. The factory did what it was told. The factory is not the failure mode. The factory is the amplifier. The judgment that was missing is the failure mode. The factory just made the missing judgment load-bearing.

+

The factory did not break. The factory did what it was told. The factory is not the failure mode. The factory is the amplifier. The judgment that was missing is the failure mode. The factory just made the missing judgment load-bearing.

What "encode taste" actually means

There is a school of thought that says you can encode taste into the spec. The spec gets more detailed. The spec captures the shape. The spec becomes the artifact's design doc, and the design doc gets reviewed.

-

That school of thought is half right. You can encode some taste into the spec. The encodable taste is the part that is visible: naming conventions, layering, error handling, the explicit invariants. The encoded taste transfers through the factory. The factory produces artifacts that respect the encoded taste. The encoded taste shows up in the artifact's diff.

+

That school of thought is half right. Some taste encodes. Naming conventions, layering, error handling, the explicit invariants. The encoded taste transfers through the factory. The factory produces artifacts that respect the encoded taste. The encoded taste shows up in the artifact's diff.

The other half of taste does not encode. The other half is the part the senior engineer carries and cannot articulate. It is the part that knows the function should be three things, not one, before the function is written. It is the part that knows this commit is going to cause the next three. It is the part that knows the customer is asking for X but they need Y.

@@ -55,25 +53,17 @@ import BaseLayout from '../../layouts/BaseLayout.astro';

What provenance does is make the missing judgment legible. When the artifact fails, the provenance graph tells you which spec element the failure traces back to. The auditor can then ask: was that spec element the right call? The question is answerable. The answer is in the spec, not in the artifact.

-

That is not a solution. That is a microscope. A microscope does not cure disease. A microscope tells you what disease you have. In a dark factory, the microscope is the only thing that keeps the missing judgment from being invisible until it is too late.

+

A microscope does not cure disease. A microscope tells you what disease you have. In a dark factory, the microscope is the only thing that keeps the missing judgment from being invisible until it is too late.

The two kinds of taste

-

I have come to think of taste as two things, and I treat them differently.

+

Encoded taste is the part that goes in the spec. It transfers. It is reviewable. It is the part the factory can carry forward. The senior engineer's job is to push as much taste as possible into this category. Most of what we call code review is the act of pushing taste from the implicit category to the explicit one.

-

Encoded taste is the part that goes in the spec. It transfers. It is reviewable. It is the part the factory can carry forward. The senior engineer's job is to push as much taste as possible into this category. Most of what we call "code review" is the act of pushing taste from the implicit category to the explicit one.

+

Unencoded taste is the part that does not go in the spec. It does not transfer. It is reviewable only by other senior engineers who carry the same unencoded taste. It is the part that gets lost when a senior engineer leaves. It is the part that no dark factory, no matter how sophisticated, can carry.

-

Unencoded taste is the part that does not go in the spec. It does not transfer. It is reviewable only by other senior engineers who carry the same unencoded taste. It is the part that gets lost when a senior engineer leaves. It is the part that no dark factory, no matter how sophisticated, can carry.

+

The dark factory does not have to carry the unencoded taste. The dark factory has to make the unencoded taste visible, so the human's audit can find the missing pieces.

-

The dark factory does not have to carry the unencoded taste. The dark factory has to make the unencoded taste visible, so the human's audit can find the missing pieces. The dark factory is the microscope. The human is the cure. The human is the part that does not scale.

- -

What this means for a single engineer

- -

If you are a single engineer running a dark factory, the leverage is real and the constraint is real. The leverage is that you ship at the speed of policy, not the speed of typing. The constraint is that every spec you write is now load-bearing in a way it was not before. The factory amplifies your taste. The factory also amplifies your lack of taste, in places you did not know you lacked it.

- -

The single engineer's job is to make the implicit explicit, in every spec, in every policy, in every audit. The single engineer's job is to be the human in the loop on the part of the loop that no machine can do. The single engineer's job is the unencoded taste.

- -

You cannot hire your way out of this. You cannot vendor your way out of this. You can only become the kind of engineer whose taste transfers through the spec, and accept that the rest of it has to stay in the audit. That is the job. That is the work.

+

You cannot hire your way out of this. You cannot vendor your way out of this. You can only become the kind of engineer whose taste transfers through the spec, and accept that the rest of it has to stay in the audit.

diff --git a/src/pages/blog/what-is-a-dark-factory.astro b/src/pages/blog/what-is-a-dark-factory.astro index c93811a..f04c7ca 100644 --- a/src/pages/blog/what-is-a-dark-factory.astro +++ b/src/pages/blog/what-is-a-dark-factory.astro @@ -4,7 +4,7 @@ import BaseLayout from '../../layouts/BaseLayout.astro'; -

The phrase is everywhere. The meaning is gone.

+

The phrase is everywhere. The working meaning is gone.

-

I sat down to write about software factories last week, and the first thing I had to do was take the word back. Every vendor has a "dark factory" slide. Every DevRel team has a "lights-out deployment" demo. The phrase has been stripped of its working meaning, the way "platform engineering" was stripped of its working meaning two years ago, the way "DevOps" was stripped of its working meaning before that.

+

I sat down to write about software factories, and the first thing I had to do was take the word back. Every vendor has a "dark factory" slide. Every DevRel team has a "lights-out deployment" demo. The phrase has been stripped the way "platform engineering" was stripped, the way "DevOps" was stripped before that.

-

So before I can tell you why I am writing a book about this, I have to tell you what I mean by the word.

+

The marketing version: AI writes the code, the pipeline ships the code, humans go home early. That is a fantasy. It assumes the spec is correct, the spec is complete, the spec captures every invariant the running system has to preserve. None of those are true today. None of them will be true next year.

+ +

The working version is a smaller claim, and a more useful one.

The working definition

-

A dark factory is a build pipeline that ships production software with no human in the critical path. The spec goes in. The artifact comes out. The intermediate steps; specs compiled, code generated, tests run, security scans, SBOMs, signing, deploy, rollback wiring; happen without a person approving each one.

+

A dark factory is a build pipeline that ships production software with no human in the critical path. Spec in, artifact out. The intermediate steps happen unattended.

-

The opposite is a light factory: same inputs, same outputs, but a human signs off between each major stage. The human is the gate.

+

The opposite is a light factory. Same inputs, same outputs, but a human signs off between every major stage. The human is the gate.

-

Most teams in 2026 are not running either. They are running a partial factory. Some stages are dark (CI runs unattended). Some stages are light (the merge to main requires a reviewer). The work is figuring out which stages should be which, and what the seams look like.

+

Most teams in 2026 are not running either. They are running a partial factory. Some stages are dark; CI runs unattended. Some stages are light; the merge to main requires a reviewer. The work is figuring out which stages should be which, and what the seams look like.

-

What the marketing version gets wrong

+

The five positions

-

The marketing version: "AI writes the code, the pipeline ships the code, humans go home early." That is a fantasy. It assumes the spec is correct, the spec is complete, the spec captures every invariant the running system must preserve. None of those are true today. None of them will be true next year.

- -

The working version: the dark factory is not "AI replaces engineers." It is "the engineer's judgment gets encoded once, at the spec layer, and the pipeline carries it forward without re-interpretation." That is a much smaller claim. It is also the only claim that actually scales.

- -

The naming

- -

I borrow the term from manufacturing. A dark factory in the literal sense is a building with no lights on because no humans are inside; robots do the work. The analogy breaks in one place and one place only. In a physical dark factory, the parts are made of metal. In a software dark factory, the parts are made of decisions. And decisions are the one thing that does not transfer cleanly across a handoff.

- -

That is the book. The next several posts are the load-bearing pieces.

- -

The spectrum, named

- -

Rather than treating "dark" and "light" as a binary, I find it useful to name five positions on the spectrum:

+

Treating "dark" and "light" as a binary loses the part that matters. The positions, named:

    -
  1. Manual. A person runs the commands. No factory yet; the engineer is the build system.
    2. -
  2. Assisted. CI exists. Tests run. A person merges. Most teams in 2026 live here and call it "DevOps."
    3. -
  3. Light. Merge is gated by policy, not by reviewer. The pipeline is the gate. A human is the auditor, not the approver.
    4. -
  4. Dark, gated. The pipeline ships. A human is the only one who can promote from staging to production. The dark factory runs, but it cannot cross a particular seam without a key turn.
    5. -
  5. Dark, full. The pipeline ships. Promotion is policy-driven. The human's role is to write the policy and the spec, then to inspect the artifacts the factory produces.
    6. +
  6. Manual. A person runs the commands. No factory yet; the engineer is the build system.
    7. +
  7. Assisted. CI exists. Tests run. A person merges. Most teams in 2026 live here and call it DevOps.
    8. +
  8. Light. Merge is gated by policy, not by reviewer. The pipeline is the gate. A human is the auditor, not the approver.
    9. +
  9. Dark, gated. The pipeline ships. A human is the only one who can promote from staging to production. The factory runs, but it cannot cross a particular seam without a key turn.
    10. +
  10. Dark, full. The pipeline ships. Promotion is policy-driven. The human's role is to write the policy and the spec, then to inspect the artifacts the factory produces.
-

Most serious engineering organizations in 2026 are between three and four. The interesting question is not "how do we get to five" but "which of our stages should be at which position, and why."

+

Most serious engineering organizations in 2026 sit between three and four. The interesting question is not how to get to five. The interesting question is which of your stages should be at which position, and why.

-

Why the term matters

+

What the framing changes

-

Calling it a factory changes the questions you ask. You stop asking "who is on call for the deploy" and start asking "what is the failure mode of the deploy step." You stop asking "how fast can my engineers merge" and start asking "how fast can the policy express a correct merge." You stop asking "is the code reviewed" and start asking "is the spec from which the code was generated auditable."

+

Calling it a factory reallocates where the engineering effort goes.

-

The factory framing is not a metaphor. It is a reallocation of where the engineering effort goes. A factory is designed, not grown. A factory is maintained, not improvised. A factory has a failure budget, not a heroism budget.

+

You stop asking who is on call for the deploy and start asking what the failure mode of the deploy step is. You stop asking how fast your engineers can merge and start asking how fast the policy can express a correct merge. You stop asking whether the code was reviewed and start asking whether the spec the code was generated from is auditable.

-

If you have been calling a pile of bash scripts and a Slack channel "the build system," the factory framing will be uncomfortable. That is the point.

+

A factory is designed, not grown. A factory is maintained, not improvised. A factory has a failure budget, not a heroism budget.

-

What comes next

+

If the build system you are running today is a pile of bash scripts and a Slack channel, the factory framing is going to be uncomfortable. That is the point.

-

The next post in this series is Light vs Dark Factories; why most teams will end up in the middle, and why the middle is harder to run than either end.

+

The next post in this series is Light vs Dark Factories.

@@ -105,13 +95,6 @@ import BaseLayout from '../../layouts/BaseLayout.astro'; font-weight: 600; } - .post-content h1 { - font-family: var(--font-heading); - font-size: clamp(2rem, 4vw, 3rem); - color: var(--color-brown-dark); - margin: 0 0 1rem; - } - .lead-question { font-size: 1.25rem; font-weight: 600; diff --git a/src/pages/blog/where-dark-factories-are-already-running.astro b/src/pages/blog/where-dark-factories-are-already-running.astro deleted file mode 100644 index f45c897..0000000 --- a/src/pages/blog/where-dark-factories-are-already-running.astro +++ /dev/null @@ -1,150 +0,0 @@ ---- -import BaseLayout from '../../layouts/BaseLayout.astro'; ---- - - - -
-
-
- July 16, 2026 - AI -
-

Where Dark Factories Are Already Running (and What to Learn From Them)

-
- -
- -

Three companies are running pieces of a dark factory in production right now. None of them call it that. The vocabulary is missing, and the vocabulary is the part you need.

- -

I spent the spring talking to platform teams at four companies that are visibly running some version of an unattended build pipeline. Three of them were willing to talk about it on background. The fourth one said no, and that is fine, because what they are doing is not what I want to write about anyway.

- -

What I want to write about is the shape of what is actually working. The marketing version of "the dark factory" is a fantasy. The working version is a smaller claim, and the smaller claim is the one you can actually learn from.

- -

Company 1: a hyperscaler you have heard of

- -

This is the one that surprised me. I expected a small team with a clever pipeline. What I found was a 60-person platform org running a fully dark build pipeline for one of the company's internal product surfaces. The pipeline takes specs from a domain-specific tool the company built. The pipeline produces signed, attested artifacts. The pipeline ships them to a production cluster without a human in the loop.

- -

The pipeline has been running for three years. The pipeline has had four outages. The first outage was caused by a bad spec. The second outage was caused by a spec that was good but a tool that was bad. The third outage was caused by a network policy that was too aggressive. The fourth outage was caused by a human, in the audit, pushing a button they should not have pushed.

- -

The interesting detail: the policy operator is the only operator in the system that is written by hand. Everything else is generated. The policy operator is 4,000 lines of Go, and the engineer who wrote it said, "I will rewrite it next year. It is the part I am least proud of, and the part I most need."

- -

Company 2: a fintech that does not want to be named

- -

This is the company I thought would be running a dark factory. They are not. They are running a light factory with a dark middle. The spec-to-artifact step is fully unattended. The deploy-to-production step requires a human key turn. The key turn is the policy.

- -

The reason is regulatory. The company has a regulator. The regulator wants a person to be the gate. The regulator does not care whether the person is right; the regulator cares that the person is the gate. The company designed the system around the regulator, not around the engineering ideal.

- -

This is the most honest dark factory I have seen. The factory is shaped by the constraint, not by the marketing. The factory ships 50 times a day. The factory has a key turn. The key turn is the audit surface the regulator needs. The factory runs the way the business needs it to run, not the way the conference talk would have it run.

- -

Company 3: the agentic platform you are using

- -

This is the one most people will recognize, and the one I have written about before. A major agent platform is, in effect, running a dark factory for the prompts-to-actions translation layer. The user writes a prompt. The platform produces an action. The action is signed. The action is logged. The user did not approve the specific action; the user approved the prompt that led to the action.

- -

The pattern is the same as a spec-to-artifact build pipeline. The prompt is the spec. The action is the artifact. The signature is the provenance. The platform is the factory. The user is the policy.

- -

The interesting failure mode: the platform is good at the translation. The platform is bad at the policy. The user writes a prompt, and the platform produces an action, and the user has no audit surface to ask "what did the platform do, and why." The audit surface is the part nobody has built. The audit surface is the part that is going to define whether these platforms survive the next regulatory cycle.

- -

What the three have in common

- -

All three are running a version of the same shape:

- -
    -
  1. A spec layer. The human writes the spec. The spec is the contract.
    2. -
  2. A build layer. The build is fully dark. The build produces signed artifacts.
    3. -
  3. A deploy layer. The deploy is dark, gated, or light, depending on the regulatory environment.
    4. -
  4. A policy layer. The policy is the human's signature. The policy is the audit surface.
    5. -
  5. An audit layer. The audit is a separate step. The audit is run by a different team than the build. The audit asks "did the policy work this week," not "is this artifact correct."
    6. -
- -

None of them call it a "dark factory." All of them are running one. The vocabulary is missing. The vocabulary is the part that lets the rest of us talk about what is actually happening, instead of either pretending it is a marketing demo or pretending it is impossible.

- -

What the three get wrong

- -

All three are bad at the audit layer. The audit is treated as a separate concern. The audit is not integrated into the build. The audit is something the compliance team does, not something the factory does. This is a category error. The audit is a seam. The audit belongs to the factory, not to a different org chart.

- -

The second thing they get wrong: the spec is treated as a documentation artifact, not as a code artifact. The spec lives in a wiki. The build lives in the cluster. When the spec changes, the build does not know until someone updates the build to read the new spec. The spec and the build are two systems. The spec and the build should be one system, with the spec as the source of truth and the build as the derived artifact. The provenance graph is the link.

- -

What you can do this quarter

- -

Pick one part of your pipeline that is currently gated by a human review. Pick the part that is the most boring, the most repeated, the least interesting. The one your senior engineer resents reviewing. Make that part dark.

- -

Write down the spec. Make the spec a file. Make the file versioned. Make the build take the file. Make the artifact signed. Make the policy check the signature. Make the deploy gate on the policy. Watch what happens.

- -

You will find two things. You will find that the part you made dark is faster and more reliable than it was. You will find that the human review you removed was catching a small number of real problems, and those problems now need a new home. The new home is the spec. Move the judgment from the review to the spec. Make the spec load-bearing. Make the build respect the spec. Make the policy enforce the spec.

- -

That is the lever. That is the part you can do this quarter. The rest is scale.

- -
-
- - - - diff --git a/src/pages/blog/why-kubernetes-for-dark-factory.astro b/src/pages/blog/why-kubernetes-for-dark-factory.astro index fa1ee36..b226577 100644 --- a/src/pages/blog/why-kubernetes-for-dark-factory.astro +++ b/src/pages/blog/why-kubernetes-for-dark-factory.astro @@ -4,7 +4,7 @@ import BaseLayout from '../../layouts/BaseLayout.astro'; A dark factory is a control loop. Kubernetes is the most boring control loop in production. That is the reason.

-

I have built dark factories on Nomad. I have built them on bare systemd. I have built them on a single VM with a cron job and a Slack webhook. The smallest one ran on a Raspberry Pi in a closet in Texas for nine months without paging anyone. The largest one ran on a managed Kubernetes service and handled 400 deploys a day across four regions.

+

What the substrate has to do

-

Of those, the Kubernetes one is the only one I would recommend to a single engineer in 2026. Here is why.

- -

What a dark factory needs from its substrate

- -

A dark factory has four requirements from whatever it runs on:

+

A dark factory has four requirements from whatever it runs on.

    -
  1. Declarative desired state. The system has to be able to say "I want N copies of this artifact running, and the latest signed version, behind this network policy," and have the substrate reconcile to that state without the engineer babysitting.
    2. -
  2. Restart and replace as a primitive. Failures are not exceptions. Failures are the steady state. The substrate has to make "kill it and start a new one" cheap, automatic, and observable.
    3. -
  3. Operator-extensible control plane. The factory needs to add new resource types as it grows. It needs the substrate to treat those resource types as first-class, not as jobs squashed into the lowest common denominator.
    4. -
  4. Boring upgrades. The factory runs unattended. The substrate upgrades cannot require a person to read release notes on a Saturday.
    5. +
  5. Declarative desired state. The system has to be able to say "I want N copies of this artifact running, signed, behind this policy," and have the substrate reconcile to that state unattended.
    6. +
  6. Restart and replace as a primitive. Failures are the steady state. "Kill it and start a new one" has to be cheap, automatic, and observable.
    7. +
  7. An operator-extensible control plane. The factory grows new resource types as the system grows. The substrate has to treat those types as first-class, not as jobs squashed into the lowest common denominator.
    8. +
  8. Boring upgrades. The factory runs unattended. The substrate cannot require a person to read release notes on a Saturday.

Kubernetes is not special because it is fast, or because it scales, or because it is cloud-native. Kubernetes is special because it is the only widely-deployed substrate that makes all four of those requirements boring.

Why the alternatives fall short

-

Serverless (Lambda, Cloud Functions, Workers) gets close on restart-and-replace. It fails on declarative desired state. The platform owns the loop, not you. The moment you need a custom resource type, the platform tells you no. The factory that wants to define a BuildRun resource cannot do it on Lambda.

+

Serverless (Lambda, Cloud Functions, Workers) gets close on restart-and-replace. It fails on declarative desired state. The platform owns the loop, not you. The moment you need a custom resource type, the platform tells you no. A factory that wants to define a BuildRun cannot do it on Lambda.

-

Systemd on a VM gets close on boring upgrades. It fails on operator-extensible control plane. The minute you want a custom resource type, you are writing your own reconciler. You are now maintaining a control plane in addition to the factory. The single engineer does not have that time.

+

Systemd on a VM gets close on boring upgrades. It fails on the operator-extensible control plane. The minute you want a custom resource type, you are writing your own reconciler. You are now maintaining a control plane in addition to the factory. The single engineer does not have that time.

Nomad is the most underrated of the three. Nomad is what Kubernetes would look like if HashiCorp had cared more about operator patterns and less about service mesh. Nomad is excellent. Nomad is also a single-vendor system, and the ecosystem of pre-built operators and CRDs does not exist. You will build everything from scratch.

-

Docker Compose on a single host works until the host dies. Then the dark factory is dark in a different sense; the lights are off, the artifact is gone, and you find out what your backup posture actually is.

+

Docker Compose on a single host works until the host dies. Then the dark factory is dark in a different sense. The lights are off. The artifact is gone. You find out what your backup posture actually is.

The single-engineer argument

-

The argument I want to make is the leverage one. Kubernetes is the only substrate that gives a single engineer the same leverage a 20-person platform team gives a thousand. That leverage comes from the operator pattern.

+

An operator is a reconciler. The operator watches a custom resource. The operator makes the cluster match the resource. The operator is written in Go and packaged as a container. The operator runs the same way the artifact runs. The operator is part of the factory.

-

An operator is a reconciler. The operator watches a custom resource. The operator makes the cluster match the resource. The operator is written in Go (mostly) and packaged as a container. The operator runs the same way the artifact runs. The operator is part of the factory.

- -

A platform team of 20 cannot build a custom resource type faster than a single engineer with an operator template. The single engineer's bottleneck is the spec, not the infrastructure. The infrastructure is the easy part. The hard part is figuring out what the factory should do, and that is the same problem in both cases.

+

A platform team of 20 cannot build a custom resource type faster than a single engineer with an operator template. The single engineer's bottleneck is the spec, not the infrastructure. The infrastructure is the easy part. The hard part is figuring out what the factory should do, and that is the same problem at any team size.

What Kubernetes does not give you

-

Kubernetes does not give you a factory. Kubernetes gives you a substrate on which a factory can be built. The gap between "I have a cluster" and "I have a factory" is roughly the size of the next post in this series; the operator pattern.

+

Kubernetes does not give you a factory. Kubernetes gives you a substrate on which a factory can be built. The gap between "I have a cluster" and "I have a factory" is the size of the next post in this series: the operator pattern.

-

Kubernetes also does not give you provenance. The cluster will run whatever artifact you give it. The cluster does not know where the artifact came from. The cluster does not know whether the artifact was generated from a spec, or by a human, or by a model with no spec. The cluster does not care. Provenance is a layer above the substrate, and the layer above Kubernetes is where the dark factory actually lives.

- -

The choice

+

Kubernetes also does not give you provenance. The cluster will run whatever artifact you give it. The cluster does not know where the artifact came from. The cluster does not know whether the artifact was generated from a spec, or by a human, or by a model with no spec. The cluster does not care. Provenance is a layer above the substrate, and that is where the dark factory actually lives.

If you are a single engineer in 2026 trying to build a dark factory, the choice of substrate is not really a choice. Kubernetes is the answer. The interesting work is what you build on top of it.

-

If you are a platform team of 20, the choice is also Kubernetes, but the leverage argument is reversed. The platform team has the leverage to consider Nomad, or a managed control plane, or a custom substrate. The single engineer does not. The single engineer needs to be the platform team's customer, not the platform team's peer.

-