One pipeline, four clients: rebuilt from scratch with Claude Code

· 5 min read

I inherited a document pipeline that had been built for one client. By the time it reached me, it was broken and stale. I got it working again first, then re-architected the entire solution from scratch and built it with Claude Code in VS Code. The result was one deployed C# application serving 4 clients and moving 12–15 GB of documents a week, more than 100,000 documents, from zip files in each client's environment into their data warehouse.

No single step was exotic. The hard part was making every step dependable at that volume, for four clients with different rules, while nobody was watching it run.

in the client's environment1 of 4 client configscreates the folders as neededso every file can be mergedrules from the client confighandles timeouts, exceptionsand delays; never loads twiceprocessed, to a person, failed;with charts made in Pythonprocess 1prepareprocess 2load, reportZip lands in the incoming folderUnzip, pick the client configRegex sort into foldersConvert other file types to PDFMerge by account numberLoad to the data warehouse (SQL)Send a readable summaryAudit trail: a timestamped entry for every step

One deployed application. Process 1 prepares each client's files, process 2 loads them into the data warehouse, and every step writes a timestamped audit trail entry.

1. Stabilize first, then re-architect

A broken pipeline still has people waiting on it, so my first job was to get documents moving again. Then I stepped back and designed the replacement as a solution architect: what the four clients had in common, where they differed, and which failures to expect at that volume.

2. Split the work into two processes

Process 1 prepares the files. It watches each client's incoming folder for new zip files, unzips them, picks that client's config, sorts every file into its folder with regex rules, converts other file types to PDF, and merges the PDFs by account number. Process 2 goes through the merged files and loads each one into the client's data warehouse with SQL commands.

Keeping them apart means a slow load never holds up new files coming in, and each process can be rerun on its own.

3. Put every client difference in config

Four clients share one codebase. What changes per client lives in its config: where files arrive, the patterns that sort them, which file types to convert, how to merge, and where to load. Adding a client becomes a config change rather than a new build. A simplified sketch:

{
  "client": "client_b",
  "incomingFolder": "D:/incoming/client_b",
  "sortRules": [
    { "pattern": "^STMT_(?<account>[0-9]{10})_", "folder": "statements" },
    { "pattern": "^NTC_(?<account>[0-9]{10})_",  "folder": "notices" }
  ],
  "convertToPdf": [".tif", ".jpg", ".docx"],
  "mergeBy": "account",
  "warehouse": {
    "table": "dbo.ClientDocuments",
    "timeoutSeconds": 120,
    "maxAttempts": 5
  }
}

4. Expect the warehouse to push back

At 12–15 GB a week, SQL timeouts, exceptions and delays are part of normal operation, not rare events. So every load retries with a pause, records each attempt, and checks the audit trail first, so a rerun never loads the same file twice. When the retries run out, the file is marked failed and shows up in the summary instead of disappearing quietly. A simplified sketch:

// Simplified sketch: load one merged PDF, retrying timeouts and delays
async Task LoadAsync(MergedFile file, ClientConfig cfg, AuditTrail audit)
{
    if (audit.IsLoaded(file.Id)) return;            // reruns never load twice
    for (var attempt = 1; ; attempt++)
    {
        try
        {
            await using var conn = new SqlConnection(cfg.Warehouse.Connection);
            await conn.OpenAsync();
            await using var cmd = new SqlCommand(cfg.Warehouse.InsertSql, conn);
            cmd.CommandTimeout = cfg.Warehouse.TimeoutSeconds;
            cmd.Parameters.AddWithValue("@account", file.AccountNumber);
            cmd.Parameters.AddWithValue("@pdf", file.Bytes);
            await cmd.ExecuteNonQueryAsync();
            audit.Write(file.Id, "load", "loaded", attempt);   // timestamped
            return;
        }
        // on the last attempt the error goes up and lands in the summary
        catch (SqlException ex) when (attempt < cfg.Warehouse.MaxAttempts)
        {
            audit.Write(file.Id, "load", $"retry: {ex.Message}", attempt);
            await Task.Delay(TimeSpan.FromSeconds(30 * attempt));
        }
    }
}

5. Leave a trail and a summary

Every workflow step writes an audit trail entry with a timestamp. If anything goes wrong, the engineering team can go straight to the file and the step where it happened. After each run, a human-readable summary goes out as a notification, with charts built in Python: how many documents were processed, which ones need a person and why, and anything that failed. The client sees what happened without having to ask, and triage starts from facts.

6. How I built it: architect-led, agent-built

I built the entire solution with Claude Code in VS Code: multiple C# projects for the pipeline, and Python for the charts in the summary. I call the approach architect-led, agent-built. I owned the architecture, the client rules and the final call on every change, and Claude Code worked as the build agent, writing and refactoring the sorting, conversion, merge and load code with me. Every change went through the same four-step loop:

  1. Context. A CLAUDE.md file at the root of the solution described the architecture, the two processes, the conventions and each client's rules, so every session started from the same picture.
  2. Plan. Before any code, Claude Code and I agreed on a plan for the change: which files, which steps, and what could break.
  3. Build. Claude Code made the change in small steps, and I read every diff.
  4. Verify. Tests ran the regex and merge rules against sample file names and batches before the change was kept.

The structure of the solution followed the workflow, one project per stage, so each part could be tested and changed on its own. A simplified view:

DocumentPipeline/
  CLAUDE.md               architecture, conventions, client rules
  clients/                one config per client: a.json ... d.json
  src/
    Pipeline.Intake/      watch the folder, unzip, pick the client
    Pipeline.Prepare/     regex sort, convert to PDF, merge by account
    Pipeline.Load/        SQL loads with retries
    Pipeline.Common/      config, audit trail, notifications
  charts/                 Python charts for the summary
  tests/
    samples/              sample file names and batches per client
    Pipeline.Tests/       regex, merge and load tests
From the field

Other ways teams approach this

Every company handles document pipelines differently, and there's no single right answer. Some teams buy a document capture product, some build on a cloud workflow service, and some call a model at runtime to classify documents when file names aren't reliable. Regex and rules work well when naming is consistent. Teams also use AI coding tools in different ways, from quick prototypes to production systems. What carries across all of them: config per client, loads that are safe to retry, and an audit trail people can follow.

A checklist for rebuilding a pipeline

  1. Stabilize what's running before you re-architect it.
  2. Keep intake and loading in separate processes that can each be rerun.
  3. Put every client difference in config, not code.
  4. Treat timeouts and delays as normal: retry, record each attempt, never load twice.
  5. Write a timestamped audit trail entry for every step, and send a summary people can read.
  6. Build with Claude Code in a loop: context, plan, build, verify.

All field notes