Collected thoughts on building an internet-scale web service

October 11, 2026 (3w ago)

Lessons from building and shipping an internal build system at a large company.

I’ve been at my current job1 for over a year now, and in my time, I’ve had the unique opportunity and challenge of building and deploying our internal build system and distribution network. In the process of doing so, I have learned some lessons from both the successes and failures2 and these are a compilation of them.

The lessons mix advice and heuristics for building (hopefully lasting) product at a large company.

1. Nothing is shipped until they say we shipped

In his article How I ship projects at big tech companies Sean Goedecke argues that shipping is a social construct.

Concretely, that means that a project is shipped when the important people at your company believe it is shipped. If you deploy your system, but your manager or VP or CEO is very unhappy with it, you did not ship.

Keeping this mindset central to the product development strategy and communication is the easiest route to build trust capital and having a portfolio of value-driving products. The way to kickstart this process is to start small feedback loops. “Does this new feature help ?” “Hey, I saw you had this problem with the current system; what’s your ideal workflow?” “If we did it this other way, would you use it then ?” Act on this input and then collect more input. Small feedback loops compound. Eventually you move from making patches to shipping product.

Engineers who think shipping means delivering a spec or deploying code will repeatedly engineer their way into failed ships.

Interestingly, universities never broach this topic. While coursework is an amazing way to act on your curiosity and delve into material, it is not a primer for what matters in the real world.

2. You have no product until you have numbers.

This goes hand-in-hand with point #1. Communication about your product and what you shipped, especially to leadership that doesn’t understand software, is a different ball game compared to communicating to a manager or colleagues. I’ve learnt this the hard way working in an industry (automotive) where executives have neither the background nor the eagerness to adapt to the paradigms of software.

Leadership does not know what a Kubernetes cluster on the cloud is or what the hell Cloudflare does, nor do they particularly care. You’ve got to get on their level and say:

  • “We served X users today to deploy Y apps. There have been Z builds on our remote build farm since we’ve been up.”
  • “We are working on a caching system that would reduce a build time for users from T_1 to T_2.”
  • “We’ve introduced Alertmanager, and that has reduced our incident response time to under T_3”
  • “We’ve introduced pytest, a framework for unit testing, and in X min we have achieved YY% coverage. ”

I would go further and say that if you don’t know how to reduce what you’ve deployed into the lines above, you haven’t shipped product.

On that note, it's best to have these systems (observability and metrics) in place before you claim that you have shipped. (See #5 in Appendix A).

3. Figure deployment pipelines as early as possible

A deployment pipeline is the steps it takes to make a feature available to the public right after you’ve developed it.

This goes 2 ways:

  1. A fully or semi-automatable series of steps that deploys your code to staging/prod instances - these can be done via GitHub Actions, Artifactory, an internal rollout system, etc. It is paramount that you build and set these up once you’ve got something working and you have people using them.
  2. A development environment representative of what you see in production. - If you’re deploying with Docker, use a prod-representative docker-compose.yaml; if you’re deploying with Kubernetes, have a prod-representative cluster.

No. 1 reduces the time and effort to showcase your work as soon as you’ve done developing and testing it. (See #4 of Appendix A). No. 2 reduces the gap between how it’s like on your machine and how it’s like in production, and it irons out some kinks in CI/CD. (See #2 of Appendix A)

The major questions to ask yourself are:

  • “How can I simulate a production environment as quickly as possible locally ?”
  • “How can I automate rollouts in a timely and controlled manner?”
  • “If someone requests a feature, how can I reduce time outside development to get said feature in their hands?”
  • “If an incident or outage were to occur, outside of patching the fix, what are the other hurdles I have to jump through to show that it's fixed?”

Having a daemon in your brain, repeatedly asking these questions is a valuable exercise and building / documenting systems that help you solve them increases overall velocity.

4. Dissociate identifiers from artifacts

This is a nugget of data engineering learnt the hard way. Identifiers are any markers that can be linked to a user of your system. Artifacts are the data your user stores outside the database. For eg. a profile picture or gallery is an artifact that is stored in an S3 bucket with the key stored in the database.

In my case, with a C++ / Docker build system, the artifacts were .so files and Docker images. The takeaway is not to save these like johndoe/version1/build_for_x86.tar.gz or save with any user-specific information. Instead, save them like builds/884d92ba22b64a4c9a0e5a9f7d11602b. The reason is twofold:

  1. Flexibility - the artifacts can be remapped, transplanted, transferred, etc., and no one is the wiser. Having artifacts with user identifiers makes it harder to manipulate.
  2. Single source of truth - the database is the only place that understands and encodes the mappings between users and their data. If and when some of these mapping structures change, a single migration of the database is sufficient.

5. Low-hanging high-visibility fruit are Golden Apples.

Low-hanging high-visibility fruit (or ‘LHHV fruit’ if I’m being an obnoxious engineer) are products or tools that seem so obviously helpful, but no one has made them yet. Stop to make them. These are Golden Apples.

The beauty of these are that you get to showcase your work at a small scale (internal development group vs target customer base) and build trust capital. Whether its an internal tool to save people time or an observability integration to ease people’s efforts in debugging, once people see that you’ve done something well they start trusting you with more. This compounds over time and you become a DRI3 for bigger product.

These opportunities don’t come along all that often and its paramount that you jump on them as soon as you can. So stop your race to pick up these Golden Apples.

6. Simple systems get adopted. Complex systems break.

The best part is no part. -Elon Musk4

There has never been a more harmful oxymoron in engineering than “beautifully complex”. History has shown us time and time again, and companies/products that don’t respect it have paid for it.

In his phenomenal article The Grug Brained Developer, Carson Gross refer to Complexity as the mortal enemy of a developer. This is something I agree with wholeheartedly. An addendum that I’ve boiled it to is:

 Simple and reliable > Complex and efficient \text{ Simple and reliable } > \text{ Complex and efficient }

This is not a hall pass to write something poorly, but rather a heuristic that an engineer should keep in mind when defining and implementing solutions. It is better to have something simple that users can trust to run 24/7 than something efficient that breaks down every now and then. This is true for any software and especially true of an internal build system.

The modern developer landscape is a Red Queen's Race. You’re racing to meet requirements that are racing away from you. To act fast and to stay nimble, it behooves oneself to use the 80/20 principle i.e try and achieve 80% of the results with 20% of the changes you make in code.

7. Get it onto paper.

Seriously, get it onto paper. Take a pencil and some paper and draw out the feature and its interactions with the various systems. How users invoke it. What the design ramifications are. The constraint of paper not only forces you to develop a synopsis of your idea, but also helps prune and refine your proposed solution to the necessary bits.

With code now becoming a commodity, an engineer’s job shifts to owning tradeoffs when building product. The necessary friction of expressing your idea out loud (or out soft onto paper) is very useful if you’re building customer-facing products in a team.

8. Code is easier to write than read.

Everyone worth their salt in engineering knows this quote. But I want to break it up into 2 parts:

  1. Code is easier to write than to read. Programmers always want to rewrite things. This is a costly fallacy.
  2. Prioritise human readability over line-count or smart tricks when programming. This helps prevent failures that stem from #1.

“Code is easier to write than to read” is an adage that has stuck around in the community since software became an industry. In his seminal article, Joel Spolsky talks about this and how programmers always want to bulldoze old systems and rewrite them from the ground up. Ground-up rewrites almost never go well. More often than not, these leave the company in a place where they can neither build new features from customer demand, nor patch bug fixes. Eventually the company cedes market share and gets on a fast-track to oblivion.

Remember that code is not for machines. The computer only understands binary. The code is for humans. Code is simply a translation layer of idea to text. The compiler, linker and assembler take that text and inform a computer of what to do. So if you want your idea preserved and used, hopefully for a long time, it is better to write code that transmits this idea to other people and to future you. i.e write code for humans.

Donald Knuth, a programming Hall-of-Famer, said this about writing code for humans in his essay Literate Programming - Donald Knuth

Literature of the program genre is performable by machines, but that is not its main purpose. The computer programs that are truly beautiful, useful, and profitable must be readable by people. So we ought to address them to people, not to machines. All of the major problems associated with computer programming—issues of reliability, portability, learnability, maintainability, and efficiency—are ameliorated when programs and their dialogs with users become more literate.

9. It takes a tough man to make tender chicken.

The MIT guy pointed out that the implementation was simple, but the interface to the functionality was complex. The New Jersey guy said that the right tradeoff has been selected in Unix – namely, implementation simplicity was more important than interface simplicity.

I learnt this point from Richard P. Gabriel's essay Rise of Worse Is Better. My point here is as a DRI of a project, to develop a reliable value-generating product that users like to use, difficult decisions need to be made. You will need the grit and discernment to say ‘No’ to both managers and the feature-devil on your shoulder. To ensure that your product lasts, discipline gets you further than raw intellect.

Learning to say ‘No’ as an engineer is an important skill, both at a micro and macro level. Micro being fighting scope creep. Macro being telling Management (who often do not understand engineering concerns) ‘No’. This goes hand in hand with #6.

Through both my job and my milieu, I’ve noticed that programmers/teams that have a healthy mix of discipline and intellect go the furthest, especially when building customer-facing product.

Appendix A - On responsible scaling

Scaling something from a prototype to a full-fledged product is a long-winded task5. While working on our internal build system, I devised a plan for how to achieve it. The plan most pertinently applies to web development but can be mapped onto systems programming with some modifications. The following list is what you will need to progress through to ensure a reliable internet-scale web service.

  1. Formatting and Linting - to maintain code quality and reduce developer toil.
  2. Seed Data - should be as close to production data as possible.
  3. Automated Testing Suites (unit + integration)
  4. Automated Deployments / Testing - automated via scripts you run or hooks or GitHub Actions.
  5. Observability - logging, tracing, basic metrics to see how your
  6. Load Testing + E2E testing - should be done ideally on a copy of prod data or at least a representative slice of it.
  7. Benchmarking and Profiling - this and L6 should ideally go hand-in-hand. These are more granular than the metrics of L5.
  8. Security and Static Analysis - to prove to Big Brother Corporate that your product is secure.
  9. Automated Backups and Data Recovery
  10. Disaster Response and Runbooks

Appendix B - On AI usage

I’ve been keeping up to date with the latest AI tricks and use cases and I’ve honed in on a workflow that has served me pretty well so far. Some of them are:

  • Do one thing and do it well. - I don’t believe in running multiple Claude agents to run multiple different tasks at the same time. The coding flow that has brought me the most reward has been where I work on one issue and have it augment my understanding of the issue and help with research.
  • Plan, plan, plan and then implement - It comes from the adage “Measure twice cut once”. I will never one-shot a feature or big code change. I usually ask it to generate specs and tell me what needs to change and why. After iterations of this, I will get it to do it one part at a time.
  • Whiteboard defence - as described by Mitchell Hashimoto in this tweet. “Can I justify the decisions my product has made at any point in time?”. I try to live by this.
  • Welcome questioning - I always tell the agent to ask me questions for clarification. Always for a feature. Sometimes for a bug fix. We humans are poor information compressors, so a single prompt (or even a CLAUDE.md) will not be sufficient to translate your thoughts. Welcome questioning.
  • Rules of engagement - define clearly how Claude (or any coding agent) should interact with your repository and where information can be found. If it changes X part, it needs to run Y tests. A and B in the codebase are coupled and if things change in one, they need to change in the other, etc.

Some useful plugins for Claude Code

Appendix C - On Ownership and CTFOIGT

Ownership isn’t about being territorial or closed-minded. It shouldn’t limit our willingness to collaborate or invite criticism. It doesn’t even mean doing all the work yourself. It just means taking responsibility for the work getting done or, if it cannot be done as planned, communicating broadly to reset expectations.

CTFOIGT stands for ‘Chill The Fuck Out I Got This’. At the end of the of day, after all the cake and watermelon, what most managers, and some teammates, are looking for is assurance. They want to work with people who define a plan and say “I will get it done”. Better yet, they want someone who ‘gets it’. Someone they don’t need to break down the problem for.

Ownership and CTFOIGT go hand in hand. In my experience, a product ships when one person takes on the role of shipping it. Tip of the Spear. The default state of teams and products is to not ship. Making the product go 0 →1 requires one person to be directly responsible and say “CTFOIGT”.

It pays dividends to say CTFOIGT. It pays even more dividends to know when to say it.

References

Footnotes

  1. Software Engineer at Martinrea International ↩

  2. More from failures than successes tbh. ↩

  3. Directly Responsible Individual ↩

  4. There is some “debate” about whether he actually invented this line or it comes from a line of wise engineering sages. ↩

  5. No matter the farce Base44 sells you ↩