Make Evil Plans that Don’t Work Out All of the Time
And Make Sure you Tell them to the Good Guy Before Implementation… More on the Trust Assembly Stuff and Reasonable Expectations
The Next Right Question
So, there I was…
Picture yours truly on a conference call with forty or so people, trying to understand an entirely new area of the business. My job? Simply to find a way to increase control performance over a process I hadn’t even known existed until that morning. For the roughly two years I did this, my job was basically to be MacGyver plus fill out endless amounts of paperwork.1
I didn’t have a Swiss Army Knife that worked on Zoom, so I started asking dumb questions instead.
The error our monitoring team found was mysterious on the surface but really stupid at its root.2 Basically, a user interface was pushing an update through a microservice and then the host system was returning a success message confirming the update was accepted. Sounds great, right? Even better, when the user interface displayed the success message it showed the updated piece of information right there! You could update a field value to “ABCDEFG” and see right on the screen that “ABCDEFG” had been updated. The only problem was that the update never actually happened.
When you have something like this break and nobody knows what is wrong right off, your best course of action is to pull in every development team involved. Then you push a test transaction and make each of the teams confirm what it looked like going into their custody and what it looked like as it exited their custody. Eventually, the game of telephone breaks and you’ve found your error. You can try going to one team and hope you get it right, but this almost never works out in practice and you usually get caught in a game of blame-shifting. That’s how this call started.
About eighteen months previously, a developer had introduced a typo to a configuration file when moving their code from the test environment to the production environment. They’d added a “ ” at the end of several field names read by the microservice that carried these transactions to our host system. Then that microservice, in effect, asked the CRUD service at the host to create a transaction to a field that didn’t exist.
In the computer world, “privacy_option” and “privacy_option ” are wildly different things. The CRUD service responded that it had successfully completed an update to a field that didn’t exist. Then the user interface, to save bandwidth and increase service level agreements on host responses, combined that success message with a cache file of the values sent to display the confirmation to agents and customers. The whole thing had been built for speed instead of accuracy, which is actually completely normal.
But at the end of the day, the systems were basically saying: “Congratulations! You asked us to do nothing and we did!”
This was an eighteen month old typo that was probably going to cost the company a good chunk of change in remediation. I needed to not only identify everyone this had happened to and make them whole, but I also had to make sure that this provably wouldn’t happen ever again.
“Right now, it’s like you throw something over a fence and someone says ‘I caught it!’ There’s no information about what was thrown or what was caught. Can we add more information to that call and response confirmation so we say ‘I threw a small red ball over the fence’ and the CRUD responds ‘I caught the small red ball!’ or is that not possible?”
In the controls world, we called this A/B reconnaissance, developers called it some sort of handshake, but there were non-technical business people on so I used the analogy instead.
I asked the Product Owner for their thoughts. They gave me a run down of what it would take to make that level of change. If we wanted to change the way the user interface was displaying these success messages or the CRUD service was sending confirmation messages, we could submit a funding request, but it was likely to be declined for not being economic. Complicated vendor relationships were involved that would further slow the change. We were also outside the window of time where we could get the funding and make the change before internal deadlines applied. These were root features of the tech stack and would be very expensive to alter. I asked several more questions to make sure all those responses fit together.
I already knew touching anything at the host layer was probably a non-starter but I wanted to check my assumptions. My goal had been to kickstart a conversation about sensitive transactions in general and I wanted to use this as a leverage point. We treated all updates equally, optimizing for speed over accuracy, but I’d wanted to build a special architecture and design standard for changes relating to regulatory processes. Prioritize accuracy over speed for lower volume but sensitive updates. I figured if it has to absolutely be correct, you can spend a few more milliseconds on it and just display some kind of animation to the customer while it confirms. No dice.
“So when the testers are looking at this in the test environment, why can’t they test to see the changes at the host at the same time? What could we do to make the testing more end to end?”
The test manager gave a long speech about how the testing tools didn’t have the appropriate integration. You could only test one application at a time. I wasn’t the first person to ask for this kind of comprehensive end to end testing. If the testers had to do the necessary login/logouts their capacity would greatly diminish. There were batch dependencies at play as well, because anything we wrote to host actually went to an intermediate file first that didn’t actually truly update to the system of record until 5pm EST. There’d always be some kind of blind spot in our testing. He had the highest level of hostility of anyone on the call. This was understandable as testing can sometimes bear the brunt of the other resource groups needing to move work around, given that testing tends to be the least expensive part of an organization to operate. He would never get funding, he said, to do any kind of system integration in their test environment.
Other than fixing the configuration file that caused this whole problem, this took away all of the tech-first fixes that I could easily imagine. That left me with… gulp… process updates. These are expensive as well but operate under a leaky faucet principle. The spend only happens a bit at a time, it gets hidden within the normal operations budget, it’s not that much, but over years and years it adds up. Especially if everyone fixes everything through process updates. Suddenly, you end up with something like thirty full time employees who are plugging holes in a dam with their fingers. That isn’t necessarily as bad as it sounds if the cost to plug those holes is higher than the cost of those employees, but it’s unstable ground.
“When we reload an account, even if it’s been loaded before, does the agent or customer have any sort of cache file to load the host information from the first session or will they be getting a fresh pull for the new session?”
I’m jumping over some high level questions, and jumping right to the hyper-specific one. They’d already demonstrated someone reloading an account with a failed transaction and it showing the true state on the reload. I asked the user interface engineering team to weigh in on this, and they confirmed each session was unique and the pull would always be fresh. I asked for a slightly different demonstration of the user experience and I suddenly knew the answer.
End to end testing enhancements had been requested, apparently, for years. And now some random guy was going to solve it for this group in about an hour. My first gut feeling was that this was going to be hard news to deliver. Sometimes you walk into an area with fresh eyes and perspective, you get all the right people involved to understand the problem fully, and you can see an answer right away. It happens. You may already see the answers as well.
“So when the testers are working on validation, and they perform updates, can we have them reload the account to make sure the update stuck? Not asking them to login and then load it in the host screens. I’m asking if they can hit that reload button in the user interface. It looks like that only takes a few seconds to execute.”
I am a very nice, professional person in real life and I know how easy it is to be the person who didn’t see an obvious solution. I have had some cringeworthy experiences. You need to take those losses on the chin and roll with them. Don’t let your ego get in the way of doing the smart thing. Don’t turn down the good idea because it came from someone you don’t know. Just keep doing that over time, keep admitting your weaknesses and shoring them up. Do that and you can become formidable rather quickly.
This particular test manager had yet to learn this, but I was also so transparently correct that by the end of the conversation he couldn’t tell me no. He had been looking, for years, to spend several million dollars to integrate his testing systems end to end. Maybe that had value for a larger variety of problems he faced that I was not aware of, but on this particular one his pain point could be solved with a simple account reload. A simple button click. That would get him all the end to end testing he would ever realistically need at least between the primary host system and the user interface. And after we talked more that was most of what he’d needed. He’d gotten so locked into the frame of needing system enhancements to do logins across multiple systems that he couldn’t see the good he could get done with the tools he already had. He hadn’t taken the step of thinking about how his piece of the puzzle fit into the whole.
It all worked out in the end, and I made sure to compliment and his team who had a pretty thankless position. This was primarily engineering’s fault for not having better maker/checker processes on their code deployments, but they were harried and it was understandable why they hadn’t done this. Developer time is extremely expensive. Testing was going to be responsible for the ongoing fix. It’s seriously hard to just “go figure out everything that might go wrong” with any given code change which is the position a lot of testing teams are left with. I also made sure the other teams doing post-implementation regression testing every month received the same information and made the same updates. The other ugly part about process changes is you can’t make them in one place and trust that they’ll keep getting done. You need to have a game of suspicion where if one group fails another group catches it. That’s another part of why it gets so expensive over time.
I remembered this experience when I later became a Product Owner myself. I remembered how easy it could be for all of these transactions to fail in the middle and for everyone to still get a success message. If you work at a big organization, people build over the top of you without ever noticing you’re there. Instead of saying something naive to myself like “I will build so well, this will just never happen” I planned for it to happen. I knew it would happen. I was right.
I made sure that all of my systems had next day recon processes so that we could see anything that fell into the category of “timing issue” or “bad field name” as soon as it happened and fix it right away. I had to fight developers on this, as their feeling was that we should find specific errors and fix them specifically. I told them, in my experience, you will always have errors you weren’t counting on and that it doesn’t even matter if you execute perfectly. Someone else, in a whole other area, might build a process that conflicts with yours and never tell you about it. It’s better to address the category broadly, and cheaply, so you can find those mistakes and then fix them right away. Plus, it makes you look like a genius when you fix your own breaks right away and no one notices.
My processes were among some of the most sensitive and highly monitored in the institution, but all the other surrounding monitoring support teams had a three month lag. I wanted to fix things before a customer even noticed. I wanted a feedback loop so tight that before anyone sent me an email three months in the future about this one whacky transaction, out of the millions that we processed, that I’d already have a long chain of evidence on how we found it remediated it, and permanently fixed it. As a strategy, this was incredibly successful.
I learned a lot in that role. I learned how to build reporting that gives you samples of passing volume as well as your failing volume, so you could tell right away if your report has broken. I knew to do that because we’d had a report that only showed errors, and when it stopped working altogether nobody noticed for something close to a year. The only error mechanism was a checksum and if all the records were blank, the checksum never caught it. I learned how to build exception handling loops as a matter of course. I learned how to build from “failure up” for anything customer facing so you never make a promise to a customer that you don’t actually have a way to keep. I learned to always give simplicity first to your customers and then only secondarily to yourself. And I fought all kinds of architects on that until I explained how easy it is for a customer to break something. Designs might be cheaper if they’re simple, but remediation is expensive and time-consuming. Plus, it looks awful to auditors. If a customer presses a button in one of my flows, about a dozen things happen behind the scenes to make sure what is promised to happen will actually happen. It’s one thing for one of your agents to notice something is wrong and work around it.3 Customers have no tolerance for this. Regulators have even less tolerance.
I’ve built things. I’ve learned a lot of lessons. I’m good at it. I have weird lucite trophies in my office.4
I attribute all of this to the Socratic method, or what I call “Finding the Next Right Question” so that it doesn’t sound pretentious. It’s how I train the people on my team to arrive at conclusions. You will very seldom know enough to immediately find a definitive answer. So if you don’t know the answer, do you know enough to formulate the correct question that will lead you to the answer?
The Next Right Question here is probably:
So why am I not doing any of that with Trust Assembly stuff?
A Box Full of Evil Plans
I like to plan all the time, with the understanding that my plans will never be perfect or implemented in full. I know that even while I’m planning. But my goal is always to build up a box of “things that we can do” that I can draw from in the event we get the opportunity. This has worked out for me multiple times, including the recent past. When you’re at a big org, nothing moves quickly. It’s like steering a giant battleship. You’re always politicking,5 putting together decks, and call-hopping. Plus doing anll the work that is approved to do right now.
Eventually, you become so good at bug-fixing, that they don’t let you do it anymore and you get to talk about budgeting instead. Eventually, you figure out budgeting and you evolve to the next stage, which is formatting Power Point decks to pitch people ideas.6 The Power Point evolution is the longest phase of the career. You need the giant box of plans so that when opportunity arises, and you basically have two days to come up with a hyper complex system diagram and pitch to secure funding, you can actually get it done in time. The only possible chance you have at succeeding in that is to have a giant box of plans.
To me, the failure mode of planning is, “I want to specifically do this and have no flexibility whatsoever. When this doesn’t work out exactly the way I want, then I will begin to weep hysterically and shut down.”
The success mode of planning is, “I can do one of these twelve things. I’m going to try to do this specific thing, and if I can’t I’m going to pivot to one of these variants. I won’t let the perfect be the enemy of the good.”
The benefit you get from this is an ideal state. You get something to aim toward. For instance, “I want a shared user interface across all channels that shares one common architecture that I can update at any time using a very small amount of money.”
What you’ll probably end up getting is “We got two of these four channels on the shared architecture. We ran out of money. It will be slightly less expensive to maintain cohesion going forward.”
Doing that however, makes the next request a little bit easier to approve. You show a strategy. Bit by bit, you wring order out of chaos. The better picture you can paint, the closer it comes to realization, the happier people are to fund your projects.
I’ve built new stuff before, but only in the context of an existing organization. Even there, to get it done right, it takes something like eighty hours a week7 for something like eight or nine months. If you’re the bottleneck on the vision, you need to be available at all hours to remove roadblocks and provide clarity. If everyone is waiting on you for “What next?” your ability to define that, quickly, is what allows that team to effectively execute. The whole time you’re doing that, you’re also politicking and making sure you keep the trust of the people you’re doing the work for.
I’m realizing that I’ve let this experience tarnish my thinking outside of big organizations. Also, that I am kind of a jackass. Which I already knew.
I’ve been approaching Trust Assembly stuff in the same manner. I show up in random comments on X, I post about it on Notes, write long vision documents, and I harass the entire substack team.8 I’ve been on a weird journey of consensus building via whisper campaign. And this is kind of stupid, I’ve realized.
Although, it’s kinda worked? The vibes are pointing the right way. For those who know who Shivon Zilis is, if anyone would know what’s on the roadmap, she would, right? I’ve tried to gently point at the concept to other people. Including some of the fine folks here at substack.
You have to be more bold if you want to build without an organization and, this is dumbest of all, I don’t know that anyone has ever successfully just kind of... convinced an organization to build something outside of actually working there. So what am I doing?
My gut feeling is that you’d need at least one really, really good back-end engineer, one decent designer, and one decent front-end engineer, to deliver something like a Trust Assembly for just one desktop browser, maybe two. Mobile is a whole other, much harder thing. If all of your workers were part-time and semi-volunteers, you focused on just doing the headline replacement feature, and threw all of the other pieces onto a roadmap, you could probably launch that for a reasonable sum of money. I’m thinking the low tens of thousands of dollars. Maybe even less than that, but to get your workers motivated, you’d really have to do something like a full company with shares and the prospect of future profits. Phase one would be to get a customer base interested, get them bought in on the vision, and then keep iterating.
You’d have to be super careful on the order you executed the roadmap to push out he complexity as long as possible. If you’re going to start moving money around, building a decision engine for your payouts, and start doing reputation scoring with another decision engine, you’re going to need more engineers and legal counsel. One because payments get finicky, and two because some of the news sites you do this to will probably be unhappy and file spurious lawsuits. So you really only want to do that when you have enough revenue to cover those costs. That’s where you bump up the price of playing the game and I see a burn rate of several million dollars being more realistic.
Let’s say you do some subscription service and you’re bringing in $3 per customer every month. That’s $36 annually. Let’s say this is $6million dollars a year to run and do a lot of hand-waving over how much of that is initial development cost versus salaries, etc. Round up, you need about 170,000 customers to be breakeven. And while you’re doing this, if it’s successful, other people with deeper pockets and established networks are going to copy you. Facebook and X will be on your heels. You’ll be more nimble and agile, you’ll execute quicker, but you’ll also have less money to spend. They’re the warship slowly changing course. You have to be a bigger warship by the time they hit you.
I tell myself that I can’t do this because I have young children, very few chunks of time longer than five minutes, my wife won’t let me risk steady employment to do a startup, and general gutlessness. Being honest, people have risked more on less with less experience.
The other part is that I’m pretty sure I’m right that this would work9 and that someone for sure needs to this as quickly as possible. I want this because I have kids. AI Slop is going to take over the internet even more than culture war spam has. The least objectionable thing you could do is make a Trust Filter for just that. Separate the sheep from the goats. This is going to be needed in the near future even if people will argue over whether it’s needed right now.
Demand Signal
I’m trying to stay focused on a mission here, even as I divert to tell funny and crazy stories about my life. The best way I can serve that mission is by creating a demand signal. The best way to create a demand signal is to become popular. The thing I know I can do that is popular with people is to write stories about my life. I’ve done it before, successfully. This also does other things, like give people a sense of my motives, a sense of who I am so they can feel better about buying in to the idea, and well… it’s just going to get me more eyeballs. I just know that stuff is more popular than me talking about theoretical sense-making systems.
I do also find myself wanting to spout off about stuff in general, like building an Olympic Stadium on the moon, so it might as well be to a purpose. Plus, I think one of the most ugly sterile things about our present culture is that nobody is honest as a matter of course about being “Some Guy.”10
We live in a world of peacocks spreading their plumes. Which, to be clear, I am also doing. But my plume is “look how normal my plume is!” The folksy meta plume! Look at how folksy that regular guy knows that he regularly is!
I don’t think people realize this, but you can be in an organization, everyone can want to do something and think it’s a good idea, but if there’s no demand signal you just can’t do it. Like, if your customer base of subscribers, or your customer base of creators, balks at the idea… you have to pass on it.
You need to spend money to make money. Your goal is to spend the least money to make the most money. You can’t spend money to lose money. It destroys you.
Parallel to the demand signal is an “I’m serious about this” signal.
So here’s what I want to do. Enable a paid option for my substack. Then I’ll see how much I can grow that.
I can’t make any promises. I want to be very clear on that. It may end up being the case that it’s only enough to subscribe to a couple more substacks and do some glad handing. If it’s more than that, I’ll see what I can do. I also feel like I should do this just because I feel kinda bad about how supportive all the people at substack are and I’m not vibing along with their vision.
I am a crazy goofball on here most days because I burn up my “I’m a serious adult” neurons during the day. My tired, low-energy state, is goofball. This is also an accountability mechanism for me.
I won’t paywall anything relating to the mission, but might do it for some of the more sensitive personal stuff.
Anyhow, feel free to call me a sellout in the comments and/or express desires and concerns?
I may have decided the MacGyver part on my own
Most problems are like this
This isn’t true if it’s something you have to change all the time. If it’s a build once kind of thing, yes, redundancy. Otherwise your maintenance costs blow up.
There’s a less prestigious internal award where you get a bobble head doll that looks like you. I’ve never won this and it bothers me sometimes.
I mean “making friends” by this. People get way too into win/lose situations and win/win situations are usually very abundant. You make friends by having conversations like “If we did this thing, wouldn’t both of our lives be easier?” And then wait a few years, keep saying that, and when the internal political environment is right you run like hell to say “we just thought of this. What if we did xyz?”
The last stage of this is “Look at Power Point decks formatted by other people.”
With spikes going up to 100 hours per week. There have been times where all I did was work and sleep.
Substack has all the right pieces in place, in my opinion, to make something like this work perfectly. It has independent journalists who would be most incentivized to use such a system as it equalizes their power against larger institutions. It has normal people, under those journalists, grouped into different trust blocks already. And it has paying customers, who are breathtakingly normal in my experience, who could provide a good jury pool for the review system. I also think this could be good for them long-term. If Bari Weiss starts to making even more money, what is the technology piece she’s not going to be able to replicate on her own? What’s the network flywheel she can’t take away and decide “having this much money is enough?”
This is essentially the same mechanism as prediction markets, but sidesteps all the legal problems and allows people to produce work rather than just place bets. I think prediction markets have their place, but a Trust Assembly gets you most of what you want, quicker, and better. You need a people coordinator. A Trust Assembly would do that.
This has definite downsides and is why I stopped writing these kinds of stories ten or so years ago. You end up having these very one-sided relationships with people over time where they know all your most formative life-experiences and they’re total strangers to you. Also, some people start to think of you as a literary character instead of a real person who happens to be writing stories. This was the weirdest thing about it, by far.
Did I get this right - you want to finance your "mission " project through the income from substack subs? Isnt it a bit sorta roundabout way ?
I am kinda intrigued by how different this post is from your life stories. Your idea about having truthful headlines and news really belongs in web 3.0 space ( I am sure you know this already). In something like farcaster ecosystem ( or in that direction).
Strangely I am in kinda similar position. i do have an idea I really want to build ( ideally as web 3.0 dapp). But I can't do it alone, nor have funds to get a team. Seems a common problem in start up space: )
P.s. do you know about Network State community/project?
Really enjoyed this. Loads of useful business wisdom in here, some of which I've discovered for myself as a business manager.
Especially enjoyed this bit for instance:
'I learned how to build from “failure up” for anything customer facing so you never make a promise to a customer that you don’t actually have a way to keep. I learned to always give simplicity first to your customers and then only secondarily to yourself.'
Cheers man