[{"content":"Platform and infrastructure engineer with 12+ years building the systems that ship software at scale. Now leading DevOps engineering at Palo Alto Networks, after architecting an enterprise DevSecOps platform supporting 2,000+ developers.\nI write about DevOps, platform engineering, and running AI in production.\n","date":"September 21, 2026","externalUrl":null,"permalink":"/","section":"Cliff Hults","summary":"Platform and infrastructure engineer with 12+ years building the systems that ship software at scale. Now leading DevOps engineering at Palo Alto Networks, after architecting an enterprise DevSecOps platform supporting 2,000+ developers.\nI write about DevOps, platform engineering, and running AI in production.\n","title":"Cliff Hults","type":"page"},{"content":"Four projects, picked for outcomes rather than technology. Each links to the full write-up; this page is the five-second version.\nTrack record: ~20x artifact-processing throughput (25 → ~500/min) · SaaS deploy time cut from ~90 min to 25 min, success rate ~50% → ~80% · teams of 5–9 engineers led · FedRAMP High–compliant pipelines built and run in AWS GovCloud.\nPrompt Engineering in Production — Hours-to-Days Down to Minutes # Built an AWS Bedrock pipeline giving developers in a FedRAMP High environment autonomous access to their own build-failure analysis, without a human in the loop or direct production access. The old process — a ticket to a US-based team — took hours to days per failed build; the new one surfaces sanitized analysis in an internal developer portal within minutes. A Bedrock Knowledge Base of known failure patterns cuts hallucinations and token burn.\nStack: AWS Bedrock, AWS GovCloud, SQS, Lambda, CI/CD platform, internal developer portal\nRead the full case study →\nAutomated Artifact Factory — Turning a Recurring Support Pattern Into a Platform Boundary # Traced a recurring cluster of pipeline support tickets back to a structural problem: every team was independently wiring signing, vulnerability scanning, and antivirus checks into its own CI pipeline. Designed (not yet built) a centrally-owned, asynchronous artifact factory that runs these checks in parallel and gates promotion through a single status entity — so the next tool swap is one change behind a boundary, not a migration across every team.\nStack: AWS, Kubernetes/EKS, artifact repository/registry, CI/CD platform, internal developer portal\nRead the full case study →\nAxios Supply Chain Incident — Finding the Blind Spot in a Trusted Cache # When a malicious axios release stayed live on npm for roughly three hours, our artifact repository cache kept serving it long after npm pulled the package. Traced exposure through a shared CI service account — which erases per-team attribution in access logs — by manually correlating build logs, then contained affected build nodes on the container orchestration platform and purged the cache. Fixes now in flight: dependency-pinning enforcement, a publication delay window, and automated malicious-package scanning.\nStack: Artifact repository/proxy, CI/CD platform, container orchestration platform (Kubernetes), npm\nRead the full case study →\nSCA Tool Outage — Making the Freeze-vs-Bypass Call # An SCA tool went down for about five hours, taking every required security gate with it and freezing all pull request merges org-wide. Within forty minutes, made the call — jointly with business leadership and security — to lift the gate and keep teams shipping, then closed the gap with a retroactive scan of every PR merged during the ~4.5-hour bypass window. Nothing was found, but the retrospective is now driving two efforts still in progress: more granular per-check bypass controls, and an evaluation of tooling redundancy so one vendor outage can\u0026rsquo;t freeze the org again.\nStack: SCA tool, Git platform (required status checks), CI/CD platform, pipeline metrics/reporting\nRead the full case study →\n","date":"September 21, 2026","externalUrl":null,"permalink":"/projects/","section":"Cliff Hults","summary":"Four projects, picked for outcomes rather than technology. Each links to the full write-up; this page is the five-second version.\nTrack record: ~20x artifact-processing throughput (25 → ~500/min) · SaaS deploy time cut from ~90 min to 25 min, success rate ~50% → ~80% · teams of 5–9 engineers led · FedRAMP High–compliant pipelines built and run in AWS GovCloud.\nPrompt Engineering in Production — Hours-to-Days Down to Minutes # Built an AWS Bedrock pipeline giving developers in a FedRAMP High environment autonomous access to their own build-failure analysis, without a human in the loop or direct production access. The old process — a ticket to a US-based team — took hours to days per failed build; the new one surfaces sanitized analysis in an internal developer portal within minutes. A Bedrock Knowledge Base of known failure patterns cuts hallucinations and token burn.\n","title":"Projects","type":"page"},{"content":"","date":"September 20, 2026","externalUrl":null,"permalink":"/tags/artifact-management/","section":"Tags","summary":"","title":"Artifact Management","type":"tags"},{"content":"","date":"September 20, 2026","externalUrl":null,"permalink":"/tags/automation/","section":"Tags","summary":"","title":"Automation","type":"tags"},{"content":"","date":"September 20, 2026","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"September 20, 2026","externalUrl":null,"permalink":"/tags/ci/cd/","section":"Tags","summary":"","title":"CI/CD","type":"tags"},{"content":"Every team wiring the same three security checks into its own pipeline isn\u0026rsquo;t really a security problem. It\u0026rsquo;s a platform problem wearing a security costume.\nThis is the story of how we traced that pattern back to a structural issue, and the design we landed on to fix it: an automated artifact factory that turns signing, vulnerability scanning, and antivirus scanning from something every team bolts onto their own pipeline into something the platform just does for them.\nWhere This Started # The signal came from support data, not a single dramatic incident. Code signing had quietly become a disproportionate share of our support ticket volume, with a recurring cluster of \u0026ldquo;signing job failures\u0026rdquo; that ate up a lot of engineering time across many tickets. Separately, a single offline signing node had silently queued far more jobs than anyone expected before anyone noticed and filed a ticket; that\u0026rsquo;s the kind of blast-radius pattern that\u0026rsquo;s easy to miss until you actually look at the aggregate data.\nNeither of those, on its own, was an emergency. Together, they pointed at something structural: every team was independently gluing signing, vulnerability scanning, and antivirus checks into their own CI pipeline, and every one of those integrations was a slightly different, slightly fragile thing.\nWhy We Looked at It Now # There was also a forcing function. We have a code-signing tool migration in discovery right now, and that\u0026rsquo;s where the support-data pattern turned into an actual design question. Swapping a signing tool without a platform boundary means every team touches their own pipeline again. Swapping it with one means the change happens behind the scenes.\nThat reframed the whole problem. This wasn\u0026rsquo;t really about signing specifically; it was about whether we had a boundary at all between \u0026ldquo;the checks that gate an artifact\u0026rdquo; and \u0026ldquo;the pipelines that produce artifacts.\u0026rdquo; We didn\u0026rsquo;t. So every future tool swap, in signing or anywhere else, was going to cost us a full migration across every team, forever.\nWhy the Current Approach Doesn\u0026rsquo;t Scale # Today, artifact signing, vulnerability scanning, and antivirus scanning are steps individual teams inject into their own CI pipelines using shared-library snippets. That gets you repetitive per-team setup, inconsistent coverage depending on who copied which version of the snippet, and security gating that\u0026rsquo;s coupled to whatever each pipeline happens to implement (or forgets to).\nThe failure mode compounds it. These steps are error-prone, and when they\u0026rsquo;re not built to fail gracefully, one failure forces the whole pipeline to restart from the beginning instead of resuming from where it left off. That\u0026rsquo;s expensive in engineering time, repeatedly, across every team that hits it.\nThe Design We Landed On # We designed a centrally-owned, asynchronous artifact factory: uploading an artifact to the staging tier of our artifact repository triggers signing, vulnerability scanning, and antivirus scanning automatically, in parallel, without any team wiring anything into their own pipeline. Results land on a single status entity per artifact version in our internal developer portal, and that status gates promotion through the pipeline.\nflowchart LR A[\"Volatile tiers\\n(dev / latest)\\nout of scope\"] --\u003e B[\"Staging tier\\nupload triggers event\"] B --\u003e C[\"Orchestrator + developer portal\\ncreates pending status entity\"] C --\u003e D[\"Code signing\"] C --\u003e E[\"Vulnerability scan\"] C --\u003e F[\"Antivirus scan\"] D --\u003e G{\"All required\\nchecks pass?\"} E --\u003e G F --\u003e G G -- yes --\u003e H[\"RC tier\\nauto-promoted\"] G -- no --\u003e I[\"Quarantine + alert\\naccess revoked, team notified\"] H --\u003e J[\"Release\\nhuman-gated promotion\"] A few decisions shaped that design more than anything else:\nThe factory only engages once an artifact stabilizes. Artifacts are still volatile in the earliest tiers, actively changing and not yet meant to be checked. We deliberately kept the factory out of that stage entirely: no event, no status entity, no checks. It only begins once an artifact reaches staging, where it\u0026rsquo;s meant to hold still long enough to be evaluated.\nPromotion out of staging is automatic, not another approval queue. Once every required check passes, the artifact moves itself from staging to the release-candidate tier. No human clicks anything. If a required check fails, the artifact is quarantined (held back, download access revoked, the responsible team notified immediately). A failed optional check doesn\u0026rsquo;t block anything; it\u0026rsquo;s recorded on the status entity but doesn\u0026rsquo;t stop the artifact from moving forward. We haven\u0026rsquo;t settled which checks are required versus optional yet, or whether we need a waiver path for someone to override a required-check failure deliberately. That\u0026rsquo;s an open question, not a design gap we glossed over.\nWe drew the boundary at \u0026ldquo;checks,\u0026rdquo; not at \u0026ldquo;everything downstream.\u0026rdquo; What a team does with an artifact once it reaches the release-candidate tier (further validation, integration testing, manual QA) stays entirely theirs. We considered pulling more of that into the platform and deliberately didn\u0026rsquo;t: the value of this project is in owning the checks, not in owning every team\u0026rsquo;s release process.\nPromotion to release stays human-gated. We\u0026rsquo;re not automating the last step. That promotion uses infrastructure we already have; the factory\u0026rsquo;s job is only to make sure the status entity has good enough data that a human making that call actually has what they need.\nThe Parts We Deliberately Didn\u0026rsquo;t Solve Yet # A design case study is more honest if it says what\u0026rsquo;s still unresolved, so:\nWe haven\u0026rsquo;t yet confirmed that the event mechanism we\u0026rsquo;re relying on is actually reachable from our own infrastructure against our specific systems, or what its real delivery and retry guarantees look like in practice. That needs verification before this is buildable, not just designed. The download-revocation mechanism we picked for quarantine needs to operate per artifact version, not per repository; we ruled out an alternative that only worked at the repository level. Whether the per-version mechanism is fully licensed and enabled on our own systems is still unconfirmed. Antivirus tooling for non-Windows artifact kinds doesn\u0026rsquo;t have an answer yet. The exact schema for the status entity (what it tracks, how it represents which tier a version currently sits in) is still a discussion, not a spec. We haven\u0026rsquo;t decided how existing per-team pipeline steps get migrated onto the factory and then retired: all at once, gradually, team by team. That sequencing question is intentionally deferred until the core of the factory is built and proven. None of these are fatal to the design. All of them need answers before it\u0026rsquo;s a spec someone can build against.\nWhat This Taught Us # The obvious payoff here is developer experience: upload an artifact, and the checks just happen, consistently, without every team maintaining their own copy of the logic.\nThe less obvious payoff is architectural, and it\u0026rsquo;s the one that actually justified the effort. By centralizing these checks behind one platform boundary, swapping any underlying tool (the signing tool, the vulnerability scanner, the antivirus engine) becomes a change made once, behind that boundary, instead of a migration that touches every team\u0026rsquo;s pipeline. That\u0026rsquo;s the property we were actually missing, and it\u0026rsquo;s the reason a signing-tool migration already in discovery is what turned this from \u0026ldquo;would be nice\u0026rdquo; into \u0026ldquo;worth designing now.\u0026rdquo;\nThere\u0026rsquo;s a quieter efficiency argument too: today, the same artifact often gets pulled and re-checked multiple times across multiple places. A factory that pulls an artifact once and runs every check against that single pull is a smaller cost on both infrastructure load and cloud spend, not the headline reason to do this but a real one.\nWhere It Stands # The open questions above are the actual next step: resolve them, verify the two that need hands-on confirmation against our own infrastructure, and turn this from a design into something implementable. The interesting part of this project was never the individual checks; it\u0026rsquo;s whether the boundary underneath them holds up the next time we need to swap a tool without anyone else having to notice.\n","date":"September 20, 2026","externalUrl":null,"permalink":"/posts/2026-09-20-automated-artifact-factory/","section":"Posts","summary":"Every team wiring the same three security checks into its own pipeline isn’t really a security problem. It’s a platform problem wearing a security costume.\nThis is the story of how we traced that pattern back to a structural issue, and the design we landed on to fix it: an automated artifact factory that turns signing, vulnerability scanning, and antivirus scanning from something every team bolts onto their own pipeline into something the platform just does for them.\n","title":"Designing an Automated Artifact Factory: A Case Study in Turning Pipeline Pain Into a Platform Boundary","type":"posts"},{"content":"","date":"September 20, 2026","externalUrl":null,"permalink":"/tags/devops/","section":"Tags","summary":"","title":"DevOps","type":"tags"},{"content":"","date":"September 20, 2026","externalUrl":null,"permalink":"/categories/engineering/","section":"Categories","summary":"","title":"Engineering","type":"categories"},{"content":"","date":"September 20, 2026","externalUrl":null,"permalink":"/tags/platform-engineering/","section":"Tags","summary":"","title":"Platform Engineering","type":"tags"},{"content":"","date":"September 20, 2026","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"September 20, 2026","externalUrl":null,"permalink":"/tags/security/","section":"Tags","summary":"","title":"Security","type":"tags"},{"content":"","date":"September 20, 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"May 1, 2026","externalUrl":null,"permalink":"/tags/change-management/","section":"Tags","summary":"","title":"Change Management","type":"tags"},{"content":"","date":"May 1, 2026","externalUrl":null,"permalink":"/tags/devsecops/","section":"Tags","summary":"","title":"DevSecOps","type":"tags"},{"content":"","date":"May 1, 2026","externalUrl":null,"permalink":"/categories/security/","section":"Categories","summary":"","title":"Security","type":"categories"},{"content":"","date":"May 1, 2026","externalUrl":null,"permalink":"/tags/security-program-design/","section":"Tags","summary":"","title":"Security Program Design","type":"tags"},{"content":"This is the final part of a series. Start with Part 1 — giving the backlog teeth, then Part 2 — the waiver trap.\nThe waiver experiment was a failure in the best possible way; it was diagnostic. It told me exactly where the real bottleneck was: fixing a vulnerability was hard to do, not hard to decide. Teams weren\u0026rsquo;t ignoring the work because they were lazy or uncaring. The work to fix a dependency vulnerability was slow, manual, and painful. Every fix was a research project.\nSo I stopped trying to apply more pressure and started asking the question that changed everything:\nWhat if fixing a vulnerability was the easiest possible thing to do?\nThe insight: friction is a security decision # I\u0026rsquo;ve come to believe this is one of the most important ideas in security program design: people choose the most convenient safe option available to them. If doing the secure thing is harder than doing the insecure thing, you have not built a security problem. You have built a UX problem. Nobody deliberately does the wrong thing; they do the thing that\u0026rsquo;s closest to hand.\nThis reframes the entire job of a security leader. You\u0026rsquo;re not just setting policy and measuring compliance. You\u0026rsquo;re designing the paths people will actually walk.\nMaking the secure choice the easy choice # Instead of tightening the enforcement further, which would have just created more waiver demand, I invested in making the remediation itself nearly effortless. The core idea was automation that hands teams almost-complete work rather than more work:\nContinuous, automated discovery of new dependency versions, running across the fleet on a cadence. Ready-to-review changes — the automation opens a change with the fix already applied, so the team\u0026rsquo;s job collapses from \u0026ldquo;research and implement\u0026rdquo; to \u0026ldquo;review and approve.\u0026rdquo; Assistance on the long tail — teams with custom needs got guided support to configure the automation for their specific setup, rather than being told to handle it manually. The result was that the secure action became the default action. Teams adopted it not because they were forced, but because it was the path of least resistance. And critically: they kept it. It became part of their own planning, something they carried themselves rather than a mandate imposed on them.\nWhat changed when friction was removed # The numbers spoke for themselves:\nThe remediation feedback loop went from a quarterly cycle to effectively biweekly; a step-change in responsiveness, not a marginal gain. Adoption of the automated path climbed steadily, and it spread on its own merit because it was easier. Teams moved vulnerability management onto their own roadmaps, which is the signal I value most. When a team chooses to keep doing the secure thing unprompted, you\u0026rsquo;ve changed their culture, not just their compliance. The lessons for security leaders # If adoption is a fight, look at your friction, not your people. Resistance to security tooling is almost always a signal about ease-of-use, not about intent. Automation that hands people almost-finished work beats automation that adds more work. The goal is to shrink the effort to comply, not pile on. Sustained adoption is the only metric that matters. A rollout that reverts the moment you stop pushing is coercion, not adoption. Sequence matters: make it easy before you make it mandatory. Enforcement creates urgency, but it only works sustainably when the easy path exists first. Don\u0026rsquo;t fall in love with your first design. This whole program is the story of three iterations — enforce, waive, pave. The last one worked because the middle one taught me what to build. The takeaway # The most effective security programs aren\u0026rsquo;t the ones with the strictest enforcement; they\u0026rsquo;re the ones where doing the right thing is the easiest thing. When you remove the friction, secure behavior stops being a negotiation and becomes the obvious choice that engineers make on their own.\nThat\u0026rsquo;s the real job of a security leader. Not to force compliance, but to design the conditions where compliance is the default.\nNext steps: the next-generation layer # The paved path got us to fast, sustainable remediation, but it was deliberately built with the ceiling in view. When I designed the system, I chose not to automate every part of it at once, specifically so I could watch how teams actually used the base loop before layering intelligence on top. That was the point of saving the major-dependency-updates automation for later.\nThat\u0026rsquo;s the thread Part 4 picks up. The next-generation opportunity is AI-driven remediation that complements the current-gen automation rather than replacing it, at a cost that actually scales.\nThe shape of the opportunity:\nThe hardest remaining work isn\u0026rsquo;t the routine bump; it\u0026rsquo;s major dependency upgrades, where a version change can carry breaking changes, migration effort, and real risk. That\u0026rsquo;s where teams still stall. The current generation of known, proven tooling already handles the bulk cheaply: token-efficient, low-cost workflows that cover the common cases reliably. The next-generation layer, AI, sits on top of that known-functional baseline. It\u0026rsquo;s not about swapping the proven solution out; it\u0026rsquo;s about using AI to handle the judgment-heavy tail that a fixed rule base can\u0026rsquo;t, on detection of a vulnerable dependency. The design principle that matters: use the low-cost, known-good solution for everything it can do, and let AI extend it only where it adds value. Done that way, adoption of AI across an enterprise isn\u0026rsquo;t a big-bang project; it\u0026rsquo;s a complement. You get new functionality (automating major upgrades at the moment they\u0026rsquo;re detected) while reducing the total cost of the program, because the expensive AI token spend is only spent where the cheaper solution can\u0026rsquo;t already do the job.\nThe trap: \u0026ldquo;AI all the time\u0026rdquo; # There\u0026rsquo;s an opposite risk to this, and it\u0026rsquo;s the one I see most enterprises running toward: \u0026ldquo;AI all the time.\u0026rdquo; The default posture in a lot of organizations right now is to assume every problem is an AI problem, and to throw a model at anything that moves, on every interaction, everywhere.\nThat\u0026rsquo;s a recipe for blowing your budget. It\u0026rsquo;s also usually worse at the task.\nHere\u0026rsquo;s the thinking that\u0026rsquo;s missing from most of it: the majority of the tooling we already run is still genuinely good at what it does. It has been for a decade. These are known-functional, battle-tested solutions that solve real problems reliably and cheaply. The instinct to rip them out and replace them with an AI-driven version for the sake of it being AI isn\u0026rsquo;t modernization; it\u0026rsquo;s paying a premium to reintroduce fragility into something that already worked.\nSo the optimization that actually matters isn\u0026rsquo;t \u0026ldquo;how much AI can we use.\u0026rdquo; It\u0026rsquo;s optimizing for cost versus the specific deficiency you\u0026rsquo;re actually trying to fix. The right question isn\u0026rsquo;t can AI do this?; it\u0026rsquo;s does AI do this better than the tool we already have, given that it costs more?\nThat discipline is what separates a program that uses AI wisely from one that burns money to look current:\nIf the proven solution already solves it well — keep it. Don\u0026rsquo;t pay for AI to do what the tool already does at near-zero marginal cost. If there\u0026rsquo;s a real gap — a judgment-heavy case, a breaking change, a decision the rule base can\u0026rsquo;t make reliably — that\u0026rsquo;s where the AI token spend earns its keep. Replace the proven solution only when the AI version is actually better, not when it\u0026rsquo;s merely different or newer. Most enterprises are going to learn this lesson the expensive way. The ones that get it right are the ones that ask the hard question up front: where does AI genuinely extend what we have, versus where would it just be an expensive substitute for something that\u0026rsquo;s already working? The answer to that is where you deploy next-gen capability — and where you keep your budget intact.\nIn other words: the future isn\u0026rsquo;t AI instead of the paved path; it\u0026rsquo;s AI on top of it, deployed surgically, so the marginal cost of each new capability stays low while the functionality keeps expanding.\nThat\u0026rsquo;s the story of Part 4: how we started layering next-generation, AI-driven automation onto a current-gen foundation, and what it took to do it without blowing up the economics.\nRead Part 4: where next-generation, AI-driven remediation meets the paved path, and how to make the upgrade-on-detection opportunity scale.\nI\u0026rsquo;d value your take on enforcement vs. friction — and on where AI belongs (and doesn\u0026rsquo;t) in a security program. What\u0026rsquo;s worked in your organization, and what taught you the most when it didn\u0026rsquo;t?\n","date":"May 1, 2026","externalUrl":null,"permalink":"/posts/2026-05-01-the-paved-path/","section":"Posts","summary":"This is the final part of a series. Start with Part 1 — giving the backlog teeth, then Part 2 — the waiver trap.\nThe waiver experiment was a failure in the best possible way; it was diagnostic. It told me exactly where the real bottleneck was: fixing a vulnerability was hard to do, not hard to decide. Teams weren’t ignoring the work because they were lazy or uncaring. The work to fix a dependency vulnerability was slow, manual, and painful. Every fix was a research project.\nSo I stopped trying to apply more pressure and started asking the question that changed everything:\nWhat if fixing a vulnerability was the easiest possible thing to do?\n","title":"The Paved Path: Making the Secure Choice the Easy Choice at Scale","type":"posts"},{"content":"","date":"April 12, 2026","externalUrl":null,"permalink":"/tags/incident-response/","section":"Tags","summary":"","title":"Incident Response","type":"tags"},{"content":"","date":"April 12, 2026","externalUrl":null,"permalink":"/tags/sca-tool/","section":"Tags","summary":"","title":"SCA Tool","type":"tags"},{"content":"","date":"April 12, 2026","externalUrl":null,"permalink":"/tags/supply-chain/","section":"Tags","summary":"","title":"Supply Chain","type":"tags"},{"content":"Security tooling exists to protect your organization. But what happens when the security tool itself becomes the outage?\nThat\u0026rsquo;s the situation we found ourselves in when an SCA tool experienced a service disruption that lasted approximately five hours. For us, it wasn\u0026rsquo;t a degraded experience; it was a complete development freeze.\nHow We Use the SCA Tool # The SCA tool is integrated into our standard CI/CD pipeline as a mandatory gate on every pull request. It handles three things: static code analysis for insecure patterns, dependency scanning for known vulnerabilities and licensing issues, and container image scanning. These checks are enforced as required status checks in our Git platform; a PR cannot merge until they pass.\nThis isn\u0026rsquo;t optional tooling. It\u0026rsquo;s business compliance and security enforcement baked into the development workflow. Every repository is enrolled, and historical analysis is maintained for audit purposes.\nHow We Found Out # We didn\u0026rsquo;t find out because a developer filed a ticket. We found out because our backend monitoring started firing: communication to the SCA tool\u0026rsquo;s API had failed, and failed status checks were spiking across the board. Developers hit it almost simultaneously as they tried to merge work during normal business hours.\nWithin minutes, it was clear this wasn\u0026rsquo;t a flaky test or a misconfigured repo. The SCA tool was down.\nThe Catch-22 # Forty minutes into the outage, we had a decision to make.\nThe technical situation was straightforward: the SCA tool\u0026rsquo;s service was degraded, our required status checks couldn\u0026rsquo;t complete, and no pull requests could merge. The business situation was also straightforward: development had stopped entirely. Not slowed; stopped.\nThe harder question was what to do about it.\nKeeping the gates up was the safer security posture, but it meant accepting a full development freeze of unknown duration. Bypassing them meant restoring velocity, but it also meant any code merged during that window had skipped mandatory security scanning. In a regulated environment, that\u0026rsquo;s not a decision one person makes unilaterally.\nWe looped in business leadership and the security team. The joint decision, made with full awareness of the tradeoff, was to temporarily remove the SCA tool as a required status check and allow teams to continue operating. The outage resolved approximately five hours after it started. We had introduced the bypass about forty minutes in, so roughly four and a half hours of merges occurred without the SCA tool gating.\nClosing the Gap # When the SCA tool came back online, we didn\u0026rsquo;t just re-enable the gate and move on. We pulled the full list of PRs merged during the bypass window (something we were able to do quickly because we maintain pipeline metrics and can generate reports against them) and enforced a retroactive scan of every affected repository.\nNothing was found. The window was short enough that the exposure was limited. But \u0026ldquo;nothing was found this time\u0026rdquo; is not a risk management strategy.\nWhat the Retrospective Surfaced # During our retrospective, the conversation shifted from \u0026ldquo;what do we do next time this specific thing happens\u0026rdquo; to a broader question: how do we design our security posture to be more resilient to SaaS tool availability?\nTwo themes emerged:\nGranular bypass controls — When the SCA tool went down, our only option was a binary one: all checks required, or none. We\u0026rsquo;re investigating whether we can build more surgical controls (for example, temporarily disabling only the dependency scanning component while keeping code analysis active, or allowing bypasses scoped to specific repo classifications). The goal is to have options that don\u0026rsquo;t require choosing between full security and full productivity.\nTooling redundancy — Most organizations don\u0026rsquo;t self-host their security tooling, and we\u0026rsquo;re no exception. That means our security posture has an availability dependency on third-party SaaS providers. One tool going down shouldn\u0026rsquo;t mean zero coverage. We\u0026rsquo;re evaluating whether distributing our scanning responsibilities across multiple tools provides meaningful resilience, not just redundancy for its own sake but genuine defense-in-depth that can survive a single vendor outage.\nThe Uncomfortable Reality # The axios supply chain incident and this SCA tool outage happened within the same operational period. Back to back, they illustrate the same underlying tension: we build security controls into our pipelines, and then we discover that those controls can themselves become the risk.\nA cached malicious package is a supply chain problem. A security tool outage blocking all development is a single point of failure problem. Both are worth taking seriously, and both require thinking beyond \u0026ldquo;add more tools\u0026rdquo; toward how those tools behave when something goes wrong.\nThe goal isn\u0026rsquo;t to have more security gates. It\u0026rsquo;s to have security controls that are resilient, observable, and designed with failure modes in mind.\nThis post is part of an ongoing series on real-world engineering decisions — not the clean version, but how things actually go.\n","date":"April 12, 2026","externalUrl":null,"permalink":"/posts/2026-04-12-sca-tool-outage/","section":"Posts","summary":"Security tooling exists to protect your organization. But what happens when the security tool itself becomes the outage?\nThat’s the situation we found ourselves in when an SCA tool experienced a service disruption that lasted approximately five hours. For us, it wasn’t a degraded experience; it was a complete development freeze.\n","title":"When Your Security Tool Goes Down: Surviving an SCA Tool Outage","type":"posts"},{"content":"","date":"April 11, 2026","externalUrl":null,"permalink":"/tags/ai/","section":"Tags","summary":"","title":"AI","type":"tags"},{"content":"","date":"April 11, 2026","externalUrl":null,"permalink":"/categories/ai-in-the-trenches/","section":"Categories","summary":"","title":"AI in the Trenches","type":"categories"},{"content":"","date":"April 11, 2026","externalUrl":null,"permalink":"/tags/aws/","section":"Tags","summary":"","title":"AWS","type":"tags"},{"content":"","date":"April 11, 2026","externalUrl":null,"permalink":"/tags/bedrock/","section":"Tags","summary":"","title":"Bedrock","type":"tags"},{"content":"","date":"April 11, 2026","externalUrl":null,"permalink":"/tags/container-orchestration/","section":"Tags","summary":"","title":"Container Orchestration","type":"tags"},{"content":"","date":"April 11, 2026","externalUrl":null,"permalink":"/tags/fedramp/","section":"Tags","summary":"","title":"FedRAMP","type":"tags"},{"content":"","date":"April 11, 2026","externalUrl":null,"permalink":"/tags/npm/","section":"Tags","summary":"","title":"Npm","type":"tags"},{"content":"","date":"April 11, 2026","externalUrl":null,"permalink":"/tags/prompt-engineering/","section":"Tags","summary":"","title":"Prompt Engineering","type":"tags"},{"content":"Most prompt engineering content is written by people who have never shipped AI into a real system. This post is different.\nI\u0026rsquo;m a Staff DevOps Engineer. I\u0026rsquo;ve spent time integrating AWS Bedrock into actual production workflows inside a FedRAMP High environment, not demos, not notebooks, not prototypes. Real systems with real constraints and real failure modes.\nHere\u0026rsquo;s what I learned.\nThe Problem Nobody Talks About # We had a compliance wall. Our developers live and work outside the United States. In a federal environment, that means no direct access to production systems, period. When a deployment pipeline failed, a developer couldn\u0026rsquo;t just click into the CI/CD platform and read the logs.\nThe old process: open a ticket with the Federal Production Engineering team. An engineer would log in, read the output, write up troubleshooting steps, and either hand them over asynchronously or schedule a screen share to walk the developer through it hands-off. Given timezone differences, scheduling conflicts, and team capacity, the feedback loop was hours to days. For every failed build.\nThis wasn\u0026rsquo;t a people problem. The engineers were doing everything right. It was a structural bottleneck: a compliance requirement created a hard dependency on a small group of US-based humans to act as interpreters between developers and their own pipeline output.\nThe Solution: AI as a Secure Intermediary # The insight was simple. Developers didn\u0026rsquo;t need access to the system; they needed information from the system. If we could extract the relevant content, sanitize it, and surface it through a channel they could already access, the human-in-the-loop bottleneck disappears.\nAWS Bedrock was the right tool here for a few reasons: it\u0026rsquo;s a managed service that runs inside our AWS GovCloud environment, it never sends data outside our compliance boundary, and it integrates cleanly with the rest of our stack.\nHow It Works # The flow is straightforward:\nThe CI/CD platform fires a webhook on every build completion — pass or fail — into an SQS queue A Durable Lambda processes the queue. If the job failed, it captures the console output The console output is sent to AWS Bedrock with a structured prompt for analysis The summarized, sanitized output is forwarded to our Internal Developer Portal (IDP), where the developer can read it directly No ticket. No screen share. No waiting for a US engineer to wake up. The developer gets actionable failure analysis in their IDP within minutes of the build completing.\nWe also integrated a Bedrock Knowledge Base maintained by my team. It contains known errors, common failure patterns, and platform-specific context. When Bedrock analyzes a failure, it can reference this knowledge base to match known issues and return consistent, reliable guidance rather than generic AI output.\nThe Hard Part: Console Log Chaos # The architecture is clean. The reality was messier.\nThe biggest challenge was data quality, not AI. Console logs are not designed for machine consumption. Some teams let their pipelines print everything: download progress bars, package manager install lines, verbose debug output that nobody reads. In a 50,000-line console log, the actual error is buried somewhere in the noise.\nThis matters because LLMs have context limits. Sending a 50,000-line log to Bedrock either hits the token limit, costs a fortune, or both.\nWhat we did:\nChunked the console output — rather than sending the whole log, we split it into windows and process the most relevant sections, prioritizing the tail where failures typically appear Advised development teams to reduce noise — download progress lines, package manager output, and repetitive status messages don\u0026rsquo;t help humans read logs, and they don\u0026rsquo;t help AI either. We started advocating for cleaner pipeline output as a general practice Added a \u0026ldquo;LOOK HERE\u0026rdquo; pattern — teams can embed explicit signal phrases in their pipeline output. When the system prompt sees one of these markers, it directs the model\u0026rsquo;s attention accordingly. If a developer knows their build sometimes fails for a specific known reason, they can annotate their pipeline output to steer the AI toward the right conclusion That last one is worth expanding on. The \u0026ldquo;LOOK HERE\u0026rdquo; pattern isn\u0026rsquo;t a hack; it\u0026rsquo;s a deliberate design. You\u0026rsquo;re essentially making your pipeline output model-aware: structuring it so that both humans and AI can navigate it efficiently. It forces teams to think about what information is actually signal vs. noise, which improves debugging for everyone.\nThe Red Herring Problem # Early on, we fielded complaints: the AI was identifying the wrong error. A build would fail for reason X, but Bedrock would confidently explain reason Y, an earlier warning in the log that looked suspicious but wasn\u0026rsquo;t actually the cause.\nThis is the \u0026ldquo;red herring\u0026rdquo; problem. Console output is full of warnings, deprecation notices, and non-fatal errors that look alarming but are harmless. A model without context reads all of them as potential causes.\nThe fix was better input, not a better model or a smarter prompt. By guiding teams to print cleaner, more intentional output, and by using the \u0026ldquo;LOOK HERE\u0026rdquo; pattern to explicitly flag the failure point, we reduced the surface area for the model to misinterpret. The quality of the analysis is directly proportional to the quality of the input. That sounds obvious in retrospect. It wasn\u0026rsquo;t obvious at the start.\nWhat It Changed # Since shipping, ticket volume to the Federal Production Engineering team has dropped significantly. More importantly, developers now have autonomy they didn\u0026rsquo;t have before. A failed build goes from \u0026ldquo;open a ticket and wait\u0026rdquo; to \u0026ldquo;check the IDP and iterate,\u0026rdquo; a feedback loop measured in minutes, not hours or days.\nThe trust in the output has also improved substantially as we\u0026rsquo;ve tuned the system. Engineers on my team spend far less time as log-reading intermediaries and more time on work that actually requires their expertise.\nWhat\u0026rsquo;s Next # We\u0026rsquo;re actively looking for more use cases as we grow the Federal footprint across more teams. The core pattern (capture, sanitize, analyze, surface) is portable. Pipeline failure analysis was the first application, but anywhere you have compliance-gated information that developers need access to, there\u0026rsquo;s a version of this problem worth solving.\nThis is the first post in the \u0026ldquo;AI in the Trenches\u0026rdquo; series — practitioner-level writing about using AI in real infrastructure, without the hype.\n","date":"April 11, 2026","externalUrl":null,"permalink":"/posts/2026-04-11-prompt-engineering-in-production/","section":"Posts","summary":"Most prompt engineering content is written by people who have never shipped AI into a real system. This post is different.\nI’m a Staff DevOps Engineer. I’ve spent time integrating AWS Bedrock into actual production workflows inside a FedRAMP High environment, not demos, not notebooks, not prototypes. Real systems with real constraints and real failure modes.\nHere’s what I learned.\n","title":"Prompt Engineering Isn't Magic — Here's How I Did It in Production","type":"posts"},{"content":"On March 25, 2025, a malicious version of the axios npm package (version 1.14.1) was published to the npm registry. It contained a bundled dependency called plain-crypto-js@4.2.0, which was later confirmed to be malicious. The package was identified and removed from the npm registry within roughly three hours.\nFor most organizations, three hours sounds manageable. For us, it wasn\u0026rsquo;t that simple.\nHow We Found Out # I came across John Hammond\u0026rsquo;s breakdown of the incident through Huntress. After watching it, the severity was clear: this wasn\u0026rsquo;t a theoretical risk. Axios is one of the most widely used HTTP client libraries in the JavaScript ecosystem, and our organization had pipelines using it across multiple teams.\nI immediately looped in our product security team and central security architects. We aligned quickly on the need to investigate our exposure and got to work.\nThe First Problem: Artifact Repository Doesn\u0026rsquo;t Know What It Doesn\u0026rsquo;t Know # Our organization uses an artifact repository as a proxy and cache for third-party npm packages. The intent is to reduce external dependency, improve build reliability, and give us a controlled chokepoint for what enters our supply chain. In theory, it\u0026rsquo;s exactly the right architecture for situations like this.\nIn practice, it had a blind spot.\nWhen axios@1.14.1 was published, some of our pipelines pulled it through the artifact repository during that three-hour window. It cached the package. When npm yanked the package, the artifact repository didn\u0026rsquo;t get the memo; it continued serving the cached version to any pipeline that requested it. From the artifact repository\u0026rsquo;s perspective, it was just doing its job.\nWe were able to check the download count for axios@1.14.1 in the artifact repository relatively quickly. The numbers confirmed it had been pulled. The harder question was: by whom?\nThe Second Problem: A Service Account Tells You Nothing # Our CI/CD pipelines authenticate to the artifact repository using a shared service account. That means every download from every pipeline shows up under the same username in the artifact repository\u0026rsquo;s access logs. The logs are effectively web server access logs: they tell you what was downloaded, but not which team or pipeline requested it.\nTo answer that question, we needed to look at the build logs in our CI/CD platform. Those logs weren\u0026rsquo;t piped into a SIEM or centralized logging system. Ingesting them at scale would be expensive, and potentially introduce additional data exposure risk. So our DFIR team did it the hard way: manually searching build logs for specific log lines indicating the axios@1.14.1 download and the presence of plain-crypto-js@4.2.0.\nWhat We Found # The DFIR team searched for npm install log lines referencing axios@1.14.1 and plain-crypto-js@4.2.0. They found hits.\nThe affected pipelines weren\u0026rsquo;t directly depending on axios; they were pulling datadog-ci and npm-groovy-lint, both of which list axios as a dependency. Critically, neither package had pinned an explicit axios version in their package.json. They used a minimum version constraint, meaning npm would resolve to the latest matching version available at install time.\nDuring that three-hour window, the latest matching version was 1.14.1. Any pipeline that ran an npm install during that period (or afterward, while the cached version remained available in the artifact repository) pulled the malicious package.\nContainment # Once we confirmed which build agents had been affected, we traced them to their specific worker nodes on our container orchestration platform. We treated those nodes as compromised.\nThe containerized workload model worked in our favor here. Because the build agents were isolated containers, the blast radius was constrained. We quarantined the affected nodes, captured the volumes for forensic inspection, removed the nodes from the cluster, and terminated them after completing our investigation. No malicious files or indicators of active compromise were found, but we weren\u0026rsquo;t willing to leave that to chance.\nWe also manually purged axios@1.14.1 and plain-crypto-js@4.2.0 from the artifact repository cache to eliminate the ongoing distribution risk.\nWhat We\u0026rsquo;re Doing About It # This incident exposed a few gaps we\u0026rsquo;re now actively closing:\nDependency pinning enforcement — We\u0026rsquo;re implementing automated checks across CI/CD pipelines to flag unpinned dependency versions as a policy violation. Using minimum version constraints (^, ~, \u0026gt;=) is a common practice that most developers don\u0026rsquo;t think twice about. This incident is a good illustration of why it matters.\nPackage publication delay — We\u0026rsquo;re investigating a configurable delay in the artifact repository before newly published package versions become available to pipelines. A 24-hour hold would have meant axios@1.14.1 was already removed from npm before any of our pipelines could pull it. It adds friction to adopting new releases, but that tradeoff is worth it for the reduction in zero-day exposure.\nAutomated malicious package scanning — We\u0026rsquo;re evaluating solutions that can scan packages in the artifact repository for known-malicious indicators and automatically quarantine them, rather than relying on manual response after the fact.\nThe Broader Lesson # The axios incident is a good case study in how supply chain attacks actually work in practice. The malicious package was live for three hours. That\u0026rsquo;s fast enough to fly under the radar of most security teams, but slow enough to get cached everywhere.\nThe real risk isn\u0026rsquo;t the registry. It\u0026rsquo;s everything downstream of it: your artifact proxy, your package cache, your pipelines that were running during the window. By the time npm acts, the package may already be sitting in your artifact repository waiting for the next build to pull it.\nIf you use a package proxy or cache (and you should), make sure you have a plan for what happens when that cache becomes the attack vector rather than the defense.\nCredit to Huntress for their rapid analysis and public disclosure of the axios supply chain compromise.\n","date":"April 11, 2026","externalUrl":null,"permalink":"/posts/2026-04-11-axios-supply-chain/","section":"Posts","summary":"On March 25, 2025, a malicious version of the axios npm package (version 1.14.1) was published to the npm registry. It contained a bundled dependency called plain-crypto-js@4.2.0, which was later confirmed to be malicious. The package was identified and removed from the npm registry within roughly three hours.\nFor most organizations, three hours sounds manageable. For us, it wasn’t that simple.\n","title":"Supply Chain Attacks Start in Your Build Cache","type":"posts"},{"content":"Platform Engineering Leader · DevSecOps · Cloud · AI\nI\u0026rsquo;m a platform and infrastructure engineer with 12+ years of experience building the systems that ship software at scale. I currently work at Palo Alto Networks as Manager of DevOps Engineering, leading a team of 8 that owns the security toolchain, policy enforcement, and org-wide security posture strategy. Before moving into management, I architected an enterprise DevSecOps platform there supporting 2,000+ developers. I spent the years before that at CyberArk, most recently as Staff DevOps Engineer, leading teams and driving platform modernization across cloud and on-prem environments.\nWhat I Do # I specialize in the full lifecycle of platform engineering: designing cloud infrastructure, building CI/CD pipelines, ensuring compliance, and extracting performance out of complex distributed systems.\nCore skills:\nCloud: AWS (including GovCloud), Kubernetes/EKS, Terraform, Pulumi CI/CD: GitHub Enterprise, ArgoCD, Jenkins Observability: Grafana, Prometheus, DataDog Automation: Ansible, Docker, Python, Go Security \u0026amp; Compliance: FedRAMP High, DevSecOps AWS Certified Solutions Architect – Associate (expires March 2027)\nHighlights # ~20x throughput improvement on an automated artifact migration system (25 → ~500 artifacts/min) Reduced SaaS deployment time from ~90 minutes to 25 minutes; improved success rate from ~50% to ~80% Led engineering teams of 5–9 engineers across full-stack platform and DevOps initiatives FedRAMP High compliance — built and maintained compliant delivery pipelines in AWS GovCloud AI in production — engineered prompt tooling with AWS Bedrock, including a knowledge base layer to reduce hallucinations and token burn Find Me # LinkedIn GitHub Email ","date":"April 5, 2026","externalUrl":null,"permalink":"/about/","section":"Cliff Hults","summary":"Platform Engineering Leader · DevSecOps · Cloud · AI\nI’m a platform and infrastructure engineer with 12+ years of experience building the systems that ship software at scale. I currently work at Palo Alto Networks as Manager of DevOps Engineering, leading a team of 8 that owns the security toolchain, policy enforcement, and org-wide security posture strategy. Before moving into management, I architected an enterprise DevSecOps platform there supporting 2,000+ developers. I spent the years before that at CyberArk, most recently as Staff DevOps Engineer, leading teams and driving platform modernization across cloud and on-prem environments.\n","title":"About","type":"page"},{"content":"","date":"February 13, 2026","externalUrl":null,"permalink":"/tags/compliance/","section":"Tags","summary":"","title":"Compliance","type":"tags"},{"content":"Enforcement worked. Critical vulnerabilities were suddenly getting fixed on deadlines, and the backlog that had sat ignored for years started to clear. But it worked too well in one sense: the volume of new work it created across thousands of developers turned enforcement itself into a blocker.\nThis is the part of the story almost nobody writes about, because it\u0026rsquo;s the part that looks like a failure. It was the most instructive decision I made.\nThe friction problem # Every enforcement program generates friction. When you add a hard deadline to a category of work developers had been deferring, you create a surge of demand. Teams that were coping by ignoring vulnerabilities suddenly had to confront all of them at once.\nThe result: enforcement worked, but it became very difficult to manage across an org of thousands. The friction was real, and it threatened to burn people out.\nThe instinct: add a relief valve # The natural first response is to give people a temporary way around the pressure (an exemption, a waiver, a business-justified exception). I introduced exactly that: a process where a team could temporarily bypass an enforcement block, with the requirement that they commit to repairing the underlying issue later.\nOn paper it was clean. In practice it taught me a hard lesson.\nWhat the waiver actually did # The waiver relieved the pressure, but it also relieved the priority. Once a team knew they could get a temporary bypass, the urgent work quietly became the deferred work again. The exemption I designed as a safety valve became a deferral mechanism.\nThe uncomfortable truth: exceptions shift problems; they rarely solve them. Every waiver was a moment where \u0026ldquo;fix it now\u0026rdquo; became \u0026ldquo;fix it later,\u0026rdquo; and \u0026ldquo;later\u0026rdquo; has a very long half-life in a busy engineering org.\nWhat I tell leaders designing exception processes # Assume the exception will be overused. Whatever fraction you think will use it, it will be higher. Design for that. Make exceptions expensive enough to be rare. If bypassing is as easy as complying, you\u0026rsquo;ve built a path of least resistance to not fixing things. Every exception should have an owner and a deadline. An open-ended waiver is a permanent vulnerability with extra paperwork. Watch what happens with the exceptions; it\u0026rsquo;s your most honest signal. The waiver rate isn\u0026rsquo;t a compliance stat; it\u0026rsquo;s a UX stat for your security program. Why this failure unlocked the real answer # The lesson wasn\u0026rsquo;t \u0026ldquo;remove the waivers.\u0026rdquo; It was diagnostic: the waivers revealed the fundamental problem with enforcement-based security. Enforcement creates urgency, but urgency alone doesn\u0026rsquo;t make remediation easy. If the only two options are \u0026ldquo;suffer the friction\u0026rdquo; or \u0026ldquo;use a loophole,\u0026rdquo; people will pick the loophole, and you can\u0026rsquo;t blame them.\nThat\u0026rsquo;s when I stopped asking how do we enforce harder and started asking why is fixing this so hard in the first place?\nThe answer to that question changed everything, and it\u0026rsquo;s the subject of Part 3.\nThis follows Part 1 — giving the backlog teeth.\nRead Part 3: the paved path — how making the secure choice the easy choice did what enforcement alone never could.\n","date":"February 13, 2026","externalUrl":null,"permalink":"/posts/2026-02-13-the-waiver-trap/","section":"Posts","summary":"Enforcement worked. Critical vulnerabilities were suddenly getting fixed on deadlines, and the backlog that had sat ignored for years started to clear. But it worked too well in one sense: the volume of new work it created across thousands of developers turned enforcement itself into a blocker.\nThis is the part of the story almost nobody writes about, because it’s the part that looks like a failure. It was the most instructive decision I made.\n","title":"The Waiver Trap: When Your Compliance Safety Valve Delays Real Remediation","type":"posts"},{"content":"","date":"February 13, 2026","externalUrl":null,"permalink":"/tags/vulnerability-management/","section":"Tags","summary":"","title":"Vulnerability Management","type":"tags"},{"content":"","date":"February 6, 2026","externalUrl":null,"permalink":"/tags/ciso/","section":"Tags","summary":"","title":"CISO","type":"tags"},{"content":"Every security leader has stared down a backlog of ignored vulnerabilities. You know the shape of it: thousands of open findings, tracked diligently, remediated rarely. The problem wasn\u0026rsquo;t that people didn\u0026rsquo;t care; it was that nothing was at stake for not fixing them.\nI led a vulnerability-management program across a large engineering organization of thousands of developers and repositories. Early on I learned that you can\u0026rsquo;t improve what you don\u0026rsquo;t enforce. Here\u0026rsquo;s what actually happens when you give the backlog teeth, and the uncomfortable truth about what enforcing it really takes.\nStart by asking: where\u0026rsquo;s the accountability gap? # Before touching any tooling, look for the structural reason things don\u0026rsquo;t get fixed. In our case, teams weren\u0026rsquo;t refusing to remediate. They were prioritizing. But with no forcing function, \u0026ldquo;fix it eventually\u0026rdquo; quietly became \u0026ldquo;fix it never.\u0026rdquo; Vulnerability work always loses to feature work when both are optional.\nThe gap wasn\u0026rsquo;t awareness. It was that nothing forced a decision on a deadline. That\u0026rsquo;s where enforcement enters the picture, not as a punishment mechanism but as the accountability structure that makes prioritization honest.\nEnforcement is a product decision, not a policy decision # When I introduced remediation SLOs (critical vulnerabilities had a fixed window to be fixed before the team was blocked), I deliberately treated it less like a compliance rule and more like a product launch. The reasoning:\nAn SLO nobody can meet is just noise. If the window is unrealistic, teams stop believing the system, and the metric becomes fiction everyone quietly ignores. Enforcement changes behavior only if the consequence is real. A block on day N that the system never actually fires is theater, and teams sense it immediately. The design has to survive contact with reality. This is where most enforcement programs die, not on principle but on the day-to-day friction of operating at scale. The uncomfortable truth: putting enforcement in place is the beginning of the work, not the end. It surfaces problems everywhere, some you predicted and some you didn\u0026rsquo;t. And your willingness to watch the backlash, learn from the friction, and iterate on the design is what separates an enforced program from a performative one.\nWhat I\u0026rsquo;d tell a security leader starting this # Pick a metric that can\u0026rsquo;t be gamed. Time-to-fix is better than raw count, because it measures behavior, not volume. Expect the friction. Enforcement reveals the friction in your process. That\u0026rsquo;s data, not a failure. Measure before you enforce. You need the baseline to prove later that you moved the needle. Be ready to iterate. Your first enforcement design will be wrong in some way. The fix is learning from it, which is exactly where the next part of this story begins. When we compressed the average critical-vulnerability lifecycle from weeks to days and cleared thousands of backlogged findings, the enforcement timer was the catalyst. But it wasn\u0026rsquo;t the solution. The solution took two more iterations to find, and the middle one was the most instructive.\nRead Part 2: the safety valve that taught us why enforcement alone isn\u0026rsquo;t enough.\n","date":"February 6, 2026","externalUrl":null,"permalink":"/posts/2026-02-06-give-the-backlog-teeth/","section":"Posts","summary":"Every security leader has stared down a backlog of ignored vulnerabilities. You know the shape of it: thousands of open findings, tracked diligently, remediated rarely. The problem wasn’t that people didn’t care; it was that nothing was at stake for not fixing them.\nI led a vulnerability-management program across a large engineering organization of thousands of developers and repositories. Early on I learned that you can’t improve what you don’t enforce. Here’s what actually happens when you give the backlog teeth, and the uncomfortable truth about what enforcing it really takes.\n","title":"Giving the Backlog Teeth: What Enforcing Remediation SLOs Actually Requires","type":"posts"},{"content":"","date":"December 16, 2021","externalUrl":null,"permalink":"/categories/info/","section":"Categories","summary":"","title":"Info","type":"categories"},{"content":"","date":"December 16, 2021","externalUrl":null,"permalink":"/tags/java/","section":"Tags","summary":"","title":"Java","type":"tags"},{"content":"","date":"December 16, 2021","externalUrl":null,"permalink":"/tags/log4j/","section":"Tags","summary":"","title":"Log4j","type":"tags"},{"content":"Lately, everyone has been talking about Log4Shell (CVE-2021-44228) and likely, if you\u0026rsquo;re reading this, you\u0026rsquo;re looking for info for what to do. Most people attempted to utilize Huntress\u0026rsquo;s Log4Shell detection tool to show connections to a LDAP server they were hosting. Some people had issues with this as it was overburdened with requests (rightfully so) or didn\u0026rsquo;t want to, or aren\u0026rsquo;t allowed to send outbound traffic to a server they didn\u0026rsquo;t own. At our organization, we were part of the latter group. In order to comply with our rules, we needed to find a reliable way to scan our devices for the vulnerability and look for requests to a domain that we owned.\nIn order to know what to look for, you have to understand what happens with this vulnerability.\nWhen the vulnerability is executed, the log4j module navigates to the address provided in the payload.\n${jndi:ldap://log4shelltest.company.pvt:1389/somerndmstr}\nIn the above payload, the targeted host that receives the data will attempt to resolve \u0026ldquo;log4shelltest.company.pvt\u0026rdquo; to make a connection. If you own the domain/subdomain you can track for DNS query logs for that domain.\nIn order to achieve this, we setup 2 DNS servers (internally and externally) and utilized nmap scripts found here. Scanning each host and sending the payload for our domain with a unique subdomain address, we were able to isolate the calling host to the query made to the server and could easily track down the offending host.\nTo prove the initial concept, we used a vulnerable application provided by the community. There are many like it but the one we used was kozmer/log4j-shell-poc.\nGood luck to all of you out there and hope this could be helpful to some.\n","date":"December 16, 2021","externalUrl":null,"permalink":"/posts/2021-12-16-log4j/","section":"Posts","summary":"Lately, everyone has been talking about Log4Shell (CVE-2021-44228) and likely, if you’re reading this, you’re looking for info for what to do. Most people attempted to utilize Huntress’s Log4Shell detection tool to show connections to a LDAP server they were hosting. Some people had issues with this as it was overburdened with requests (rightfully so) or didn’t want to, or aren’t allowed to send outbound traffic to a server they didn’t own. At our organization, we were part of the latter group. In order to comply with our rules, we needed to find a reliable way to scan our devices for the vulnerability and look for requests to a domain that we owned.\n","title":"Log4j Scanning and Detection","type":"posts"},{"content":"","date":"December 16, 2021","externalUrl":null,"permalink":"/tags/vulnerability/","section":"Tags","summary":"","title":"Vulnerability","type":"tags"},{"content":"","date":"November 20, 2020","externalUrl":null,"permalink":"/tags/adfs/","section":"Tags","summary":"","title":"Adfs","type":"tags"},{"content":"Microsoft recently performed a patch for Kerberos and the KDC service on domain controllers. This would patch a heavy vulnerability in the Kerberos signing structure. However, this presented a problem with our domain joined NAS appliance, and disallowed any users from authenticating to SMB shares. In order to alleviate ourselves of the issue, we followed the instructions to disable the following registry key within HKLM\\SYSTEM\\CurrentControlSet\\Services\\Kdc\\ by adding PerformTicketSignature set to DWORD 0.\nUpon making this change, settings worked back to normal and we followed with the week as normal. However, we didn\u0026rsquo;t know what else we broke\u0026hellip;\nToday, we had our help desk call and mention that our self service ticket portal, which leverages ADFS for authentication, no longer lets people login. The ADFS logon page would kick users back to the logon page with no error message.\nChecking within the Applications and Services Logs\\ADFS of the Event Viewer, the event logs displayed the following error upon every login:\nThe digitally signed Privilege Attribute Certificate (PAC) that contains the authorization information for client USERACCOUNT in realm \u0026lt;NETBIOSDOMAIN\u0026gt; could not be validated. This error is usually caused by domain trust failures; Contact your system administrator. Enabling PerformTicketSignature in HKLM\\SYSTEM\\CurrentControlSet\\Services\\Kdc\\ allowed ADFS to properly sign tickets again and allow users to login once again.\nMoral of the story, pay attention to Microsoft\u0026rsquo;s Patch Tuesday notes and stay ever vigilant.\nMore information on the patch can be seen here: https://msrc.microsoft.com/update-guide/vulnerability/CVE-2020-17049\n","date":"November 20, 2020","externalUrl":null,"permalink":"/posts/2020-11-20-adfs-cve-2020-17049/","section":"Posts","summary":"Microsoft recently performed a patch for Kerberos and the KDC service on domain controllers. This would patch a heavy vulnerability in the Kerberos signing structure. However, this presented a problem with our domain joined NAS appliance, and disallowed any users from authenticating to SMB shares. In order to alleviate ourselves of the issue, we followed the instructions to disable the following registry key within HKLM\\SYSTEM\\CurrentControlSet\\Services\\Kdc\\ by adding PerformTicketSignature set to DWORD 0.\n","title":"ADFS \u0026 CVE-2020-17049","type":"posts"},{"content":"","date":"November 20, 2020","externalUrl":null,"permalink":"/tags/cve/","section":"Tags","summary":"","title":"Cve","type":"tags"},{"content":"","date":"November 20, 2020","externalUrl":null,"permalink":"/tags/windows/","section":"Tags","summary":"","title":"Windows","type":"tags"},{"content":"","date":"October 20, 2019","externalUrl":null,"permalink":"/tags/ansible/","section":"Tags","summary":"","title":"Ansible","type":"tags"},{"content":"In my last post, I opened with the fact that my company has decided to dive into the world of GCP to get ahead of most of the market in our space. With a few of us being tasked for this initive, I decided to take it upon myself to look into Packer. The goal would be to roll our homebrewed software/OS image into an automated build process to make images in GCP (or AWS, vSphere, etc.).\nOur current image is based on CentOS 6.8, so I needed to create kickstart scripts, some provisioning scripts, and then create some post processing steps to get it all into GCP without interaction.\nTODO # GRUB changes # Google Guest environment challenges. # ","date":"October 20, 2019","externalUrl":null,"permalink":"/posts/2019-10-20-gcp-image-build/","section":"Posts","summary":"In my last post, I opened with the fact that my company has decided to dive into the world of GCP to get ahead of most of the market in our space. With a few of us being tasked for this initive, I decided to take it upon myself to look into Packer. The goal would be to roll our homebrewed software/OS image into an automated build process to make images in GCP (or AWS, vSphere, etc.).\n","title":"Building Images for GCP","type":"posts"},{"content":"","date":"October 20, 2019","externalUrl":null,"permalink":"/categories/devops/","section":"Categories","summary":"","title":"DevOps","type":"categories"},{"content":"","date":"October 20, 2019","externalUrl":null,"permalink":"/tags/packer/","section":"Tags","summary":"","title":"Packer","type":"tags"},{"content":"","date":"October 20, 2019","externalUrl":null,"permalink":"/tags/terraform/","section":"Tags","summary":"","title":"Terraform","type":"tags"},{"content":"","date":"September 11, 2019","externalUrl":null,"permalink":"/tags/vault/","section":"Tags","summary":"","title":"Vault","type":"tags"},{"content":"Being a good SysAdmin requires some sense of laziness. In the spirit of that approach, I\u0026rsquo;ve spent some time looking into Hashicorp\u0026rsquo;s Terraform and Red Hat\u0026rsquo;s Ansible tools in my organization\u0026rsquo;s route to Google Cloud Platform. I wanted a method to create reproducible Compute Engine VMs that would allow us to easily create multiple hosts with minor changes quickly and easily.\nI decided that with the use of Terraform, Vault (also from Hashicorp) would allow me to templatize the configs, as well as, create secrets/passwords that would be randomized, able to be called at will, and even automatically rolled with a configured frequency.\n","date":"September 11, 2019","externalUrl":null,"permalink":"/posts/2019-09-11-vault-journey/","section":"Posts","summary":"Being a good SysAdmin requires some sense of laziness. In the spirit of that approach, I’ve spent some time looking into Hashicorp’s Terraform and Red Hat’s Ansible tools in my organization’s route to Google Cloud Platform. I wanted a method to create reproducible Compute Engine VMs that would allow us to easily create multiple hosts with minor changes quickly and easily.\nI decided that with the use of Terraform, Vault (also from Hashicorp) would allow me to templatize the configs, as well as, create secrets/passwords that would be randomized, able to be called at will, and even automatically rolled with a configured frequency.\n","title":"Vault Journey","type":"posts"}]