This year, I used Claude Code to fill my tax return.

I know. Taxes. How is that relevant to Software engineering. Bear with me.

This article isn’t really about taxes. Taxes are just the problem that pushed me to discover something much bigger.

The real story is about connecting tools in ways you didn’t expect. About turning a painful, weeks-long process into a few hours of work.

If you’re a software engineer, this matters to you. Not because of taxes. Because this pattern works for anything that happens in a browser. Batch-fill job applications. Automate web scraping. Test web applications. Scrape data at scale. Whatever repetitive browser work is eating your time, this unlocks it.

No AI browser was harmed in this process. Instead, I used Playwright MCP to give Claude Code direct control over my browser.

The tax return was just the catalyst. The real pattern is much bigger.

The Problem: Why Tax Returns Feel Impossible

Every January, I spend weeks filling out HMRC forms. Hundreds of questions. Complex income streams from stocks, side projects, and savings. The math is easy. The form is not.

I looked up accountants. Some were expensive, some might have actually helped, but it felt like overkill for someone who could probably figure this out themselves.

I tried tax software next.

But here’s the thing: most UK people don’t need to fill tax returns at all. If you work for a single employer, the system handles it for you.

Tax software is designed for the majority, not for someone like me with multiple income streams. I had stocks from my previous company, side income from mentoring, Substack revenue, sand avings interest.

My situation was slightly more complicated than most, but it didn’t fit the simple templates tax software was built for.

Last year, I got Claude Code to write Python code to calculate my total income, handle capital gains, all of it. That worked great.

But here’s the thing. The math was only 5% of the problem.

The other 95% was filling out the actual form. Understanding what each question meant. Figuring out which questions applied to my situation. Typing the right answers in the right boxes. That’s where I was losing weeks every year.

The Solution: Playwright MCP

Why AI Browsers Failed

I wanted Claude to actually read the form. See the questions in context, understand what each box was asking for. I tried Claude’s Chrome extension. Seemed perfect at first. Claude looking at my browser, helping me fill forms.

One problem. It only supported the older Haiku model. I’ve been using Opus, which is way smarter.

I looked at other AI browsers like Comet by Perplexity and others. Same issue. They locked me into weaker models. No Opus. No choice.

But there was a bigger problem I didn’t see at first.

These AI browsers had no access to my local data. My income calculations. My savings files. My bank statements. They could only see the form itself. That meant I had to manually copy-paste my data, which defeated the whole purpose of automation.

How Playwright MCP Works

Then I realised the approach was backwards.

Instead of bringing Claude Code into the browser, I needed to bring the browser to Claude Code. That meant finding a way for Claude to control my browser through proper automation, not through a limited Chrome extension.

I looked it up and found it. Playwright MCP. MCP stands for Model Context Protocol, Anthropic’s way of letting Claude access external tools. There’s a Playwright MCP that bridges Claude and Playwright together.

The setup is straightforward. Install the tool, add configuration to your Claude Code config file, tell it where Playwright is, and you’re done.

Suddenly Claude could see your browser, read the content, fill form fields, navigate pages, take screenshots. No copy-pasting. Claude actually looks at the form and fills it.

But the real magic? You keep access to everything on your machine. Your local files. Your income data. Your calculations. And you’re not locked into a weaker model. You can use Opus for complex reasoning. Switch to Haiku for fast form filling. You’ve got options.

That’s the difference. AI browsers give you sight into the form. Playwright MCP gives you full context plus full control.

In Practice: My Tax Return

This year I tested it on my tax return and the difference was night and day.

I went from spending weeks wrestling with forms to just a few hours, spread over a couple of days.

Here’s what actually happened. I asked Claude to navigate to the HMRC form, read each question, and fill in my answers based on my income data. Claude could see the form. It could read what each field was asking for. And because I had all my calculations locally, Claude had full context.

The best part wasn’t even the automation.

It was the conversations. Because Claude had access to every question on the form, I could ask it to explain things. Why does this field matter? Does this apply to my situation? It was like having a private accountant sitting next to me. One that costs nothing beyond a Claude Pro subscription. Not even Max. Just Pro.

Voice + Browser Automation: Combining Tools for Power

Do you want to know the best part of this experience?

Actually, while I was filling my taxes this way, I was also using my voice to ask Claude questions.

No typing. Just speaking. “Claude, why does this field matter? What does this mean?” All while Playwright was filling forms in the background. That’s when I realized how powerful combining these tools really is.

If you’re interested in learning how I stopped typing most of my AI prompts, check out my article at Talk to your AI agent: the ultimate vibe coding.

What You Can Do With This Pattern

I built my tax filling learnings into a reusable Claude Code skill I can trigger next financial year.

If you want to learn more about why Claude Code skills are a game changer, check out my article on Claude Agent Skills: Teaching Your AI Agent to Wear Multiple Hats.

But taxes were just the starting point. Think about what’s happening in your business. Where do people spend weeks filling out forms? Job applications. Expense reports. Data migration. Customer onboarding. Any repetitive web task that requires human judgment.

Now you have a tool for it. Claude Code plus Playwright MCP. Web scraping. Batch form filling. Testing. Data automation. Anything that happens in a browser.

You get full automation with human reasoning. Full access to your local context and data. And you’re not locked into a weaker model. You can use whatever Claude model makes sense for the task.

Conclusion

That’s the pattern. Voice. Browser automation. Local files. Three tools that weren’t designed to work together, now solving problems that seemed impossible.

Innovation isn’t about writing code anymore. It’s about connecting existing tools creatively. Playwright MCP proved that to me. A library I’ve used for years, suddenly powerful when connected to Claude Code.

The tax return was just the catalyst. This pattern works for anything that happens in a browser. Web scraping. Form filling. Job applications. Data automation.

What creative tool combinations have you found? What problem have you solved by wiring two things together that weren’t meant to work?

Drop a comment or reach out on LinkedIn. I’d love to hear what you’re building.

References

• Claude Code - Anthropic’s official CLI for Claude with integrated tool access

• Playwright MCP - Model Context Protocol server that bridges Claude Code to Playwright browser automation

• Model Context Protocol (MCP) - Open protocol for connecting Claude to external tools and data sources

• Playwright - Cross-browser automation library for web testing and form automation

• Claude Agent Skills: Teaching Your AI Agent to Wear Multiple Hats - How to build reusable Claude Code skills and why they’re transformative for automation

• Talk to your AI agent: the ultimate vibe coding - How voice-to-text unlocks a faster, smarter way to work with AI through natural conversation

When you are too lazy to type your prompts, use your voice instead.

Jokes aside, you don’t have to be lazy to benefit from this article. You just need to be smart enough to recognise that even seemingly silly suggestions, like using your voice instead of typing, can make a massive difference in your career over time. These things compound.

Once you realise you can speak to your agent instead of typing, regular typing feels like you’ve been using only 10% of your brain.

Why does voice work so well for vibe coding?

Because you’re not dictating code. Typing code requires precision—brackets, punctuation, indentation, whitespace—all the things your voice struggles to convey.

But speaking conversational prompts to an AI agent that’s designed to handle imperfect input? That’s a completely different game. The agent filters out your “umms” and pauses. It corrects your spelling. It catches the meaning of what you’re saying even when you’re rambling or going into too much detail.

In this article, I’ll share a personal story that connects you to my grandfather 30 years ago. I’ll show you why small daily improvements compound into real career gains. And I’ll give you the exact setup that works, even in a noisy office.

Beyond AI interaction

Voice-to-text isn’t just for vibe coding either.

I voice-typed notes for this article in about half an hour. Normally, writing those notes down by typing would’ve easily taken me a couple of hours. That’s not a small difference.

The reason? When you use your voice, your thoughts flow naturally. You’re not slowed down by your fingers trying to keep up with your brain. And that matters more than you’d think. Because the moment you slow down, you lose momentum. You lose your train of thought. You start second-guessing yourself instead of just getting the ideas out.

With voice, you just... talk. The ideas stay fresh. The momentum keeps going.

And here’s the real magic: you’re not staring at a blank page anymore. You’ve got rough notes to work with. Something tangible to mould into a full article.

It takes some practice, but once you’ve done it for a while, it feels like talking to a colleague.

You’d be surprised how much you can accomplish just by whispering to your AI agent.

Personal story: Full circle after 30 years

I remember my grandad in 1997 (when I was 13 and he was 69). He was looking up articles on Encarta Enciclopedia, burning CDs with Nero Burning ROM, and reading passages aloud from Pinocchio to train Dragon NaturallySpeaking to understand his voice so he could dictate text to his computer.

I’m sharing this because I'm using AI to convert my speech to text. Just like my grandfather did more than 30 years ago.

No time seems to have passed, except that now I use my voice to interact with an AI that actually understands me.

Small improvements compound: Typing with 10 fingers

I learned to touch-type more than 15 years ago.

When someone at the university suggested that using all ten fingers was a must.

How would that actually make me a better engineer?

I thought about it, decided it was worth a shot, and started training. Honestly, it took me probably a couple of months to get it right. It was boring. There was a program that asked me to type the same letter combinations repeatedly. It seemed pointless.

But you know what? It paid massive dividends over time.

The same lesson: Typing with no fingers

A month ago, a colleague suggested I use my voice to type text into my laptop. My first thought?

That’s silly. It’ll be slow. It’ll be uncomfortable. I’ll feel awkward talking to my laptop at work.

Plus, I was already quite fast at typing. Why would I need this?

Then I saw the same colleague using voice-to-text at his desk.

Something clicked.

It was the same feeling I had when someone first suggested touch-typing. I realized it was worth the time investment to give it a genuine try.

The 1% better every day principle

Both touch-typing and voice-to-text are perfect examples of how getting 1% better every day compounds into real change.

This reminds me of the book Atomic Habits by James Clear.

I have a terrible memory, so I rarely remember passages from books, but one specific story from it stuck with me: the UK cycling team.

They were really struggling until they changed coaches. The new coach implemented a radical strategy: improve everything by 1% every single day. They didn’t focus on massive changes. Instead, they worked on tiny things. Better tyre grips. Washing hands to avoid the flu. Hundreds of small improvements that individually seemed pointless.

Together? They compounded into something massive. That cycling team went from underperforming to consistently winning gold medals at the Olympics.

If you want to dive deeper into this concept, James Clear (author of Atomic Habits) has a great breakdown on marginal gains.

I think about this principle all the time.

Using voice-to-text or mastering touch-typing, these are the kinds of silly things you can do that have massive effects over 5, 10, or 15 years.

Every mentorship session I have now, I tell people: find that 1%.

What makes you 1% better every day?

It could be voice-to-text, touch-typing, or something completely different. But get better every day.

My voice-to-text setup

I was sceptical about using a microphone at work. Our office is shared with hundreds of others and has background noise from the heating units. But honestly? It works really well.

I can whisper into my microphone (positioned right in front of me), and it captures everything. I don’t have to shout or bother my colleagues. I just whisper, and it works.

I’m using an app called MacWhisper (available on Gumroad). Here’s how it works:

The app runs in the background. When you press a specific key combination, it captures your speech and converts it to text using a speech-to-text model running locally on your machine. You can choose from different models depending on your needs and hardware. I’m using “Large Turbo v3” on my M1 MacBook, but you have a range of model options.

After the initial transcription, there’s an optional second step where you can customise the text cleanup. You can use another model (either running locally in Ollama or through an online AI service) to refine your dictation. You can customise the prompt to handle tasks such as removing pauses, correcting spelling, or restructuring your text.

MacWhisper already provides some cleanup prompts, but you can customise them. If you always mispronounce a word, add it to the prompt. If you need specific spelling corrections, add instructions. I honestly haven’t explored 10% of this app’s features yet, but what I’ve seen convinced me to upgrade from the free version to the paid version for access to additional features.

MacWhisper isn’t the only option. Flow is more famous, but it’s subscription-based. Since I’m already overwhelmed with subscriptions, I prefer MacWhisper’s one-time purchase model with a lifetime license.

Why this matters: It started with my manager

I need to credit my manager at Loveholidays for introducing me to MacWhisper. He was using it at his desk, which inspired me to try it. Since then, I’ve invested in a professional podcaster microphone for home and set up the same system at work.

Now we both sit at our desks facing each other, both whispering into our laptops. A couple of colleagues have asked about my setup and how I’m using voice to interact with AI. That’s the main reason I’m writing this article. I wanted to share what I’ve learned.

Thank you to my manager for introducing me to voice-to-text, and to my curious colleagues for pushing me to document this.

Conclusion

This article is about recognising that small improvements compound. Find your 1%. It could be voice-to-text, touch-typing, or something completely different. But once you commit to getting 1% better every day, you’re in the same league as that UK cycling team that went from struggling to winning gold medals.

The best part? Once you’ve done it for a while, it feels as natural as talking to a colleague. You’ll wonder how you ever lived without it.

If you’re interested in trying this, start small. Whisper one prompt to your AI agent today. Better yet, tell someone about what you’re learning. My boss showed me. My colleagues asked me questions. That’s how these things spread.

What’s your 1%? What small thing could you do every day that would compound into something massive over 5, 10, or 15 years?

Drop a comment or reach out on LinkedIn.

I’m genuinely curious what you’re working on.

References

• Atomic Habits: Tiny Changes, Remarkable Results by James Clear - The foundational book on how small daily improvements compound into remarkable results

• Marginal Gains: The Coach Who Improved Every Tiny Thing by 1 Percent - James Clear’s article on the UK cycling team’s success through incremental improvements

• Mac Whisper - Speech-to-text application for macOS with local Whisper V3 Turbo model and customizable post-processing

• Flow by Wispr - Alternative voice-to-text application with subscription-based pricing

• Ollama - Run large language models locally for text cleanup and post-processing

Here’s a situation you might recognise: You want an AI agent that can help you with both Kubernetes troubleshooting AND Python development. So you load it with docs for both. Add some guidelines for how to approach each. Throw in examples, templates, edge cases. Before you know it, your system prompt is thousands of tokens. It becomes a bloated mess that confuses more than it clarifies.

This is the fundamental problem with how we traditionally extend AI agents: prompt stuffing. We take everything the agent might ever need and cram it into system prompts or CLAUDE.md files. The result? Higher costs, slower inference, maintenance headaches, and an agent that struggles to apply the proper context at the right time.

Claude Agent Skills solves this by flipping the approach entirely. Instead of loading everything upfront, skills use progressive context loading: the agent gets just enough information to decide if it needs a skill, then loads only what’s relevant when it’s needed.

This article introduces Claude Agent Skills: what they are, how they work, and why they’re a game changer for anyone building with AI.

How to Interact with Claude

Before diving into skills, let’s quickly map the landscape of ways to extend Claude:

Slash Commands (/commands) are saved user prompts. They provide quick automation for tasks you run manually and are great for repetitive workflows you already know you need.

Output Styles are saved system prompts that change how Claude formats responses and behaves.

MCPs (Model Context Protocol) have been getting a lot of hype lately. Every company rushed to write them. But here’s the reality: MCPs are powerful but complex. Learn more at modelcontextprotocol.io.

Claude Agent Skills are the focus of this article. They combine Markdown-based instructions with optional Python code, designed for complex workflows and automatic invocation.

Claude Agent Skills: Wearing Different Hats

Imagine an agent as a person with multiple jobs. One day, they’re a system designer. Next, they’re a code reviewer. Then they’re a documentation specialist. A single Claude instance can do all of this. You need to teach it how, one skill at a time.

That’s what Claude Agent Skills do. They let you extend Claude with specialised capabilities without rebuilding the entire agent. To understand the deeper architecture and design patterns, read Anthropic’s Equipping Agents for the Real World with Agent Skills article. It explains the reasoning behind how skills are structured.

The Structure of a Skill

A skill is surprisingly simple. It’s just a folder under .claude/skills/ that contains:

• A SKILL.md file: Markdown-based instructions (required)

• Optional reference files: Detailed guides, templates, examples

• Optional Python code: Logic that executes when needed

The reference files, docs, and code can be organised however you want, as long as they’re referenced in the SKILL.md. Claude will find them. For a deeper dive into the technical setup and how to manage skills in Claude Code, see the Claude Code Documentation. That’s it.

A skill is just a folder with some Markdown and some optional Python code. You can copy the skill from one laptop to another, and it works. Move it from one project to another, and it works. No special installation, no dependencies to configure, no marketplace publishing required (though you can do that later).

Just a portable folder of instructions and code.

How Progressive Context Loading Works

This is where skills get powerful. Context loads in stages:

Level 1 - Front Matter (Always Loaded)
The description at the top of SKILL.md is always present in context. It’s short (usually one or two sentences) and answers the question: “Should I use this skill?” Claude reads this and decides if the task matches.

Level 2 - SKILL.md Content (On-Demand)
The full skill instructions load only when Claude decides they’re needed. The agent asks to confirm loading the skill before proceeding. If the task doesn’t match, the skill never loads.

Level 3 - Reference Files (Selectively Loaded)
Detailed guides, templates, and reference docs stay in separate files. They are referenced in SKILL.md and loaded only when required. Unless you explicitly need them, they don’t consume context.

Level 4 - Source Code (Execute, Don’t Load)
Python code executes in the background but isn’t loaded into the context. Claude sees only the input and output. Complex logic stays external, keeping your context clean.

The beauty of this design: the agent only carries the context it actually needs, when it needs it. No bloat. No distraction.

Why This Matters

Code Execution: Unlike slash commands or output styles, skills can run Python code. This means complex workflows, calculations, and integrations happen outside the context window.

Rich Documentation: Unlike basic prompts, skills can include detailed reference materials that load on demand.

Problems Addressed by Skills

Skills aren’t just technically clever. They solve real problems. Here’s what they fix:

Problem A: Slash Command Fatigue
With slash commands, you have to remember what /analyse-code does and when to use it. You’re responsible for invoking the correct command at the right time. Scale this to 10 commands, 20 commands, and you’re perpetually context-switching.

Skills flip this: Claude automatically invokes them when it recognises a matching task. You don’t need to remember anything.

Problem B: MCP Tool Memory Bloat
MCPs are powerful, but they have a hidden cost. Every tool description from every installed MCP loads into context all the time. If you have 5 MCPs with 10 tools each, that’s 50 tool descriptions constantly taking up tokens, whether you need them or not. This bloats your context window and drives up costs.

Skills solve this through progressive loading. Only the brief front matter description loads initially. The full skill content loads on demand.

Problem C: CLAUDE.md Context Overload
CLAUDE.md is great for repo-wide rules, but it breaks down at scale. You might have instructions for “How to bootstrap a Python project” that load every single time you interact with Claude, even when you’re writing Golang. In monorepos, different areas need different rules, but CLAUDE.md can’t distinguish between them. It gets bloated quickly.

Skills let you apply focused rules to specific tasks. Context appears only when necessary.

Problem D: Prompt Distribution Challenge
How do you share a working prompt, workflow, or skill with your team? There’s no standard way. CLAUDE.md is repo-specific. Slash commands are personal. MCPs require publishing infrastructure.

Skills solve this through Claude Plugins, a Git-based marketplace. Build once, share everywhere.

Quick Comparison: When to Use What

In practice: I use slash commands for very short prompts I remember. Agent Skills are for complex workflows where I can also benefit from code execution and better context management.

Learning to Build Skills

You don’t need to start from scratch. Anthropic built tools to help:

The Skills Creator Skill: Anthropic’s own interactive skill-building tool. Find it at github.com/anthropics/skills. That repo contains instructions for adding the Anthropic marketplace to your Claude Code instance. Once the marketplace is added, install the required plugins, then ask Claude to use the skill-creator skill to create a new skill. Claude will ask for confirmation to load that skill and then guide you through building your first skill interactively.

Deep Dives (I won’t repeat what’s already well-documented):

• Anthropic Engineering Blog: Equipping Agents for the Real World with Agent Skills - The canonical explanation of what skills are and how to design them

• Agent Skills Best Practices - Comprehensive guide to building production-quality skills

Read those for the deep technical guidance. This article introduces the concept.

(Bonus point) Sharing Skills: Marketplaces and Plugins

Once you’ve built a skill, you can share it with the world through Claude Plugins. However, it’s entirely optional.

Remember: as explained earlier, a skill is just a folder. You can copy it from one project to another, from one laptop to another, and it works immediately. No special setup required. If that’s all you need, you’re done. Sharing is as simple as putting your skills folder in a Git repo and sending the link to your team.

If you want broader distribution, Marketplaces and Plugins provide a discoverable way to share:

A Marketplace is just a public GitHub repository with a .claude-plugin folder containing a marketplace.json file. That’s it.

See an example here: github.com/anthropics/skills/.claude-plugin/marketplace.json

A Plugin is a container for one or more Claude Agent Skills. Group related skills together, publish the whole plugin, and others can discover it and add it to their Claude environment.

The key requirement: your GitHub repo must be public so others can add your marketplace.

Again, this is entirely optional. Most of the time, copying the skill folder is enough.

Why This Matters (And Why I’m Excited)

I’ve been using Claude Agent Skills since they launched in mid-October, and they’ve been a genuine game changer.

They’ve fundamentally changed how I extend Claude’s capabilities. Instead of fighting with bloated prompts, I build focused skills with progressive context loading. Maintenance is easier. Sharing is easier. The agent is smarter about when to use what.

This is the first article about skills because I’ll reference them often in future articles. We’ll explore real-world use cases, including system design, code review, documentation generators, and more. This foundation matters: understanding why skills exist and how they work differently than everything else.

If you’re building AI agents or extending Claude for your team, invest time in learning Claude Agent skills. They’re worth it.

Get in Touch

If you enjoy this content, please repost it, share it online, and give it a like. Substack is a place for everyone to learn from each other, so please let me know about your experience with Claude Skills, or share a link to an article about Claude Skills that you found helpful. I’ll make sure to read it.

Since I’ve already been using extensively Claude Agent Skills both at work and in my personal life, if you need some help with Claude Agent Skills, please feel free to reach out to me both on Substack or on my Mentorcruise profile page at https://mentors.to/gsantoro where I provide paid mentorship to anyone interested in learning about AI, system design, cloud-native concepts, or anything that’s been discussed in this newsletter.

Since this is a free newsletter, all the content is accessible to everyone.

Mentorship is how I keep this newsletter up and running.

References

• Anthropic Skills Marketplace - Open repository with Anthropic’s official skills and the Skills Creator tool

• Equipping Agents for the Real World with Agent Skills - Anthropic Engineering blog explaining skill architecture and design patterns

• Agent Skills Best Practices - Official documentation on building production-quality skills

• Claude Code Documentation - Claude’s official CLI tool for working with skills and agents

• Model Context Protocol (MCP) - Alternative approach to extending AI capabilities with tools

GitHub Actions are typically referenced by version tags (e.g., actions/checkout@v5), but version tags can be moved, force-pushed, or accidentally updated. For better reproducibility and security, you can pin actions to specific version SHAs (immutable references that never change).

This is the approach that tools like Renovate (and Dependabot) use behind the scenes to keep your dependencies secure and consistent. If you’re managing GitHub Actions dependencies at scale, Renovate remains the preferred way to automate version updates and SHA pinning. But understanding where these version SHAs come from and how they work is valuable in several ways:

• Debug Renovate behaviour: If you’re wondering why Renovate is pinning to a specific SHA, or if it’s not updating when you expect, understanding the mechanics helps you troubleshoot.

• Implement the logic yourself: Sometimes you need custom logic for dependency management. Knowing how SHAs relate to version tags lets you build your own tooling.

• Understand the fundamentals: Version pinning is a core security and reproducibility practice. Understanding the why behind it makes you a more thoughtful infrastructure engineer.

Note: SHA pinning isn’t unique to GitHub Actions. The same principle appears everywhere in the infrastructure and dependency management world. Most programming languages use SHA hashes in lock files (npm, pip, cargo, etc.) to pin transitive dependencies. Docker images are pinned to SHAs for the same reasons (security and reproducibility). Understanding this pattern at the GitHub Actions level helps you recognise and apply it across your entire stack.

In the Quiet: Where I’ve Been

If you’ve been wondering why I went quiet back in February, I owe you an explanation.

Earlier this year, I left Elastic after three great years. I needed time to step back and recharge. I spent five months travelling, reconnecting with family in Italy, and just absorbing things at a slower pace. It gave me space to think clearly about what came next.

In July, I joined Loveholidays as a Senior Platform Engineer. Loveholidays is a UK-based travel booking platform that helps customers find and book holidays across Europe. The role was a significant shift: much heavier on the DevOps and infrastructure side. I had to quickly absorb Terraform, Grafana, Google Cloud, and a whole ecosystem of tools I’d never worked with deeply before. Those first months weren’t about writing; they were about listening, learning, and taking in everything I could. I was quietly observing patterns, asking questions, and building foundational knowledge.

There’s also the reality of life outside of work: I’m raising a two-year-old who’s in full toddler mode. Most weekends are spoken for, and writing demands the kind of mental space that wasn’t available.

But here we are at four months in, and something’s shifted. The fog is lifting. I’ve absorbed enough to start seeing clearly, and more importantly, I’ve been quietly accumulating ideas: real, concrete things I’ve learned about infrastructure, platform engineering, and cloud-native practices that I genuinely want to share.

So here I am, breaking the quiet. I’m back, and I have things to say.

Why Pin to SHAs?

Version tags are convenient, but they’re mutable. A maintainer can re-release a tag, move it to a different commit, or even force-push to it. This means your workflow could run different code tomorrow than it did today without you changing anything.

Here’s the key insight: a version tag like v5.0.1 is just a pointer to a specific commit. When you pin to the commit SHA that tag currently points to, you’re locking in that exact version. If the tag ever gets moved or re-released to point to a different commit, your workflow stays pinned to the original commit. You’re insulated from any future changes to that tag.

Commit SHAs are different. A 40-character SHA like 93cb6efe18208431cddfb8368fd83d5badbf9bfd points to one specific commit, forever. It can never change, never be re-released, never be overwritten.

For production workflows and security-sensitive operations, this immutability is invaluable.

How to Find the SHA

Prerequisites

You’ll need:

• GitHub CLI installed:

https://cli.github.com/

• Authenticated with GitHub: Run gh auth login

The Command

For any GitHub Action release, use the GitHub CLI to get the commit SHA that the version tag currently points to:

gh api repos/actions/checkout/commits/v5.0.1 --jq ‘.sha’

Replace actions/checkout and v5.0.1 with your action and desired version. See the GitHub Actions documentation for more details on security best practices.

Result: You’ll get back a 40-character commit SHA like 93cb6efe18208431cddfb8368fd83d5badbf9bfd. This is the commit that the v5.0.1 tag is currently pointing to: your immutable lock.

That’s it. One command, one immutable reference.

Using SHAs in Your Workflow

Replace the version tag with the full commit SHA:

# Before: Version tag (can change)
- uses: actions/checkout@v5

# After: Immutable commit SHA with version comment for readability
- uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5.0.1

It’s common practice to include a comment with the version tag (e.g., # v5.0.1) next to the SHA. This gives human readers a quick reference to which version you’re pinned to, while the SHA itself provides the immutable guarantee. The comment doesn’t affect functionality, it’s purely for documentation and clarity in code review.

The behaviour is identical to the tag, but now you have a guarantee that the exact same code will run every single time.

The Benefits

Immutability
SHAs never move. Version tags can be re-released, updated, or accidentally modified. A SHA pin locks you to a specific point in git history forever.

Security
You can explicitly control which version of an action runs in your workflow. No surprises, no unexpected updates that could introduce bugs or vulnerabilities.

Auditability
When you audit your workflows, you know exactly which version of code ran. The 40-character SHA serves as an immutable audit trail.

Reproducibility
Your workflow behaves identically across runs and across time. This is critical for debugging, compliance, and understanding system behaviour.

When to Use This

Use commit SHAs for:

• Production workflows where stability is critical and unexpected updates could break deployments

• Security-sensitive operations where you need complete control over dependency versions

• Compliance requirements that demand audit trails and version pinning

For development workflows, version tags may be convenient enough. But for production, SHAs provide the security and predictability you need.

Ready to pin your actions? Start with your most critical workflows and expand from there. The small effort of pinning to SHAs pays dividends in reproducibility and peace of mind.

References

• GitHub Actions Documentation - Official guide to GitHub Actions workflows, syntax, and capabilities.

• GitHub Actions Security Hardening Guide - Best practices for securing GitHub Actions workflows, including SHA pinning recommendations.

• GitHub CLI - Command-line tool for interacting with GitHub repositories and retrieving commit SHAs.

• Renovate Documentation - Comprehensive guide to automating dependency updates and SHA pinning across your repositories.

• GitHub Dependabot Documentation - GitHub’s native tool for dependency updates and security vulnerability alerts.

• Container Image Digests - Guide to pinning Docker images using immutable digest SHAs.

Welcome to Cloud Native Engineer, where we explore the tools, patterns, and practices that power modern cloud-native systems.

Who This Is For

This newsletter is for software engineers, platform engineers, and DevOps practitioners who want to:

• Master Kubernetes and container orchestration

• Design scalable systems that can grow with your users

• Implement observability to understand what’s happening in production

• Discover modern tools and DevOps practices that make your life easier

Whether you’re preparing for the Certified Kubernetes Administrator (CKA) exam, designing your first microservices architecture, or optimizing your deployment pipeline, you’ll find practical, hands-on content here.

About Me

I’m Giuseppe Santoro, a Platform Engineer at Loveholidays based in London, with over 12 years of experience in cloud infrastructure and platform engineering. I’ve worked with companies like Elastic and PlayStation, specializing in high-scale distributed systems, Kubernetes, and cloud-native technologies.

I hold a Certified Kubernetes Administrator (CKA) credential and focus on system design, DevOps practices, and making complex topics accessible.

My mission is to guide engineers through modern system design and cloud-native engineering with hands-on technical expertise and real-world industry insights.

Through this newsletter, I share lessons learned, tools I’ve discovered, and techniques that have made me more productive as an engineer.

Get In Touch

I offer one-on-one mentoring on system design, Kubernetes, and cloud-native architecture. Whether you’re preparing for system design interviews, building your startup’s infrastructure, or looking to optimize your cloud deployment, I’m here to help guide you through the challenges.

Book a mentoring session →

Top Performing Articles (as 2025-11-30)

These are the most-read articles across the entire newsletter, handpicked based on views and reader engagement:

1. Taskfile: a modern alternative to Makefile (DevOps & Tooling) - 13,404 views
   Modern approach to task automation with Taskfile as a user-friendly alternative to Makefile.

2. How to publish a post with the LinkedIn API (DevOps & Tooling) - 3,407 views
   Automate posting on LinkedIn by using their API. Learn about OAuth2.0 access tokens and Open Graph meta tags.

3. How to deploy Fleet and Elastic Agent on Elastic Cloud Kubernetes (Observability) - 1,505 views
   Deploy Fleet and Elastic Agent on ECK for centralized management and access to 400+ Elastic integrations.

4. Dev environment with Elastic Cloud on Kubernetes (ECK) (Observability) - 1,490 views
   Set up a scalable development environment for Elasticsearch using Elastic Cloud on Kubernetes.

5. Powerful Load Balancing Strategies: Kubernetes Gateway API (System Design) - 1,293 views
   From ClusterIP to Ingress and Gateway API. Discover the most common strategies to load balance services in Kubernetes with practical examples using Traefik and K3d.

6. Maximise Your Productivity: Harness Hot Reloading in Kubernetes (Kubernetes & Containers) - 1,207 views
   Accelerating Kubernetes development workflows with Tilt and hot reloading for near-instantaneous feedback loops.

7. From Zero to K8s Hero: 5 Must-Have Tools for Kubernetes (Kubernetes & Containers) - 1,154 views
   Essential tools for Kubernetes development and operations, from K9s terminal UI to advanced debugging with node-shell.

8. Enhancing Software Design with Diagrams as Code (System Design) - 896 views
   Create version-controlled, automated architecture diagrams using PlantUML, Mermaid, and Diagrams for AWS, Kubernetes, and system design.

Explore by Category

Navigate to content organized by topic:

Kubernetes & Containers

Master container orchestration with practical guides, exam preparation, and advanced Kubernetes patterns. Learn everything from CKA exam strategies to production-ready deployment techniques.

Go to Kubernetes & Containers →

System Design

Learn the principles and practices behind scalable, resilient systems. From understanding HTTP fundamentals to designing architectures that serve millions of users, these articles cover the architectural foundations of modern applications.

Go to System Design →

Observability

Build systems where you can see what’s happening. Learn how to implement effective logging, structured logging, and log collection strategies to understand and troubleshoot your applications and infrastructure.

Go to Observability →

DevOps & Tooling

Discover modern tools, automation practices, and development techniques that make your workflows more efficient. From command-line tools to infrastructure automation, learn how to work smarter.

Go to DevOps & Tooling →

Community & Milestones

Behind-the-scenes stories, wrap-ups, and milestone celebrations. These articles share the journey of building Cloud Native Engineer, lessons learned, and gratitude to the community.

Go to Community & Milestones →

A warm welcome to the 191 new subscribers who joined the Cloud Native Engineer newsletter in the last three weeks.

It's unbelievable how things can change with a single Substack note.

If you are new here and you don't know what I am talking about, feel free to read my story at A Thank You to my 1K Subscribers (And How Being Seen Changed Everything).

Are you a Python developer?

Are you frustrated with your development tools?

Have you ever wished there was a better way to handle dependencies in Python?

I have the answer you have been looking for.

I've been a Python developer for the last 10 years and have never been so excited about programming in Python.

Years of frustration are finally gone.

In this article, I will introduce you to Uv, the latest package manager revolutionising Python tooling.

This article is not meant to be an alternative to the UV documentation.

The documentation by Astral (the company behind UV) is excellent, and repeating it here would be a waste of your time and mine.

I aim to introduce this tool, compare it to the Python ecosystem, and explain how I use it daily.

The outline of this article is as follows:

• The story so far

• UV adopts the industry's best practices

• Why is UV better than other solutions?

• How do I use UV in my development tools?

• Bonus feature: Inline dependencies

I have published a cheatsheet for UV on Gumroad at UV Python Package Manager - The Essential Cheatsheet.

In a previous article, A Thank You to my 1K Subscribers (And How Being Seen Changed Everything), I have described my decision to remove my paywall from my newsletter and go completely free.

So, if after reading this article, you have enjoyed my content, please donate by buying a cheatsheet from Gumroad, donate via Buy me a Coffee or simply like and reshare this article so it can reach a broader audience.

Your donation will help keep this newsletter free for everyone.

Thank you for being so supportive.

The story so far

I have written in a previous article Effortless Python Development with Nix how I have already abandoned pipenv, pyenv and virtualenv in favour of a tool called Poetry.

If you have never heard of any of those, do not worry. You don't have to learn them or even be aware of what they used to do.

Poetry was already a huge step forward in the history of Python development tools.

But there is now a better alternative.

A while ago, I wrote another article Goodbye Dependency Headaches: Discover the Power of Pipx for Python Package Management where I introduced pipx, a great tool that allows you to install each Python tool in an isolated virtual environment to avoid dependency conflicts.

UV combines the best characteristics of Poetry and Pipx in a single tool that is both fast and easy to use.

Speed and usability are usually at the top of my mind when I choose a tool to use.

UV adopts the industry's best practices

From the official Astral website, you can read about UV:

An extremely fast Python package and project manager written in Rust.

🚀 A single tool to replace pip, pip-tools, pipx, poetry, pyenv, twine, virtualenv, and more.
⚡️ 10-100x faster than pip.

If you have ever written any Python code, you are already familiar with virtual environments.

If you haven't, you can learn about it from this great article Python Virtual Environments: A Primer – Real Python. If you read that article, bear in mind that with UV, you will have a much simpler way to create virtual environments than what is suggested there.

Each tool I have used in the past (aka pyenv, virtualenv, pipenv) has its own way of creating virtual environments. Some prefer creating them in a global fixed location; others prefer to make them close to your code.

The approach that UV follows is my favourite one.

UV creates a hidden directory called .venv in the same root folder where you have your Python code.

The benefits of this approach are many.

Keeping code and virtualenv next to each other makes development a lot easier when you have to activate the virtualenv from your shell or if you want to eliminate the entire project and the associated venv.

You will always know where the venv for your project is stored.

For managing dependencies, Uv adopts a lock file approach similar to Poetry.

There is a file called pyproject.toml where you add your dependency rules (greater than this version or lower than that version) and another file, uv.lock, where the dependencies are pinned to fixed versions.

This approach allows the developer to define their high-level dependencies in a human-friendly file and separate it from the file used by the machine to create reproducible environments.

Forget about requirements.txt; using a single file for humans and machines was never a great idea.

Uv is compatible with pip and requirements.txt, but you don't have to use those features. Those are there primarily for backward compatibility.

Finally, Uv adopts the same approach as Pipx for installing tools in isolated virtual environments.

Please have a read at Goodbye Dependency Headaches: Discover the Power of Pipx for Python Package Management for more information about Pipx and installing tools in Python.

Why is UV better than other solutions?

By now, you should have a good idea of how Uv adopts best practices from established tools and combines multiple tools into one (namely Pipx and Poetry, for me).

But what is the fuss about UV? Why is everyone using it?

UV is extremely fast compared to other solutions.

While speed is usually not at the top of your mind when you think about Python development tools, speed is very beneficial in many contexts.

Speed is a great benefit when you have to install dependencies many times a day for all your developers as part of a CI/CD pipeline. You might save lots of money each month in cloud costs by running faster pipelines and countless hours in development time.

Furthermore, UV uses a global caching mechanism that avoids storing the same dependency multiple times on the same machine. This can also have enormous implications for storage savings in a cloud scenario.

There are lots more UV features that make it a great solution.

I have barely scratched the surface of what you can do with UV, but I am already in love.

How do I use UV in my development tools?

If you have read my article Effortless Python Development with Nix, you might wonder by now how I am combining Nix with UV.

I discovered UV in the first place because I recently had an issue at work where Python installed by Nix would break when installing a specific dependency while compiling Python from the source code.

After looking around for a more straightforward setup, I discovered UV can not only be used to install dependencies but also to install Python itself.

Instead of compiling Python from source, as Nix does, Uv uses binaries from Astral for a quicker setup.

What if you combine UV with the Nix dependency manager for non-Python tools?

I have been using UV for anything about Python and Nix for everything else.

Furthermore, direnv integrates with UV to activate the virtual environment.

Instead of running the command source .venv/bin/activate every time I want to run a Python project, I now use the following two lines in my .envrc:

layout python-venv
PATH_add .venv/bin

The second command is particularly useful if Python is installed by Uv or other binaries are installed in your virtualenv. All those binaries will be available in your shell with no extra effort.

Bonus feature: Inline dependencies

There might be a case where you have a single Python file that you want to distribute as a tool to your colleagues.

I'm sure you would like to avoid publishing a package on Pypi or creating a docker image if it's just in the initial development.

UV provides an easier alternative to the previous two options, avoiding writing long instructions on creating virtualenvs and installing dependencies.

You can add inline dependencies to your Python script like this.

uv init --script example.py
uv add --script example.py 'requests<3' 'rich'

The content of your example.py will look like this:

Finally, you can run this script using uv run example.py.

More information on how to use scripts is available at Running scripts | uv.

I only discovered this feature with UV, but it is also available in Pipx.

Happy New Year!!!

I know it's almost the end of January, and not even my first article of the year.

But, while I usually keep my article strictly technical, I wanted to try a different format this week.

First, I want to give a big WELCOME to almost 480 new subscribers who have joined this newsletter since last week.

I still can't believe that this is really happening.

It all started when I announced this paid newsletter was going free... but I'm getting ahead of myself. I will elaborate on the decision to remove any paywall for my content later.

This article will primarily focus on how we got here and what happened last week.

I'm going to post a follow-up article in a couple of days (or a week), where I'll discuss in more detail what the future of this newsletter looks like.

The outline of the article is the following:

• The Story so far: slow growth and mostly talking into the void

• Last week's update: going viral with Substack Notes and boost sessions

• Conclusion: what is the lesson learned

The story so far: slow growth and mostly talking into the void

I never really believed those stories of people reaching thousands of subscribers in a few months. I thought they might have just paid for ads or were just pure lies trying to attract new subscribers.

It didn't make sense to me.

My own experience was very different.

Slow growth and mostly talking into the void.

In my mind, I have always "blamed" my very narrow niche, but this is what I am interested in, and I won't change it for anything in this world.

I always thought I could make it a bit broader to attract a bigger audience, but I firmly believe that I would lose part of who I am in the process. So, no, thank you.

At the same time, when you spend enough time on Substack, you see wild success stories like ByteByteGo Newsletter by Alex Xu and you think, one day that will be me.

Fair enough, he wrote two books, has a YouTube channel, and has an entire team of writers, marketers, and a considerable budget. You can't really think of competing with him. That still won't stop me from trying to get there either.

Anyway, I am diverging from my main focus.

The growth of my newsletter has always been top of my mind, but I have never really done much about it.

As every other creator here, I have read all the tips about Substack growth:

• Writing on Substack Notes daily

• Writing consistently each week

• and so on and on

I always thought that my content would speak for me.

Even if I was never very consistent.

I was so wrong!!!

Consistency alone is nothing if nobody sees your content, no matter how good it is.

Last week's update: going Viral with Substack Notes

During Christmas, I had time to think about where I wanted to get my newsletter and what I was doing wrong.

I wasn't very happy about the current situation, even if I could consider myself lucky with 15 paid subscribers and almost 1000 free subscribers.

The thing about paid subscribers has always been that I felt the extra pressure of delivering very high quality. That meant taking many hours to write an article. Countless hours of research, coding, and testing were needed to get to the point where I really felt like an expert in the field before I could write about it.

My articles usually hit the limit of the email size, and I struggled to cut off bits of it.

Furthermore, while trying different strategies (like only putting a paywall 3 months after an article was published), I always felt that paywalling my content could reduce my engagement and visibility.

As many people have said before me, I was probably already in the top 20% of newsletter creators, given that most people give up after 6 months.

If I still had to do something drastic about growth or otherwise, I would have given up writing entirely.

A drastic change of direction. That's what I needed. I thought I'd do something that not so many people do.

So on January 15th 2025, I decided that I had enough of slow growth. I decided to remove any paywall from my newsletter and go free.

What is the worst that can happen, I thought!!! I could lose all my paid subscribers overnight, but at least I'll get that motivational boost to write shorter articles more often.

I might lose my reader's trust other than just the monthly revenue. But if I don't do something drastic now with 15 paid subscribers, I might never get out of this situation.

This was when I could average 3-7 daily free subscribers, but I struggled to reach 1000 subscribers.

My numbers were going up and down every day during the Christmas break.

So what did I do?

I created a note describing my decision and shared it in the "Wednesday Boost Session" chat by .

Best decision ever!!!

I think that my message really resonated with lots of readers since that Substack note alone has since then generated (in no particular order):

• new free subscribers: 320

• clicks: 2740

• new followers: 1

• Shares: 20

• likes: 229

• comments: 29

• restack: 4

My free subscriber's chart went totally vertical, as you can see below:

My current count of subscribers is:

• free subscribers: 1,476

• paid subscribers: 15 (I haven't lost any yet)

Conclusion

Of course, I had no idea that this would happen.

I had no expectations!!!

The best part of this newly found engagement is that I spent most of the weekend:

• chatting with many fellow creators

• exchanging ideas about content

• learning about their experience

• creating multiple opportunities for further collaboration

• learning about various different ways to monetise your free content

My main realisation was... "That's why they call it social networks; you are supposed to talk to people, not just talk at them". 😂

So, what is the lesson here.

I meant to write this article not for bragging but as my personal life lesson that maybe might inspire new content creators to not give up on their journey.

Quality content is essential, but you will never get a sizeable audience if nobody sees your content.

In the next article, I'll discuss the future of this newsletter and what I plan to offer my paid subscribers to avoid loosing them.

There are many ways to expose HTTP applications running in Kubernetes.

A typical setup involves creating a deployment and an associated service.

The type of Service decides the visibility of your application.

The most common types of Kubernetes Services are:

• ClusterIP: default value (if no type is provided) that only exposes an application internally to other services in the same Kubernetes cluster.

• NodePort: primarily used for testing to expose a service externally at every cluster node at a chosen port.

• LoadBalancer: used for production use cases in the cloud to expose the service outside your cluster.

This is it, right?

That's what I thought for a while since that is what I learned when I took the Certified Kubernetes Admin (aka CKA) exam more than two years ago.

In this article, we will briefly introduce CluterIP and LoadBalancer Service types and discuss the alternatives of Ingress and Gateway API.

We will briefly discuss each option and its pros and cons and provide some code samples to get you started.

This article has the following sections:

• A small caveat

• ClusterIP: exposing a service internally

• LoadBalancer: exposing a service in the cloud

• Ingress: a single gateway for all services

• Gateway API: the modern alternative to Ingress

Want to connect?

👉 Follow me on LinkedIn and Twitter.

If you need 1-1 mentoring sessions, please check my Mentorcruise profile.

A small caveat

The code samples in this article are working examples, even if just a toy application.

They are kept as simple as possible since there is a lot to cover, and we only want to focus on the load-balancing strategies.

We will use a container image called traefik/whoami that returns the content of an HTTP request with all its headers and parameters as output.

This container image is used by the Reverse Proxy Traefik in their official documentation. Since Traefik is my Reverse Proxy of choice (since it comes as default in K3d), this is what we are going to use in this article.

Even if those code samples have been tested with K3d and Traefik, they should be working with any Kubernetes distribution or any Reverse Proxy.

More info on setting up those code samples is not part of this article. I'll write about it in the future.

This article doesn't mean to be an exhaustive tutorial to load balancers in Kubernetes. There would be too much to cover.

I would like to introduce you to the concepts so that you know those things exist in the first place.

I'll provide some external resources to keep the learning going.

ClusterIP: exposing a service internally

The bare minimum to create and expose an HTTP application is to create a Kubernetes Deployment and an associated service.

Throughout the article, we will use the same Kubernetes Deployment, the code provided in the following image.

As discussed above, this is just a toy application that is useful for debugging HTTP network routing.

We will see what the output of this application looks like later in the section for Ingress.

Aside from the boilerplate code for a deployment, you should notice:

• we are creating three replicas of the same pod. This is so that when running the client, you will be returned a different internal IP for each pod responding.

• The container port used by the application is configured by the environment variable PORT and then referenced by the containerPort: 80

• We will use the Kubernetes label label: my-app for deployment, service, and pods.

Once you have a deployment, you can create an associated service with no type.

By default the type of the Service is ClusterIP, and the Service will only be accessible from other services and pods inside the same Kubernetes cluster.

There is not much to notice from this Service except that it exposes an HTTP application at port 80.

As per the above deployment, the Service above will be used (mostly as it is) in the following chapters.

LoadBalancer: exposing a service in the cloud

As discussed at the beginning, if you want to expose a service externally to the internet for a production use case, you need to create a Service of type LoadBalancer.

The cloud provider where you are running your Kubernetes cluster will provide a Load balancer associated with this Service.

A different load balancer will be created for each Service you want to expose.

As you can notice, the only difference from the Service in the previous section is the type of Service in spec.type = LoadBalancer.

Pros and Cons

While this solution is straightforward, it starts to be a problem when you have many microservices and don't want to create a single Load balancer for each of those external services.

While this solution is perfect when running in the cloud, it doesn't work when running your Kubernetes cluster locally on your machine, for example, with K3d.

Suppose you, like me, firmly believe that a local Kubernetes cluster should be able to allow you to replicate a production use case. In that case, you are out of luck here.

Alternatively, you could use a Service Mesh provider like Istio. Still, for simple use cases, there is a much simpler alternative.

Ingress: a single gateway for all services

A while ago, Kubernetes decided that a good alternative to a single Load Balancer for each Service was to create a resource called Ingress that would allow Reverse Proxies like Traefik or Nginx to create complex routing rules.

This Ingress resource points to the service definition that we defined in the section ClusterIP: exposing a service internally and the related deployment. For brevity, the code is not duplicated here.

To reach the Service from your laptop, you will have to run the following curl command:

curl -H "Host: my-app.example" http://localhost:8080

As you might have noticed, my-app.example is the hostname defined in the Ingress resource at spec.rules.host.

As a reference, the output of the curl command is the following.

The example provided in this section is officially called Host-Based Routing (Virtual Hosting) and it is only one of the many use cases for Ingress:

• Path-Based Routing

• SSL/TLS Termination

• Load Balancing and Traffic Management

• Authentication and Access Control

Pros and cons of using Ingress API

Between the pros of using Ingress:

• Well-established API supported by all major Reverse Proxies (e.g. Traefik, Nginx)

• Works on a local Kubernetes cluster with K3d as well as in the cloud

Between the cons:

• Only support L7 Load balancing. There is no support for L4 protocols like TCP and UDP

• From the L7 network stack it only supports HTTP/S. It doesn't support GRPC. This might be inconvenient if you want to support microservices.

• Ingress is not a standard API between all Reverse Proxies. There might be some differences in the spec and ways to implement some routing strategies.

Gateway API: the modern alternative to Ingress

The problem with Ingress is that most Reverse Proxies have their way of implementing those routing rules.

Each one of the Reverse Proxies would require the user to provide either a custom Kubernetes annotation, a custom middleware extension or some custom Kubernetes Custom Resources (CRDs).

This was the case until a new Kubernetes API was introduced called Gateway API.

As you can see from above, instead of an Ingress you can replicate the same outcome with a HTTPRoute.

In this scenario, you will be able to hit the backend service with a command like:

curl -H "Host: load-balance.example" http://localhost:8080

Pros and cons of Gateway API

Between the pros:

• Similar concept to Ingress. A single network endpoint for all services instead of a single load balancer per Service.

• Support both L4 and L7 network protocols (eg. UDP, TCP, HTTP/S and GRPC).

• Standard API across multiple Reverse Proxies (eg. Traefik, Nginx, etc).

Between the cons:

• it is a much newer API, so not all features are implemented by all Reverse Proxies. As an example, for the complete list of features supported by the latest version of Traefik v3.2.2 visit gateway-api/conformance/reports/v1.2.1/traefik-traefik/experimental-v3.2.2-default-report.yaml at main · kubernetes-sigs/gateway-api.

Resources

• Getting started with Kubernetes Gateway API and Traefik - Blog - Traefik Labs Community Forum

• Gateway API vs Ingress: The Future of Kubernetes Networking | Kong Inc.

In computer science, it is common to see programming languages emerge out of the blue every 5-10 years.

Every new language promises to improve your life for the better.

Keeping up to date as a software engineer is a challenging life.

I'm sure that engineers are still making a living by still programming in COBOL. For the rest of us, if you still want to be relevant and change jobs every 4-6 years, you need to learn the latest flavour of programming language.

However, if there is a single language that has yet to go out of fashion for over 50 years, that is SQL.

Over the years, many different flavours of databases and various SQL dialects have emerged, but at its core, SQL has stayed the same since it was created.

So, in my opinion, learning SQL is an excellent long-term investment for your career.

What if I tell you that you could use SQL to query the status of your cloud infrastructure?

Today, I would like to introduce you to Steampipe, a tool I recently added to my arsenal of DevOps power tools.

Steampipe lets you query your cloud infrastructure and many other APIs (Twitter, GitHub, Kubernetes, to name a few) via a simple SQL interface.

This article has the following sections:

• What is Steampipe?

• SQL to query everything: OSQuery and Cloudquery

• How to query Hackernews via SQL

Want to connect?

👉 Follow me on LinkedIn and Twitter.

If you need 1-1 mentoring sessions, please check my Mentorcruise profile.

What is Steampipe?

Steampipe is an innovative open-source project that allows users to query cloud services and APIs using SQL.

Between the main features of Steampipe, we can find:

• Wide Range of Plugins: Steampipe has an extensive library of plugins that support popular cloud services and data sources. Currently, you can count more than 140 plugins.

• Zero-ETL Data Access: Steampipe provides Zero-ETL data access, allowing you to fetch data directly from cloud APIs without complex integrations.

• Open-Source: Steampipe is an open-source tool that ensures data security and integrity and is free for developers.

Notable Steampipe integrations are the following:

• all major cloud providers: AWS, GCP, Azure

• Kubernetes

• OpenAI

• GitHub

You can use Steampipe in different ways:

• Default CLI: a solution with its own PostgreSQL database and plugin management.

• Postgres FDWs: native Postgres Foreign Data Wrappers that translate APIs to foreign tables.

• SQLite extensions: virtual tables that translate your queries into API calls.

• Export CLIs: each exporter is a stand-alone binary that extracts data from Steampipe without a database.

Read more at: https://steampipe.io.

OSQuery: SQL to query your OS

Steampipe is not the only project that provides an SQL interface for APIs.

Osquery provides a SQL interface to query OS resources like processes, open network connections and much more.

Its website describes the project as:

osquery exposes an operating system as a high-performance relational database. With osquery, SQL tables represent abstract concepts such as running processes, loaded kernel modules, open network connections, browser plugins, hardware events or file hashes.

An example query with Osquery is as follows:

With the above query, you can list the open ports on your machine at localhost.

How to query Hackernews via SQL

Among the many integrations provided by Steampipe, Hackernews is an interesting one.

I chose this integration for my demo since Hackernews' API doesn't require an API token, which makes it easy to query.

You only need to install the Hackernews plugin via the command steampipe plugin install hackernews and then use Steampipe binary to query that API via steampipe query.

Each plugin provides multiple tables. You can find the full list of tables at Tables in Hacker news.

In this example, you can use the table hackernews_top to query the top stories for today sorted by their score.

I made a recording of a live session with Steampipe at https://asciinema.org/a/nnBWKsoQLpd0qFvhdkRaSjg1Y.

The same recording is available below as an animated GIF.

In my 20 years of experience with Linux command tools, I always felt frustrated with having to learn so many different text-processing tools like awk, grep, and sed, every time I wanted to parse the output of command line tools like ls, ps or df.

If only those commands were using a structured format like JSON as their output, you would only have to learn a single processing tool like Jq.

I recently discovered Nushell, a modern alternative to the Bash terminal that rewrites some command line tools to export structured data instead of just plain text.

Nushell makes it extremely easy to work with structured data with its human-friendly programming language and modern approach to the terminal.

As you might expect, though, since Nushell is a relatively new project, it only rewrites some of the most common command line tools.

For those tools that are not covered, you either have to hope they expose a --json flag or you can integrate with another command line tool called Jc (to not confuse with Jq, the famous JSON parser) that converts 100s of command line tools outputs to JSON format.

So after years of struggling, I finally got rid of awk, grep, sed, and jq and significantly improved my life.

I can use Jc to convert legacy Linux command output into structured data and then Nushell to process, filter, update or display that structured data with a human-friendly language.

In this tutorial, I want to share my experience and the lessons I learned as a Platform Engineer.

Want to connect?

👉 Follow me on LinkedIn and Twitter.

If you need 1-1 mentoring sessions, please check my Mentorcruise profile.

Problem statement

As a Software Engineer, you interact daily with myriads of Linux command line tools (e.g. ps, grep, dig, ls), each with its output format and list of flags.

Mastering those tools can take years.

Most people, myself included, only master the basics and then spend hours consulting the documentation or StackOverflow whenever they encounter a new use case.

ChatGPT can really help here, especially when you want to create complex scripts. The only caveat is that you are left at the mercy of trusting a machine to do the work for you.

Furthermore, the more tools are necessary in your script, the harder it will be to make your code portable to other machines and the variety of Operating systems available.

Sometimes, when I get frustrated by the limitations of Bash (and my knowledge of the tool), I wish I had started directly from a Python script or another compiled language like Golang when things get complicated.

However, with those languages come other complexities.

There should be an easier way to use those command line tools in a powerful and human-friendly way.

A little bit of personal history

I got my first Linux distribution almost 20 years ago, in my first year of university.

A terminal was a scary place to be coming from a Windows desktop.

So many tools to learn!!

You had to do anything just by typing commands on a black background.

Somehow, I managed to survive university and my first few jobs without being a ninja in text processing tools like grep, awk, sed and so on.

I knew enough to get by but was never fast by any metrics.

For the young people reading this article, this was long before ChatGPT was even a remote idea.

You had to invest a lot of upfront time to learn those text processing tools or countless hours on Stackoverflow asking for help whenever needed.

So, almost 10 years ago, I decided to invest some time into learning those tools and picked up an excellent book called Data Science at the Command Line, 2e. You can now read this book online for free.

While reading this book, I got so frustrated that each command had a completely different output, and making them work together was so tedious. So, while I finished the book, I never retained big chunks of its content.

If only those commands had a standard interoperability format that would make it easier to filter, search, and pass along data from one command to another.

Why has nobody thought about it?

Fast forward to today.

ChatGPT can do a great job at crafting a complex pipeline for you. The only thing that you need to provide is the expected input and expected output.

Except that even ChatGPT makes mistakes, and you don't want to trust it completely.

If only there was a better way.

JC - Convert legacy command line output into JSON

Since JSON is the de-facto standard for machine-readable data, it would be nice if any Linux command had a --json flag to output its results in JSON.

Instead of rewriting tools written in the '70s, JC parses the old and well-establish Linux commands output in a JSON format that can be easily processed by tools like Jq.

Examples

In the example below, we can see how to get the results of pinging an IP address in Json format by using jc -p ping command.

Jc can also be used as a Python library, as seen in the example below.

Documentation

More about JC at:

• jc - official docs

The author of Jc has written extensively about the philosophy behind Jc at:

• Bringing the Unix Philosophy to the 21st Century - Brazil's Blog

• Tips on Adding JSON Output to Your CLI App - Brazil's Blog

Nushell: A modern shell with a friendly programming language

Now that we have most of the Linux commands output in JSON format, what about processing this data?

Suddenly, most data processing tools like awk, sed, and grep are completely irrelevant, and you can just use jq.

But how easy is it to use jq for everything?

It is easy to start but it becomes pretty complex very soon.

Discover Nushell, a modern alternative to Bash conceived for the 21st century.

Rather than thinking of files and data as raw streams of text, Nu looks at each input as something with structure.

In Unix, it's common to pipe between commands to split up a sophisticated command over multiple steps. Nu takes this a step further and builds heavily on the idea of pipelines

Examples

In the example below, you can see how to filter the command ls results by size greater than 10 Megabytes and then sort the results by the modified timestamp.

In this other example, we can see how the output of the command ps to list all the running processes in your machines are filtered and sorted in reverse order by the amount of CPU.

As you can see from the screenshots above, Nushell is quite a powerful language.

Pros and cons

Between the benefits we can list:

• Nushell is a fully-typed scripting language. Similar support to compiled programming languages. Errors (like type inconsistencies) are detected before the script execution.

• Built-in commands: an extensive list of built-in functions to work with Sqlite, Excel, HTTP endpoints and so much more

• Cross-platform support: scripts written in Nushell can run without changes in all major operating systems: Windows, Linux, and Mac.

• Autocomplete: for all commands (even your custom-made ones) and programming language functions.

• Verbose error management.

• Fast: since it's written in Rush.

• Good support from the community: it integrates with lots of other tools.

Negative bits:

• Nushell is not Posix Compliant. Bash scripts won't run unmodified in Nushell. Replaced commands (like ls or ps from above) don't have the same parameters. Nushell prefers to have commands with fewer parameters and uses Pipes to combine multiple commands instead.

Documentation

You can find more information about Nushell at:

• Introduction | Nushell

• Command Reference cheatsheet

• Cookbook | Nushell

Conclusion

I plan to still use Zsh as my favourite Posix-compliant shell to collaborate with other developers at work and progressively move to Nushell for any automation or personal use.

I published my first article on Elastic Cloud on Kubernetes (ECK) almost a year ago at Dev environment with Elastic Cloud on Kubernetes (ECK).

Several of my readers and mentees have asked for an extended article to help them with installing integrations in Elastic.

So here in part 2, I will expand on the Kubernetes manifests to explain setting up Fleet and Elastic Agents.

But first, why do we need Fleet and Elastic agents?

If you want to observe resources that are scattered on different machines, you need to have one Elastic agent for each node. You also need policies configuring which packages to install and many other configs.

Those configs can change over time. Managing this complexity by hand can be overwhelming.

Fleet allows you to centrally manage all your Elastic agents, their configurations, and the installation and upgrading of integrations.

Why should you bother with integrations?

Elastic has almost 400 integrations on its page Elastic integrations.

You can observe anything from the same Kubernetes cluster where Elastic is running to Nginx servers, AWS cloud resources and anything in between.

For example, I wrote an integration for collecting access logs and metrics from Istio.

Installing and configuring Fleet and the Elastic agents is quite a complex setup when done manually but very simple when running on ECK.

In this article, I will present a simple setup that can be used in a dev environment to get started and experiment with Elastic.

Moving to a production cluster needs some improvements to those manifests that can be discovered from the official documentation.

This article is split into the following sections:

• Source code, with links to a GitHub project and other learning resources

• Elasticsearch, where I configure a single Elasticsearch node

• Kibana, where I set Kibana to create the Agent policies used by Fleet and the other Elastic agents

• Fleet and Elastic agents, where I configure the Fleet servers and the other Elastic agents.

Want to connect?

👉 Follow me on LinkedIn and Twitter.

If you need 1-1 mentoring sessions, please check my Mentorcruise profile.

Source code

The complete source code for this project is available at demos/eck.

Bear in mind that I assume that you are already familiar with Takskfile, direenv, devenv, Helm, K3d and Kustomize.

If you have never used Taskfile and you would like to learn it, head to Taskfile: a modern alternative to Makefile and then later to Supercharge Your Automation Workflows with Taskfile: Say Goodbye to Makefile.

If you want to learn how to use DevEnv to install 3rd party dependencies like kubectl, kustomize, helm and so on, head to Effortless Python Development with Nix.

If you have Taskfile, direnv and devenv already installed and configured, you should be able to setup the entire project by just running cd envs/eck; task.

You can find more information at demos/eck/Readme.md.

As I mentioned already in the past, I prefer k3d for setting up a multi-node Kubernetes cluster on my laptop. You might want to learn more about other options at Exploring Kubernetes Dev Environments in 2023.

If you need some extra help getting started with those tools or, in general, with Elastic products, feel free to reach out for some mentoring sessions at Giuseppe Santoro - Kubernetes Mentor on MentorCruise.

Elasticsearch

The Elasticsearch manifest is almost identical to the one from the previous article.

The only thing that changed:

• increased memory and CPU requests and limits to speed up bootstrapping. It used to take up to 10 minutes to bootstrap this setup before I changed those limits. It now takes less than 2 minutes.

• the version of the stack is now parametrised with an environment variable $ELASTIC_STACK_VERSION. This placeholder is replaced with the actual value of the env variable by the tool envsubst before the manifest is applied by kubectl.

Kibana

The manifest for Kibana is now much larger and too big to fit in a single screenshot, so we have split it into 3 parts even if the actual source code is still a single file.

Please refer to the full manifest at Kibana manifest.

Part 1 is the same manifest as in the previous articles. We have only changed the resource limit and parametrised the version similarly to the Elasticsearch manifest.

In part 2, we configure Kibana to talk to the Fleet server and Elasticsearch and we install some required Elastic packages.

Both Elasticsearch and Fleet server are referenced here by the Kubernetes services deployed by the ECK operator.

Part 3 is where most of the changes are. Here, we set the agent policies for the fleet server called eck-fleet-server and the other elastic agents called eck-agent.

As you notice, both of those policies have monitoring enabled to collect logs and metrics from the nodes where they are running.

While the Fleet server will have the fleet_server package installed, the other elastic agents will install the system package.

Fleet and Elastic agent

Fleet server is just a normal elastic agent with some extra packages and configuration.
That's why it uses the same apiVersion and kind of the other Elastic agents.

Some settings to keep in mind for the Fleet server:

• mode: fleet

• fleetServerEnabled: true

As mentioned in the Kibana manifest, Fleet server uses the policy id eck-fleet-server.

Since we only need a single Fleet server, this pod is deployed as a Kubernetes deployment with a single replica.

Also the Fleet server needs to have pointers to Kibana (here called kibanaRef) and Elasticsearch (here called elasticsearchRef) via the respective Kubernetes resource names to correctly communicate with the other components.

Similar to the Fleet server, the Elastic agents use the same apiVersion and kind.

From a configuration perspective, we can notice:

• mode: fleet

As mentioned in the Kibana manifest, Fleet server uses the policy id eck-agent.

The Elastic agents are deployed as a DaemonSet, which means that Kubernetes will deploy a pod per node in the cluster.

In this setup, we have changed the Kubernetes cluster in K3d from the previous article to create a 3-node cluster to see multiple Elastic agents.

Each elastic agent needs to have pointers to Kibana (here called kibanaRef) and to the Fleet server (here called fleetServerRef) via the respective Kubernetes resource names to correctly communicate with the other components.

Finally, each Elastic Agent must have a volume mounted to store logs and metrics. Here, we are just using a locally mounted volume called agent-data created via the Kubernetes config emptyDir.

What is hot reloading?

Developers can use hot reloading to instantly see changes in their code reflected in the running application without needing to restart or rebuild. This speeds up the development process and boosts productivity.

If you have ever programmed in React or any other front-end framework, you might have used hot-reloading before.

Hot reloading is less common in compiled programming languages like Golang, and it is definitively hard to achieve without Tilt when running applications in Kubernetes.

You might be asking:

Why would I want hot-reloading in Kubernetes when I can test my application locally without the extra complexity?

With the advent of cloud-native technologies, micro-services must interact with many other services like databases, APIs, distributed caches, etc.

If you want to develop in such a complex environment, you have a couple of options:

• use Testcontainers in your functional tests.

• use mocks and unit tests. This solution requires quite a large amount of code, and it is not for everyone.

• "develop in production" with hot-reloading. Among the other solutions, this is the only one that doesn't require to write extra code.

Beware, I'm not suggesting you shouldn't write functional or unit tests with mocks.

Hot reloading is exclusively for those instances where you want to test something quickly and don't want to mock the rest of the system architecture to test your changes.

This article will explain how you can achieve hot reloading of a Golang application running in Kubernetes with Tilt.

You can apply the insights you'll gain from this article to any compiled or interpreted programming language.

Also, you can extend this solution to using Kubernetes on any cloud provider, remote Docker registry and more complex applications with lots of different Kubernetes resources.

We have split this article into the following sections:

• K3d and ttl.sh

• What is Tilt?

• A Golang sample application

• Hot reloading with Tilt

  • Golang compilation

  • Docker build

  • Kubernetes apply and port-forwarding

K3d and ttl.sh

This article is primarily about Tilt, but I would like to introduce two other tools that can contribute to achieving a fast feedback loop: K3d and ttl.sh.

K3d is the easiest and fastest way to create a Kubernetes cluster on your machine. It's a more lightweight solution both in terms of CPU and memory consumption when compared to Minikube or Kind since it uses SQLite instead of Etcd to store the state of your Kubernetes cluster.

A lightweight Kubernetes cluster with K3d means I can fit more containers in my laptop memory.

I have already mentioned K3d in a past article Exploring Kubernetes Dev Environments in 2023.

The other tool in this article is ttl.sh, dubbed on their website as an Anonymous & ephemeral Docker image registry.

From their website, we can read:

Anonymous: No login required. Image names provide the initial secrecy for access. Add a UUID to your image name to reduce discoverability.

Ephemeral: Image tags provide the time limit. The default is 24 hours, and the max is 24 hours (valid time tags :5m, :1600s, :4h, :1d)

I prefer ttl.sh to the default Docker registry when developing with Tilt. It avoids the extra complexity of the Docker login, and there is not need to delete intermediate images created by Tilt. Given its ephemeral nature, ttl.sh will delete those images automatically after 24h.

What is Tilt?

A great description of Tilt is provided on their official website.

Tilt automates all the steps from a code change to a new process: watching files, building container images, and bringing your environment up-to-date. Think docker build && kubectl apply or docker-compose up.

You might never hear about Tilt, but Tilt was acquired by Docker in May 2022.

Tilt implements its logic in a file called Tiltfile using syntax from a programming language called Starlark, a simplified version of Python designed for configuration files.

You don't have to be a Python programmer to create a Tiltfile. However, if you want to master more advanced features, it helps if you know Python programming basics.

To learn more about Tilt, check their documentation.

A Golang sample application

I have created a straightforward Golang application to show you how to implement hot reloading in Kubernetes.

Here is the Golang application.

package main

import (
	"os"

	"github.com/gin-gonic/gin"
)

func main() {
	r := gin.Default()

	r.GET("/", func(c *gin.Context) {
		c.JSON(200, gin.H{
			"message": "Hello, World!",
		})
	})

	r.Run(":" + os.Getenv("PORT"))
}

If you want to skip ahead and look at the final solution, check out the GitHub repository at tilt-go-sample-app.

There is much more in that repository, which I am not explaining here. In particular, I use two tools: Taskfile and Devenv (UI for Nix).

If you want to learn more about those tools, make sure to check out the articles below:

Hot reloading with Tilt

Below is the full code snippet of the Tiltfile I used to implement hot reloading in my application:

# -*- mode: Python -*-

load('ext://restart_process', 'docker_build_with_restart')

local_resource(
  'compile',
  'CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -o build/hello cmd/hello.go',
  ignore=['build'],
  deps=['cmd/hello.go', 'go.mod', 'go.sum'])

docker_build_with_restart(
  'ttl.sh/tilt-go-sample-app',
  '.',
  entrypoint=['/app/hello'],
  dockerfile='deploy/docker/Dockerfile',
  only=[
    './build'
  ],
  live_update=[
    sync('./build', '/app')
  ]
)

k8s_yaml('deploy/kubernetes.dev/pod.yaml')

k8s_resource('tilt-go-sample-app', port_forwards=8085,
             resource_deps=['compile'])

When trying to replicate the results in this article, make sure you have the above Tiltfile and all the resources mentioned in this article from tilt-go-sample-app.

Then, you only need to run tilt up from the root folder where you have your Tiltfile.

There are three stages in this Tiltfile:

1. Golang compilation

2. Docker build

3. Kubernetes apply and Port forwarding

Let's analyse each of the stages in detail.

Golang compilation

Here it is the snippet that we are analysing in this section:

local_resource(
  'compile',
  'CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -o build/hello cmd/hello.go',
  ignore=['build'],
  deps=['cmd/hello.go', 'go.mod', 'go.sum'])

To build the Golang application, you must define a local resource and assign it a name and a command to run.

Here, we used the name compile and the command CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -o build/hello cmd/hello.go.

The command is executed by default from the folder where the Tiltfile is located.

We are compiling the Golang code to run on linux/amd64, the architecture we chose for our Docker base image.

Any changes to the file or directory paths specified in the deps array will kick off a rerun of the command. We only have one Golang file cmd/hello.go, and its dependencies are defined in go.mod and go.sum.

Docker build

Before explaining the Tiltfile command to build the Docker image, let's analyse the content of the Dockerfile.

FROM cgr.dev/chainguard/wolfi-base:latest@sha256:3490ac41510e17846b30c9ebfc4a323dfdecbd9a35e7b0e4e745a8f496a18f25

RUN mkdir /app

COPY build/ /app

CMD ["/app/hello"]

Here, we use a distroless container from Chainguard called wolfi-base, a minimal image with very little in it.

We have pinned both the tag and the digest of the base image to fully replicate the build in the future.

The rest is straightforward. We are copying the previous stage's output from the folder build to the path /app. We will use the entrypoint /app/hello to run our application from inside the container.

load('ext://restart_process', 'docker_build_with_restart')

docker_build_with_restart(
  'ttl.sh/tilt-go-sample-app',
  '.', # context = current directory
  entrypoint=['/app/hello'],
  dockerfile='deploy/docker/Dockerfile',
  only=[
    './build'
  ],
  live_update=[
    sync('./build', '/app')
  ]
)

The command docker_build_with_restart is loaded as an extension to the docker_build command. It provides a crucial part of the hot reloading functionality.

When the compilation stage changes something in the build folder (specified in the only parameter), this command will:

1. build a docker image called tilt-go-sample-app using the Dockerfile deploy/docker/Dockerfile with the entrypoint /app/hello.

2. copy that container image to the ttl.sh container registry at the location ttl.sh/tilt-go-sample-app

3. at any subsequent change to build, Tilt will skip creating that container image if it doesn't detect any changes in the Dockerfile. Instead, it will just live update the content of the ./build/ folder to /app/ inside that container. As you can imagine, this logic dramatically speeds up the development since it happens almost instantaneously for such a simple application.

Remember that the container image name ttl.sh/tilt-go-sample-app needs to match the container name in the Kubernetes manifest, explained in the next section.

Kubernetes apply and port-forwarding

Finally, the last stage in this Tiltfile involves deploying our Kubernetes resource and port-forwarding the port 8085 from the container to the developer machine.

The following snippet from the Tiltfile achieves exactly those things.

k8s_yaml('deploy/kubernetes.dev/pod.yaml')

k8s_resource('tilt-go-sample-app', port_forwards=8085,
             resource_deps=['compile'])

Remember that tilt-go-sample-app in the command k8s_resource is not the container name but the name of the Kubernetes resource. I have fallen in this trap quite a few times before figuring out what was wrong with my Tiltfile.

You can see that in the pod.yaml definition at metadata.name

apiVersion: v1
data:
  HOSTNAME: 0.0.0.0
  PORT: "8085"
kind: ConfigMap
metadata:
  name: app
  namespace: default
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: tilt-go-sample-app
  name: tilt-go-sample-app
spec:
  containers:
  - image: ttl.sh/tilt-go-sample-app:latest
    name: tilt-go-sample-app
    envFrom:
    - configMapRef:
        name: app
    ports:
    - containerPort: 8085
  dnsPolicy: ClusterFirst
  restartPolicy: Always

As you can see from above, the image name ttl.sh/tilt-go-sample-app matches the first parameter of docker_build_with_restart from the previous section.

This is the most basic Kubernetes manifest that you could create. Just a pod with a single container and a ConfigMap to pass the port to expose to the application.

As you can recall from the beginning of this article, this application takes an environment variable to configure the port at which the REST application is listening.

Are you ready to take your skills to new heights? 🚀

🚢 Let's embark on this journey together! 

👣 Follow me on LinkedIn and Twitter to receive valuable content on AI, Kubernetes, System Design, Elasticsearch, and much more. 

🎓 If you want personalised guidance, I am here to support you. Book a mentoring session with me at https://mentors.to/gsantoro on MentorCruise, and let's work together to unlock your full potential. 

♻️ Remember, sharing is caring! If this content has helped you, please re-share it with others so they can benefit from it. 

🤩 Let's inspire and empower each other to reach new heights!

This article continues our journey into Observability and logs.

So far, I have only examined logging from an application developer's perspective.

Here, we will analyse the problem from the perspective of a DevOps engineer who wants to aggregate logs from multiple distributed sources into a single log analysis tool, like Elasticsearch.

The main focus here is on "distributed."

If you were analysing a single log file from a local application, you wouldn't even need a "log collection" process. You could use command line tools to tail a log file and grep for what you need. I might consider writing an article on this topic, but it won't be our primary focus.

You can swap the word "distributed" with microservices or "cloud-native applications running on Kubernetes over many nodes".

My over two years of experience in the Cloud Native Observability team at Elastic, the company behind Elasticsearch, has equipped me with the knowledge to guide you through the complexities of log collection in distributed applications.

In this article, we will explain how Kubernetes gathers logs from a container and makes them available to possible log collection agents in an effective manner that mitigates some of the typical challenges.

The insights you'll gain from this article can be easily applied to various solutions, regardless of their relation to Kubernetes or Elastic Observability tools.

The sections of this article are the following:

• Observability and logging

• Log collection in distributed applications

• Pitfalls of log collections

• Log writing in Kubernetes

• Nginx's logs in Kubernetes

• Log collection in Kubernetes

• Node logging agent architecture

• Logging agents

Observability and logging

If you are new to Observability and logging, I have already written several articles on this topic in this newsletter.

In Exploring Structured Logging with Slog in Go, I have already introduced the concept of Observability as a modern version of monitoring.

I have also discussed how it's best to abandon print statements and move to proper Observability with logs, metrics, and traces in Observability 101: A Beginner's Journey Free of Print Statements.

Finally, in Master Observability with Logs: An In-Depth Guide for Beginners, I have debated how logging fits in the picture of Observability, how it compares to other signals (aka Metrics and Traces) and how to get started with logging in Golang.

Log collection in distributed applications

What does log collection in distributed applications mean in this article?

The assumption here is that:

You have a non-trivial amount of logs constantly generated by a distributed application, distributed microservices, or many different applications running on a platform like Kubernetes that you want to analyse in a central location.

Let's unwrap the previous sentence:

• distributed logs are generated in many different machines

• non-trivial amount, more than just a few hundred megabytes.

• continuously, you have a constant stream of data

• central location is the most important of the previous assumptions. Unless you centralise your logs, you won't be able to correlate them to investigate a root cause analysis.

If even one of those assumptions is not valid in your use case, you might not need the solution described here.

If you have, for example, a few hundred-megabyte files as part of a support case, you could use some CLI tools or ingest your logs as a one-off into an SQL database. There is no need for a complex log collection process.

Also, if you didn't need to correlate logs from many sources, you could store each application's logs in the node where the logs are generated. You could SSH into the node with the faulty application and inspect the logs where they leave.

Pitfalls of log collection

Before discussing the clever pattern used by Kubernetes, I will briefly mention some of the problems with log collection.

Filling local disk

We have been writing logs into local files since application logging was a thing.

The biggest problem was implementing log rotation based on time or size to avoid filling the machine disk.

You could keep the last 30 days or 500MB of logs, depending on which limit was hit first.

Log rotation on local files has been a problem that has been solved for many years, but it still requires some logging agent to keep truncating the log files.

Pushing logs to a log sink

It doesn't matter if your application sends logs directly to blob storage, a remote log analyser or any intermediary like a log collection tool.

The problem is still the same.

Your application needs to be "aware" of the log sink.

This means that you might need an application library to deliver log messages to that particular sink or some config management (or maybe just a config file) to store the location of the log sink.

What if the log analyser tool is temporarily not available? You will lose logs or need to implement some in-memory buffer to account for this inevitable situation. A retry logic with an exponential back-off will be advisable in this case.

You might want to change the location of the log sink or the type of sink, requiring maybe a restart of your application.

All the previous requirements affect the application in terms of performance both on CPU (sending the logs) and memory usage (storing them temporarily in a memory buffer).

To avoid blocking your application's main logic (which is definitively not advisable), you will have to delegate sending logs to a parallel async thread so that it becomes an asynchronous operation.

As you can see, sending logs directly from your application to a log sink makes the application unnecessarily complicated.

3rd party apps

To make things worse, you might need to handle log collection from 3rd party software differently from applications where you have control over the source code.

You won't be able to push logs from 3rd party applications, so you will need to collect them first and then move them to the log sink.

Log writing in Kubernetes

There must be a better way to deal with log collection.

Kubernetes has adopted quite a clever trick to avoid some of the pitfalls presented above.

Kubernetes expect your application (or any 3rd party application) to write its logs to the standard output (aka stdout) and standard error (stderr) device files.

If you are not already familiar with what stdout and stderr are from Linux, you might want to understand those topics at Standard streams before moving forward with the rest of the section.

By default, Kubernetes redirects the content of these two stream files for a running container to the same file /var/log/containers/<container_id>.log on the node where the application is running.

At the same time, Kubernetes allows a user to access those same log files with the command:

kubectl logs <pod-name> -n <namespace>

You can even stream the content of log files as they are written with the following command writes it:

kubectl logs -f <pod-name> -n <namespace>

Thanks to a Kubelet, Kubernetes also implements log rotation based on either time or size to keep each container log to a limited size and avoid overfilling the local disk.

Nginx's logs in Kubernetes

What if instead of writing to stdout/stderr, the application that you want to collect logs from writes to one or more local files (like, for example, access.log and error.log in the case of Nginx),

In this case, the container image must be modified to redirect those files to stdout and stderr when running on Kubernetes.

Let's analyse how this is done in Nginx's Dockerfile.

ln -sf /dev/stdout /var/log/nginx/access.log
ln -sf /dev/stderr /var/log/nginx/error.log

As you can see, two symlinks are created to redirect access.log and error.log, respectively, to the stdout and stderr device files.

The above modifications are done so that Kubernetes can treat every application in the same way.

Log collection in Kubernetes

Kubernetes effectively separate the log writing step from the log collection, solving most of the pitfalls of log collection that we analysed above.

The application writes logs to a file synchronously without any performance problems or a need to handle complex retry logic and in-memory buffers; then, logs can be collected and shipped asynchronously by another tool to a log sink.

Log collection can be delegated to any 3rd party tool and handle both your applications (the ones you have control over the source code) in the same way that it handles 3rd party applications.

As you can imagine, the benefits are enormous in terms of performance, application complexity and application size.

Moreover, logs don't have to be streamed at the same speed as they are generated. They can be batched for improved performance.

Any failure handling technique we discussed above still needs to be handled by a log collection tool, but that's not your application problem.

There are different ways in which log collection can happen.

Node logging agent architecture

The most famous cluster-level logging architecture in Kubernetes is called node logging agent.

One pod per node runs separately from your application and has full access to the underlying Kubernetes node where it is running.

This pod needs full read access to the location in the Kubernetes node where container logs are stored (aka /var/log/containers/).

Given that you need one (and only one) log agent per node, this requirement can be easily implemented in Kubernetes as a DaemonSet.

The log collection agent will likely use some form of watermarking to record the last log event correctly delivered on disk. If the node agent fails and is restarted, it can start from where it left off.

Logging agents

Historically, Elastic has created or contributed to many tools to collect logs from different sources and dispatch them to Elasticsearch for storage and analysis.

In order of when those tools were developed:

• Logstash, a server-side data processing pipeline that ingests, transforms, and ships data to various destinations.

• Elastic Beats, lightweight data shippers in Golang that send data from edge devices to Elasticsearch or Logstash.

• Elastic Agent, a single, unified way to add monitoring for logs, metrics, and other types of data to a host. Under the hood, it uses Elastic Beats but also provides extra features for host security.

• Otel log collector is a vendor-agnostic way to receive, process, and export telemetry data. Elastic did not originally write this tool but has contributed heavily in recent years.

The Otel collector is likely the future of log collection, not just at Elastic but also in the Observability field.

OpenTelemetry (OTel) is more broadly an open-source observability framework that provides vendor-agnostic APIs, libraries, agents, and instrumentation to generate, collect, process, and export telemetry data (traces, metrics, and logs) for analysis in observability backends.

In this article, I'll introduce 5 must-have tools to start your journey with Kubernetes.

The first three tools in this list are absolutely indispensable for beginners. The last two make you stand out as a beginner and look like an advanced user.

I'll provide my point of view on what tools can give you an edge over your colleagues and which ones are not worth learning.

What qualifies me to give you such suggestions?

I have spent countless hours in the last few years trying all the Kubernetes tools I could get my hands on, either during my working hours or during my free time for my side projects.

I have to admit it. I am a tools nerd. I love to play with all the shiny tools, and the Kubernetes ecosystem has plenty of them.

Given that new tools pop up every day, like mushrooms, I have always thought I needed advice on which ones are worth learning and which are not worth my time.

So, I tried as many as possible, and after years of experience, I am now ready to mentor others.

In this article, I want to give the guidance I missed when I first picked up Kubernetes years ago.

So please don't waste any more time, and let's dive deep into this article.

1. Browse your Kubernetes cluster: K9s

My favourite Kubernetes tool is K9s, a terminal-based UI that lets you interact with your Kubernetes cluster.

If you, like me, have been raised with "Bread and... Linux" " you will appreciate the beauty of using CLI tools that are both open source and highly customisable with Skins, plugins, command aliases and custom key bindings.

What are the benefits of using K9s as your primary tool for browsing your cluster:

• Support for all standard Kubernetes objects: view and interact with pods, containers, services, RBAC, volumes, events, etc.

• CRD Support: Support interacting with Custom Resource Definitions (CRD).

• Runs everywhere: Runs on any OS and can be installed with most package managers.

• Skins: Change the look and feel but also the behaviour of the UI based on the cluster and context you are working on. Between the advanced features, you can also make a read-only context, preventing any involuntary modification in your production clusters.

• Plugins: I personally played with this feature a while ago when I wanted to parse some JSON logs from a Kubernetes pod and display them in tabular form. I created a Kubectl plugin in Python that I called rich-json-logs and then plugged it into K9s. Almost straightforward.

• Command aliases: Command aliases are less powerful than plugins but can save some keystrokes.

• Custom key bindings: With one or two keystrokes, you can either view a specific type of Kubernetes resource, call a plugin, a command alias or many other features.

What are the alternatives:

• Lens: a paid alternative with a generous free personal plan. This desktop application doesn't offer more than K9s unless you have a quite expensive Pro or an Enterprise license.

• OpenLens: an open-source version of Lens. Since Mirantis, the company behind Lens, has moved away from open-source, this project won't get any more updates.

Share Cloud Native Engineer

2. Automate everything: Kubectl

If you have ever worked with Kubernetes, you will have already encountered Kubectl, the Kubernetes command line tool.

Why did I only put it second on my list?

This tool can do anything you need with Kubernetes, but it doesn't provide such a nice user experience as K9s.

When working with Kubernetes, especially if you are a DevOps and in the middle of an incident, you must move fast!

Given that I can't remember the syntax of the Kubectl commands and that each command involves a lot of typing, K9s is a much superior tool for my needs.

Why do I still suggest Kubectl?

Whenever I need to write some automation that interacts with Kubernetes, Kubectl is my tool of choice.

The best part is that you don't have to choose between these two.

You can use K9s for the speed and UX and Kubectl for its extensibility and make them work together.

Look at the command below. Without using any other tool, I'll get the version label of all pods with the label app=cassandra.

kubectl get pods --selector=app=cassandra -o jsonpath='{.items[*].metadata.labels.version}'

I took this last command from Kubernetes quick reference, a great resource if you ever want to Prepare for your Certified Kubernetes Administrator exam since you can use it at the exam.

What are the alternatives:

• K9s: we have already discussed this in the previous section.

• Kubernetes client libraries: If you need to automate interacting with your Kubernetes cluster in code and you want to avoid using Kubectl in bash, you can use the Kubernetes SDK in your language of choice.

3. Package manager: Krew

The third tool on my list is Krew, the official package manager of Kubectl.

I recently learned about this tool, but it is an excellent addition to my arsenal.

Like any package manager, there is not much to say about it other than it does its job well and is supported by all operating systems.

The tool's best part is the documentation with the list of official plugins that Krew supports.

If you go through that list, you might discover plugins you did not know existed.

4. Aggregate logs from multiple Kubernetes resources: Stern

Stern is the first in this list of tools geared towards more advanced users.

If you have looked at pod logs in Kubernetes, you may have encountered a use case when you wanted to combine logs from various pods or containers within a pod in the same output.

Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod. Each result is color-coded for quicker debugging.

5. Look under the hood: node-shell

Node-shell allows you to start a shell to access the underlying OS of a Kubernetes node.

This tool has been invaluable when developing observability tools for Kubernetes.

As I explain in my in-depth article Kubernetes: node-shell, more than a few times, I had to look at a Kubernetes node in order to tail pods/container logs that are written directly to the node host to feed into Elasticsearch.

I couldn't have done it without node-shell.

Node-shell, like many other tools in this list, can be installed via Krew and integrated into K9s.

Conclusion

The list of tools in this article is definitively incomplete.

I could have mentioned:

• Helm

• Kustomize

• K3d

• Kind

And many more.

Most of them are a bit more advanced and deserve an article on their own.

I might cover some of them in the future.

🎯 In case you missed

🗣️ Shoutout

• FinOps Weekly by my friend Victor Garcia.

In his own words

Optimize your Cloud Costs! Become a Master of understanding Billing and Optimizing Workloads. Join the FinOps Weekly and get the latest FinOps news every Sunday.

Are you ready to take your skills to new heights? 🚀

🚢 Let's embark on this journey together! 

👣 Follow me on LinkedIn and Twitter to receive valuable content on AI, Kubernetes, System Design, Elasticsearch, and much more. 

🎓 If you want personalised guidance, I am here to support you. Book a mentoring session with me at https://mentors.to/gsantoro on MentorCruise, and let's work together to unlock your full potential. 

♻️ Remember, sharing is caring! If this content has helped you, please re-share it with others so they can benefit from it. 

🤩 Let's inspire and empower each other to reach new heights!

This is the second article I wrote about the basics of observability for beginners.

Previously, I have discussed how it's best to abandon print statements, or whatever primitive your programming language supports for writing to standard output, and move to proper observability with logs, metrics, and traces.

To learn more about this topic, head to Observability 101: A Beginner's Journey Free of Print Statements.

I've also already introduced the concept of observability as a modern version of monitoring at Exploring Structured Logging with slog in Go.

Here is an excerpt to refresh your memory

You can think of observability like the 2020s version of monitoring, where the sheer amount of logs and metrics introduced with cloud-native distributed applications pose enormous scaling challenges on what you can collect, how much you can store, and what is just noise and can be discarded. These new challenges made us rethink the tools we use, introduce new ones or adapt the old ones.

In this article, I would like to dive deeper into one of the three pillars of Observability: Logs.

Why start with Logs?

Logs are the most primitive form of observability.

You have already implemented some basic form of logging in your very first application and never called it observability.

In this article, you will discover the basics of logging, best practices and code snippets in Golang from the real-world knowledge I have collected during my career.

This article has the following sections:

• Logs and Observability

• Logs vs Metrics

• Logs vs Traces

• Logs vs Print statements

• Unstructured Logs

• Structured logs

• Log levels

• Log context

• Conclusion

What is out of scope:

• Metrics and Traces

• In-depth discussion about the benefits of observability

• Log collection

• Log analysis

Want to connect?

👉 Follow me on LinkedIn and Twitter.

If you need 1-1 mentoring sessions, please check my Mentorcruise profile.

Logs and Observability

The challenges introduced by cloud-native applications and modern software concepts like observability have changed how we write and collect logs in 2024 compared to 10 years ago.

The biggest challenge with logging is to maximise observability by increasing the log context size and the amount of logs produced while at the same time keeping the costs down.

Since the cost is directly proportional to the amount of text produced, you only want to log the relevant events that might help a developer investigate an error.

But how do you know what is relevant and what is not?

As with anything in computer science, there is a trade-off between storage cost and increasing observability.

In recent years, since storage is getting cheaper with time, most recent optimisations on logging have focused on reducing the amount of processing required to extract useful contextual information from logs more than the storage cost.

Hence, there is a shift from unstructured logs that need to be parsed and processed to machine-readable logs in JSON format ready to be stored.

Given the vast amount of logs that modern applications produce, logs are no longer something a software engineer can eyeball in real time to understand what the software is doing.

The recent trend is to use human-friendly colour-coded logs in console output for the development phase and switch to machine-readable logs in JSON format for production.

This requires that the output format might change depending on the environment the application runs in.

Changing the default log level between different environments is also common.

Those challenges require a shift from writing logs with print statements to a logging library that allows customising the behaviour according to the environment in which the application is running.

In the following sections, we will see practical implementations of the benefits of using a logging library over print statements by using code snippets in Golang.

Logs vs Metrics

How do Logs compare to the other pillars/signals of observability?

Metrics are collected at regular intervals (like with polling) even if the metric value hasn't changed from the last measurement.

Logs, instead, can happen at any time (similar to event-based systems).

From an observability point of view, log events (like for example errors) might happen less frequently than metrics measurements.

Unless you sample the logs, or logs go missing by mistake, no other event happens between two log events.

With metrics instead, you have no visibility of what happened between two consecutive metric collections.

You need to set the metrics frequency high enough not to miss important information but low enough to avoid high storage costs.

Metrics are generally more compact than logs.

Metrics have a predictable cost since if you know the size of a metric and its frequency, you can easily calculate the storage requirements in a given time frame. A small caveat is that the metric storage cost can get out of hand if you have metric labels that might cause a metric cardinality explosion.

More on this topic in a future article about metrics.

Logs at the right logging level and verbosity can be a lot cheaper than metrics since they are event-based, but log frequency is a lot less predictable.

You might have spikes in log cost exactly when you need it the most, meaning when errors occur.

The total cost of logging depends a lot on the verbosity of each log entry, the configured log level, and the frequency of events at that log level.

Logs vs Traces

Traces can be considered an enhanced version of logs with a specific fixed structure.

Traces are mainly used to track the duration of operations and how operations are nested within each other.

Distributed traces are an extension of such logs where you can connect a trace from an application running on a node with another application running on another. This way, you can have a more complete picture of what is involved in a user transaction in a microservice architecture.

Logs vs Print Statements

Logging has been used in history for many purposes, from a simple form of debugging to a primitive form of tracing (like getting the duration of a piece of code) to error reporting or even exposing system or application metrics.

Some common uses of logs are:

• application logs, a log entry is created at every event in the application

• audit logs, capture events by recording who performed an activity, what activity and how the system responded (e.g. exit code or response code)

• infrastructure log, record events that happen to the infrastructure (e.g. lambda invocations on AWS cloud or a pod starting in Kubernetes)

As I already discussed in a previous article Observability 101: A Beginner's Journey Free of Print Statements, you should abandon print statements in favour of one or more of the three pillars of observability.

In the following sections, I want to discuss in a bit more detail the benefits of a logging library over just using Print Statements with some real-world code snippets in Golang.

We will use code snippets that use a standard library called Slog, which I previously covered in my article Exploring Structured Logging with slog in Go.

Unstructured logging

Even in its simplest form of unstructured logging, using a library significantly simplifies the user experience.

The standard format for unstructured logs is as follows:

{Timestamp} {log_level} {message}

These three fields are present in various forms in every log entry.

The timestamp is automatically calculated from the library at the time of the function call, while the log level is encoded in the function call name.

In this specific library, the default log level is Info, so it can be omitted.

package main

import (
    "log"
    "log/slog"
)

func main() {
    log.Print("Info message")
    // 2024/01/03 10:24:22 Info message

    slog.Info("Info message")
    // 2024/01/03 10:24:22 INFO Info message
}

There is no standard format for the message field.

Given that each application has its format, if you want to extract useful bits of information from a Nginx web server logs, you should consult the application documentation to understand what regex to use to extract fields from plain text.

If you ingest Apache Web server logs, the regex format will be totally different.

For expert only:
Elastic, the company where I work, provides a complete Observability solution to collect, parse, and ingest unstructured logs from various famous applications thanks to integrations.

Each integration is aware of the log format of the application that is observing and can extract useful bits of information from the message field.

Unstructured logs are automatically parsed by integrations into JSON logs to be ingested into Elasticsearch.

In my two years working in the Observability department at Elastic, I wrote an integration for Istio to parse proxy logs and Istiod logs. You can find this integration at Istio.

Structured logging

Using a library instead of print statements allows the output format to be switched from unstructured to structured (in this case, JSON format) in a single place.

Logging, traditionally meant for the human eyes, has historically been unstructured.

In recent years, given the vast amount of logs generated by cloud-native applications and the need for more observability, it is slowly becoming more machine-friendly.

package main

import (
    "log/slog"
)

func main() {
    opts := &slog.HandlerOptions{
        Level: slog.LevelDebug,
    }

    handler := slog.NewJSONHandler(os.Stdout, opts)

    logger := slog.New(handler)
    
    logger.Info("Info message")
    // {"time":"2023-03-15T12:59:22.227408691+01:00",
    // "level":"INFO","msg":"Info message"}
}

Applications that historically have using unstructured logging are transitioning to support both structured and unstructured logging, with JSON logging being the default option.

For experts only:
Elastic, can collect and ingest JSON logs without any parsing necessary. No need for integrations to parse the message field.

In support of what I mentioned above about the current trend to move to structured logs, Kubernetes has "recently" (in 2020) moved its application logs from unstructured text logs to structured JSON logs. More info at Kubernetes introduce structured logging.

Logging levels

Most logging libraries support changing the default logging level in a single statement.

The logging levels in order of importance from the most important are:

• Error

• Warn or Warning

• Info

• Debug

• Trace

It is common to set the default logging level in production to WARN or WARNING in order to reduce the logging cost and use a Trace or Debug logging level while in development.

If you set the default logging level to WARN, any call to lower logging levels will be handled as a NoOp and print nothing to the standard output.

package main

import (
    "log/slog"
)

func main() {
    opts := &slog.HandlerOptions{
        Level: slog.LevelDebug,
    }

    handler := slog.NewJSONHandler(os.Stdout, opts)

    logger := slog.New(handler)
    logger.Debug("Debug message")
    logger.Info("Info message")
    logger.Warn("Warning message")
    
    logger.Error("Error message")
    // {"time":"2023-03-15T12:59:22.227472149+01:00",
    // "level":"ERROR","msg":"Error message"}
}

Log context

Logging libraries can avoid code duplication when some logging context needs to be shared between multiple log events.

As described below, you can create a child logger with the common context and use that logger instead for the different events (e.g. Info and Warn events here). Both events will have fields shared from the common context.

package main

import (
    "log/slog"
)

func main() {
    handler := slog.NewJSONHandler(os.Stdout, nil)
    buildInfo, _ := debug.ReadBuildInfo()

    logger := slog.New(handler)
    child := logger.With(
        slog.Group("program_info",
            slog.Int("pid", os.Getpid()),
            slog.String("go_version", buildInfo.GoVersion),
        ),
    )

    child.Info("image upload successful", 
        slog.String("image_id", "39ud88"))
    child.Warn("storage is 90% full",
        slog.String("available_space", "900.1 mb"),
    )
}

Conclusion

Various Logging libraries can provide more features like hiding sensitive information (e.g. passwords or API keys) or adding stack traces when logging errors.

You can read more about Slog for Golang at Logging in Go with Slog: The Ultimate Guide.

I hope you enjoyed learning about logging and observability.

I might write future articles about the other pillars of Observability.

It doesn't matter what stage of your career you are currently in.

At some point, you might have used Print statements to debug software.

While you might have thought that was fine, I suggest that that's not a good practice you should adopt in 2024.

This article should help abandon Print statements in favour of some form of Observability.

Before we go any further, what is Observability?

Observability pertains to how a system’s internal state can be understood by examining its external outputs, especially its data.

Wait a second. You might argue that you have used logs and metrics in the past when it was still called Monitoring.

Is Observability just a different name? Is it just semantics? Why do we need a different term for Monitoring?

Logs are still logs, no matter what you call the practice of using them in your applications. Right?

We will discuss all of that in this article and more.

This article's audience spans from Junior Software Developers who have only used Print statements to more experienced Senior Software Engineers who want to brush up on best practices and get pointers to up their Observability game.

In this article:

• Meaning of the front page image

• Print statements as a Poor Man's Observability solution

• Application Observability

• Short history of Observability

• Conclusion

What is out of scope:

• A deep dive into Logs, Metrics and Traces. I plan to cover those three pillars of Observability in future articles.

• Error tracking, application profiles, crash dumps and other more advanced forms of Observability.

• Libraries and tools to implement Observability.

Want to connect?

👉 Follow me on LinkedIn and Twitter.

If you need 1-1 mentoring sessions, please check my Mentorcruise profile.

Meaning of the front page image

A software engineer is sitting on the bank of the river, watching wood logs floating down the river.

Some logs get stuck on the river bank and never reach the bottom of the valley.

This is a metaphor for how not all logs are useful for getting to the root cause of an incident.

I generated the above picture with Midjourney using the prompt below.

A software engineer on his laptop is immersed in nature on the bank of a river, watching wood logs being transported by the river current down to the valley. Use a comic style.

Print statements as a Poor Man's Observability tool

Countless articles for beginners suggest using Print statements to debug software.

In the rest of this article, by Print statement, I assume that in your language of choice, there is a way to print text to the standard output. It doesn't matter what the API looks like; for me, this will always be a practice to avoid as much as possible.

I don't suggest using Print statements, even if you are starting your journey into Software Engineering now.

The reason is that, especially at the beginning, once you learn to use a tool, you might use it everywhere. The famous phrase if all you have is a hammer, everything looks like a nail couldn't be more accurate in this case.

Instead of wasting time with print statements, they should teach some form of logging straight away.

As I have already discussed in my article Exploring Structured Logging with slog in Go, the bare minimum in logging should be using a logging library for structured logs. We will discuss logging more in the future.

Spending extra time implementing some form of Observability or even just using a debugger with breakpoints always pays dividends in the long term.

I find it even more troubling when Senior or even Principal Engineers still suggest using Print statements because it is the easiest option over more complex setups like debuggers or logs.

Observability is still hard to master.

Hopefully, with this article, we can make things clearer.

There are many ways a print statement should be replaced with better Observability options:

• Tracking a specific event (e.g. an error or a warning) => Logs

• Counting the number of times an event occurs (e.g. number of times a web application receives a request) => Metrics

• Timing the duration of a function (e.g. how long does it take to query a database?) => Traces

• Printing error Stacktraces => Error tracking software

• Checking the internal state of variables while developing => Debugger with breakpoints

Leave a comment

Application Observability

In this and future articles about Observability, we will primarily focus on application observability, even though the basic concepts are the same for other forms of Observability, like infrastructure observability.

Every Cloud-Native Engineer should be very familiar with application observability and at least have a basic understanding of infrastructure observability since he/she will probably be involved in deploying apps on the cloud.

The concept of application observability was popularised by Etsy’s John Allspaw and Paul Hammond in 2003, but 20 years later, it is still somewhat misunderstood.

I briefly discussed application observability in a previous article titled Exploring Structured Logging with slog in Go.

A quote from that article:

You can think of Observability like the 2020s version of Monitoring, where the sheer amount of logs and metrics introduced with cloud-native distributed applications pose some huge scaling challenges on what you can collect, how much you can store, and what is just noise and can be discarded. These new challenges made us rethink the tools we use, introduce new ones or adapt the old ones.

Short History of Observability

I can imagine that it all started with a Print statement, the most basic way to observe an application's internal state.

As discussed in this article, before Observability was a thing, we used to call the practice of using logs and metrics in our applications Monitoring.

So where does Observability come from, and why does it differ from Monitoring?

An in-depth discussion of the reasons Observability is different from Monitoring is outside the scope of this article. Here, I just want to mention some of those reasons that are more relevant to my audience.

Before the cloud-native movement started, there was a much clearer distinction between developers and System Administrators.

Once you were done writing the application code as a developer, you would metaphorically throw it over the fence to be deployed and monitored in production by a different set of people.

With the arrival of cloud-native applications, developers are now responsible not just for writing the code but also for deploying it to production, keeping it running via on-call rotations, and investigating issues via root cause analysis.

Things get very complicated when the application is not a process running on a single machine but a bundle of microservices running on multiple machines on the cloud. This is what is now called a cloud-native application.

It was about time that I covered observability in this newsletter.

What worked in the past with monitoring, doesn't scale anyone with tens, hundreds or even thousands of microservices talking to each other over the network.

What was before a local syscall is now a network call.

New types of applications call for different challenges and different ways of doing things. Hence, the move from monitoring to observability.

It's not just semantics. It is a different discipline, with its own practices, challenges, and tools.

The essential components, the three pillars (also called the three signals) logs, metrics and traces might be the same, but how they are implemented, interconnected, and processed in different ways.

You must keep observability in mind from the beginning when designing your applications.

Otherwise, if you are not careful, you may experience a cardinality explosion and some hefty bills at the end of the month.

Observability can't be an afterthought; otherwise, it will bite you when an incident happens, as you won't have all the information you need to investigate the root cause.

For more information about Observability, I highly recommend the book Observability Engineering.

If you are instead into a more practical approach to learning about the three pillars of observability and how to implement them into your application, have a look at the Observability whitepaper by the CNCF.

Share Cloud Native Engineer

Conclusion

In this article, we just scratched the surface of what observability is and why it is necessary.

I plan to cover each pillar of observability (Logs, metrics, and Traces) in its own article so that I have enough space to suggest best practices and tools.

Furthermore, I would also like to discuss some other forms of observability that go beyond those three pillars, like debugging, application profiles, and crash dumps.

In the interest of brevity, I prefer to split the discussion into multiple articles.

If you have been a Software Developer as long as I have (more than ten years now), you might have encountered dependency hell at some point in your career.

If you haven't (lucky you), let me introduce the concept:

Dependency hell describes the complex web of software dependencies that can arise, making it challenging to install, update, or run applications due to version conflicts and compatibility issues. It is a common problem in software development and deployment, especially across different platforms and ecosystems.

As the term describes, it is not a great state of mind.

Software Engineers have written entire programming languages and countless tools out of the frustration of dealing with some sort of dependency hell.

In this article, I would like to describe my recent experience of dependency hell in Python and how I solved the problem once and for all—at least, I hope.

I'll introduce a tiny tool called Pipx, which allows you to install a CLI Python application (and its dependency) in a totally isolated virtual environment, avoiding any dependency conflict.

I’ll explain how to use Pipx and what are its benefits over Pip and other similar tools.

This article is specifically tailored for Python developers, both experienced and inexperienced.

Whether you're a seasoned pro or just starting out, the insights and solutions I'll be sharing can be invaluable in your Python development journey.

As a Python developer, I hope you can gain valuable insights from this article that you might apply at some point in your career, further enhancing your skills and knowledge.

In this article:

• Meaning of the front page image

• 💰 (paid content) My encounter with Pipx

• 💰 (paid content) What is Pipx?

• 💰 (paid content) Pip vs Pipx

• 💰 (paid content) How to get started

• 💰 (paid content) Example usage

• 💰 (paid content) Pipx run

Meaning of the front page image

When exploring a Midjourney prompt for this article, I wanted to convey the concept of "Dependency Hell".

What better metaphor for Dependency Hell than a sea of lava?

So, a Python snake avoids the sea of lava (the dependency hell) by crawling over a precarious tree log hanging above it.

I generated the above picture with Midjourney using the prompt below.

A vibrant, comic-book inspired image of a green Python snake slithering across a weathered log, its head tilted curiously as it navigates a fiery, lava-filled chasm beneath.

Want to connect?

👉 Follow me on LinkedIn and Twitter.

If you need 1-1 mentoring sessions, please check my Mentorcruise profile.

My encounter with Pipx

My first (and most horrific) experience of dependency hell goes back to the time that I was writing Java code early on in my professional career.

Things didn't change much later when I began writing Scala code (mainly for Spark) but it got a lot better when after a few years I started writing Python code.

I love creating virtual environments in Python to isolate each project I am working on.

So, I stayed away from dependency hell for a while.

Last week, I had my first experience of dependency hell in Python. I had to install at least four different CLI applications in Python.

As usual, I added them all to my Poetry config. Soon, Poetry started complaining that those tools were using incompatible library versions.

As years of experience solving similar problems in Java taught me, I started checking if there was a way to solve those conflicts, but after 30 minutes of struggling, I gave up.

Since I knew the maintainers of those tools, I reached out to check if they were willing to update one ancient legacy dependency library. I even opened a public GitHub issue to get this fixed.

To my surprise, they suggested I use a separate virtual environment for each tool.

My first thought was... Are you crazy? One virtual environment for each of those? It will take me ages to set everything up. Are you sure you cannot upgrade this tiny dependency? I could fork the tool and update it myself; that would have been much faster, but potentially, it could still have taken hours.

After a few minutes of complaining, they finally suggested a tool I never heard of: Pipx.

What the hell is pipx? was my first reaction. I have never heard of it.

Surprisingly, this tool did precisely that: installing each Python CLI application in an isolated virtual environment.

Here I am, with this new piece of knowledge, trying to teach you about the benefits of Pipx for your Python development.

What is Pipx?

Pipx simplifies your life by enabling you to install CLI applications written in Python (and their dependencies) in isolated virtual environments.

It's as easy as pipx install <package>.

It creates one virtual environment for each CLI application to avoid dependency conflicts.

It looks similar to brew or other package managers you might have used in different languages.

Between the benefits, you can count on a clean uninstall of your packages, no need to use sudo to install global packages and the ability to list packages already installed by pipx.

Behind the scenes, pipx uses the Python module venv to create a virtual environment and pip to install those dependencies.

Those virtual environments are installed under a fixed location, ~/.local/share/pipx/venvs/<package>, and the Python applications installed have symlinks added to the ~/.local/bin.

Those two locations differ on Windows but are the same on macOS and Linux.

More information about pipx can be found at the pipx—GitHub repository.

Some GitHub stats

Pipx is open source and actively developed at the time of writing.

It counts 146 contributors and 8.9k GitHub stars.

The number of stars has grown exponentially since the beginning of the year.

The latest version 1.5.0 has been released on Mar 29, 2024.

Pip vs Pipx

As the name implies, "Pipx" has a similar purpose to Pip.

The main difference is that while Pip installs a package globally or in the current virtual environment, Pipx instead creates a new virtual environment for each package and then adds a symlink to the package entry point to a dedicated location.

Similarly to Pip, Pipx installs packages from PyPi by default, but it can optionally install from Git repositories, folders, and any other place Pip can install from.

How to get started

Getting started with Pipx is quite simple.

You install the tool first, then add the default location ~/.local/bin to your PATH, and finally, install your packages.

Installing pipx varies for all operating systems.

If you, like me, are a MacOS user, you can use brew to install pipx. Otherwise, for Linux, you can use one of the many package managers for your operating system.

Warning: Installing pipx with pipx is not suggested.

The pipx official documentation provides more information on how to install pipx.

Example usage

Let's say you are installing the famous Python formatter called Black.

How would you install this package? And what does it get created under the hood?

You can install Black with Pipx using the command:

pipx install black

Behind the scenes, Pipx will go through these three steps:

1. Creates a virtual environment via the Python module venv at ~/.local/share/pipx/venvs/black

2. Install the package and its dependencies in that venv

3. Create a symlink at ~/.local/bin/black to the command installed at ~/.local/share/pipx/venvs/black/bin/black

Since pipx ensurepath has already added the location "~/.local/bin" to your PATH, the command black should be available immediately.

Pipx run

Pipx allows you to avoid installing a package if you only need to run a command once.

Instead of using, pipx install <package>, you can use pipx run <package> <package params>.

This command will create a temporary virtual environment that will be destroyed when the command finishes running.

I love this feature since it reminds me how Nix shell works.

If you want only run a command in Nix once without installing it, you can run nix-shell -p <command> --run <command and params>

nix-shell -p jq --run jq .

If you have never heard of Nix before I have written an article about it at Effortless Python Development with Nix.

This article explains how I use Nix to install Python, create a virtual environment, install Poetry and much more.

I am writing to inform you that I will be taking a break from writing until the end of the month in order to recharge mentally.

Thank you for your understanding.

I have not been very active in the past few weeks as I have been busy with the technical review of a book being written by my colleagues, Huage Chen and Yazid Akadiri.

The book should be available in a couple of months at Elastic Stack 8.x Cookbook.

It took me a considerable amount of time to do this technical review, so I am quite exhausted.

You might ask. What is involved in a technical review?

I didn’t know either before starting this one.

It required me to read the entire book, verify the accuracy of the content, and ensure that all of the code was functioning properly.

As this particular book is a cookbook, it involved practical applications and required me to run the Elastic stack on the cloud while learning about almost all of the services within the stack.

📚Recommendations

• The Ramen Bowl - If you are a foody like me, you can’t miss this food-related substack from my friend . He doesn’t talk about Ramen only. He has been branching off to sweets and ingredients from Asian cuisine.

The Ramen Bowl

All things Ramen and Japanese-inspired cuisine from a foodie nerd. Get bi-weekly posts with stories, recipes, and experiments - ◡ -

🎯In case you missed it…

🔮Future articles

I’m planning to write more about:

• Elastic Stack (given that I learned some nice tricks from my book review)

• DevOps tools like Crossplane.

• Container security tools like Snyk.

Want to connect?

👉 Follow me on LinkedIn and Twitter.

If you need 1-1 mentoring sessions, please check my Mentorcruise profile.

This article is my second piece on Taskfile.

My first article Taskfile: a modern alternative to Makefile went viral (at least for my standards).

Here you have some stats:

• 95 points and 212 comments on HackerNews

• 10k views and 31 new subscribers on Substack (this newsletter).

• 3.6k views and 2k reads on Medium where I cross-posted the article.

Makefile is a fascinating and controversial topic that sparks passionate debates.

Individuals have diverse and firm opinions on the matter.

The debate on HackerNews became quite intense very quickly. Both sides presented their hot takes—the supporters of Makefile and those who dislike it but have yet to discover a better alternative.

This article is directed towards the latter group.

If you strongly favour Makefile and are unwilling to consider counterarguments, please refrain from reading further.

However, this article may interest you if you dislike Makefile and have been searching for a suitable alternative for years.

Want to connect?

👉 Follow me on LinkedIn and Twitter.

If you need 1-1 mentoring sessions, feel free to check my Mentorcruise profile.

What is Taskfile?

Task (the name says already) is a task runner written in Golang that adopts a YAML-like syntax called Taskfile for writing tasks and their dependencies.

I understand that Task is the name of the tool and Taskfile is the format, but I use the terms interchangeably as people do with Make and Makefile. So keep that in mind when reading the rest of this article.

I already know what people are going to say about Taskfile.

YAML again! YAML is everywhere and... I hate it!

I am not a huge fan of YAML either, but in this case, YAML doesn't get in the way.

Some stats about Task's adoption

• 9.8k stars. When I wrote my first article, there were 8k stars. As you can see from the graph below, they are increasing exponentially in time.

• 159 contributors (including me). I have created some GitHub issues in the past and started discussions.

Taskfile vs Makefile

I have to say that comparing Taskfile and Makefile is somehow unfair.

Makefile has been around for almost 50 years and is widely used by a vast community. Taskfile has only been around for about 6 years and has a much smaller user group.

What is terrible about Makefile?

• Hard to write and read.

• Not user-friendly for beginners.

• Documentation is extensive but very hard to search.

• Cannot easily list targets.

• It doesn't support attaching a help description to a target. You need to look into the Makefile source to understand what the target is meant to be doing.

• Hard to break a Makefile into multiple files.

• If a target is not a filename, you need to label that target as PHONY.

• Targets (hence filenames of the source code to build) have some legacy limitations. No spaces, no colons, etc.

Makefile was initially conceived to be a build system and is very good at it, but when used as a task runner, it is not a very ergonomic tool.

When compared to more modern building tools like Grunt, Cargo, or Maven, respectively, for Javascript, Rust, and Java, Makefile feels like ancient history and should be replaced with something better.

What do Makefile and Taskfile have in common?

• Single binary. Any other solution based on interpreted languages (like Python) is more challenging to use.

• Runs everywhere on all Operating Systems thanks to its single binary nature.

• Small wrapper around command line tools.

Why is Taskfile better than Makefile?

Some reasons in no particular order:

• Simple to read/write and friendly to newcomers.

• Great documentation. Not extensive, but good enough.

• Simple to break a Taskfile into multiple files and import other Taskfiles as dependencies

• You can list the available tasks or see a short description of a task or a more extended summary.

• Easy templating with Go templates.

• Dry-run mode allows printing the shell commands that make a task and easily reproducing the results elsewhere without a Task.

• Cross-platform shell interpreter (it works for Windows, too).

• Public vs private (aka internal) tasks.

• You can defer a command to run after the end of the task.

• Watch feature. It can rerun a task when it detects that a list of files has changed. More on this later.

Feedback from my first article

As stated already, my article Taskfile: a modern alternative to Makefile raised quite a few comments on HackerNews.

I'm sorry if I upset some people in the process.

The passage highlighted below caused most of the discontent.

As anyone that writes or interacts with software these days, I need to run automation scripts every day but I am still stuck with a tool that (according to Wikipedia) has been written 47 years ago. Don't get me wrong, Makefile was really useful 20 years ago but I think we can do better than that in 2023.

Someone commented:

[Makefile is...] ...Proof tested because it has been used for 47y.

I agree with that point, but that is not motivation enough to not develop a more modern alternative that better serves the needs of the 21st century.

I realise now that almost 50 years of use have created pockets of fanatic users that are hard to convince.

Even though, I got some positive feedback from HackerNews as follows:

Make is much the same way. It gets the job done, that doesn’t mean it’s good at getting the job done when better alternatives exist.

The attitude of “well it should be hard, the shouldn’t even learn C if they can’t handle make” is explains why my students love Rust so much. Rust has a sane and standardised build system they prefer to use. Make is the Stone Age to them.

I’m just saying Make is an issue for a lot of reasons like accessibility and portability to multiple toolchains and systems

I won't mention the more negative comments here since some are not very productive, but I suggest you have a look if you're interested.

Here is my strong opinion about the topic of hardcore fanatics about Makefile. I believe people who are unhappy about Makefile but still use it somehow suffer from Stockholm syndrome.

Stockholm syndrome is a proposed condition or theory that tries to explain why hostages sometimes develop a psychological bond with their captors

My article on Taskfile inspired my friend Edoardo Tenani to write a piece on the same topic titled To make or not to make.

I share most of the opinions in that article, and I think Edo made some excellent points.

[Makefile...] ...may also be ... affected by “we always do it like that” or “no one has ever been fired for buying IBM” and this is a culture I’m more than happy to abandon.

Make is far from perfect when used as a task runner, something it has not been designed for.

More recently, I watched a video by Viktor Farcic on the benefits of Taskfile vs Makefile called Say Goodbye to Makefile - Use Taskfile to Manage Tasks in CI/CD Pipelines and Locally

The video highlights the importance of being able to run the same tasks on both your laptop and your CI system.

To achieve consistency, the suggestion is to use Taskfile for all your tasks and wrap it around a GitHub action that sets up Taskfile itself.

This is an alternative to tools like Act that allow you to run GitHub workflows locally. In fact, these tools can only partially replicate GitHub actions on your laptop.

Get 20% off for 1 year

Resources

I have mentioned Taskfile in one of my latest paid articles, Unlock SQLite: Easy Data Analysis for Busy Software Engineers.

You can find the relative taskfiles to install SQLite extensions and analyse some CSV files at the GitHub repository Demos.

In the same original article about Taskfile, I also wrote about other automation tools like DevEnv and Direnv that together with Taskfile are part of all my projects. If you are interested more in those technologies instead, look at Effortless Python Development with Nix.

I have a GitHub repository with some generic taskfiles that I reuse in most projects at Taskfiles.

I have also used Taskfiles to set up a development environment to run Elastic Cloud on Kubernetes. You can find the paid article at Dev environment with Elastic Cloud on Kubernetes (ECK) and the relative taskfiles at ECK taskfiles.

If you are a busy software engineer, you can't spend weeks thinking about the best solution for the job and then learning those tools.

Let me help make this choice much more manageable.

In this article, we will discuss SQLite, which I believe is the best tool for data analysis for relatively small datasets.

Get 20% off for 1 year

The article is divided into the following sections:

• When is SQLite the best tool for the job?

• How to get started with SQLite

• Extensions and tools for SQLite

Want to connect?

👉 Follow me on LinkedIn and Twitter.

If you need 1-1 mentoring sessions, please check my Mentorcruise profile.

When is SQLite the best tool for the job?

You are a busy Software engineer who needs to analyse a relatively small dataset (e.g., 30-100MB of CSV or JSON logs) for a new project.

What tools should you use?

Any big data tools like Spark or Hadoop would be an overkill for such a small dataset.

Setting up a Python project with a Pandas data frame is the best option in most cases, however, depending on the complexity of the analysis, that might even be overkill.

As I explained in a previous article called Effortless Python Development with Nix, setting up a Python project comes with some complexity.

You need to set up a virtual environment and install dependencies in a reproducible way. If you are sharing this project with someone else, you need to provide documentation on how to get started.

Using Nix makes onboarding new people, or even automating the setup process, a lot easier, but at the cost of installing Nix on your laptop.

Using command-line tools like awk and grep is definitively more straightforward to set up, but those tools may not provide enough flexibility. You also need to install those tools. Nix can also help you in this case, but you still have to install Nix.

I'm a huge fan of command-line tools; I use them all the time in my daily job.

I found a fascinating book on this topic called data science at the command line.

The book is now free, so I recommend reading it. While I enjoyed reading it, I don't agree with all the tools presented in this book. For example, I am not a fan of Makefile. If you want to learn more about this topic, I wrote an article at Taskfile: a modern alternative to Makefile. Having said that, knowing those tools is highly beneficial when learning about command line tools.

Given that, as most software engineers, you know SQL, you could probably load all your data into a MySql or PostegreSQL database. You could install those databases locally or use managed services. You are trading installation time for the cost of paying for cloud services.

Is there a more straightforward solution that doesn't involve complex setup or cloud costs?

What else can you do?

If only we could come up with a solution that is powerful, flexible, easy to onboard, doesn't require cloud costs, and doesn't feel like an overkill for such a small dataset and simple analysis.

Am I asking too much?

I think I have a solution to this problem.

Discover SQLite:

• SQLite is the most common DBMS in the world, shipped with all popular operating systems.

• SQLite is serverless.

• For developers, SQLite is embedded directly into the app.

• For everyone else, there is a convenient database console (REPL), provided as a single file (sqlite3.exe on Windows, sqlite3 on Linux / macOS).

Those quotes are from an excellent article called SQLite is not a toy database.

How to get started with SQLite

The console is a killer SQLite feature for data analysis: more powerful than Excel and simpler than pandas. One can import CSV data with a single command, and the table is created automatically:

> .import --csv city.csv city
> select count(*) from city;
1117

The quote and code are from this excellent article SQLite is not a toy database.

There are plenty of resources on how to get started with SQLite. I don't think I'll provide helpful information by repeating the same information here, so I'll give some good starting points below.

Some other learning resources:

• SQLite official documentation

• Unlocking the Power of SQLite: A Comprehensive Guide (paid article)

Extensions and tools for SQLite

As we discussed before, SQLite is not a toy database. Sometimes, though, it can feel like it is not as powerful as well-known alternatives like MySQL or PostgreSQL.

In the last few years, many extensions and tools have been developed to extend SQLite's functionality.

To name a few extensions:

• sqlite-xsv: fast extension for CSV files

• sqlite-url: functions to work with URLs.

• sqlean: extension pack with some missing functions like math functions, cryptographic algorithms, regex support and much more.

All those extensions are libraries that must be downloaded locally and then loaded into the database at runtime. For more information on how to load a SQLite extension, see Runtime Loadable Extensions.

Other than extensions, there are also some tools which can improve your experience working with SQLite:

• sqlite-utils: library and command-line utility that helps create SQLite databases from an existing collection of data

• sqlpkg-cli: manages SQLite extensions, just like pip does with Python packages or brew does with macOS programs. This tool allows you to install those SQLite extensions that we mentioned before (and many more) from a SQLite package registry.

I have created a GitHub repo with some Nix files, Taskfile and a Readme to make the experience of installing those extensions much more pleasant.