Programming

My proposal for a talk I delivered at Velocity Amsterdam, 2016. Slides are included at the bottom of the page. For the full transcript of the talk, visit https://sudarkoff.blog/p/distributed-ops-for-distributed-apps (paid subscription, free trial available).

Description

In the world of microservices, when things are moving fast and constantly breaking, the accepted wisdom is that teams must own the whole stack and operate their services themselves. But how much stack is “the whole stack”? How do we ensure that operational standards are consistent across the organization? What’s the right balance between consistent standards and the ability to move fast?

This was originally shared with the engineering org of a company I worked for at the time. It’s my stream of consciousness about the type of system my team was building at the time. Reproducing here without modifications, enjoy the chaos of my unedited remblings.

Team,

It became clear to me in the last few days that there’s not enough clarity about what we’re trying to build.

We keep talking about relatively small and specific things, but we never discuss the philosophy behind it all. What type of system are we building? Can’t we just install Puppet and Terraform and call it done? And focus instead on something more glamorous than spinning VMs?

Understanding a few key concepts will help you make sense of how Git works and how to use it effectively.

Full history

With Git, every repository contains a full copy of the project history, and no repository is inherently more important than any other.

Content-addressable

Git stores content in files with names that are 40-hexdigit (160-bit) SHA1 hashes. This has some interesting implications. It means for example that Git will never store the same data twice within a repository because identical data would have identical SHA1 hashes and therefore be stored in the same file. It also means that the contents of a Git object will never change because that would change the name the object is stored under.

This is a proposal I made for improving the disk selection algorithm of a distributed file storage system.

Current Disk Selection Algorithm

The current algorithm randomly selects a disk among the top N percent and assigns it to a given equivalence class. This strategy is suboptimal for a couple of reasons:

1. Even when there’s a great variability of fitness values among the top N percent of the disks, the most suitable disks will have the same probability of being selected as the least suitable disks, thus leading to unbalanced placement. Simply decreasing the value of N is not sufficient to resolve the problem, as it is always possible to have a situation when the fitness variability is great regardless of how small the N is.
2. When nearly all disks in the pool have similar fitness values, they all deserve equal chance, yet only the top N percent will be considered. It’s tempting to assume that the next round of placements will favor a different set of disks. But the selection of the next eligible set is determined by the weights that each fitness parameter is assigned. And it is possible that the same group of disks will be used for placement for a while, thus, once again, leading to unbalanced placement.

Undoubtedly, we would like to preserve some randomness in the selection algorithm to minimize the chances of overwhelming the same disks and nodes with placements and IO. But for reasons stated above, a fitness proportionate selection algorithm is more desirable.

I was cleaning up the address book on my PDA the other day when I suddenly realized that Palm OS is going to die pretty soon! Yes, I think its days are counted and here’s why.

Programming for the Palm OS platform has a very distinct “embedded” smell to it. Memory management is a mess, supporting different screen resolutions is a major pain in the butt, and instead of a real file system developers have to deal with this stupid notion that everything is a database. There might have been reasons for it a few years ago but not anymore. And Palm fails to recognize this fact and deal with it appropriately.

A meditative piece in which I explore the notion of productivity as applied to the craft of a software developer (and other such crap).

Yesterday I spent twelve hours hunting down a most bizarre bug. I flipped through literally thousands of lines of code, I set dozens of breakpoints only to realize that those places were being hit hundreds of times, and it is virtually impossible to debug it that way. I tried everything I could think of, but nothing helped and by the end of the day I was totally exhausted and pretty much ready to choke somebody.