MICHA.ELMUELLER

I’ve left the company I was previously working for at the end of 2018. In this post I want to reflect on some technical things I learnt there which I found really valuable.

• Have a style guide and enforce it with a linter
  A linter is a program which checks that your code follows certain style rules. You can e.g. hook this up to the CI or as a pre-commit hook. It should not be possible to merge if the style is violated. It’s common to catch quite some bugs early this way.
• Rebase heavily, group commits semantically.
  Don’t e.g. put a refactoring change in a commit with a bug fix. Split commits if they contain multiple semantically different changes, reorganize a PR until it fits semantically. Don’t squash on merge, but rather keep individual commits to retain this semantic order in the master branch.

  This is how Git works best. It makes it a lot easier to e.g. track bugs. This is because the commit message explains what the intention behind the change was and one can verify that the commit does exactly this and nothing else. Also you can use git internal tooling (like git bisect).

• Do not squash on commit
  This enables the possibility to prefix a commit with something like Revert me: Your message. This commit can easily be git revert-ed later on to only revert this specific change. We used this often to merge temporary fixes, which were supposed to be ephemeral. By indicating the short-lived intention of this change it’s clearly stated that this needs to be removed again. Also, you can always search for the corresponding Revert: "Revert Me: Your message" commit, to verify that something was actually thrown out again.
• Use git blame
  …to find out about why something was merged (works only when the commit message describes the intention). I used this often to look up the original PR and read through the discussion there to better understand why a decision was made (GitHub links the originating PR in the blame view).
• Make errors visible
  In the case of an error rather than continuing to run and fallback on some default behavior it can oftentimes be much better to just let the program crash and report the error. Otherwise you quickly run into the case where errors happen, but are not recognized or acted properly upon. And oftentimes the default behavior you defined might be suitable for one class of errors, without you having thought about other error classes which can also happen.
• Return early
  Don’t do something like

  if foo {
    bar();
  } else {
    return ...;
  }
  

  instead do

  if !foo return ...;
  bar();
  

  Of all style guide things this is a really significant one that makes code a lot clearer (and keeps the indentation levels low).

• Most of the time spent when implementing something is reading code
• Rollbacks should be easy
  This refers to deployments. At mbr it was quite easy to build an arbitrary commit and deploy it. This makes you feel a lot better about changes where it’s not entirely clear if they might cause issues.
• A feature commit must contain a test for that feature
• A bug fix should contain a test which would have catched the bug

Another thing which we didn’t do at mbr, but which I find highly valuable is to put a lot of effort on test coverage. A number of open source projects have integrated e.g. codecov.io into their CI pipeline in a way that it’s only possible to merge if the test coverage stays either the same or goes up.

During my time at mbr I wrote a couple of Slack Apps which I included into the internal company channels.

I’m quite happy with the way they turned out and how actively they were used. They all contributed to the great company atmosphere — either in functional or entertaining ways.

Weekly interview with a random colleague

I wrote a Slack App which interviewed a different employee of the company each week. The questions were the same for everybody:

Hi there!
This app interviews everybody in the team subsequently. The idea is to get to know your colleagues a bit better and maybe find out some things which you didn’t know before. This time you have randomly been chosen to be interviewed.

If you choose to answer the following questions, the answers will be posted for others to see.
These are the questions:

• Who are you and what is your job here?
• What was your nicest moment at mbr?
• What is your favorite place nearby?
• What was the last thing you discovered which excited you?
• What have you learned recently?
• When are you most productive?
• What was the last book/show/movie that you recommended to someone?
• My happy place is …?

The app offered the choice of answering the interview, skipping it this time, or skipping it forever.

Overall I am very happy with how it turned out. It’s one of the things I wrote of which I’m very proud. I would say the App was a success and it was always very interesting to read the interviews.

The team was only slightly de-centralized and I would imagine this to have even more benefit in a decentralized company where more people work remotely.

Bi-Weekly histogram of most used reactions

Slack offers the possibility of pinning emoji reactions to any message. This is (not only) a fun feature, but also reduces a lot of noise. Instead of “I agree”, “I join”, “Great idea!”, … people just pin a fitting emoji.

Each two weeks this integration posted a histogram of the recently most-used reactions. It was interesting and entertaining to watch how the reactions changed over time — e.g. during Christmas time or when there was some incident in the data center. The data center incidents were always clearly visible in the reactions :-).

Slack offers the possibility of creating custom emojis and Thomas had created an emoji for everybody with a photo of the person. So you could observe people showing up in the histogram, when they had e.g. achieved something nice. Or when somebody had a trademark thing that could be clearly attributed to her/him. We had also added a lot of logos as emojis — e.g. for the data center operator company. So whenever they caused chaos their logo was showing up high in the list.

I have put the script on GitHub.

Weekly histogram of on-call alarms issued by our monitoring system

I was working as a DevOp and part of the job was to be on-call during nights, on weekends, etc.. This meant that whenever our monitoring system detected failures or anomalies the system would notify the person on-call; first with a push notification and then escalating further…up to a robot calling the person if he didn’t react at all. It was hard to keep track of the alarms which were raised, since the on-call shift was changing weekly. So I wrote this App which after every shift of a person would post the night and weekend alarms the person had to deal with.

This really helped improve awareness for the stack stability and the number of problems we had to deal with.

I would say that this definitely also had a huge impact on how the DevOps people were seen by non-DevOps colleagues. It raised awareness for stability issues and helped other people understand how much work being on-call was. It also offered a way for other people to show appreciation of how a person managed an incident (e.g. by talking to them, since this histogram was how they even found out that there was some incident in the first place).

After a while we started to explain the cause of the incidents in a thread below the post. This helped a lot communicating what was actually going on in the stack. We also later used these notes when discussing possible improvements and reflecting on what could have been prevented.

Oftentimes these histograms were the start of a longer discussion of how something could have been prevented with the whole team being involved. Before my integration this was something which only happened when there were major incidents, but not in this structured and holistic way.

I think this integration — simple as it may seem — was a big contribution to the company. The main take-away was that it made incidents visible, not only to the directly affected engineers.

Status information about active campaigns

Another simple integration I wrote posted a notification whenever manual intervention was necessary into something running on the stack. The nice thing was that — since Slack offers threads — people were able to discuss the issue immediately in Slack and coordinate on who would take over.

I had fun playing with the messages and giving the App a bit of character. So the App would always choose a random message when alerting people about mishappennings or successful fixes.

I spent a couple months of 2018 diving into Rust, a modern low-level programming language with high-level language features — functional programming, asynchronous programming, closures, ….

The language is a bit out of the family of languages that I usually work with and so a lot of stuff had to be learnt.
Also, I characterize Rust as an expert language. It is very explicit since one of its design goals was to not have hidden performance costs of functions without the programmer being aware of it.

I learnt the language the same way that I learnt stuff during university and it has proven to work very well for me:

• I read a fundamental book on the topic from cover to cover and made sure that I understood every single line in the book. The book was “The Rust Programming Language”.
• After having read a large part of the book, I summarized each chapter that I had read so far in my own words on paper. To me it’s important to not immediately summarize each chapter after I’ve finished it, but rather gain an elementary idea of the domain first.

  For this summary I re-read each chapter and wrote up the most important things. During this “second reading” I very often suddenly understand things that I haven’t before or suddenly notice some detail that I haven’t before. Also it often suddenly clicks and I see the connection to something that appears only later in the book.

• I immersed myself into the community, subscribing to the r/rust subreddit and reading the weekly “This Week in Rust“.
• I watched a number of YouTube talks by the leading people in the field.
• I coded up an own project: this was a problem that I faced and Rust was a perfect fit. The project was definitely challenging in its goal and I had to use a number of different language features. So it was no easy walk, but the result took use of a broad set of features the language offers. I open-sourced the project and published it as a package to cargo (the Rust package manager).
• I provided Pull Requests for projects which I respect which also use Rust. The feedback was really helpful and I think this is an excellent way to learn a language. One basically gets a mentor and feedback for free. In programming languages there are often idiomatic ways and patterns to do things and this is a handy way to get to know them.

  Also, while fixing bugs for those projects I had to read the source code of bigger Rust projects. This way I saw the idiomatic way to structure/design large projects in this language.

I would say the idea of summarizing the chapters of the book in my own words is the most important idea from the list above. To me, the core idea is that it helps me find my own view on the material. It’s especially important to me that this process is done without any computer, I’m too distracted otherwise.

I have followed the NSA leaks very closely from the beginning. When the leaks started and Snowden firstly revealed himself I had been in Berlin and some tech-savy people involved in the Bitcoin scene told me about the leaks. Following the leaks closely, watching Snowdens first video and the upcoming revelations have definitely influenced me. I curiosly read Glenn Greenwalds book and watched the Laura Poitras movie “Citizenfour” soon after they were available.

In the summer of 2014 I got the possibility of creating a tool which, in my opinion, was a reasonable and necessary step in the direction of making the supervision of authorities more accessible. The german organization netzwerk recherche is a journalists association with a focus on investigative journalism. German journalists have certain extended rights to question official institutions (such as secret services) on data which has been saved about them. This is meant to strengthen their journalistic role within a democracy. These rights, however, are seldomly invoked. Concerning the secret services, a reason could be that there are actually 20 (sic!) intelligence agencies/secret services in Germany — one for each state and four for the entire country. Each service has different requirements for answering requests — one might e.g. require a copy of the passport while another might demand other documents. Additionaly each service has a different address, which is not always easy to find. So to ease these hurdles the idea came about of developing a simple PDF generator, where one could just click the agencies one wants to inquire. A PDF containing the necessary inquiry text, attachment information and the address would then be generated.

A special requirement for such a tool was that the PDF document generation had to happen on the client-side — no server should hold any state. It must not be possible for any person having access to the server to monitor who wants to take use of ones informative rights. Also one should not have to trust the organization, instead one should have the possibility of downloading and deploying the generator himself. Making the source code available (as free software) was a natural conclusion.

I have created this tool and it has now been deployd since July or so. I just didn’t get around to do a proper writeup.
The public instance is available at the netzwerkrecherche.org website. The source code is available via GitHub (under the MIT license).

Das netzwerk recherche ruft Journalisten auf, bei den Geheimdiensten anzufragen, ob diese Daten über sie gespeichert haben. Um das zu vereinfachen, stellen wir einen Generator für die entsprechenden Anträge bereit.

Ziel dieses Projektes ist, in Zeiten der zunehmenden Massenüberwachung den Diensten zu zeigen, dass ihr Handeln von der Öffentlichkeit kritisch beobachtet wird. Die Aufmerksamkeit für das Problemfeld Geheimdienste im demokratischen Rechtsstaat muss erhöht werden.

Insbesondere für investigativ recherchierende Journalisten ist eine Überwachung durch Geheimdienste und eine damit einhergehende Ausforschung ihrer Informanten und Kontakte nicht hinnehmbar.

Anlass des Projektes ist der Fall Andrea Röpke. Beim niedersächsischen Verfassungsschutz wurden offensichtlich rechtswidrig Daten über sie gesammelt. Als sie einen Antrag auf Aktenauskunft stellte, vernichtete die Behörde ihre Akte und behauptete, es gäbe keine Akte über sie. Erst ein Machtwechsel in Hannover offenbarte die Vertuschung – die politische Aufarbeitung läuft bis heute.