About "Programmer as wizard, programmer as engineer"

Programmer as wizard, programmer as engineer

Programmer as wizard, programmer as engineer - Tedinski.com

We have different design goals and constraints when we’re doing one [wizarding] or the other [engineering]. With engineering, we’re tasked with the long term maintenance of a piece of software, and we also become concerned with many lower-priority properties, such as its performance. With wizarding, we’re often best served by designing for ease of throwing code away rather than maintaining it. Well-engineered code is irreplaceable, but well-cast spells should be transient.

In this post, Ted Kaminski talks about two different styles of programming: wizarding and engineering. At first, all startup prototypes/Minimum Viable Products are created as fast as possible: we need to see if our startup idea is viable first. If the business grows, the developers need to transform this prototype to a stable product, able to grow and include new features. Developers need to shift from the wizarding attitude (everything is temporary) to an engineering mindset (long term maintenance, the software needs to work for a long time and is used by many users).

These two "styles" -and the transition between them- reminds me of the Makers vs Menders difference. A Maker is a developer who love to create stuff quickly and are excited by deadlines and time pressures. On the other hand, a Mender likes older codebases, is satisfied by fixing longstanding bugs and making the whole system more stable and more secure.

How do we do switch from one style to the other, though? The author suggests we should write enough tests to be sure everything is working, separate big applications into microservices when it makes sense, and rewrite critical parts of your system in languages that can offer static (or gradual) typing and good tooling.

Separating concerns into their own services -when it makes sense and does not lead to a distributed monolith- can help reason about the system as a whole, and making each service faster and more stable, especially when paired with the right dose of testing.

I'm also a fan of static languages such as Rust (even in the browser!), that can feature good tools such as cargo, and an expressive language that helps developers implement critical systems with fewer errors.

I'd also like to suggest to recognize when the project should move from wizarding to engineering. Failure to understand when the project needs to transition to the engineering state, and plan for it, is probably the most unappreciated problem I encountered in my career so far. If the team can't slow down and fix growing pains, they will only get stronger and mutate into outages, and the team won't stop firefighting production issues. My suggestion is to keep note of small issues, or tasks that are taking longer than expected for technical reasons, and expose them during a retrospective. Having senior engineers in the team also helps, especially if they are experienced with the project's technology stack.


So, I hope you liked the first post of Interesting Links, a new column where I highlight interesting articles, with my own comments and thoughts. If you liked this post, you should totally share it on Twitter, or support me on Ko-fi.

Rust and WebAssembly - what I learnt from porting hrm-interpreter to wasm

Some time ago, I wrote a compiler for a fictional assembly-like language. As this compiler (named hrm-compiler) performs some optimizations on the code, I wanted to know if the optimizations were breaking the compiled code. To test them, I wrote a small interpreter (hrm-interpreter) to run a JSON version of the compiled code, and then embedded the webassembly version in the bare-bone debugger I wrote as a web application.

If you want to try, the best way to get started is the Rust and Webassembly Book , a tutorial that lets you write a Rust crate that can interact with Javascript and publish it to npm. I followed the tutorial, and managed to create a wasm wrapper of hrm-interpreter, named (such fantasy! amaze!) hrm-interpreter-wasm. Let me explain what I learnt while creating it.

Write a different crate to support wasm

If you need to convert Javascript types to Rust ones, I'd suggest to write a specific crate. It's better to think about it as if you're writing a wrapper. This new crate lets you write wasm-specific code, without modifying your library to support this new platform.

Not every library needs to create an ad-hoc crate, though: handlebars-rust offers users a flag to turn off filesystem-based I/O. This may work if your main library does not expose complex types, or you are already handling platform-specific stuff.

Don't force file I/O in the main library

What does an interpreter do?

  1. read a file
  2. convert its content to a list of operations
  3. execute each operation until the program ends or some error is raised
  4. writes the output of the program to a file (or a terminal)

Unfortunately, you cannot read or write files in a wasm environment - you cannot perform I/O! So we need to find another way: let's look at the list again.

  1. read a file and extracts its text as a huge string
  2. convert the content of the string to a list of operations
  3. execute each operation until the program ends or some error is raised
  4. captures the output of the program into a string
  5. writes the output string to a file (or a terminal)

So, we may notice that the focus is now on strings, not files.

As javascript can perform I/O (read files via FileReader, generate new files via FileWriter, or even do HTTP requests!), our wrapper crate may just take a string, pass it to the main crate and return the output string. In my case, I had to create some new methods to sidestep I/O from files.

There are other cases where you may need to modify your crate to port it to wasm: I suggest reading Which Crates Work with Wasm? to learn more.

try to expose as little state as possible - use functions instead

Javascript cannot manipulate complex stuff, such as options, so you cannot export whatever you want. Before wasting several hours trying to understand why you cannot "export" your beautiful struct with your awesome Option<Result<Beatiful<Type>>>, check out wasm-bindgen's Supported Types documentation chapter. Trust me :).

To work around it, in hrm-interpreter-wasm...

  • the main structure has no public fields
  • the internal state is serialized to a JSON-formatted string
  • data from Javascript are also JSON-formatted strings

This is not the best interface, but it works for now. "Now" means that I'll fix it once I integrate some work-in-progress stuff, such as the new Rust-based compiler and sourcemap support... If you can, try to avoid serialization, as it's pretty expensive... and we're using Rust in the browser to run code faster! ;)

Browser-side: use webpack 4!

If you want to test your new crate in a real application, you may want to use webpack. Starting from version 4, Webpack ships an internal wasm loader: it makes integrating wasm-based libraries a breeze!

react-create-app offers a very simple way to get started. If you have an older project, I recommend you to switch to react-scripts ^2.1.1 and webpack ^4.26. I am not sure you need to do this in new projects, but I had to "eject" the webpack configuration (webpack.config.dev.js) and exclude ".wasm" files from the extensions managed by the "file" loader.

// "file" loader makes sure those assets get served by WebpackDevServer.
// When you `import` an asset, you get its (virtual) filename.
// In production, they would get copied to the `build` folder.
// This loader doesn't use a "test" so it will catch all modules
// that fall through the other loaders.
{
// Exclude `js` files to keep "css" loader working as it injects
// its runtime that would otherwise be processed through "file" loader.
// Also exclude `html` and `json` extensions so they get processed
// by webpacks internal loaders.
// PLEASE NOTE THE .wasm REGEX HERE!!!
exclude: [/\.(js|mjs|jsx|ts|tsx)$/, /\.html$/, /\.json$/, /\.wasm$/],
loader: require.resolve('file-loader'),
options: {
    name: 'static/media/[name].[hash:8].[ext]',
  },
}

If you found those notes useful, or you want to suggest something else, please tell me on Twitter , or consider offering me a Ko-fi !

New release: ScreenOn v1.1

A new version of ScreenOn is now available! ScreenOn is an application to measure and evaluate how much are you using your smart phone.

The main change in this version is a revamp of the "distribution view", as seen in the screenshot below. I decided to replace the ugly scatter chart of the previous version: it was hard to read, and the "logarithmic/normal" stuff was annoying and unclear.

This new chart separates usage intervals into several classes. It lets the user understand some patterns of its phone usage: a lot of very short changes may mean they were anxious, and several long sessions (over 5 minutes / 300 seconds) may mean that they spend a lot of time reading on their phone, scrolling timelines/feeds or watching videos.

/images/screenon_v1.1_new_distribution_chart.png

This version also features some bug fixes in the data manipulation functions, and handles the "no data" case better. There is some work left to do - we have to teach new users how to feed data to the application from the Automate applet - but it's a first step nonetheless!

We also have an Hacktoberfest contribution! A new contributor integrated ScreenOn with TravisCI, a platform for continuous integration. It will help us understand if some new code introduces regressions or does not build, catching annoying problems before they have a chance to slip into releases!

Changelog

Introduced by Pull Request #2 (thanks @angelbarrera92):

  • TravisCI integration, based on Docker

Introduced by Pull Request #3:

  • (database) Add index on _screenon_ table
  • (frontend) automatically load data from a week ago to today
  • (frontend) handle "no data" case in distribution view and daily usage view
  • (frontend) write tests for client-side data manipulation functions
  • (frontend) rework distribution view: replace scatter chart with horizontal bar chart to show classes of usage (under 5 seconds, 5-10 seconds, 10-30 seconds, 30 seconds to one minute, 1-5 minutes, over 5 minutes)

If you want to test ScreenOn, read its source code and contribute bugfixes... head over to Github! You can also help this project by offering me a Ko-Fi

How to set timezone in px-vis-timeseries

/images/px-vis-timeseries-example.png

px-vis-timeseries is one of the most used web components in the Predix toolkit set. This must-know component is essential, as it lets chart time-based data and events! Unfortunately, it's not so easy to configure it to show the correct timezone. Let's see how to set it right, then.

Disclaimer: in this post, we are discussing px-vis-timeseries, version 4.0.0. This post may not apply to future versions of the components discussed here.

Gotcha #1: the default x-axis timezone is UTC

The default timezone of the X axis is UTC. While it may be the norm in engineering disciplines or technical users, it may confuse normal users: they are used to the timezone they live in, and UTC may confuse them (if they're even aware of it!)

To see the time in the browser's timezone, you need to set x-axis-type="timeLocal". By default, it's set to x-axis-type="time".

<px-vis-timeseries
    x-axis-type="timeLocal"
    chart-data="[[chartData]]"
    height="450"
    show-tooltip
    hide-register
    series-config="[[seriesConfig]]">
</px-vis-timeseries>

Gotcha #2: tooltip and register timezones can be configured separately!

The tooltip and the register are independent components, and px-vis-timeseries does not configure them automatically, nor offers a single configuration point.

While this design choice makes the parent implementation simpler and allows for more flexibility, the users have to remember to configure each sub-component.

The configuration can be provided by setting tooltip-config and register-config, to set the tooltip and the register.

<px-vis-timeseries
    tooltip-config="[[timezoneConfig]]"
    register-config="[[timezoneConfig]]"
    chart-data="[[chartData]]"
    height="450"
    show-tooltip
    hide-register
    series-config="[[seriesConfig]]">
</px-vis-timeseries>

To set the timezone, you can either use a timezone string (such as "Europe/Rome") or use the browser-provided Intl global object to retrieve the current timezone.

this.timezoneConfig = {
    // timezone: "Europe/Rome"
    timezone: Intl.DateTimeFormat().resolvedOptions().timeZone
};

Proof of Concept

I've prepared a minimal proof of concept to experiment with. This PoC is hosted on wintermade.it: it lets you test the code for both gotchas, all in one component. If you want to modify it, there is a live-editable version on Glitch, a code editing service that lets you host your experiments/small websites. In both cases, you need to enable javascript to see the example :D.

Did these notes save your day? Offer me a Ko-Fi ! Or drop me a tweet if you find errors or typos!

ScreenOn: Measuring phone usage with Automate and Python

Some time ago, I was a bit nervous, so I started to browse twitter on my phone. I didn't start using it because I received a notification or a friend sent me an interesting post, but to divert my attention to the task I should have been working on. After some minutes spent scrolling the timeline, it seemed to me that I "woke up" and understood what I was doing. I understood that it was not the first time it happened, but it was just the last occurrence of a long series of "procrastination attack".

I needed to stop wasting my time reading rants of randos on the internet. Before working on a solution, though, I wanted to know how much time I actually was spending on the phone.

So, I started working, and created ScreenOn. The solution is composed of three parts:

  1. an Automate flow to read screen status and save every change to a local sqlite3 database
  2. an Automate flow to push screen changes from sqlite3 to a Python web application
  3. a web application (Python+javascript) to store and see visualizations/stats

The code and the applet are available on Github.

Automate flows

/images/screenon_automate_flow.jpg

Automate is an Android application that lets you develop your own programs via a visual language. To create a program, you add, configure and connect blocks, and each block corresponds to a single action or condition. In a single program, you can create different flows, that are executed in parallel.

As you can see, there are different flows:

  1. Start: detect screen status changes (when it turns on or off), and save them
    to an sqlite database
  2. Backup: save screen changes records to a Python webservice on the same network
  3. Config: configures the database

When the screen turns on or off, the applet saves the current time as a unix timestamp, and the status of the screen (1 = on, 0 = off).

Web application

ScreenOn has a web application, with a Polymer frontend The data are provided by the Python webservice that is also invoked by the Backup Automate flow.

The web application lets us interact with some charts, so let's see them!

/images/screenon_usage_per_day.png

Usage per Day is the default view: for each day in the selected period, it shows the time spent on the phone and how many times the screen was turned on and off that day.

/images/screenon_screen_changes.png

Screen Changes is a cumulative chart of the phone usage time in the selected day, and how many changes triggered the changes. The "screen status changes" line should help users understand short-timed phone usages.

/images/screenon_distribution.png

Distribution of Usage aims to highlight the duration of the usage, using a xy-scatter chart. On the x-axis, we can find the interval of time when the phone screen was turned on. On the y-axis, we can find how many times that interval "happened".

There are two "modes": the default one (Logarithmic) aims to highlight the short-timed usages, by applying the log operation to the time intervals. This operation compresses the larger values, while leaving more space to the smaller values.

The other one, Normal does not apply any compression to the data, and is better suited when we need to analyze longer time intervals.

Analysis

/images/screenon_analysis_usage_april_to_june.png

This is the daily usage chart, from April to June 2018. We can see that the behavior is more or less the same every week: on work days (from Monday to Friday) I use the phone less than during the week-end (Saturday and Sunday). The only exception is 25th of April, that is a national holiday in Italy.

/images/screenon_analysis_screenchanges_20180606.png

This is an example of a Wednesday, a normal work day. There are some things I can show you:

  • My work hours are from 9 AM to 6 PM, so you can see that I don't use the phone very much during that time. The increase near 2 PM is my lunch break: I scroll Twitter, check my private email account, and show small videos to friends.
  • At the time, I was using Youtube as my music player when on the road. It was before the Premium stuff, so you had to keep your screen on even if the phone was in the pocket and you weren't looking at it.
  • Every Wednesday evening, a streamer I follow on Twitch.tv streams a lore walk-through of the whole Metal Gear Solid saga, and every stream is ~3 hours long...
/images/screenon_analysis_screenchanges_20180617_masszoom.png

This screenshot will help us see how the Screen Changes chart looks like when there are a lot of small changes.

As we can see, between 11 AM and 1 PM I used the phone several times in a row for short period of times. We can understand that by seeing that the black line (the time the screen was turned on) grows slowly, but the orange line (the number of changes) grows very fast in the same interval.

Next Steps

No software is ever finished. ScreenOn can benefit from additional work. Some of the next steps are purely technical, others aim to capture more data in order to better understand why the user is using their phone and how they're spending their time.

1) CSV Exports

Sure, an user could just stop the web application, dump the database and restart the application, but why should they do that? What if you want to perform some other kind of analysis on the data? I want to empower the users and let them create their own analyses.

2) Handle timezones better

Right now, the Automate flow that records the screen changes only records an unix timestamp. It may works in a narrow set of conditions, such as the phone never changes the timezone and the browser is in the same timezone of the phone. ScreenOn can do better: the user should be able to set a preferred timezone, or set the chart to use data's timezone!

3) Twitch/Youtube is not Twitter/Mastodon scrolling!

In the analysis, we saw that the phone was used for long period of times during evenings and nights. Personally, I prefer to turn on Youtube or Twitch on my mobile device, either as TV-like device or to actively see my favourite content creators on those platforms. I don't know how to implement it yet, but it should not be very difficult.

4) Ask why you are using the phone, if using it for more that X minutes or X times in an hour

There is a reason if someone is using their phone so much. Maybe they're stuck waiting in traffic, or waiting for the build to finish. What if they just could be anxious or don't want to think about something that makes them nervous?

If this is the case, "setting a goal to use the phone less" is just treating the symptoms instead of curing the disease: the phone became an anxiety-relief device, nervous people will use something else instead. A tracker like ScreenOn cannot register context, so the Automate applet may periodically ask the user how they feel and why are they using the phone.

If you want to propose a new kind of chart, or point out some errors and typos, don't hesitate and let me know! Contact me on Twitter. If you liked this article, why don't you offer me a Ko-Fi ?