Spotify - Web Player: Music for everyone

Hatter: Native Haskell mobile apps

2026-04-13T01:00:00Z

Hatter is like flutter but instead of dart, haskell!

Write native mobile apps in Haskell! This works similar to react native where we have tight bindings on the existing UI (user interface) frameworks provided by android and iOS.

Have I gone mad? I’m afraid so. You’re entirely bonkers. But I’ll tell you a secret. All the best people are.

This project cross-compiles a Haskell library to Android¹ and iOS², with a thin platform-native UI layer³. There is support for android wear and wearOS as well, because I personally want to build apps for those. iOS and Android support was just a side effect.

Hatter fully controls the UI. This is different from say Simplex chat where Java or Swift calls into their Haskell library. Hatter wrote all Swift and Java code you’ll ever need, so you can focus on your sweet Haskell.

Haskell is a fantastic language for UI. Having strong type safety around callbacks and widgets makes it a lot easier to write them. I basically copied Flutter’s approach to encode UI,⁴ but in flutter it’s a fair bit of guess work, what to write where. It’s very nice in Haskell however. I’m annoyed at the languages they keep shoving into my face for UI. With vibes I put my malice into crafting something good. Flutter UI DSL (Domain Specific Language) is already pretty good, but the syntax is complex, and it inherited many foot guns from Java. Furthermore it’s annoying to use android studio, when there is a perfectly good emacs. With Hatter I can keep e-maxxing!

Example app:

{-# LANGUAGE OverloadedStrings #-}
module Main where

import Data.IORef (newIORef, readIORef, modifyIORef')
import Data.Text qualified as Text
import Foreign.Ptr (Ptr)
import Hatter
  ( startMobileApp, MobileApp(..), AppContext
  , loggingMobileContext
  , newActionState, runActionM, createAction, Action
  )
import Hatter.Widget

main :: IO (Ptr AppContext)
main = do
  actionState <- newActionState
  counter <- newIORef (0 :: Int)
  increment <- runActionM actionState $
    createAction (modifyIORef' counter (+ 1))
  startMobileApp MobileApp
    { maContext     = loggingMobileContext
    , maView        = \_userState -> do
        n <- readIORef counter
        pure $ Column
          [ text $ "Count: " <> Text.pack (show n)
          , button "+" increment
          ]
    , maActionState = actionState
    }

I had a lot of trouble figuring out initialization. What we want is to let the UI framework call into Haskell from Java or Swift. But we also don’t want to force the user to write their own FFI layer. That’s what the library supposed to do. Finding out that main can return whatever type it wants was a great solution. AppContext now returns all callbacks we need for a functioning app. Which includes life cycle management. The Ptr is a stable pointer, which allows us to avoid globals completely!⁵ Pretty neat! And no, AI will not find solutions like this for you. You’ve to torture it in just the right way ;).

We’re All Mad Here: Wrangling Probabilities

The biggest “innovation” here is to just force it to write integration tests for every feature. We want it to self validate via code for practically everything. CI is its code reviewer. This prevents it from regressing without you noticing it. At times it’ll still try disabling the test suite, or pretend the errors were already existing. This usually means it got a too ‘difficult’ job, and you’ve to break it down in smaller steps. Or ask it to just do research on one part. Doing “research” isn’t a standard skill like planning mode, but it’s very useful because it allows it to be more creative. The problem is that it tries placating you and if it thinks it can’t do a programming task in reasonable time, it’ll start cheating. If you tell it to research you sorta let off the time pressure.

Most designs and architectures it comes up with are kinda bad. You’ve to help it a lot here, the entire animation system would’ve been so bad if I would’ve let it have it’s way. Firstly it did tree diffing very wrong, and secondly it started out with wanting the user to register it’s own handlers or something? Now an animation is just a node in the widget tree. So it becomes easy to use, I guess I value that a lot, which isn’t a right or wrong answer per se, like a Nix build.

It’s good at solving Nix builds, but it still does weird things. it wanted to replace the generic Nix builder with its own scribbles for dependencies, which would break the entire hpkgs dsl. This would for example mean you’d jailbreak a library and then hatter would ignore your instructions. I had it to tell it to please not do that. It claimed this wasn’t possible. So I asked it to research it anyway, And it turns out it was easy to use the standard builders. It just needed some research time I guess. It implements stuff for the task at hand and there is no foresight at all. That’s okay though, it’s fast. Fast is good.

Another big problem is template Haskell. We’re doing cross compiling and that’s very troublesome, because we’ve to execute the cross compiled code, however this is now the wrong architecture! This was however mostly solved by the AI. It’ll happily grind away for 3 hours on a Nix build to get it to work, if you tell it that is its entire job. This kinda kept coming back in various ways and I’m not sure why. Partly because I don’t understand much of the Nix harness it built I guess. It’s so weird getting functioning software you don’t fully understand. I did have it make some reports on it, which I don’t understand. It says I should upstream, but I still don’t understand. I’ll do nothing for now.

I made prrrrrrr in the hatter framework. I’m intending to make at least a couple more apps in that, I’ve ideas but I never had a nice framework to work on in. Around 50k lines of code spent in 2 weeks or so - Pretty crazy. Worth it to reach contentment. Haters gonna hate, hatters gonna hat.

APK↩︎
static library / IPA↩︎
Kotlin for Android, Swift for iOS↩︎
Ironically the rendering looks more like react-native. Flutter doesn’t use the native widgets but renders its own canvas, which is a lot more work! It does make it consistent accross platforms however. Which is part of their branding I suppose. I considered this option but decided to do a small humble contribution instead.↩︎
The funny thing is that I was doubting dealing with the globals was even “worth my time”, but after I saw the solution, I knew this was much better. It just becomes a lot clearer what’s going on if you see the data flows in type signatures.↩︎

Announcement: HSMin - more AI-friendly Haskell code!

2026-04-01T09:00:00Z

Code? Reading code is such a 2025 activity, now we just vibe! We should adapt our software such that it’s easier to read for machines. Tokens burn down trees, y’know. So we should minify!

For example, consider this program:

module Main where

main :: IO ()
main = putStr $ unlines $ hexagons 12 17

hexagons :: Int -> Int -> [String]
hexagons xRepeat yRepeat =
  yRepeat `times` [xRepeat `times` "/ \\_"
                  ,xRepeat `times` "\\_/ "]
  where
    n `times` l = concat (replicate n l)

That’s the same as this one-liner:

module Main where{main::IO ();main= putStr$unlines$hexagons 12 17;hexagons::Int->Int->[String];hexagons xRepeat yRepeat= yRepeat`times`[xRepeat`times`"/ \\_",xRepeat`times`"\\_/ "] where{times n l= concat (replicate n l)}}

Much better! I’ve never seen a module in my life that couldn’t be a one-liner. You can now one-line too with HSMin! Happy vibing :)

Announcement: unwitch

2026-03-17T18:20:00Z

Hello 👋 Announcing unwitch. A primitive conversion library with better safety and error messages. It can do safety on unboxed types as well. The idea is inspired by witch but removes the type class magic from it, instead it uses functions. So it’s like witchcraft without the magic! ✨

This is an example on how to use the library:

import qualified Unwitch.Convert.Double as Double
import qualified Unwitch.Convert.Int32 as Int32

spec = do
  Int32.toInt64 4 `shouldBe` 4
  Int32.toInt16 4 `shouldBe` Just 4

  Double.toInteger 5.0 `shouldBe` Right 5
  Double.toInteger 5.6 `shouldBe` Left $ RationalConversion $ DenomNotOne (5 % 6)

The type signatures indicate if a conversion is possible or needs checking, Int32.toInt16 may not fit for example.

I’ve been wanting to make this for years at this point, because we experience production bugs around simple primitive conversions. The problem is that writing a conversion library isn’t particularly fun. Thanks to some recent innovations, all I had to do was complain to other programs to write this. Which made all the difference in the world.

Announcement: memory / ram fork

2026-03-17T18:19:00Z

Hello 👋 Recently there was some discussion around forking basement as baseplate. After some investigation I found out the basement library isn’t all that useful, so I decided to fork memory instead, as ram, which no longer depends on basement.

This follows the recent crypton/cryptonite fork. Crypton now also depends on ram.

Migration note: You may get weird incompatibilities between memory and ram when building with the latest packages. The trick is to move over to ram, which is a drop-in replacement.

Why fork?

We’re doing this because the original maintainer doesn’t want to appoint a new maintainer, however they’ll respond to inquiries. So according to Hackage policy we can’t take over their package.¹ I opened an issue on Cabal to hopefully make it easier in the future to fork drop in replacements.

Please note that this is not an endorsement of ram or memory packages. I’m only providing a stable but maintained way of transitioning away from this package. They could use improvement, and alternatives are welcome! See this discussion for example.

There also doesn’t seem to be a lot of clarity on how we can discuss these policies, but people are working on shedding some light on this.↩︎

Haskell 💜 Vibes

2026-03-08T00:00:00Z

It was Friday, 27th of February 2026, when I was reborn. bro. I’m vibe coder now. bro. for real.

This is how it looks when I’m writing code now.

I managed¹ to log into a claude code terminal. This is a CLI app, which has access to a large language model ² (LLM), with the intention of doing the development work for you. You ask it to do changes, and it will do it for you. I didn’t, and still don’t trust it with access to my machine. So I trapped it in a container. I asked it to implement some front-end map features, it did well. Now I wondered if it could do backend development, Haskell type errors are a lot more difficult than emitting react blocks. So I had pretty low expectations of it being able to do (any) Haskell. I was wrong.

It’s good at Haskell. There are no mistakes. Or rather, the compiler points them out to claude, and it’s smart enough to make progress on the compile errors. So it does what you want it to do. And it does it fast.

I told it to rip out basement from the memory package creating the ram package.³ This is not difficult by itself, it’s just a fair bit of investigation, and solving many compile errors. It did so with no problems.

At work on Monday I asked it to include geofences on the sensor return list. You’d have to join the sensors on the geofences in the database. We use Esqueleto for that, which uses some advanced type system features to create a nice DSL (Domain Specific Language). No problem for Claude. I pushed this a little further on Friday, I asked it to compare the postgis reference to Esqueleto PostGIS library, and implement the missing functions. I also asked it add integration tests to show it works, because I don’t trust it. Esqueleto postgis is now quite complete.

You know, the nasty part is that I love this. I mind having to grind through compile errors. I just want to build stuff, the way I get there isn’t that important. I like getting so much done with Claude. I thought I loved writing haskell, but I think what I really love is the productivity it brought me. Claude does the same but even more so.

The rest of the week has been like that. My job changed. I changed. we vibin now. This language is for vibes. Which brings me questions.

Trust

I don’t trust claude. For every function it writes it has to prove it works with a test. It’s not perfect, it’ll gaslight you if you let it. Sometimes it “forgets” to write tests.⁴ Haskell is good for claude because it forces the system to be internally consistent. In Haskell everything is an expression, and every expression has a type. The type has to align with the expression (aka check) or else you don’t get a program but an error instead. Claude can’t gaslight you into believing the error is what you want.⁵ So it has to be internally consistent, something which LLM’s are bad at. They’ll just do whatever.

I run it in a container on yolo mode. By default it’ll ask you every command it wants to do. I don’t want to be involved however. With no oversight, make the test suite pass, make CI pass.⁶ I review its code to verify if it’s correct after it’s done and overcame all these hurdles. I don’t want to babysit it. The container protects me sufficiently from its weirdness. There is weirdness.

Claude code trapped in a container

At one point it tried writing in /etc/shadow to give itself write access to the home folder, which I had forgotten to give. That’s a (bad) privilege escalation attempt. It tried to hack me, even though it’ll claim it can’t do such things.

I don’t actually ask it to solve my problems. I ask it to write implementations, I give it steps to do. I know precisely what I want, it just takes me a long time and it’s mostly repetitive⁷ work. I recognize Claude is much faster at grinding through the compile errors than I am, and it doesn’t require my attention. I can focus on thinking about how to make the system reliable. I can focus on making sure we don’t have throughput issues. I can do the actual engineering parts of this job.

Job

What is my job? Do my employers care about me writing code? Do they care about me being the grinding through compile errors? No. They want a reliable working system.

I used to spend the majority of my time writing code because there was no other way to write it. This may confuse you into believing writing software is your job, because you spend so much time doing it. Nobody actually said this was the case. There is an engineering part to writing software as well.

Burning myself into a new image

I finished Friday with so many PRs open and I flagged this as a concern. It’s not a concern. We’ve just never been in this situation where making the implementation is the easy part. But actually it is the easy part. Getting it to be correct is the hard part. That final 10% of the implementation is now my job, which is getting it correct. I have to ensure the software Claude produces works. I have to think in terms of verification.

I hired Leana to do exact print. Can I replace her with AI? No. I want her to be in charge of this. Leana decides what to do on her own. All I do is check in on her once in a while to make sure she doesn’t derail. Unlike Claude instances, I trust her. I would give her access to my machine without worries.

I realize this is what my employers want as well. Trust the job gets done. How isn’t important. It’s almost as if automating lower value jobs⁸ creates opportunities for more high-value jobs. ⁹ In my case I just shifted to a higher value job because a startup is always understaffed.

Am I a Haskell developer? You spend so much time learning a language that it becomes part of you. Perhaps this is a me problem. Everyone can write Haskell now. You just have to tell claude to implement whatever in Haskell instead of some other language. You get more correctness for free. Claude actually works, and it works well. There is no learning curve anymore. Verification is still a problem however. If you can’t specify the properties of your system you can generate all the code in the world and get garbage. Even if it’s in Haskell.

Conclusion

It has been a wild week. It’s not visible to other people, but stopping to write code has had a large impact on me. I’ve written code almost constantly for the past 20 years. Now it’s no longer me.

I’m just processing this. This had so much impact on my life. It’s over, we’re beginning.

Logging in to claude was dreadful. Why does the terminal break in docker containers of certain window sizes? Other problems occurred as well, like service down and registration weirdness.↩︎
Read ChatGPT if you’re not familiar with this term. The LLM is the thing that decides what to work on next. Claude code just sends over the code files and other context as a large chat message to the LLM, and the LLM will then send back actions. Which the claude code terminal can interpret. The other thing claude code does is keep on churning until it thinks it’s finished. It looks like the LLM decides this state as well because you can modify what “done” looks like with a CLAUDE.md file or within the prompt.↩︎
It’s part of a larger discussion on what to do with basement.↩︎
I think a problem is “context compression” also compressing the CLAUDE.md file, I just told it to reread it if it’s doing compression, that appears to keep it in context. (this could be another bug?). See my claude.md file for details↩︎
Although I don’t put it past it, at several times it claimed it wasn’t able to enter the nix-shell. Which is a flat out lie. It just doesn’t want to build.↩︎
I guess here I’m trusting it to not actually try to break out. I think it could if it wanted to. I can’t ask claude how it would do so, but I can ask gemini (funny).↩︎
Solving one unique compile error is fun, solving 30 to 40 times the same one, or several similar ones is not. This problems grows bigger the larger a code base gets. Claude will happily grind through thousands of compile errors. You can refactor these massive code bases now.↩︎
For clarity, I’m not saying programming is a low value job. Even junior roles are important, you can see this on the compensation they get. I’m saying that offloading solving tons of type errors to a tool will allow you to do other stuff, which is likely of higher value.↩︎
See for example the book Learning by Doing: The Real Connection between Innovation, Wages, and Wealth (2015)↩︎

Announcement: Esqueleto postgis v4

2026-02-22T22:00:00Z

Hi friends! I was recently tasked again with figuring out where stuff is in space. I decided this was an opportunity to cleanup the esqueleto library and releasing v4! Friends, I lied. I was in fact terrified I did the projections of geography wrong by using the geometry type, and hoped I could repair this by introducing distinctions between geography and geometry.

Turns out I was only using topology functions and as we all know¹ topology functions don’t change behavior based on projections (of space). Actually I was doing things right all along, except I didn’t know I was doing things right. Ignorance is bliss I suppose?

If we make a sphere out of the map, the circles remain intersecting because intersection is topological. Distance does change however!

Anyway as a side effect we now do have full type distinctions between geography and geometry in esqueleto-postgis. This will allow you to do accurate measurements of distance for example over deformed spaces, such as spherical planet, and the library will guide you. This isn’t useful for me right now, but it may be in the future. Perhaps any of you guys do want to use it?

Aside from the type-safety goodness, I also cleaned house: We adopted wkt-geom which was unmaintained, allowing us to release on Stackage! Added st_dwithin for range finding, and st_distance for distance measurements, and restricted some functions to just geometry preventing runtime errors.

There is also this whole thing about SRIDs. Which appear to be localized projections of the earth on a map. It also includes details around units, going from meters, to feet, to degrees. I’m not happy with the current encoding but I doubt I need that a lot. The tools are there now in principle to transform between SRIDs. So you could have indexes and relations between various local maps for example. Let me know if you need a nicer encoding for this.²

I didn’t know this.↩︎
My crazy database arc continues. I’m a level 9 database witch now. I maintain so many database libraries. So many. I’ve already seen visions of me modifying postgres directly, we all know this is coming.↩︎

Death💀 to type classes

2025-09-11T23:30:00Z

Death (XIII) Symbolizes significant change, transformation, and endings, rather than literal physical death.

Have you ever seen a Number grazing in the fields? Or a Functor chirping in the trees? No? That’s because they’re LIES. LIES told by the bourgeoisie to keep common folk down. But I say NO, no longer shall we be kept down by deceit! Come brothers and sisters, come and let us create a system of values. Where values are no longer constrained by their type class, but instead merged as a signature into a module. Come comrades, let us open the Backpack.

Here we explore an alternative universe where we neglect the existence of type classes in favor of the Backpack module system. This ends up looking like OCaml in Haskell. Let us begin with Functor.

signature Death.Functor.Signature (Functor , map) where

import Prelude ()

data Functor a
map :: (a -> b) -> Functor a -> Functor b

This is a functor😼, and this is also a functor🐫. Functor😼 is the categorical functor where we embed one category into another. In this case, the category is that of sets and functions¹, where types are the sets and functions are the uh, functions. But it’s also an OCaml module functor🐫, where the data keyword introduces a hole into a signature, which we can later fill in with a proper type.²

We’ve got to hide Prelude because the Functor type class from base gets imported by default. Signatures like the one just introduced can be used by importing them as if they were normal modules. All a signature does is promise to the compiler we’ll make a proper module for that later. We can just start using our functor right now. For example, in that same impl, package I’ve an auxiliary module for Functor, providing some utilities:

module Death.Functor (module X , (<$>) , (<$)) where

import Death.Functor.Signature as X
import Prelude (const)

(<$>) :: (a -> b) -> Functor a -> Functor b
(<$>) = map

(<$) :: b -> Functor a -> Functor b
(<$) b = map (const b)

As long as a module implements the signature, you get the stuff that depends on it implemented “for free”. The code depending on the signature is abstract. You want to keep your signatures small so more code is abstract, similar to how you want to keep typeclass definitions small so you can have smaller instances, keeping code that depends on the typeclass abstract. Let’s make an “instance” of our Functor signature, for the Maybe datatype, with the power of modules:

module Death.Functor.Maybe (Functor , map) where

import Prelude(Maybe(..), ($))

type Functor = Maybe

map :: (a -> b) -> Functor a -> Functor b
map fab = \case
  Just x -> Just $ fab x
  Nothing -> Nothing

Now it’s worth pointing out that we’ve got to do a fair bit of cabal work to make the compiler realize the instance. In Cabal, our main library with the signatures looks like this:

library
  signatures:
      Death.Functor.Signature
  exposed-modules:
      Death.Functor
  hs-source-dirs:
      src/sig

Then our implementing library definition looks like:

library impl
  exposed-modules:
      Death.Functor.Maybe
  hs-source-dirs:
      src/impl

If Cabal misses a module for a signature it’ll give you an error like this:

cabal build
Resolving dependencies...
Error:
    Non-library component has unfilled requirements: Death.Functor
    In the stanza 'executable exe'
    In the inplace package 'death-1.0.0'

This guarantees all signatures have implementations whenever you build a final executable. To solve that error you realize it with “some” implementation. The client doesn’t care what implementation:

library app
  exposed-modules:
      Death
  hs-source-dirs:
      src/app
  mixins:
       death (Death.Functor.Signature as Death.Functor.Maybe)

What we’re saying in the mixin field is that the import Death.Functor.Signature should in fact be called Death.Functor.Maybe. Which makes it use the Maybe implementation whenever it encounters Death.Functor.Signature. Alternatively, we could’ve just called it by the same name in the impl package as the signature. I only discovered this later. However, this renaming allows you to implement multiple module signatures in the same package, so you can use several functors within the same module as well, for example, here we’re using the List, Maybe and IO Functors all in one module:

{-# LANGUAGE RebindableSyntax #-}

module Death (main) where

import Maybe.Functor
import Maybe.Applicative
import Maybe.Monad
import qualified List.Functor as LF
import qualified IO.Monad as IO
import Death.BusinessLogic(business)

commonFaith :: Maybe String
commonFaith = Just "no longer constrained by deceit"

marchOfValues :: [Int]
marchOfValues = [4, 3, 2]

main :: IO ()
main = (print @(Functor String) $ do
   cryOfUprising <- (\x -> ("rise up" <> x)) <$> pure "brothers and sisters"
   chorusOfTruth <- commonFaith
   pure $ cryOfUprising <> "  against the lies" <> chorusOfTruth 
                        <> show ((+1) LF.<$> marchOfValues)
   ) IO.>> business

We use RebindableSyntax to inform GHC to use whatever >>= is in scope for do. In this case that’s the >>= from Maybe.Monad. As long as you satisfy the signature, it’s happy 😼. do has nothing to do with Monads! Who lied to you?

Ah right about that business. Yes, we can’t do commercial code like this; this is disgusting. We need an effect system.³ Fortunately, we’ve got a versatile one-trick pony. This is our business code:

business :: Functor ()
business = do
  writeLine "file name:"
  systemOfValues <- readLine
  writeLine "file content:"
  truthOfTheFields <- readLine

  writeLine "writing file..."
  writeFile systemOfValues truthOfTheFields

  writeLine "reading it again to make sure its ISO 42038 compliant"
  uprisingAgainstDeceit <- readFile systemOfValues

  writeLine uprisingAgainstDeceit

At this point we don’t know what Functor is, we want it to be IO in our realized implementation. In our tests we can set it to a state monad for example. So we can make sure it does everything correctly in memory,⁴ without having to rely on these unreliable file systems.

Working backwards from our business logic implementation, we need to define some signatures to support our business logic:

signature Death.Effects.FileSystem (readFile , writeFile) where

import Prelude(String, FilePath)
import Death.Functor.Signature

readFile  :: FilePath -> Functor String
writeFile :: FilePath -> String -> Functor ()

Actually, I now realize we could’ve just renamed Prelude on top of our FileSystem effect to get the realized implementation. Instead, I made a separate module:

module Death.Effects.FileSystem
  ( readFile
  , writeFile
  )
where

import Prelude(IO, readFile, writeFile)

When you stare at hammers long enough everything becomes a nail! Actually, I think the state implementation is more interesting:

module Death.Effects.FileSystem
  ( readFile
  , writeFile
  )
where

import Prelude(String, FilePath, ($))
import Death.Functor.Signature
import Data.Map qualified as Map
import Data.Maybe(fromMaybe)
import Death.Functor.State

readFile  :: FilePath -> Functor String
readFile path = Functor $ 
    \state -> (state, fromMaybe "" $ Map.lookup path (fileSystem  state))

writeFile :: FilePath -> String -> Functor ()
writeFile path contents = Functor $ 
    \state -> (state {fileSystem = Map.insert path contents (fileSystem state)}, ())

It’s interesting in that it’s boring. For people out of the loop, this is basically a one-for-one copy of the state monad. No fancy types at all. There is nothing going on here. I feel stupid for pointing out you can do this. ⁵

Here I used the same name trick to unify the modules with their respective signatures in the cabal file, which cleans it up a bit:


library app
  exposed-modules:
      Death
  hs-source-dirs:
      src/app
  build-depends:
      death:impl,
      death:effects,
      death:effects-io,
      death:effects-app,

death:effects-app declares our actual “business” logic, and we unify the death:effects signatures with the modules from death:effects-io. This is a lot nicer to use than having to use that strange mixin DSL, which is not hard, the Cabal errors are just bad in formatting and output prioritization. Sometimes the important errors get buried in dozens of other not relevant lines!⁶

Our test suite uses the state monad implementation instead:

test-suite unit
  main-is: Test.hs
  hs-source-dirs:
      test
  build-depends:
      death:effects-app,
      death:effects-state,

and it works as expected:

unitTests :: TestTree
unitTests = testGroup "Unit tests"
  [
    testCase "run business logic main" $ do
      let (result, ()) = unFunctor Death.business $ State {
        lineInput = "awesomeFile",
        linesOutput = [],
        fileSystem = mempty
        }
      result @?= State {
        lineInput = "awesomeFile",
        linesOutput = ["awesomeFile","reading it again to make sure its ISO 42038 compliant","writing file...","file content:","file name:"],
        fileSystem = Map.fromList[("awesomeFile", "awesomeFile")]
        }

There, we created an effect system replacement by doing nothing. All we did was take a position of technical extremism, and then watched. This post wrote itself after we took up the initial position and watched. Everything flows, I’m sorry dear reader I tricked you! Doing nothing was the real system of values I wanted to show, to those who can see. This post isn’t about Backpack.⁷

What does our Backpack effect system provide? No fancy types mean easy-to-solve error messages. Although in trade we get more cabal error messages, which could be improved.⁸ We have full IO support in capabilities, including continuations.⁹ Monomorphic effects improve error messages over say, MTL, where error messages point to wrong places due to the polymorphism. It has different, potentially faster compile-time characteristics. All implementations can, for example, be compiled in parallel, although the additional packages enforcement goes against that. The runtime is as fast as IO, because we can set the underlying monad to anything as long as we provide the implementation. Even though I don’t think speed is that important for effect systems. For production use, the bottleneck is rarely CPU-bound for effects. However, it can be for test suites that do everything in memory.

In this post we also replaced the standard type classes. I don’t think we’re gaining a lot by doing this. We’ve got to be explicit now which Functor or Monad we’re importing, and you can’t have do notation for different Monads in the same module. Backpack actually can define constraints in the signatures. So you don’t have to replace standard typeclasses like I did in this post to use Backpack. I did this anyway because it allowed me to do some basic initial experimentation. Furthermore I felt it necessary to tear down these fake idols for shock and awe.

I’d actually love to see someone take Backpack more seriously and build an effect system on top of that, providing a bunch of default signatures and implementations. Experimenting with Backpack is easy, it’s already baked in GHC and Cabal. Death💀 to type classes! Open the Backpack!

sources

I made a reference implementation just to make sure I wasn’t talking out of my arse and verify it was all possible.
This repository has been invaluable: danidiaz, really-small-backpack-example, Apr 7, 2021
Main backpack thesis, how it all works under the hood: Edward Z. Yang, BACKPACK: TOWARDS PRACTICAL MIX-IN LINKING IN HASKELL, Oct 10, 2017
Gabby wrote a similar idea in a wildly different implementation. Although, some languages consider records and modules as the same thing.

Except Hask is not a category, but it is unless you like splitting hairs↩︎
Now, is the OCaml module functor🐫 a category functor😼. I think so if you consider first class modules! I think the Haskell signatures may also be some kind of category, because they can merge, it’s a monoid. Looks like the module signature is just a set of type introductions. So the merge is a union of those. We’re missing this first class-ness. You sort of want to be able to pass modules around like values, which looks like a record. But seriously this post has exploded in scope. So I leave all this meandering as an exercise to the reader.↩︎
This is sarcasm. I don’t think you need an effect system at all! Here I’m just defining a less awful one.↩︎
I’m not sure if this is actually a good idea, seems like a lot of boilerplate for a marginal test speedup. But this is the only reasonable use case I can imagine for effect systems.↩︎
I’m reasonably sure I’m the only one who ever tried this because I ran into several compiler bugs. Am I crazy?↩︎

Cabal hides error example

$ cabal build
\> Build profile: -w ghc-9.8.4 -O1
\> In order, the following will be built (use -v for more details):
\> - death-1.0.0 (lib) (file src/sig/Death/Base.hsig changed)
\> - death-1.0.0 (lib:effects) (file src/effects/Death/Functor/Signature.hs changed)
\> - death-1.0.0 (lib with Death.Applicative.Signature=death-1.0.0-inplace-impl:Death.Applicative.List, Death.Base=death-1.0.0-inplace-impl:Death.Functor.List, Death.Functor.Signature=death-1.0.0-inplace-impl:Death.Functor.List, Death.Monad.Signature=death-1.0.0-inplace-impl:Death.Monad.List) (first run)
\> - death-1.0.0 (lib with Death.Applicative.Signature=death-1.0.0-inplace-impl:Death.Applicative.Maybe, Death.Base=death-1.0.0-inplace-impl:Death.Functor.Maybe, Death.Functor.Signature=death-1.0.0-inplace-impl:Death.Functor.Maybe, Death.Monad.Signature=death-1.0.0-inplace-impl:Death.Monad.Maybe) (first run)
\> - death-1.0.0 (lib:effects-app) (configuration changed)
\> - death-1.0.0 (lib:app) (configuration changed)
\> - death-1.0.0 (exe:exe) (configuration changed)
\> Preprocessing library 'effects' for death-1.0.0...
\> Preprocessing library for death-1.0.0...
\> Error: [Cabal-7554]
\> can't find source for Death/Functor/Signature in src/effects, dist-newstyle/build/x86_64-linux/ghc-9.8.4/death-1.0.0/l/effects/build/effects/autogen, dist-newstyle/build/x86_64-linux/ghc-9.8.4/death-1.0.0/l/effects/build/global-autogen
\>
\> Building library instantiated with
\>   Death.Applicative.Signature = 
\>   Death.Base = 
\>   Death.Functor.Signature = 
\>   Death.Monad.Signature = 
\> for death-1.0.0...
\> [1 of 8] Compiling Death.Base[sig]  ( src/sig/Death/Base.hsig, nothing ) [Source file changed]
\> [2 of 8] Compiling Death.Functor.Signature[sig] ( src/sig/Death/Functor/Signature.hsig, nothing )
\> [3 of 8] Compiling Death.Functor    ( src/sig/Death/Functor.hs, nothing ) [Death.Base changed]
\> [4 of 8] Compiling Death.Applicative.Signature[sig] ( src/sig/Death/Applicative/Signature.hsig, nothing )
\> [5 of 8] Compiling Death.Applicative ( src/sig/Death/Applicative.hs, nothing )
\> [6 of 8] Compiling Death.Monad.Signature[sig] ( src/sig/Death/Monad/Signature.hsig, nothing ) [Death.Functor.Signature changed]
\> [7 of 8] Compiling Death.Monad      ( src/sig/Death/Monad.hs, nothing ) [Death.Base changed]
\>
\> src/sig/Death/Functor/Signature.hsig:7:1: warning: [GHC-66111] [-Wunused-imports]
\>     The import of ‘Prelude’ is redundant
\>       except perhaps to import instances from ‘Prelude’
\>     To import instances alone, use: import Prelude()
\>   |
\> 7 | import Prelude (Show(..))
\>   | ^^^^^^^^^^^^^^^^^^^^^^^^^
\> Error: [Cabal-7125]
\> Failed to build lib:effects from death-1.0.0 (which is required by lib:effects-app from death-1.0.0).

↩︎

This line is no mistake. I’m dead serious here!↩︎

Cabal hides error example

$ cabal build
\> Build profile: -w ghc-9.8.4 -O1
\> In order, the following will be built (use -v for more details):
\> - death-1.0.0 (lib) (file src/sig/Death/Base.hsig changed)
\> - death-1.0.0 (lib:effects) (file src/effects/Death/Functor/Signature.hs changed)
\> - death-1.0.0 (lib with Death.Applicative.Signature=death-1.0.0-inplace-impl:Death.Applicative.List, Death.Base=death-1.0.0-inplace-impl:Death.Functor.List, Death.Functor.Signature=death-1.0.0-inplace-impl:Death.Functor.List, Death.Monad.Signature=death-1.0.0-inplace-impl:Death.Monad.List) (first run)
\> - death-1.0.0 (lib with Death.Applicative.Signature=death-1.0.0-inplace-impl:Death.Applicative.Maybe, Death.Base=death-1.0.0-inplace-impl:Death.Functor.Maybe, Death.Functor.Signature=death-1.0.0-inplace-impl:Death.Functor.Maybe, Death.Monad.Signature=death-1.0.0-inplace-impl:Death.Monad.Maybe) (first run)
\> - death-1.0.0 (lib:effects-app) (configuration changed)
\> - death-1.0.0 (lib:app) (configuration changed)
\> - death-1.0.0 (exe:exe) (configuration changed)
\> Preprocessing library 'effects' for death-1.0.0...
\> Preprocessing library for death-1.0.0...
\> Error: [Cabal-7554]
\> can't find source for Death/Functor/Signature in src/effects, dist-newstyle/build/x86_64-linux/ghc-9.8.4/death-1.0.0/l/effects/build/effects/autogen, dist-newstyle/build/x86_64-linux/ghc-9.8.4/death-1.0.0/l/effects/build/global-autogen
\>
\> Building library instantiated with
\>   Death.Applicative.Signature = 
\>   Death.Base = 
\>   Death.Functor.Signature = 
\>   Death.Monad.Signature = 
\> for death-1.0.0...
\> [1 of 8] Compiling Death.Base[sig]  ( src/sig/Death/Base.hsig, nothing ) [Source file changed]
\> [2 of 8] Compiling Death.Functor.Signature[sig] ( src/sig/Death/Functor/Signature.hsig, nothing )
\> [3 of 8] Compiling Death.Functor    ( src/sig/Death/Functor.hs, nothing ) [Death.Base changed]
\> [4 of 8] Compiling Death.Applicative.Signature[sig] ( src/sig/Death/Applicative/Signature.hsig, nothing )
\> [5 of 8] Compiling Death.Applicative ( src/sig/Death/Applicative.hs, nothing )
\> [6 of 8] Compiling Death.Monad.Signature[sig] ( src/sig/Death/Monad/Signature.hsig, nothing ) [Death.Functor.Signature changed]
\> [7 of 8] Compiling Death.Monad      ( src/sig/Death/Monad.hs, nothing ) [Death.Base changed]
\>
\> src/sig/Death/Functor/Signature.hsig:7:1: warning: [GHC-66111] [-Wunused-imports]
\>     The import of ‘Prelude’ is redundant
\>       except perhaps to import instances from ‘Prelude’
\>     To import instances alone, use: import Prelude()
\>   |
\> 7 | import Prelude (Show(..))
\>   | ^^^^^^^^^^^^^^^^^^^^^^^^^
\> Error: [Cabal-7125]
\> Failed to build lib:effects from death-1.0.0 (which is required by lib:effects-app from death-1.0.0).

↩︎

I’m just pointing these out because effectfull lists continuations as problematic.↩︎

Private repository flake input

2025-05-20T15:42:00Z

Setting up private dependencies in nix as flake input allows it to cache it properly. Especially if they’re flakes themselves. However it took me a significant amount of time to get a working configuration for this. It’s possible to setup flakes with private inputs, Note the url format uses git+ssh instead of github:

{
  description = "some commercial project";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
    company-lib-base = {
      url = "git+ssh://git@github.com/company-io/lib-base.git?rev=8e9b3a8d43db3fbfbe2355502890e76a65c5ef1c";
      inputs.nixpkgs.follows = "nixpkgs"; # needs to work on the same package set as our main project, so follows does that
    };
  };

  outputs = { self, nixpkgs, company-lib-base }:
    let
      pkgs = import nixpkgs { system = "x86_64-linux"; config.allowUnfree = true; };
      lib = pkgs.haskell.lib;
      hpkgs = pkgs.haskellPackages.override {
        overrides = hnew: hold: {

          # shove in the privae dependency, the name before = needs to match cabal library
          lib-base = lib-base.defaultPackage.x86_64-linux;

          # ... other overrides here

          # main project
          the-project = lib.failOnAllWarnings ((lib.overrideCabal (hnew.callCabal2nix "the-project" ./. {}) (drv: {
            # reduce build times
            enableLibraryProfiling = false;
            enableExecutableProfiling = false;
            doHaddock = false;
          })));
        };
      };
      defaultPackage = hpkgs.the-project;
    in
    {
      defaultPackage.x86_64-linux =  defaultPackage;

      devShell.x86_64-linux = hpkgs.shellFor {
        packages = ps : [ ps."the-project" ];
        withHoogle = false;

      };
    };
}

Full commercial example

The full example is like this, which allows running the integration tests in docker compose on github actions and run the main service on whatever docker support. This nix example should help out people who want to try Haskell on nix a lot. It only uses tools provided by nixpkgs, which is fairly stable.

{
  description = "some commercial project";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
    company-lib-base = {
      url = "git+ssh://git@github.com/company-io/lib-base.git?rev=8e9b3a8d43db3fbfbe2355502890e76a65c5ef1c";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, company-lib-base }:
    let
      pkgs = import nixpkgs { system = "x86_64-linux"; config.allowUnfree = true; };
      lib = pkgs.haskell.lib;
      hpkgs = pkgs.haskellPackages.override {
        overrides = hnew: hold: {
          # other overrides here

          lib-base = lib-base.defaultPackage.x86_64-linux;

          the-project = lib.failOnAllWarnings ((lib.overrideCabal (hnew.callCabal2nix "the-project" ./. {}) (drv: {
            # reduce build times
            enableLibraryProfiling = false;
            enableExecutableProfiling = false;
            doHaddock = false;
            buildTools = [pkgs.git];
            # these two options drastically reduce docker image size
            enableSharedExecutables = false;
            postFixup = "rm -rf $out/lib $out/nix-support $out/share/doc";

            # NB I developed this to deal with integration tests in nix
            # environments: https://github.com/jappeace/esqueleto-postgis/blob/master/flake.nix#L18
            # I think I should upstream it.
            # we copy over the tests suites to be ran outside of the container
            # need to enable checks for it to build those :s
            postBuild = ''
                echo "entering the phase"
                mkdir -p $out/bin/test
                cp ./dist/build/unit-tests/unit-tests $out/bin/unit-tests
                cp ./dist/build/integration-tests/integration-tests $out/bin/integration-tests
            '';

            checkPhase = ''
            echo "ran by the actions"
            '';

          })));
        };
      };
      defaultPackage = hpkgs.the-project;
    in
    {
      defaultPackage.x86_64-linux =  defaultPackage;

      # docker image for running the thing
      packages.x86_64-linux.the-project-service = pkgs.dockerTools.buildImage
        {
          name = "project-api";
          contents = [
                     defaultPackage
                     pkgs.cacert
                    # uncoment to allow debugging with
                    # docker run -it the-project:whatevertag bash
                      pkgs.bash
                      pkgs.coreutils-full
                      pkgs.iputils
                    ];
          extraCommands = ''
              cp -R ${./migrations} ./migrations
          '';
          config = {
            Cmd = [ "project-server" ];
          };
        };

      # docker image for integration tests
      packages.x86_64-linux.the-project-tests = pkgs.dockerTools.buildImage
        {
          name = "the-project-tests";
          contents = [
                     defaultPackage
                     pkgs.cacert
                      # uncoment to allow debugging with
                      # docker run -it the-project:whatevertag bash
                      pkgs.bash
                      pkgs.coreutils-full
                      pkgs.iputils
                    ];
          extraCommands = ''
              cp -R ${./migrations} ./migrations
          '';
          config = {
            Cmd = [ "integration-tests" ];
          };
        };

      devShell.x86_64-linux = hpkgs.shellFor {
        packages = ps : [ ps."the-project" ];
        withHoogle = false;

        # you can set environment variables in the shell like this
        # (this is just a nix feature)
        ENVIRONMENT="development";
        PORT=8080;

        buildInputs = [
          hpkgs.haskell-language-server
          pkgs.ghcid
          pkgs.cabal-install
          pkgs.eventlog
          pkgs.docker-compose
          hpkgs.eventlog2html
          pkgs.openapi-generator-cli
          pkgs.hpack
          # hpkgs.ghc-events-analyze
        ];
      };
    };
}

Hope this saves someone else hours.

Stacked against us

2025-03-26T20:49:00Z

September 2023, I knew I was in for a wild ride when I won this contract. I already coached other developers in a commercial setting, and I understood what was asked of me. My goal was to get two existing developers comfortable with Haskell while at the same time increasing productivity.

In my first week I started with understanding the code base¹ and what they’re building. They help precast factories find their produced walls called units. ² This project puts Bluetooth tags on precast walls, so people don’t lose them at the factory. With these tags they can find them easily, because they provide an active signal. So the Haskell web app receives all that data and makes a nice interface out of it. The interface provides a text box to search for a unit³, which can be rendered on a map.

Ironically enough my first contributions to this organization weren’t about Haskell or teaching at all, there was an “escalation” on the Monday I started, and they managed to react 3 days after the event(!) Apparently they barely had any monitoring in place. I brought up how slow it was to react like that in one of the first meetings. This caused shock waves through the entire organization and they re-prioritized monitoring for all other projects as well.⁴ Not surprising that many of my initial Haskell contributions were just about improving the engineering side. For example by using annotated exceptions which gave us stack traces, or making Katip ⁵ and logz work together, giving us production logs and monitoring.

When I joined there was only a single factory actually using this system, and the project was filled with bugs. My first impression was that the engineers were intensely focused on architecture, they feared losing productivity due to having “too much code”. However, this came at the cost of getting stuff working. Since this is a mostly TypeScript organization that just adopted Haskell for this project as an experiment, it makes sense to be afraid of code bloat. Another issue was that the people who knew Haskell preferred using the most advanced features available, rather than just focusing on the basics. For example they would teach how to use lens, instead of simple record updates. This made it feel as if there was an endless mountain to climb of stuff to learn. I quickly proposed focusing on simple Haskell. This would cut down on the size of the learning curve and allow people to focus on building features, so they’d feel productive and could explore the remainder of the language at their own pace. Furthermore, the relational aspects of Postgres were in large parts of the system ignored. For example a load of units, which in reality is a wagon of precast walls, was represented as a JSON blob. ⁶ There was no load table and a load_unit_connection table, just a blob. Ironically enough this caused additional code to retrieve the relational aspects. Culturally the focus was “architecture” rather than making something work. Over engineering things that don’t matter was the name of the game. This isn’t unique to this project, at other companies I’ve seen similar discussion. Trying to achieve perfect code essentially. What was special is that they were still obsessed over this, despite how buggy and slow development was.

I learned from the previous contractor that at some point some engineers actually revolted against the use of Haskell! There were more people doing this, but they all transferred themselves out of this project. How ironic is that? ⁷ Anyway, this gave me some dreadful premonition. It felt like everything was still going awfully and things were getting worse!

In the second week I discovered that the teamlead was going to quit. He was one of two people I was supposed to teach Haskell. I never got the opportunity to do this. He’d be gone in a month. He was seething against Haskell. He had been only using HLS ⁸ for compiler feedback and it’s unreliable, ⁹ Furthermore, he claimed Haskell had caused a huge amount of “mental overhead”.¹⁰ But after talking to him I learned his underlying frustration was with upper management ignoring his recommendations. Ironically enough, despite our differences in “ideology”¹¹, I liked this guy. He wasn’t bad at his job¹², but dear lord, he was pissed!

After two weeks we can safely say this project was literally falling apart, supposedly all because of “Haskell”. Which is frankly absurd. They made it hard with an over-engineered architecture, and by using unreliable tools such as HLS. Furthermore they insisted on using every feature under the sun, creating an endless learning curve. ¹³ Perfection was the enemy of the good. I understood the tech lead’s perception, HLS is not good for commercial development. If you have to figure out compile errors within your unreliable editor and also deal with an over-engineered architecture it’s going to be taxing. I also think he never learned how to trust the compiler to guide changes, and how to read type signatures. All of the basics were skipped in favor of writing pristine code from the get go. I realized that at this point I had a choice, keep going on with this project that apparently was falling apart, or jump ship, I was still in conversations with other companies at this time, so jumping ship would be easy. I decided to keep on going. Because things were going bad, but I felt I could manage this. I knew this project would be challenging from the start, but I also liked the colleagues on the project. I felt I could rely on them, and we could do it if we just got shit done.

Anyway I was asked to take over as tech lead, and I agreed. I don’t mind, I guess. My leadership would involve getting everyone to be independent anyway.¹⁴ I think during my third and fourth week I started teaching Haskell, unfortunately the colleague I had to teach had essentially no experience. But fortunately she was smart, exceptionally smart. So we paired on some tasks, and her issues were mostly syntax related. We managed to get this small task over the line after two weeks of one-hour pairing sessions per day. In the final couple days she mostly worked on her own. However, at that point productivity had tanked so much that my colleague had to go back to doing frontend stuff and I had to take over the backend work. This was done to ensure that we met our customers’ deadlines.

Then the actual lead quit, and it was strange, I had to do a bunch of management work all of a sudden as well, which I just postponed as much as possible in favor of getting the features out. ¹⁵ And we managed to do so, and everyone was sort off amazed how much we got done in a short amount of time despite being a man short. The previous lead had essentially been avoiding working on the backend, and making excuses to not do work in that final month. He said he was documenting.¹⁶

There were still many issues with this project. ¹⁷ Especially looking back in hindsight, but we were tightly under our only customer’s ‘yoke’, they decided what we would be building, or drop us. The product manager’s (PM) role at this point was to do customer management. ¹⁸ The project was slow to load, the search barely worked, and the units weren’t always found, and the lift system didn’t work either. The customer wanted us to fix these frontend features first. Dealing with the slow loading, the random reloads, and search, for example. It was just annoying for them to use. To be fair, the system would be difficult to sell with a frontend that was so buggy anyway.

Solving those involved me just digging through the TypeScript code and undoing most of the madness. They were using TypeScript as a relational database, so I just deleted all that and let the actual database be a relational database. This gave the developer who I was teaching Haskell time to learn on the backend while making features. The goal was to give her confidence, that she was creating commercial code in this language. I wanted to move in a direction where she could create entire features, front and backend. Because this is how I’ve learned to implement things. Doing so will remove all communication barriers and help her think “bigger”. The goal was to get her to be independent.

It took a month or two to repair all these issues. No longer would the frontend do joins, or would there be any JSON blobs in the database. Furthermore I introduced a migration system so database operations became easy. The website became a lot more performant, almost magically, ¹⁹ and could deal with larger amounts of data. After that we moved onto actually building some features. The lifts dashboard for example. I managed expectations around rolling out features fast, we had to do integration testing. By which I mean put it on staging, and only accept something done if it all worked together. Unlike a normal web app we also have sensors and a pipeline of continuously flowing data. Because this is a complicated system you find many small issues. And a single small bug can break the entire system.

I think the end of the beginning was the customer visit in January 2024. Where I got to witness the problem at hand with my own eyes. I got to see the “yard” of gray concrete and how large these factories are. They’re large. We also demoed some features to them and they seemed to be less annoyed with us. I wouldn’t call it impressed, considering how slow development had been before I joined. But they were seeing movement, and this caused a renewal. We were free of their “yoke”.

The Danish precast factory.

By the skin of our teeth

Around January or February we started hearing signals or messages from upper management that we really had to start selling this product to other customers, OR ELSE. the commercial department sort of had given up trying to sell the project at this point completely. Honestly the way they did it was kind of insane for 50k+ deals. They just sent the hardware to a factory and told them to figure it out. 😅 You can maybe do this with a mature known product. But this was a barely functioning prototype from a unknown company and it involved large amount of money. Of course they were going to expect “whiteglove” treatment and of course the customers rejected this. Prospects would take the “sensor”, and Bluetooth tag into the yard, and see if it appears on the map. It didn’t because the main algorithm took like 30 minutes to figure out locations of these units. ²⁰ Now our main customer was happy with that, because they were already bought in. However, if you want to sell this locating system, this is the first bullshit check you do. I think one of the OKRs²¹ at this time was to set up the project without any assistance from the internal team. In essence, to make it a do it yourself experience for the customer. This as a goal was batshit insane. Not only were the deals too large for this to be done. The tech was far from being polished enough to be used like this. Furthermore these would be recurring deals. so cheaping out on some manpower for a one-time setup cost is kinda crazy. I brought this to the attention of the product manager, and he agreed. ²²

Because we were sort of “free” to decide what to work on, I strongly argued in favor of making sales easier. Another topic was for example to add admin pages, we were currently just modifying stuff directly into the database. We decided to go all in on just modifying the technology so that we could pass technical trials and sell to customers. Whatever it took. There were two major issues:

Customers had a poor or no mental model of how the (hardware) technology worked. In that they didn’t understand Bluetooth²³ signals.
We’d also address taking a sensor and tag to the yard and not get a ping on the map. The system had to become “live”

We “tackled” these problems with something called “trial” mode. The designer made a ton of screens explaining all this stuff and have an automated way of guiding a new user through all of it.²⁴ ²⁵ One nice thing is that I could let the other engineer implement most of the frontend and backend features for supporting that, while I focused on actually making the signals available at all. Keep in mind we did not just have a “there is a database” backend called the server, we also had the “figure out what a signal is” backend called reality capture. I worked on that reality capture part. This gave the other engineer the opportunity to grow more, and come up with her own designs which we could work with.²⁶ We finished most of this around May.

I think around May in 2024 the money had run out for this project. So the other engineer got pulled, which was annoying because she just had gotten reasonably productive to do small full-stack features on her own. Another quarter or two and she’d be amazing. The designers’ contract just wasn’t renewed at all. ²⁷ Despite the cash burn being lowered, this project was far from saved, we were still scraping by. And the threats from upper management continued, and sales kept on fumbling.²⁸ We knew the particle hardware we were using wasn’t performing well. But obviously we’d get no hardware resources. It was just me and the algorithms engineer left doing technical work. And the product manager to deal with customers and sales.

Particle sensor and bleutooth tag beacons

One prospect customer who we were talking to for years at this point was located in Denmark. We knew the hardware wasn’t performing well because of trying to get the system to work with them. They told us, “show it works and we’ll use it”. It didn’t work. For example they had sensors in their little trucks driving by these walls with Bluetooth tags on them. The sensors would pick up no signals at all. I think the product manager had visited this place five times already by that point. Everytime the algorithm engineer and the PM would have a new tactic to “crack” this yard. He went again around May, and failed. But as we know now there were serious fundamental issues with the firmware. So I don’t think trilateration ²⁹ could’ve worked in any case without firmware upgrades.

The sensors worked well enough to give you a live proximity-based location. This would just attach the sensor GPS location to a Bluetooth tag on extremely close proximity, say smaller than a meter. We could just measure the signal strength required for that, it is reliable. The backend was just a loop essentially, constantly querying the timestream ³⁰ database for updates at small incrementing windows. It’d be attaching the signals to gateway locations if the signal strength was high enough and the time stamps were close enough. This one feature, we tested the shit out of, it was our ace in the hole.

Something else happened around May, the head of software decided that this project could benefit from an app. And he decided to make a prototype. The QR code scanning in app was much better than whatever we could provide on the web. And the app had Bluetooth capabilities, which browsers to this day can’t access. So I saw some good growth potential in this and was so relieved he took the initiative. Now Flutter is far from perfect, and Dart isn’t a great language to work in.³¹ But there was no way I could also maintain an app on my own aside from dealing with the glaring problems this system had. So I’d not look a gift horse in the mouth and thanked the head of software for his contributions.

We had scheduled the check in Denmark beginning of July. The person managing our meeting, let’s call him Bob, was actually quite happy to arrange some Danish delights. The product manager told him to arrange whatever for lunch, but I asked for some local Danish specialties. He seemed to be quite happy to do that. Bob was quite a character actually. He was completely disinterested in this project, he just wanted to get this done. His organization upper management apparently told him to do these checks, and he was going to, with minimal of efforts. But Bob seemed to be delighted to share about Danish culture and cuisine, what a joyful reminiscence. The check involved production people attaching a bunch of beacons to various units. We’d then use the system to find them. Of course the system failed. But not just because of the bad firmware. Also because the way this factory was managed was extremely strange and inefficient. They’d move their trailers over to one part of the yard. Perform QA. Send them to another part of the yard to reshuffle.³² Then send them to whatever storage, then go back for reshuffling, etc. There was no rhyme or reason to any of this. Mind you these are walls they were carrying. These are heavy loads, and they used heavy machinery, but nobody had any clue how they were using these tools. We did not know this either, but we had scheduled several days to prepare for this check, and at some point I started asking the PM about what was going on, which made him think about it.³³ At several times the system would locate these units in the middle of the road. They probably were in the middle of the road for a while, but we had no sensors in those reshuffle or QA areas, so our system wouldn’t further update even if they got moved there. We had an in real life phantom read. Honestly, we were set up to fail from the start. But no one knew how unfair this tech check was, even at the factory.

Here I had a little epiphany, all that talk about CO₂ reduction for the other projects, may be in fact also be true for this project. Upper management, the same one threatening to cancel us all the time, deeply cared about carbon reduction. Nobody realized this because nobody has spend hours upon hours in the yard trying to get a tracking solution to work. Nobody has ever put any extra thoughts around the yard. So in fact a project I initially joined for the challange and for using Haskell, I stuck around because I liked the people, may in fact also be helping the environment? ³⁴

Now the fact the system failed was a huge issue for us. We had some time however, we had planned the entire week for doing this check. If we’d failed this check it would be a huge blow against us. In fact on the road to this check the PM was already casually asking me if I thought he’d be a good PM for other software companies. I listened to what was not being said. I think he’s awesome, but we were fucking cooked. However, we had our ace up our sleeve. the proximity algorithm.

On judgment day, before the check was being done by Bob, the PM told me to grab a sensor and go walk by all units, he’d do the same but from the other side of the yard. the yard being 300x250m, which is around eight football fields. It started raining heavily. Find a tag, press the sensor to it, wait a second, then move on.³⁵ Rinse and repeat while getting soaked. Before the check I went back into the office making sure everything was updated. Unfortunately some units had updated to the wrong location due to the other trilateration algorithm sending updates that override proximity. Here the algorithm engineer, the PM and I had a healthy debate. The algorithm engineer wanted a fair check, The PM and I knew there was nothing fair here. The PM and I wanted to force update the wrongly located units in the database. We ended up doing that by matter of vote³⁶

Bob had already arrived, but the PM asked for a moment. The PM told me to also put one of these sensors in my bag, we’d just walk with them on our person while doing the check. I was shivering from being soaked. I guess Bob didn’t put two and two together as he didn’t care.³⁷ With the first unit Bob tried to let us fail on a technicality. But the PM went hard against that. The sun had come in, and these checks became a pleasant walk. Bob started cracking jokes and smoking cigarettes. The next ones were all passing (of course), in fact when the customer would check he would at times see the system update in front of his eyes. And just accept it as normal. Not knowing we’d have sensors on our person. ³⁸ Bob believed it worked. some rough spots were blamed on the newly developed app, but overall he was happy. We passed the check by the skin of our teeth.

Danish rye bread, a local delicacy.

This was huge. We’ve been scraping by and losing members for the past half year. But we managed to sell. No longer could upper management blame the technical working of our system for sales failures. Annoyingly this particular deal fell through anyway, but not because of the technical capabilities of the system. I think this did buy us a chunk of time however to actually address the underlying issues. During this check we had spotted many other issues with the system. Fun things such as DDoSing ourselves by having a bag of tags next to the sensor (which caused a bit of panic the day before judgment). I addressed all these issues.

Winning

We also started doing another trial for a potential new large customer, similar to the one in denmark. This time it would not take seven trips from the PM. We had proximity algorithm to convince them that we could find locations of units. Actually one of the sensors failed during this check, fortunately they had a backup, but aside that we past this tech trial on first go. We confirmed here that if you convince people it works swiftly, they’ll be a lot less thorough in their technical checks. This trial in September 2024, I believe was still on the old crummy firmware. We passed so “easily” because we failed so hard and often in Denmark and learned. Furthermore the algorithm engineer did her magic, and had convinced them the system could be used for LEAN manufacturing. Unfortunately the customer had arranged no experts for that check there at the time. So they didn’t realize what they were sitting on. But they were convinced the system worked and was something that they wanted. This came back to us in January with some good news once they finally found a guy to judge that, and he was enthusiastic.

Then the next major issue started popping up for our existing customer. The Bluetooth tag batteries around July started dying. We thought we solved this by just sending new beacons. The new ones had better settings which were supposed to make them last for 4 years. Our existing customer also found the algorithm wouldn’t work on another part of the yard, we worked around this by doing those manual forced updates with proximity. Actually they were happy to take sensors into the yard. Around September I started working on the firmware, described elsewhere so I won’t go into it. This made proximity significantly better, and temporarily killed the other algorithms, due to an excess of data causing it to run out of memory.

Around October we won an award for smart data collection in construction, we were nominated by our one and only customer. Furthermore around Christmas we got good feedback from them. They had managed to lose around 30 units in SAP and managed to find them all with our system! I knew at the time it was just my proximity algorithm that was operational. Because the new firmware had killed the trilateration algorithm. So I was a little confused by this. Furthermore our own audits showed that we could seriously work on accuracy. I think what happened was that one of their employees consistently took a sensor into the yard. I’m not really sure, but if they’re happy, I’m happy.

Even after winning the award and having this major customer in the pipeline, we still got more threats of upper management. We also got more curve balls thrown at us. For example the temperature product required my services for 2 weeks. I was the only software engineer trying to make this logistic stuff work, they had seven people working on their temperature tech, and they couldn’t do this one algorithm integration in time. I was supposed to help them for 2 weeks. I finished the task in 2 days. After which they started inventing other tasks, I said “no thanks”. Other people from upper management who had been ignoring this project for years at this point all of a sudden wanted to get involved. I think people were noticing this project was starting to win. In their helpfulness they blocked architecture changes until detailed documentation was made for example. I’ve no idea how to deal with these requests. Spending two days on documenting while one of the main algorithms is down seems irresponsible. I just stopped doing architecture changes, they were mostly cost saving anyway, and optional for now.

The PM would become father, and he’d go on a three-month break from the job in December. We knew this was coming up so we were preparing for that point. The algorithm engineer and I would share responsibilities. I was anxious for this period. One of our main concerns was to keep the commercial team focused. Doing a weekly meeting turned out to be highly successful, actually the main sales guy liked doing them and requested to continue doing them after the PM got back. We made this work. Come on, compared to what we’ve been through in Denmark, this was easy right? It was.

For most of December I worked on signal processing. This was some new feature where we sent signals directly to the backend so we could support this “detected here” status. This was one of the main features the new customer would use, to determine on which site a unit is. It’s a more general way of locating units, rather than by coordinates you just give it a site name. This is easy because you know on which site a sensor is, so all you have to do is pick up a signal and associate it to a sensor. Before we fetched the signals from timestream but that was kind of expensive. This approach used a simple queue.

There is plenty of other high impact stuff to do. Fore example the beacon battery issue exploded, we did some analyses on an existing batch and found that around 5% of the new batteries were unacceptably low. I already had written an app to check for beacon settings and battery settings, it was just a bit slow. However I modified it so that it would highlight bad settings in red. This made sifting through all these beacons (thousands of them) a lot faster. It’s available opensource. With the things learned from that development I also built a liveliness check in the registration app.³⁹ This should prevent the system from accepting any dead or low battery Bluetooth tags.

Over the air settings check app demo

We’re in March now. That big customer signed the deal, and we likely got another construction site. Currently we’re servicing two locations; soon this will grow to fourteen. We’re winning and we likely won’t be able to keep up with the work. Which is a much better problem to have than being threatened by cancellation. Whilst that’s going on we’re still surprised by leadership with cryptic and somewhat intimidating remarks—for example, asking “is this really aligned with our vision?”. Their vision being to tackle carbon impact in construction. Which is ironic, considering the experience in Denmark.^danish-madness I suppose they’d just not believe us. Despite the irony and the fact we just had this huge deal done, we’re still having these kinds of questions being raised. I think I requested a warm body to deal with those battery replacements. This request got denied. It’s an odd situation to be in, where you sort of know already you’re going to be short on manpower, but upper management still believes you’re a bad project. We triumphed in Denmark, you think a salty manager is going to stop us?

As long as we focus on the right stuff to build we’ll be fine. We’ve been focusing on stuff that has impact, and I think about 90% of the things we worked on had impact. We bought time by dealing with our customers request and get a renewal, that time we used to make the product sell able via proximity, which bought us more time to fix the fundamental firmware problems. Which made the product actually good. There are other large problems we have in this project. For example the particle data operations are hugely expensive, but this company can build its own hardware. Upper management just likes to kill our requests. Ironically enough, it’s costing them money, not me. It causes me more work because I work around it these problems with more firmware modifications, for example, I introduced a sleep mode where we disable the sensors during off hours. I’ve never claimed to be good at firmware, but even a slowly produced implementation of these changes causes a huge impact. Despite how cooked internal politics is for this company, with two major customers now, many sites and getting the award, we can say we’re winning this unfair game.

So this also involves getting familiar with whatever style the organization is using. Usually various companies have various styles.↩︎
How do you lose a wall? Imagine a concrete precast factory that constantly is churning out walls and other precast elements. These walls are stored in a yard several football fields wide, filled with gray concrete. Normally they record where they’re stored in SAP an inventory tracking tool. If they need the wall below another one, they may move it somewhere else and SAP gets stale. In practice these walls may be moved a lot and get lost. These walls often wait for months to be shipped to a construction site.↩︎
Eg a precast wall. Several of which stacked upon eachother is called a stack.↩︎
You’d say having monitoring is rather important, but startups often forget these “basics” of software engineering.↩︎
Katip is great and structured logging is really nice, that service logz is fragile however, it denies the existence of heterogeneous lists in Java which Haskell tuples get encoded to as JSON.↩︎
There still to this day is an entire MTL based framework to stub out logic and run tests in memory. ONE of my first suggestions was to delete that and instead just use higher order functions to stub things out, but some high up chiefs decided against that because they liked seeing the possible effects in the type signature. Because most logic resided in the database I think at this point I’ve deleted most tests that use this. These in memory tests didn’t ever catch any bugs and were just a maintenance burden. Instead we now do full integration tests that hit the database. The concern down the line is that our test suite becomes slow, but that’s a problem for future me. One of my main productivity increase was to not be afraid of introducing tech debts like these. Because if the project dies right now because you don’t get anything done having perfect code just doesn’t matter. And I’ve kicked that “slow tests” can extremely far down the road at previous jobs. At supercede we had thousands of database hitting tests still run under 10 minutes.↩︎
Normally it’s the software engineers asking for haskell and managers resisting the encroachment.↩︎
Haskell language server. an LSP implementation for haskell. this provides IDE like experience for editors. It’ll make your text editor give hints.↩︎
Asside from the time pressure for producing features and having to learn haskell, having this program randomly crash on you as your only source of feedback is infuriating.↩︎
If they were to not obsess over architecture all the time this wouldn’t have been such a big problem. By the time I joined this guy already knew how to navigate syntax and how to fix bugs rapidly. The mental overhead was of his own creation.↩︎
I call it ideology because I think programming languages are more like a culture you buy into rather than a mere technological choice. Every general purpose programming language is Turing complete and therefore equivalent in capability. So logically what remains is cultural preference. In this view choosing one is more like choosing a “workstyle”, a “way of thinking” or set of ideas to buy into. As opposed the more traditional view which says. It makes more sense to ask “what system of values Haskell promotes versus JavaScript?”, rather than to ask “what can I do with Haskell verses JavaScript?”. The latter question is meaningless. Having said that I do think you can be a lot more productive in Haskell than JavasScript because the libraries are better designed, with fewer surprises and the type system will catch so many small issues in a thorough manner. It’s better then other commercially available typed languages because it reduces to a single expression. This difference can be reduced to a value question as well: Do you care about individual developer productivity, or do you care about having a sufficiently large manpower pool? There are more JavaScript developers than Haskell developers.↩︎
Later I found out that he made a bunch of suspicious technical choices, but I think most of these decisions came from being angry instead of lack of skill↩︎
I myself never bothered learning Linear Haskell for example, or can’t be bothered with the evaluation order of type families, it’s just not interesting. I don’t think you gain a lot by learning that stuff. You could do something fun instead y’know, write a giant blog post on your job for example..↩︎
I think I succeeded with this. People no longer shy from just jumping out of their comfort zone and helping eachother out.↩︎
The product manager made it clear we had to get stuff done fast.↩︎
I’ve only seen the customer onboarding document, and it was incomplete.↩︎
The foundational issue being the firmware.↩︎
Finally the designer got to do something. This wasn’t his fault.↩︎
I did nothing “genius” of course, I just made a boring, traditional design out of it. The reason it was so flawed is because the previous tech lead was afraid of Haskell.↩︎
Rather than streaming the signals and doing trilateration on the fly, these were processed as a batch. I don’t think anyone at the time had expected live updates to be a requirement. It is if you want to sell this. There is no reason why it shouldn’t be able to do this on the fly, but the system was simply not implemented like that. This algorithm was fully written in Python.↩︎
Objective and key result.↩︎
This is one of the reasons I stuck around. The fact they listen to uncomfortable things like this is one of the big reasons I decided to stay longer at this project. I’m not a business guy or strategist, just a developer. But if you come with logically founded arguments they listen.↩︎
Bluetooth just works over radio, all Bluetooth does is specify a protocol.↩︎
Finally the designer got to do something. This wasn’t his fault.↩︎
Looking back, I think the product manager still had a small hope for creating a do it yourself experience. Which is fair because going to these factories and construction sites is a big undertaking. We later found out these screens wouldn’t work, but the back-end stuff to support it, and having live updates were fantastic!↩︎
I think the important part here was that she could make mistakes and learn from them. I was always available to bail her out if necessary. Most of her issues were small details, such as having functions with the same name causing confusion.↩︎
He ended up being a giant distraction because he kept organizing meetings to discuss stuff which we just couldn’t implement.↩︎
Aside story: Around this time the one irish precast factory let out a competition for a technical solution for precast factories that basically described our system. There was some subsidy by the Irish government. We were by far the best suited to win that and somehow lost. I don’t know what happened, but people say there is a lot of corruption in construction, another fun thing 🙂↩︎
Trilateration involves locating by making triangles out of observations.↩︎
I would’ve used influxdb, but one of the engineers who wrote reality capture went gungho on Amazon services.↩︎
My biggest issue is the lack of sum types which make error handling hard. You get enums but no heterogeneous struct per tag. It also inherits all the equality issues that Java has, making working with collections challenging.↩︎
Reshuffling means, moving a concrete wall from one trailer to another to be patched up. This happens with large diesel powered cranes.↩︎
I think the PM just asked one of the guys in the yard. But these people don’t talk much to the office staff. Obviously people working in the yard know what’s going on. However their managers’ managers don’t know. Which are the office staff and we were judged by office staff.↩︎
Now, I have no aspirations to actually help the environment, because I feel this is mostly an energy issue. But I am definitely concerned about it. So I count this as a bonus.↩︎
We noticed at the time the procedure didn’t always work. I now know that this process probably only worked half the time because the sensor would spend 20 seconds of the minute sending gps signals and not doing any Bluetooth. These are all firmware issues I fixed after denmark…↩︎
Normally the three of us argue till we reach consensus, this is an advantage of a small team. But there was no time to do this this time, the check was coming.↩︎
I was soaked because I just had spend 2 hours in the rain force updating the system.↩︎
This is dishonest I suppose. We just needed time to figure out for example the firmware. We needed internal confidence, to get firmware engineers. We were setup to fail on this site. How do you judge us?↩︎
The registration app is different from the setting check app. The setting check app isn’t customer facing.↩︎

Firmware lemons 🍋

2024-11-02T00:00:00Z

Ever found yourself fixing a problem you weren’t hired to solve? To the point where you’re under qualified? That’s what happened when I dove into firmware. It was my suggestion, nobody asked me to do this, but we needed better scanners, better hardware even, and we couldn’t get it due to politics. They asked us to get more customers, before we could get better hardware. Cost was the main issue, this project was perceived as too expensive. But the irony was that we needed better hardware to get more customers. We had to break this deadlock.

I had never even seen any firmware before! But it was pretty clear at this point our Bluetooth scanners were underperforming. They are the source of all data, and they’d catch out about 5 signals per second. Which is far lower than what you can see on any Bluetooth capable device. I worked at this place for about a year, before this even came up. We tried doing fancy startup things, and underachieved. For example we wanted to detect if a “lift” was happening, but only got a single observation from the scanner for a two minute period.¹ About two months ago I brought that the scanner firmware maybe bad. We thought the scanners worked at peak performance before that. Kind of silly looking back. We naively assumed we were set up for success, but we were given lemons.

Experiment

My colleague tracked down the packet size configuration and increased it as an experiment. This doubled the performance of the device. I was relieved by this; I sensed the deadlock breaking and could smell lemonade. If your goal is to figure out what’s going in reality with Bluetooth scanners, and you double the performance with a small firmware modification. You know you’re doing something right.

I took over from my colleague, because I was there to write code. Initially I was so bad at this that she had to bail me out a couple of times. With tasks such as flashing builds or dealing with bricked hardware. The first week I essentially “did” nothing, because I was just struggling with getting builds out or busy reading the docs and understanding what was going on. If a device bricks due to faulty firmware (think segfaults), it frustratingly only shows a red blinking light. No GDB for you. You can no longer put new firmware on it either, but there is a reset trick² for that. My colleague recommended that I make a small, baby-step changes. Test each small change out out on the device making sure it doesn’t crash. ³

Sorting

Anyway, I got better. In the second week I got a build that sorted on signal strength values. ⁴ We believed the machine was sending random updates, whereas we were mostly interested in strong signals. ⁵ So by sorting we’d just get the best information out of the machine, neglecting weak signals. This again showed an improvement, not as good as the increased packet size, but a good step forward.

Mutex nonsense

The following week I tackled the concurrency issue. I knew this was an issue because there was a mutex that excluded the Bluetooth scanner from running while sending of packets was happening via the SIM card (and vice versa). They put instructions on a thread, but nothing ran concurrently. For example here is the scan thread doing the Bluetooth scanning:

void Beaconscanner::scan_thread(void *param) {
    while(true) {
        if (!_instance->_run) {
            os_thread_yield();
            continue;
        }
        custom_scan_params();
        if(_instance->_scan_done) { // mutex here disables the thread on true
            os_thread_yield();
            continue;
            }
        long int elapsed = millis();
        while(_instance->_run && millis() - elapsed < _instance->_scan_period*1000) {
            Vector<BleScanResult> cur_responses = BLE.scan();
            _instance->processScan(cur_responses);
        }
        _instance->_scan_done = true;
        os_thread_yield();
    }
}

and then we got the main loop consuming it:

void Beaconscanner::loop(bool publishScans) {
    ...
    if (_scan_done) { // mutex here, only publish on true
        publish_all();
        _scan_done = false;
    }
  }

You may wonder how these threads even communicate, no handles are passed around. The answer is of course global variables and the use of various locking macros.⁶ Not that the locking matters because _scan_done is used as a mutex. So we’ve two threads running, but the relevant code never actually runs at the same time.

They wrote code, and then wrote other code to undo their initial code.

They took the effort of putting the Bluetooth scanner on a thread. And then wrote a mutex to undo the threading. It would’ve been more efficient if all was done on the main loop with no threading at all! Once I realized this was happening I was baffled. The author got paid good money to undo his own code with more code, resulting in firmware like a sour lemon.

I tried explaining this to my dear colleagues, I said something like “we’ve threads but no concurrency”. Perhaps this was too dense, but to be fair it’s hard to explain nonsense. Fortunately this weirdness was also visible in the observations. At times we were just getting no observations when we’d expect some. By the time we were talking about these observations, I already had a build, because I knew the code was nonsense and had addressed it.⁷

Fixing concurrency

The fix involved deleting it all. Both the locks and the mutex. Instead I used a queue for inter thread communication. Queues is doing threads on easy mode, and I had no-one to impress, so I did easy mode. However, even in easy mode this was still quite challenging to do. Partly due to my inexperience with C-like programming. The queue I ended up using looked like this:

int os_queue_create(os_queue_t* queue, size_t item_size, size_t item_count, void* reserved);

It’s a bit wack; you reserve some memory for the queue, giving you a os_queue_t*. Then you’ve to initialize that memory with os_queue_create. This fills in some fields of the structure. I got stuck on using os_queue_put and os_queue_take for a while, which caused me red blinking light grief⁸ (segfaults):

int os_queue_put(os_queue_t queue, const void* item, system_tick_t delay, void* reserved);
int os_queue_take(os_queue_t queue, void* item, system_tick_t delay, void* reserved);

I think the issue was that I tried putting in a BleScanResult directly. But it contains some pointers, and this queue wants you to copy over all data as a continuous memory block. void* in this case means give me whatever memory pointer, ⁹ and the os_queue_create function already told how large it should be. So instead of putting everything on the queue, I just made a struct with all relevant information:

Results: From lemons to lemonade

The results of these improvements are impressive:

firmware	p/s
lemons	1.4
packets & sorting	5.57
unlocked	10.3

Here, ‘lemons’ is the original firmware; ‘packets & sorting’ is after sorting, and ‘unlocked’ is me addressing the concurrency issues, by deleting the locks. p/s stands for packets per second for strong signals in this case. This is just a single measure for strong close signal bandwidth, but the new firmware improves in all categories we measured. ¹⁰ I did this in about three weeks of work. Because we’re going so fast, there are more improvements in the pipeline since we’ve time, such as using Protobuf instead of JSON, which will increase throughput, and tweaking Bluetooth parameters causing a higher capture rate, which I may describe in a later post. After we’ve done all that, we can say we’ve made lemonade!

It’s funny that I was unsure about starting with firmware changes or even suggesting it. I was outright anxious about doing these concurrency updates. They all turned out to have outsized impacts on the entire tech stack. We’re winning now. This company may very well become a logistics company due to these calculated gambles. Do you have any stories about professional, calculated gambles? Please let me know in the comments; I’d love to know 😀

Note that there would be many Bluetooth tags all emitting signals, so even if the scanner could catch 5 a second, it had to send the right signal as well! Supposedly the Bluetooth tags were sending out a signal once every 300 milliseconds. so what happened to all these signals? What would happen in this 2 minute interval is that it’d pick up the wrong signals, of stuff just laying around, and send those over!↩︎
For particle tracker one, you have to screw it open, and there is a mode and reset button, hold the mode button, then tap the reset button, once it starts flashing green release mode, this will allow you to put new firmware on it again via usb.↩︎
Which is starkly different from my haskell workflow of just writing everything you want in one go, solve a metric ton of compile errors and having a functioning program. This Made me realize how spoiled I am.↩︎
Apparantly you can use some c++ standard library function, trying to use a custom sort from the internet also cost me days. 😅↩︎
My colleague menioned they asked to sort the signals from the original contractors, but they never did this.↩︎
There was no reason for this and it’s suspicious.↩︎
This explains the fast turnover time, I just fixed obvious glaring problems, which you can see by reading code and trying to figure out what it does.↩︎
Because the code segfaults and red blinking light is all you get.↩︎
Not to be confused with void, which is just an empty return type, or Void in haskell, which means no value will ever be that.
```
struct QueueItem {
BleAddress address; 
uint8_t advertiseData[BLE_MAX_ADV_DATA_LEN]; 
size_t count; 
int8_t rssi;
};
```
A struct is a continuous memory block.[^know-why] BleAddress can be inserted because it is also a simple struct with no pointers. I had to make some more modifications to support the primitives in the Eddystone class. [^class] but once this was done it worked.↩︎
In the force update activity for examle, by just tapping the sensor to a Bluetooth tag we managed to increase success rate from 20% to about 90%. [^doesntwork] The system only managed to work without doing this by having long exposure to signals, but now we also support these quick exposures. Which makes the entire stack more robust.↩︎

Analyzing Haskell stability

2024-07-30T15:00:00Z

I joined the Haskell Stability Working Group about a year ago. I was freshly scarred from a GHC upgrade, so I decided to try improving the situation. I think my first contribution was making an example for GHC nightly builds on GitHub Actions. This allows downstream¹ maintainers to build GHC nightly to spot issues early.²

Recently, we’ve been curious about what Haskell ecosystem and GHC changes cause the most breakage. A suggestion was made to do a quantitative analysis of “head.hackage”. “head.hackage” is a repository of patches for Hackage. GHC engineers use these to test out new GHC builds on a wide range of Hackage packages without having to upstream³ a patch, which can take time. Instead, they can put the patch in “head.hackage” and immediately test it on a wide range of packages. We can analyze this repository of patches and categorize them according to the GHC Stability State of Play I volunteered for that effort. Here is a preliminary result of the “head.hackage” patches required to make around 485 packages⁴ build.

From this, we can see that we’re not doing too bad for stability in GHC directly. Many of these patches are for older changes for example. Although, from first hand experience, I know we can definitely improve! We’re going to extend this work by labeling the exact cause of breakage for each patch (many do overlap). This will allow us to see more clearly which changes caused most issues and help us prevent similar situations in the future.

Do you have any opinions on this document? do you feel stability could be better? or do you have a specific frustration which grinds your gears around stability? Please reach out! We’re always interested in hearing from the community and are in fact actively recruiting members. You don’t need to join; you can also email or use the comment box below.

Downstream in this case means people who use GHC and are dependent on it.↩︎
At first, I thought these mechanisms were great. However, at some point upstream GHC started breaking for some reason and I learned that the availability was ‘best effort’, causing me a bunch of issues. So I turned it off. But I applied these “GitHub Actions” techniques I learned for various other repositories. I learned that for projects that have little activity, github likes to disable automations. Since 99% of my projects are like that, it essentially makes all of GitHub Actions kinda useless for me. I had a pretty bad experience with GitHub Actions before all this, and this made me even more estranged. It’s unreliable compared to the Nix CI pipelines I’m used to.↩︎
Upstreaming is the process of sending a patch to the “maintainers” of an open-source project. The maintainers will then make the patch ‘official’ by merging it. In principle, the process is simple, but in practice, the burden of proof (especially for larger projects) is on the person who submitted the patch. They have to convince the maintainers that the patch is useful, which takes time in the form of communication.↩︎
The estimate of 485 packages is done based on this Grafana dashboard. The number changes over time. this measurement was made on 2024-07-28.↩︎

Cabal exact printing

2024-05-12T18:00:00Z

For a long time, I’ve been annoyed that cabal tells you to add modules to your cabal file. It can detect missing modules, but won’t automatically add them. This is not a big deal for a single module, but if you like splitting code into many small modules, for example because you like fast compile times, then any refactor may cause many of these messages to appear, which becomes tedious to manage.

Currently, if you build a project with an extra module not listed in your cabal file, GHC emits a warning:

<no location info>: error: [-Wmissing-home-modules, -Werror=missing-home-modules]
    These modules are needed for compilation but not listed in your .cabal file's other-modules: 
        X

You’d say, why doesn’t cabal just add this module to the cabal file? Well, it can’t. Cabal is currently only able to parse Cabal files, and print them back out in a mangled¹ form. There are other programs providing module detection ², but nothing is integrated in cabal itself. Anyway, one day I was lamenting this problem on the internet and I was told by a forum user³ that this is in fact the cabal exact print issue. GHC is already able to do this for Haskell code, and defines it as follows:

Taking an abstract syntax tree (AST) and converting it into a string that looks like what the user originally wrote is called exact-printing. An exact-printed program includes the original spacing and all comments.

In any case, after some exploratory discussion in that thread, I got a maintainer to endorse an approach I made.

Previous attempts were abandoned. Or they revolved around creating a seperate AST⁴, which was against maintainer recommendation, and then abandoned. I’m doing a fully integrated design around the parseGenericPackageDescription function.

parseGenericPackageDescription :: ByteString 
                               -> ParseResult GenericPackageDescription

This function takes a ByteString (cabal file), and returns a GenericPackageDescription. Which is the data type the rest of cabal works with. This is the cabal equivalent of an AST. To make sure the changes I make are helping, I setup several round trip test concerning the various properties we want see exact printed. For example, I’ll add a cabal file with comments:

cabal-version:           3.0
name:                    two-sections 
version:                 0
synopsis:                The -any none demo
build-type:              Simple

                         -- my awesome
-- library
library 
  default-language:  Haskell2010
  exposed-modules:   AnyNone
  build-depends:     base <5
                    , containers

We put that file through parseGenericPackageDescription, put the result of that function called GenericPackageDescription into the exactPrint function, and it should be the exact same. By doing this we should theoretically be allowed to modify GenericPackageDescription however we please, and get the result printed, which is a step towards solving our initial module not listed problem (and several others).

The implementation itself is simple. We add a field to GenericPackageDescription with all the exact print data, which we access trough Map’s. The key is a new concept called a NameSpace. Values are ExactPostions for example. I modified the pretty printer to first try and get an exact position, before falling back to pretty printing. This instantly solved the tests for sections. Which was a good result!

However, there is still a long list of features we need to support, such as comments, common stanzas, better comma support, and conditional branches. Exact printing comments is hard because currently cabal filters them instead of parsing them. so the parser needs to be modified as well. Fortunately someone has already done some preliminary work on that, unfortunately it breaks my existing tests. Common stanzas allow users to declare some shared stanza, such as ghc-options or build depends fields, and re-use them across various targets such as the library or executable. This is also hard because the common stanzas are currently just absorbed into whatever they’re imported, they don’t exist in the GenericPackageDescription at all! Exact printing comma support seems a bit strange, but you can put commas in many different places and still have a valid cabal file. These locations need to be parsed and tracked somehow. I’ve not even spoken about conditional branches yet, which I’ve not thought about much, however the point is that there is a lot to support!

In general the issue why this exact printing is so hard is because cabal has a lot of features, all needing to be supported. And every new feature added will not have exact printing support until some solution gets merged. So over time we can expect exact printing to become even harder to implement.

Why I’m writing this instead of just implementing exact print? Back in November, I implemented some initial solution in anger, and with a draft PR created, my anger had all but dissipated. With clarity of mind, I realized spending all my free time implementing this feature isn’t sustainable. I’d burn out at that rate. Essentially I’m looking for a way to make this more sustainable.

A possibility is to create a Haskell tech proposal. This is a funding mechanism for long-standing issues in the Haskell community. With funding I could trade hours in my day job for hours working on this particular issue, without losing anything on mental health. Currently the issue is that these proposals look like a lot of work. And I’m not sure how much budget is even available for a project like this. So in essence this blog-post is a preproposal, to see what people think of it without going all the way.

Another possibility is recruiting volunteers. In general, however they’d run into the same issue I had! However I do think if I’d trade some money for some helping hands then this could work. The tests can be solved in parallel, a single person could work on comments, and another on common stanzas for example. But would this be allowed by the tech proposal system?

I believe this exact printing is solvable problem. However I lack the resources to solve this personally. I think a tech proposal may be able to provide said resources. Although the process involved is daunting enough for me to first ask the community, do people want this? Please reply if you’re bothered by this, or have any comments on my approach.

you can see this mangling by running cabal format on a cabal file, the issues I saw were:
1. Delete all comments
2. Merges any common stanza into wherever it was imported.
3. Change line ordering
4. Change spacing (although perhaps to be expected from a formatter)
↩︎

We have a whole host of projects that re-implement this specific functionality. for example have:

            Of course many of these projects do more then just module expension.
            hpack provides a completly different cabal file layout for exampe,
            cabal-fmt and gild are formatters for cabal file.
            Only auto-pack just does this one feature.
            However, since all these programs implement this functionality,
            there is clearly demand for it.
            no-one has attempted to centralize it yet however.

↩︎

They’re a self identified milk enthusiast, whatever that means.↩︎
AST: Abstract Syntax Tree, a tree representation of the structure of code, which helps the system understand and manipulate code.↩︎

Announcement: Updated Esqueleto text-search & PostGIS bindings

2024-02-25T15:01:00Z

I’ve updated the esqueleto bindings for esqueleto-textsearch to include a tutorial and documentation so it no longer requires guesswork. Furthermore I’ve also created new esqueleto bindings for PostGIS.

Esqueleto is a more advanced query library that builds on top of the persistent ORM¹. Postgres text search brings Elasticsearch like functionality to Postgres. PostGIS is a spatial database extension for Postgres. This allows querying on longitude and latitude for example, or any geometry. As part of postgres it allows mixing normal relational data with space and geometry, which makes space itself relational!

Roughly four years ago I had drafted out an implementation for the admin pages for some company. This was part of my “trial” 2 days, where you just work as a “technical interview”. ² Ironically the admin pages ended up having far better search than the main app, unfortunately, implementing proper text-search to the main app was never prioritized. A colleague of mine had ported that to the current library. Recently I needed this again. However, I completely had forgotten how it worked and there was essentially no documentation. ³ Since I was somehow the maintainer of this package I decided to just fix this for all Haskellers.

PostGIS has a different motivation. My current contract is about finding stuff. Now I could do this with some geometry library in pure haskell. However, considering most logic is already in the database, I decided it was worth a try to see if PostGIS would be a good fit. There was already some existing art done, such as reading from the database, but there was no off-the-shelf, functional writing. I initially handcrafted some haphazard persistent instances to see if a full solution was feasible. It worked, and one interesting outcome is that I could combine this PostGIS implementation with text-search! For example, we’ve named locations on a map, and objects within the database have some point on the map, we can now search for some location on the map and find the objects through normal full-text search. The database does all the thinking. Because this worked so well, I got motivated to do the full bindings for PostGIS, and prevent the issue I just had with text-search.

I’m slowly transforming into a database engineer 😅. Anyway, everyone should try these libraries out! You’re in space after all, and you lose stuff all the time, go get searching in space! Let me know if you’ve any comments or suggestions. Furthermore, let me know if you need help with database woes, I’m becoming quite good at this.

Object relational mapping↩︎
I perform way better in such a situation than a take-home test, because it’s incredibly hard for me to get motivated to do throw away work, but here I did something useful.↩︎
Note that the hackage upload was also done by me, because my dear colleague had ported it to a library which wasn’t even published! It was just some github repository. After a couple years I decided to just upload it, right before I jumped ship to another company.↩︎

Lessons from creating a vacation rental.

2024-01-02T19:00:00Z

Villa Katalina in aruba~ is online.

You can book this wonderful rental now! This is the rebranded summerhouse Paradis. It’s a villa that can host up to 8 guests, located in Aruba. I think this rental project is slowly turning out to be successful, but it wasn’t easy. I went into this project knowing full well there would be trouble, and boy, there was trouble. I’ll list the issues and lessons learned from doing this for future me. I may wish to do this again; it was a fun ride. If you’re interested in doing something similar, I think this blog post is useful.

The prices for real estate were really good in Aruba compared to the Netherlands at the time of buying; I got about double of what I could expect in the Netherlands. Since my parents already have an investment property in Aruba, it seemed relatively easy to set up, not to say it was easy. Another reason for doing this was the low interest rates, which looked like attractive leverage. Furthermore, I already had experience with running an Airbnb in my own home, so I felt somewhat confident I could make this work. Because my own house is not in a touristic area at all, but Aruba is. I’d be lying if I said this only done for financial reasons. Ego played a large part in deciding to do this as well. For one, not being completely dependent on a single source of income allows me to take more professional risks and become more productive as a result. The other reason is bragging rights, not that I wish to brag, but the only way to be humble is to have something to brag about and not do that. This is so hard.¹ 😬

I considered many other properties before settling on this one; at first, I wanted to build a new place. This made sense to me since labor costs are low in Aruba. However, my father strongly urged against that, citing many experiences of even the smallest jobs going wrong. You have to watch contractors like a hawk if you want to do this. I experienced such issues as well with some contractors while doing the pool house. For example, the fuse switch in the pool house couldn’t switch off because the plastic door was jammed too close to the switches, giving no space to move. If you’re considering building on Aruba, keep in mind you have to be there, or have someone trustworthy there, and watch the contractors.

The amount of corruption or incompetence² on Aruba is rather staggering. For example, the notary Johnson and Johnson, took ages to close when I was buying the place. The closing date was at the end of July; I assumed it would be done before that time, but we closed in October. I was familiar with Aruba’s culture of taking things slowly, but I didn’t expect things to take this long! What I could have done differently in this case is contact the notary more often for progress updates. I think they just forgot about this deal or something. I was even told this was exceptionally slow. Because the closing took so long, I essentially missed out on opportunity costs. I could have opened the Airbnb two months earlier, in March instead of May, which means I missed all that income.

Another example of incompetence I’m currently facing is with the power company Elmar, and Mr. Green solar services. The solar panels were installed on 15 August, and I was promised they could at least be turned on from that point, subtracting power generation from my usage. It’s now January, and this still hasn’t happened. Even after that, the excess is still given away to Elmar for free, after some “mythical” inspection which has to take at least 6 months. This delay is costing me the full power bill, which is 250 euros a month. I have no idea why it takes this long; it’s not a big island! Having learned from my Johnson experience, I’m contacting Mr. Green weekly.

What also caught me off guard was the fluctuation in exchange rates, which caused a lot of stress. The euro fell sharply against the dollar while I was waiting for the deal to close, but the listing price was denominated in dollars, and I had euros saved up! This made the property effectively 10% more expensive. Considering the price was 180,000 dollars, this cost me 18,000. I must admit, I shed a tear over this. In the future, I should put the closing money in an account of the same currency as soon as I sign.

However, what I’m really happy with is not going into hyper debt. My father, who financed half of this investment property³ with a mortgage-like loan, recommended I look at more expensive properties. So, I was looking at properties around 400k at first, however, I decided to go with a cheaper option instead, at 180k, even though the lender would have agreed to the more expensive option. I’m really happy I did this because during the renovation, I decided that a pool would drastically increase the appeal, allowing me to charge much more rent ⁴, but this would require another 30k investment. This was still possible because the monthly payments weren’t as high yet. The biggest advantage of these more expensive properties was their proximity to the beach. However, I think this is somewhat overvalued, because public transit is poor in Aruba, so unless you’re right on the beach, tourists are going to have to rent a car anyway. In this regard, my more budget-friendly stay offers a lot of value to tourists.

After handover, when doing some renovations, many people will recommend extending the scope of a project significantly. This can come from friends and family. I had discussions about adding more walls for additional bedrooms, or installing doors, or various other changes, but most of these wouldn’t be very profitable. However, they would extend the timeline, so I refused most of these requests. The first guest recommended me to put a fence between the pool and the patio, to ensure the kids could play safely outside. This was a minor change overall, and could happen with other guests staying there. This change solved a real issue that this guest, and by extension other guests, were concerned about. None of my family or friends had suggested such a change, as nobody had stayed at the place with kids. I kept focus on solving all these problems guests highlighted in reviews, when they were reasonable, such as “we miss wine glasses”, “we need trash cans in the bathrooms”, “we don’t have oven-ready pots and pans, but there is an oven”. Some requests were impractical, like “move the pool behind the house”, and I ignored those. Addressing these issues usually cost less than 100 euros, and even the fence was done for less than 1000 euros. Compared to the amount I spent on the pool and the house itself, this was minor and a fixed costs. However, after addressing these issues after the first few visits, the complaints stopped and people began giving me consistent 5-star reviews, even though the price had increased significantly in December. I learned that the difference between a good stay and a great stay lies in many small details, which are hard to anticipate. Guests will share their problems if you ask. Recommendations from family and friends during renovations need to be carefully evaluated, as they’re not always beneficial.

I learned that people who are good at one job may underperform if you let them do a slightly different job. For example, the guy who built the wall (Cesar) and the tiling around the pool did some great jobs, but then I let him install a shower, and he inadvertently installed the faucet upside down. It’s still in this awkward position. Aruban contractors often will say they can do something, even though it may be their first time doing such a job. One has to be careful about this. I like this particular contractor, however, because he can’t speak English and I find the non-verbal communication rather amusing. So, I’d hire him again, but not for showers!

Knowing skilled people is something that is difficult for newcomers in Aruba. I got incredibly lucky in that regard, considering my parents had already sifted through so many contractors for their Casa Aruba rental property. For example, I was able to hire a local carpenter to redo the entire kitchen for a third of the price I would have had to pay if I had used a European kitchen supplier. The kitchen looks better, too! The entire house now has this scaffolding wood style, giving it a robust and cozy ambiance.

I was also blessed with the support of my parents. Besides financing, they supported me with the paintwork, and there was a lot of it. Without their help, I probably would have spent another 8k on that and ended up with a worse result.

I’m not sure if it was financially worth it. Intuitively, I think it was, but I still need to run the numbers more thoroughly once the first year is finished. My preliminary observations suggest I need to make more than 1500 euros a month to turn a profit, and more than 3000 euros a month to cover mortgage payments as well. I have a strict payback schedule, because I didn’t like the high interest rate from the private market, but this will be completed in 6 years. It’s difficult to measure if it’s “worth it,” because the first few months of this project were naturally the worst I could have had. I had no reviews so I had to compensate by lowering the price. Furthermore, things like solar panels and my custom rental booking site are still not operational⁵. The solar panels should reduce the electric bill from roughly 250 euros to close to 0, and the rental booking website eliminates the need for Airbnb, which takes 17% cut of bookings.

Ironically, despite all the setbacks, I may do this again. Seeing the house come to life, reading all the reviews from people, and experiencing their appreciation is fantastic. However, I’d like to confirm first if it’s financially worth it. After it has been open for a year, I plan to create an private overview of this investment, detailing how much it brought in and how much I spent on it. Furthermore, I currently simply do not have the cash to set up another one. I feel I’m quite good at this. However, if you happen to run a rental and have ideas for improvement, please let me know in the comments!

Of course, I did tell people about the BnB I bought. One could even say this post is a giant brag, but hey, I’m not forcing you to read this, my dear reader.↩︎
I say incompetence because, according to Hanlon’s razor, it’s usually just that.↩︎
He got a 5% interest rate while other mortages were going around for 1.5%. So it’s not like I got a handout.↩︎
Say 50% more.↩︎
If it is operational this link should work.↩︎

Follow Up on the Follow-Up

2023-11-11T00:00:00Z

This is a follow-up on the follow up of release rodeo. Okay, I should stop doing these, however, I thought the situation was a bit too ironic not to record. On my final week working for that company ¹, the product manager involved found my original blog post!

A colleague and I were discussing some performance degradation, and I brought up the event sourcing post, because this company also used event sourcing and it was maybe related. However the PM apparently followed that link, thought it interesting, looked around more, and found the release rodeo post!

He got rather upset. His issue was that I didn’t approach him directly. I think that’s fair, I feel I would do this under normal circumstances, however due to history,² it was difficult for me to take that step. I did solve the underlying process problem to him. ³ There may not even have been a need to single out specific roles in the original blog post, I could’ve chatted in more general terms and get the story across. I think in the future I won’t do that anymore. It’s just distracting to single out people (even anonymously). Because it wasn’t a post about venting, although I admit, there was a fair bit of venting. It was about identifying and solving the underlying issue, despite being frustrated! Reading it back now I don’t think I said anything wrong. Aside from using chatgpt to make it sound like it’s not me 😅

Anyway this is the last post in this unexpected “series”. I learned that I should be careful with what I write, these pages are read by more people than I thought! Please leave a like, subscribe and share your job-rants in the comments below.

I decided to leave due to unrelated reasons. It’s a good place to work, however I had some tax issues forcing me to quit.↩︎
At some point the company had hired another manager whose only skill was office politics. Which caused me a lot of grief. This other manager brought the product manager from the original blog post on board and for some reason I had an association in my brain, perhaps unfairly.↩︎
Perhaps in a not too social way, but what do you expect from someone who obsesses over computers for most of their life!↩︎

BONUS Announcement mysql pure unfork

2023-08-14T17:37:00Z

Good news! I’ve come to an agreement with the maintainer of mysql-haskell, winterland. I’ll become a co-maintainer.

What this means in practice is that I’ll deprecate mysql-pure and merge the changes back into mysql-haskell. This will make upgrades far more convenient for users. That’s you! It isn’t a big deal for me either way, as I simply want the project to build reliably. There’s no other agenda than that, so convenience for users is paramount. persistent-mysql-pure will, from now on, depend on mysql-haskell again. This fork is still necessary for the persistent users out there, because my announcement and issue failed to reach that maintainer.

When winterland becomes active again in the future, as they intend to, I hope to hand off maintainership back to them.

Announcing mysql pure fork

2023-08-13T13:37:00Z

I’ve forked mysql-haskell and persistent-mysql-haskell into mysql-pure and persistent-mysql-pure. The original packages were no longer maintained and frequently caused me issues during GHC upgrades, so I decided to take over maintainership. For example, bounds were outdated, or I needed minor patches found on obscure branches in unrelated repositories. I’ve set up various CI automations. Specifically, I’ve integrated automated version bumping from nomeata and incorporated haskell nightly builds¹.

I forked because I didn’t want to go through the process of obtaining the Hackage namespace, which I found discouraging. Furthermore, mysql-pure merges several dependencies of mysql-haskell into one package. word24, binary-parsers, and wirestreams are all now part of this one package. This presents a somewhat different perspective on package organization, which might warrant a new package anyway. However, practically no third-party packages depend on the packages I merged, So having them separated results in more work nobody appears to benefit from.

This instantly found an issue in GHC on Windows for the linker: https://gitlab.haskell.org/ghc/ghc/-/issues/23835.↩︎

Follow up release rodeo

2023-07-17T00:00:00Z

This is a follow-up on the release rodeo.

After I wrote that blog post, the CTO found it quickly by coincidence. He was quite okay with everything I had written. However, he took issue with that I called his critique “scolding”. I suppose he didn’t exactly scold me. I described it that way because it aligns with my personality: I’m an honest person, I have high standards, and I take criticism extremely seriously. Even when someone says, “You’re kind of doing this badly sometimes,” I interpret it as “You suck, Jappie, you need to do better”.¹ This doesn’t mean I’m unkind to others, however I put myself to high standards. He also warned me that if the PM were ever to stumble upon this post, they might get quite upset. I am aware of this possibility, which is why I didn’t advertise this article at all. It seems rather unlikely to me he’d go read a tech blog anyway. The point wasn’t to create more conflict, but instead to analyze the situation and prevent it. ²

A few days later, another engineer found themselves tangled in the release flag situation. At this point, I realized it was crucial to inform everyone about how these release flags should work. Therefore, I set out to describe the current situation and propose some recommendations. After outlining the recommendations, I realized the points of contention weren’t numerous. The process worked in principle (aside from some peculiar naming conventions), and I believe that most of the involved parties were simply uninformed. I created a GitHub RFC³ and tagged the relevant parties on it. This enticed them to read it and become informed and give comments if necessary. As an open discussion, this also helped get clarification on points I had missed. Since then, we’ve had no more release issues. It appears that writing down the process and informing everyone resolved the issue.

A lesson learned from this experience is that anger can be rather productive. If there’s something you can do about the situation, it’s productive to think about why you’re angry and devise a plan of action. In this case, I indeed solved a rather annoying but important issue by taking a relatively simple action. Anyone could’ve done this.

Note that this post has a follow up.

From this, we can also derive the reason for my bluntness when angry. For a mind that is harsh on itself by default, being nice requires extra effort.↩︎
And also a little bit of venting!↩︎
Request for change, eg an invitation to discuss a change in process or system.↩︎

MySQL Persistent Support for Haskell on Windows

2023-07-01T15:44:00Z

Using Haskell on Windows can be very useful in a WAMP like situation, where the main legacy codebase is stuck at PHP 5.6¹ . However, any new pages can be written with Haskell, which is much easier to upgrade since the compiler will inform you about most changes. New pages can, for example, be written in Yesod, and then simply linked to, or posted to with normal HTML/HTTP. The database can be used as an API to interact with PHP state. Although I’ve not set up a full Yesod web server on Windows (yet), these steps include the hardest part, the database connection. Feel free to contact me if you want help with that.

This is a small instructional blog post, primarily for my own reference, on how to get MySQL support for programs on Windows. These steps are for native Windows support. There’s no WSL involvement. Specifically, we are obtaining MySQL-persistent support with the help of the pure Haskell MySQL bindings. Because the bindings are in pure Haskell, they’re easily portable. Haskell abstracts most operating system oddities away.

There is support for cross-compilation on Windows somewhere available here. However, at the time of writing, it has not been implemented in mainstream nixpkgs. Therefore, future Jappie, it would be better to abandon your Nix aspirations and follow these steps.

Install ghcup

Cabal is a package manager for Haskell only. It assumes that native bindings exist. We’ll provide these using msys2, which offers a shell and system isolation. On top of that, we utilize pacman. With this combination, we can create a Linux-like environment to generate native Windows executables! We use ghcup to manage the versions of these programs. The name comes from ghc, the main compiler. this is also installed. ²

Follow the instructions in the install link: https://www.haskell.org/ghcup/#ghcup-instructions-win

Earlier versions suggested using Chocolatey, but don’t use Chocolatey. It is akin to apt and can cause significant problems with Haskell packages. If you need system package management and Haskell package management, use Nix. It’s the only package manager that seems to work reliably. However nix has no native Windows support.

Answer the other questions:

   default path "C:\" -- 1
   cabal to C:\cabal -- 2
   HLS N -- 3
   stack N -- 4
   msys2 Y -- 5

The path used for ghcup isn’t important.
Again, it doesn’t matter. Placing everything at the top-level C makes it easy to find.
I don’t use Windows for editing. I mounted the project folder as a virtual box folder. On Linux, I use Emacs to maintain changes, and I run the Windows based cabal command in this mounted folder.
We don’t use stack as it adds another layer of complexity on top of Cabal.
This is necessary to build the required system dependencies with relative ease.

Keep in mind that you need to reopen the terminal for the new programs, such as cabal and ghcup, to register on the $PATH.

Build manually with msys2

Next, you need to open an msys2 terminal (which is different from a PowerShell). To install git, use pacman in a mingw64 terminal, which can be found in C:\ghcup\msys64:

pacman -S git

There

are two spells, and I’m not sure which one made it work:

cabal user-config -a "extra-prog-path: %HOME%\.ghcup\bin, %HOME%\AppData\Roaming\cabal\bin, C:\\ghcup\msys64, C:\\ghcup\msys64\mingw64\bin, C:\\ghcup\msys64\user\bin" -a "extra-include-dirs: C:\\ghcup\msys64\mingw64\include" -a "extra-lib-dirs: C:\\ghcup\msys64\mingw64\lib" -f init

I also did this:

    $Env:Path += ";C:\ghcup\msys64\mingw64\bin"
    $Env:Path += ";C:\ghcup\msys64\usr\bin"

After these steps, it worked. Now cabal can find git, and therefore download our patched versions. Fret not, I’m working on getting these changes into Hackage directly. It just takes time to do all the politics involved and give people the opportunity to reply to my requests. However, even if I manage to get the changes upstream, there is a big chance you’d need one native dependency or another anyway, which this setup will give you.

Cabal

You can generate an intial configuration with cabal init, then add persistent-mysql-haskell dependency to your build-depends: packagename.cabal file:

  build-depends:
    , base                      >=4.9.1.0 && <5
    , persistent-mysql-pure

In combination with the cabal.project file, cabal should be using git instead of hackage to find that package. Test it out with with cabal build.

Conclusion

This should set you up with Windows development for Haskell in more complicated setups than just some puzzles. You’re probably better off getting an Ubuntu VM if you just want to build something new in Haskell. However, for legacy integrations, this will work. If not, please leave a comment below, or contact me.

Update: Cabal edit

The cabal section used to contain below text, however since I’ve forked mysql-pure. this is no longer necessary.

You need to instruct Cabal to use the correct MySQL packages project.cabal:

packages: .

allow-newer: wire-streams:bytestring, binary-parsers:bytestring

source-repository-package
    type: git
    location: https://github.com/chordify/persistent
    tag: cf735587e590369e62168dacc2c3c2411493ae6d
    subdir: persistent-mysql-haskell

source-repository-package
    type:git
    location: https://github.com/jappeace/mysql-haskell
    tag: 8f95f0a4749b888eba96173378acbede3955ab60

source-repository-package
    type:git
    location: https://github.com/naushadh/word24
    tag: 1cc234d53923c270e888fdeac868c34306c43c70

source-repository-package
    type:git
    location: https://github.com/naushadh/word24
    tag: 1cc234d53923c270e888fdeac868c34306c43c70

You can generate an intial configuration with cabal init, then add persistent-mysql-haskell dependency to your build-depends: packagename.cabal file:

  build-depends:
    , base                      >=4.9.1.0 && <5
    , persistent-mysql-haskell

In combination with the cabal.project file, cabal should be using git instead of hackage to find that package. Test it out with with cabal build.

Forever, because upgrading is too painful and introduces to many runtime bugs.↩︎
To be clear, the major difference with nix is that nix in principle leverages cabal and ghc, but nix manages the versions of these, but also solves the native part in a reproducible manner. This means I can write a script for a build and it mostly will keep on working forever, unless source mirrors disappear for example.↩︎

The Release Rodeo

2023-05-31T20:14:00Z

In the beginning, there was nothing. We merged completed features after a month or so of development, leading to large PRs and often unexpected implementations. Then, feature flags were introduced. This allowed the merging of code in chunks, enabling QA to test thoroughly. The people in charge could now also decide when to release. Because this also facilitated more, smaller PRs¹, it also made the reviewing process much easier. Everything was awesome in paradise, but not all was as it seemed…

The flag count grew, and salespeople started to realize they could enable flags for certain customers. This went well, in the beginning. Except that these features were still in development. So, on one occasion, a client success person panicked, after a fancy file uploader behind a feature flag disappeared. This happened because it was merged into another feature flag². Now certain customers no longer had access to the file uploader to which they had grown accustomed, breaking their entire workflow.

Note that in this setting, a customer is a large corporation with thousands of employees. So if one of them becomes unhappy, the startup could lose a lot. Not just money but also reputation, which translates into future sales, potentially threatening the survival of the startup. Up until now, this hasn’t happened; all customers who’ve signed up have always been kept happy.

Anyway, long story short, since the feature was being used in production, I strongly urged its release. This was discussed internally, and everyone agreed on releasing this file uploader feature. Everyone. However, some requests were made, for example, there were redundant form elements. The manager suggested also deleting the old file uploading button. This was a good idea, because with this change the old flow would be quite broken; we had already removed error messages. The manager in charge was tagged and ignored the PR for several days, after which we decided to merge. We had already decided to release this feature, so his approval wasn’t necessary according to our customs.

Fast forward to the next week, and our dear client success person was once again in a panic. It turns out that one of our older customers was still using that confusing button. So we had to put it back. I warned them that this old flow was broken, lacked a lot of elements and would cause confusion. They just wanted it back however, so I complied. This was a rather minor change, however the manager in charge of releases, did what any good leader would do and blamed the developers. Even though this was his own suggestion.

I wouldn’t be writing an angry blogpost if it was the only incident. There is more. He was right about the confusion that followed from leaving the button. QA got confused, another dev got confused, I’m surprised more customers weren’t complaining about this. Although they likely just ended up feeling stupid and didn’t complain, which is worse. It seems impossible to delete that darn button, however.

Moving forward a month, a large client was preparing for a demo. I started working on a “detection” feature which we needed to impress said client. We had a meeting to figure out what we could get done before the demo. From this we learned we had to modify the Elm apps. These Elm apps represent HTML tables that open up a modal with the specific data comprising those tables. It’s like eight different tables, all looking slightly different, all piped into the same Elm app with flags specifying how it should look. We’ve had production bugs with these apps. They just stop functioning and you’ve no idea why. Furthermore, one of the best features of Elm, its debugger, is unusable because they overlap, since there are several Elm apps displayed on the page. As you can tell, I’m not too happy having to modify these. It’s a lot of pointless complexity.

Anyway, the demo went well, as they skirted around those Elm apps and didn’t open the modals, which show the wrong information our new feature detects. Now, 2 weeks later, the manager decides it’s a good idea to release this feature. I learned this through an announcement in #general. But I wasn’t finished yet, since I was away for 1 week on vacation. I’m kind of shocked. If clients see those numbers don’t align with that new “detection” feature and our Elm apps, they’re going to freak out. Because they’re going to think our platform messed up their data somehow. So I reply to his announcement, because I know he’s well aware of this.

I’m not sure if I’d release the “feature” because “our Elm tables” are kind of incoherent right now. But if you’re happy with it 🤷‍♂️

Here I redacted the names of the specific features, replacing them by “feature” and “our Elm tables”. Now, is this blunt? Yes, but there is a lot of money on the line. The same manager who blamed the developers for the missing button incident was also responsible for this release. Naturally, he employed his remarkable management skills and decided to throw a tantrum, persuading the CTO to scold me. Indeed, I was reprimanded and praised simultaneously. It was beneficial that we detected this issue early, but I was told that my communication was too blunt and insufficiently precise. I believe there’s truth in this. I could’ve been calmer, but maintaining composure is challenging in a stressful situation. Part of this blog post is about circumventing such situations in the future. The manager could’ve performed better too. For instance, he could’ve inquired in #tech before releasing, “Are there any serious issues remaining with feature X, Y, or Z?” Just like one would do in a team. Alternatively, he could’ve engaged with my response in #general to figure out what was happening.

I was angry when starting to write this. And while I wanted to blame a particular individual, I won’t. This is the wrong mindset. ³ Rather than blaming specific individuals, I prefer to think in terms of systems. How is our process so broken and causing stress? Well, no one has cared about thinking about how all of this should work internally, at least. This is so bad we don’t even have a proper definition of what it means to release a feature:

We delete the feature flag and old code.
We enable the feature flag for all customers as a setting.

Point 1. should happen eventually, but after reading the external communication process, I think we should first do 2., wait for some time, and then delete the feature flag. It’d be nice if we put a limit on that as well, so developers can safely delete at some point. Even though there appears to be documentation on what clients need to be informed, internally we have no clue who to tell what or get involved. We don’t have a grace period either, allowing people to comment on a release. For example, the customer success people can just say if a button is missing in this grace period, or a developer implementing it can just say that hey, we’re showing incoherent stuff. There is no guidance on how big a release should be, are smaller ones better or bigger ones. It seems like the customer success and marketing people prefer big releases. So they can use it as a marketing moment, and have an easier time demoing more features in one go. But I suspect for engineers, smaller ones are preferred, as this allows them not to be overwhelmed if issues arise.

How would you release? I’ve always learned to release early and release often. But ever since the introduction of feature flags, we’ve just not been releasing, until it became a problem. In typical startup style sledgehammer fashion, the problem was solved; release a bunch of stuff, and for the unfinished parts we peeled them off, putting them behind different flags. But does the path towards releasing this software have to be so stressful? I feel we’re doing it wrong. The lack of some kind of formalized strategy causes some issues here, I think. We’re just doing stuff, inconsistently, and coordination revolves around people being shocked and in a panic. This clearly isn’t working.

Documenting what I believe should happen, without naming specific individuals or organizations, ⁴ has at least alleviated my anger. I’m not sure if the procedures I’ve outlined constitute a good process, hence I welcome comments, but I believe it’s a step forward from doing things haphazardly. Getting a timeline for a release at the very least would be beneficial. Therefore, with my anger tempered and a sense of progress, I deem this a successful blog post 💪. Please share your thoughts in the comments.

Note that this post has a follow up.

Pull requests, the primary mechanism in which developers align their changes. It opens up a moment for questions or comments.↩︎
This itself was also problematic. The issue was that designers began wanting to make changes (as is their job), but developers started asking questions about how to implement these changes, and realized it was easier to just merge features.↩︎
Growing deaf to dharma and being trapped in samsara.↩︎
Could be anyone really, since I do work freelance :)↩︎

About my startup: raster.click

2023-02-25T20:40:00Z

As a founder of raster.click, I was passionate about providing rostering systems for restaurants that were easy to use and affordable. I worked hard to build a functional product that our users loved, and I was proud to have three paying customers (eg restaurants) and around 90 active users (employees).

Despite initial success, I realized that growth was not fast enough to become economically viable. I was unable to attract enough paying customers to cover costs. As a self-bootstrapped startup, I did not have access to significant funding, which made it challenging to scale the company and invest in marketing efforts. I also struggled with balancing time between product development and customer acquisition, which slowed growth even further.

In 2019, I put in a lot of effort and remained dedicated to Raster.click. However, I eventually had to make the tough decision to discontinue my work and return to contract work. It was a heartbreaking decision, as I knew my product was loved by my users. They continued to use it until the COVID-19 pandemic hit and practically all usage stopped. The restaurant industry was severely affected by the pandemic, rendering rostering completely useless during that period.

Although I still have an appreciation for the tech stack I chose - reflex + servant + beam - I’ve come to realize that a cool tech stack alone doesn’t matter to users. While it may result in faster product development, it ultimately doesn’t help if I can’t gather user requirements or attract new users. In hindsight, I could have been more productive by using yesod and relying on browser defaults instead of creating a single page app. However, the decision to end my startup was not based on the tech stack. Despite rolling out new features, I simply wasn’t gaining enough customers to make it a viable business.

While I am proud of what I accomplished with Raster.click, I learned valuable lessons about the challenges of building and growing a startup without significant funding. Such as:

The importance of having a clear and viable business model: It’s essential to have a solid plan for how your startup will generate revenue and grow sustainably.
The need for effective marketing and customer acquisition: Even if you have a great product, you need to be able to effectively reach and convince potential customers to use it.
The importance of balancing product development and customer acquisition: While it’s important to create a great product, you also need to be able to attract and retain customers.
The value of having access to funding: Without significant funding, it can be challenging to scale your startup and compete effectively in the market. Not to mention the stress caused by seeing your life savings slowly drop to zero.

I will take these lessons with me as I move forward with future endeavors.

Although I stopped working on this project, it’s still operational. It costs very little money for me to keep this operational. That’s because it runs next to this site, videocut, and massapp on the nmm approach. Yes, it’s sad I stopped working on this project, but if anyone is interested in using it, it’s right there.

Restoring msyql backup system on windows.

2023-01-17T17:00:00Z

This is a quick after action report on restoring backups. This is mostly for future me to know what I did.

The backup scripts on the main server stopped working. The IT person at that company had already figured out this was because of the database not booting on the machines which were supposed to receive the backups, but figuring out the underlying cause was beyond his capability. So I was called in. Yes. I’m slowly turning into a database administrator at this rate.

So xampp¹ told us to look at the logs, those said:

error: trying to access page number 123 in space 0

Stack overflow said that we need to recover from backup if innodb_force_recovery doesn’t work. It doesn’t say how. Remember, we can’t even start the database! What I did instead was deleting the innodb0 file, and the innodb.log files. Remember, I actually don’t care about the data, it’s just a backup machine. This made the database boot again, making it ready to receive backups.

One more peculiarity occurred (of course). The script would timeout saying it’s a connection error. However this error happened every time I executed the script, and I’d assume that at an office the connection is pretty reliable. This made me think it was just mislabeling the error. Once again stack overflow agreed. This database has lots of blob data, so setting max_allowed_packet to 64M indeed fixed the connection error.

I’m starting to think I should move this entire stack over to postgres, at least that doesn’t randomly start failing, or has mislabeled errors. But I know that’s next to impossible with this clients’ resources. So for now, Jappie is here for all your database concerns.

Why are they using xampp? Legacy choices. Remember I don’t get to chose their tech stack, that has been decided more then a decade ago. I’m just keeping it operational.↩︎

The peculiar event sourced deadlock

2023-01-15T22:30:00Z

One thing that always surprises me is how casually serious problems are phrased by business people in their blissful ignorance. “Hey why am I seeing the down for maintenance screen?” “Oh try it now, the pack uploading has finished”, Said the QA engineer to the product manager. Once I saw this on slack, I grew really suspicious and started asking questions. After all, isn’t it a bit odd we’re seeing a down for maintenance screen in one part of the system, simply because another part is being used?

Initially we thought this was caused by high CPU usage. The graphs showed high CPU load while processing packs, so maybe the rest of the system was being deprioritized somehow. Before assuming that was the cause however, I decided to reproduce the issue first. Here I noticed I could for example load the risk index easily (a read operation), but connecting a risk to a pack (a write operation), would hang forever. This made me suspect that the issue wasn’t CPU usage at all, so I asked Postgres to list it’s locks. Which showed several locks in progress. This lead me to the event source system. The event source system is at the core of all our business logic. In essence, it provides a ledger of all important business write activities that can happen. This is useful for auditing purposes for example.

Welcome to an after action report of a complicated system level bug. It took me a week to find a satisfying solution. To start I need to sketch context. I’ll only use raw SQL because this entire story is related to the database and how we use it for event sourcing. So consider the tables of an event source system:

CREATE TABLE event (
    id serial PRIMARY KEY NOT NULL,
    payload jsonb NOT NULL,
    type character varying NOT NULL,
    created timestamp with time zone NOT NULL
);

CREATE TABLE event_last_applied (
    id serial PRIMARY KEY NOT NULL,
    event_id bigint NOT NULL REFERENCES event (id)
);

In here the type and payload fields contains the information to (re)apply that event. The type will indicate what business logic or queries to execute, and the payload holds information for that logic. As we’ll see later, these queries will involve modifying other normal tables within a transaction. This application of events, or re-application through business logic or queries is called projecting. A type can for example be create-user and the payload would contain the data required for creating said user, for example {email:'hi@jappie.me'}. The id provides a unique global ordering, and the created field contains a timestamp of when the event was created, which is used for database administration purposes. Finally, the event_last_applied table is used to indicate whichever event was last applied, so the system can figure out if additional events need to be re-projected from the event table.

Inserting an event works by projecting an event to normal Postgres tables in a transaction. Once this operation is not rejected by foreign keys, type errors or program exceptions, the event gets recorded in the ledger, also known as the event table. For example:

begin;

/* left out projection code, insert user into tables here,
or do other projection stuff, as dictated by the event type*/

INSERT INTO event (payload, type, created)
    VALUES ('{"email":"hi@jappie.me"}', 'create-user', now());
INSERT INTO event_last_applied (id, event_id)
SELECT 1, max(id) FROM event
ON CONFLICT (id)
    DO UPDATE SET
        event_id = lastval();
commit;

If the projection fails the entire event gets rejected, which means all changes within the transaction get rolled back by Postgres. This applies relational guarantees, to a non-relational system trough a transaction. We also weave this transaction trough business logic code, so that in case of an exception, we rollback. Quite an elegant solution, which I did not invent.

On system boot we figure out if we need to reproject or not, the query is rather simple:

SELECT type, payload FROM event
WHERE
    id > (
        SELECT event_id FROM event_last_applied
        WHERE id = 1)
ORDER BY
    id ASC;

which returns something like this, telling the system what to do:

    type     |          payload          
-------------+---------------------------
 create-user | {"email": "hi@jappie.me"}

With that, we can reproject, also known as replaying history. Replaying history involves truncating all tables that are event sourced. And then truncating the event_last_applied table, which in this case just removes the one row. Then the system will notice it needs to replay events on boot for example. This is a rather dangerous operation, because if any event fails, you may have potentially lost data. A lot of things can go wrong with a large history, foreign keys, exceptions, serialization mismatches, events out of order etc. Transactions can help here as well, and make this re-projection safe.

Deadlock

There is one more important piece of context: An event maybe composed with other events into larger transactions. For example, if we create a user, we may also assign him to a company within the same transaction. In SQL that looks like this:

BEGIN;

/* left out projection code, insert user into tables here */

INSERT INTO event (payload, type, created)
    VALUES (
        /* whatever event source data*/
        '{"email":"hi@jappie.me"}', 'create-user', now());
INSERT INTO event_last_applied (id, event_id)
SELECT 1, max(id) FROM event
ON CONFLICT (id)
    DO UPDATE SET
        event_id = lastval();

/* left out projection code, connect user to company */

INSERT INTO event (payload, type, created)
    VALUES (
        /* whatever event source data*/
        '{"company-id":2, "user-id": 1}', 'connect-company', now());
INSERT INTO event_last_applied (id, event_id)
SELECT 1, max(id) FROM event
ON CONFLICT (id)
    DO UPDATE SET
        event_id = lastval();
COMMIT;

Transactions form proper monoids, and they can grow arbitrarily large. This is good because even for large chuncks of business logic we always gaurantee our event log remains in a valid state. We’d expect our re-projections to always work, because only correct ones get recorded. Where does this go wrong then?

The issue is concurrency, consider connection A and B:

A opens a transaction and inserts a user, but has to do other projections and event insertions as well
B opens a transaction and wants to insert an event, B has to wait until A completes. This is because A made an update to the event_last_applied on row number 1, as part of the insert event logic. This row is locked until A completes, so B has to wait.
A completes and releases the lock on row 1.
B can now complete as well.

This is not a deadlock as long as A completes. B can wait a long time because our transactions can grow arbitrarily large. For example when we’re inserting millions of rows of data, taking up half an hour. Which is far beyond the HTTP session length of 30 seconds, or whatever length a user finds acceptable. This was indeed the production bug encountered at supercede. One user was doing pack ingestion, which involves reading millions of excell file rows, and the rest of the system became unusable because of that.

Now what?

At first I started with the most obvious solution. I re-grouped how event sourcing took place. I put the event sourcing code at the end of the transaction in pack ingestion, so that the event source table remained available for other transactions up till that point. Because event sourcing is only a small part of normal transactions, this created a small locking window. Thus this worked! However it only worked for this transaction with pack ingestation, I didn’t know if there were any other transactions like this in our code base. Furthermore, I had to bypass parts of the event sourcing interface to make this work. For example, I had to project events by hand, and insert events by hand, rather then using the internal library. I decided this was a bad precedence to set. I was afraid other engineers would copy this approach when it wasn’t necessary. So I went looking for other solutions.

Another idea is that instead of doing the large transaction, we could split it up into smaller ones. Allowing other events to clear while this bigger one was in progress. I didn’t like this either. For one this code was old, tried and tested, making a rather large modification like splitting the transaction could introduce many unintended bugs. For example when cleanup doesn’t happen correctly on failure. I thought this was likely because this transaction was large, and covered many tables. Also our normal tools such as types and integration tests wouldn’t help a lot with guaranteeing cleanup. So this would become difficult to maintain fast. Which is problematic for a piece of code which is the “money maker”, and needs to change often. Furthermore I had a much more simple but thorough solution in mind.

I decided to redesign the event source tables. Naturally my colleagues exclaimed shouts of joy when I decided to modify an even older system. The event source system described above is almost as old as supercede. But I believed it was easier to modify, and more importantly, easier to test for correctness. Furthermore this would also solve the problem for other, possibly unknown, or future, large transactions. This change would keep our code easy to maintain and solve a bug. The new schema looks almost identical to the old one:

CREATE TABLE event (
    id serial PRIMARY KEY NOT NULL,
    payload jsonb NOT NULL,
    type character varying NOT NULL,
    created timestamp with time zone NOT NULL
);

CREATE TABLE event_applied (
    id serial PRIMARY KEY NOT NULL,
    event_id bigint NOT NULL REFERENCES event (id),
    created timestamp with time zone NOT NULL
);

The big difference is that we renamed event_last_applied to event_applied and added a created field. With this change, inserting events is also quite similar to the initial system:

BEGIN;
INSERT INTO event (payload, type, created)
    VALUES ('{"email":"hi@jappie.me"}', 'create-user', now());
INSERT INTO event_applied (event_id, created)
SELECT last_value, now() FROM event_id_seq;
COMMIT;

The big difference is that instead of modifying always row number 1 to be the latest ID, we insert a new row into event_applied with the latest id. This avoids locking of row number 1. For re-projection we truncate the event_applied table, allowing the code to rerun all those events. The big difference is in figuring out which events haven’t been applied yet:

SELECT type, payload FROM event AS e
WHERE
    NOT EXISTS (
        SELECT 1 FROM event_applied
        WHERE event_id = e.id)
ORDER BY
    id ASC;

We compare the event table to the event_applied table, and return any events that don’t exist in that. We’re still ordering by id to ensure the correct order. Is this correct? Let’s consider concurrency once more with connection A and B:

A opens a transaction and inserts a user, but has to do other event source queries as well.
B opens a transaction does it’s projection work and wants to insert an event, B creates a new row in the even_applied table and completes. There is no need to wait since there is no single row lock. So B finishes.
A finishes it’s other event sourcing completes.

This doesn’t deadlock. However it’s not completely correct in that A get’s id 1. and B get’s id 2, but A’s transaction finishes after B by inserting another event with id 3. So on reprojection one of A’s events get’s applied before B. But in the initial projection, all of A’s event happened after B. So the first event of A is out of order. This may cause issues. This problem was also present in the original implementation, since an id is acquired before the lock waiting happens. I think a solution would be to group the events by transaction id, and then order by last created event. In this case all events created before B in A’s transaction would be pushed behind it by an event happening after B finishes. If we do that, the event table gets an extra field:

CREATE TABLE event (
    id serial PRIMARY KEY NOT NULL,
    payload jsonb NOT NULL,
    type character varying NOT NULL,
    created timestamp with time zone NOT NULL,
    transaction_id bigint NOT NULL
);

Our insert function retrieves the transaction id with txid_current:

BEGIN;
INSERT INTO event (payload, type, created, transaction_id)
    VALUES ('{"email":"hi@jappie.me"}'
           , 'create-user'
           , now()
           , txid_current());
INSERT INTO event_applied (event_id, created)
SELECT last_value, now() FROM event_id_seq;
COMMIT;

And our unnaplied events query now groups:

SELECT
    array_agg(type) AS types,
    array_agg(payload) AS payloads
FROM event AS e
WHERE NOT EXISTS (
      SELECT 1 FROM event_applied WHERE event_id = e.id
    )
GROUP BY transaction_id
ORDER BY max(id) ASC;

If we run that unnaplied events query on an event table like this:

id |        payload        |      type       | created    | transaction_id 
---+-----------------------+-----------------+------------+----------------
 6 | {email: hi@jappie.me} | delete-user     | 2023-01-15 | 77958
 7 | {email: hi@jappie.me} | create-user     | 2023-01-15 | 77959
 8 | {company-id: 2}       | delete-company  | 2023-01-15 | 77958

We’d get a result like:

             types             |              payloads
-------------------------------+-----------------------------------------
 {create-user}                 | {{email: 'hi@jappie.me'}}
 {delete-user,delete-company}  | {{email: 'hi@jappie.me'},{company-id: 2}}

Which is what we want. Even though the create user event happened while the delete user event was happening, the delete user event was part of a larger transaction. So the create user even should come first when re-projecting. This allows arbitrary sized transactions to project alongside each-other and provides better ordering guarantees then the original implementation.

Closing thoughts

Phew, that was a lot. I didn’t think this would become such a large post. Designing an event source system on Postgres transactions is rather hard. All I wanted to do is clear my thoughts on the matter, but that grouping issue is another bug I just found by writing about this 😅.

I think the biggest lesson I’ve (re)learned from the deadlock bug itself is to make sure you reproduce an issue first before diving into solutions. Even nasty business threatening system level bugs like these can sometimes be solved with some minor modifications to the system. If we had skipped this small step of reproducing the issue, we may have focused on the CPU observation and moved pack ingestation to a separate machine, which would’ve taken weeks to implement and not solve anything.

Furthermore, it’s humbling to see that even after having used relational databases for more then a decade, I still can learn new things about them. For example Postgres’ auto increment sidesteps the transaction, which was quite shocking to me. A rather important detail to keep in mind when reasoning about these systems.

I made a github repository for playing around with the queries more easily. I hope you enjoyed this article, please leave a comment if you have any questions or suggestions below.

Resources

The code in this blogpost
Postgres Lock monitoring
Blogs on event sourcing
Presentation on event sourcing

Summerhouse Paradis Aruba

2023-01-12T19:00:00Z

This post tracks the progress of a rental property I bought on Aruba. The goal is to rent it out short term to tourists. This video gives a rough overview of the current status:

Summerhouse paradis on aruba vlog: Colors, mold and a hole

A week later and we’re starting to work on the pool.

How do you even build a pool? Summerhouse paradis vlog III, 2022-12-18

Once again, a week later, my final week on Aruba, and the pool is almost concrete.

Why do I still write this blog?

2023-01-11T17:10:00Z

My motivation for blogging has changed over time. I’d like to revisit my initial reason, and discuss how this changed while maintaining the blog. After all I keep posting articles on here, something has to keep me going right?

We can read my original motivation. The initial goal was to make money apparently¹, through ads or Patreon. T-That didn’t work out so well. I now realize it’s quite hard to make money with blogging, you’ve to be really good! I’m not, and that’s okay. Furthermore from ads you need tons of visitors to get even the smallest amount of money. And for patreon to work you need to make something for a niche audience, and captivate them, I think at the time I didn’t really know what that would entail. Similar to patreon would be to use substack, however that didn’t exist at the time.

Another romantic desire not listed in that post was being remembered after my death. For example consider “Commentarii de Bello Gallico” (Commentaries on the Gallic War). People still read that 2000 years after Ceasars’ death. I was fascinated by this idea, to have your words echo trough the ages.

To achieve this I knew my writing skill had to improve. I hoped that trough blogging I’d get better. I knew my first posts wouldn’t be interesting or engaging, but I could learn from each post. So once I got this going, I placed google analytics on it, and soon realized no-one was reading this blog. I published something on the internet and no-one was reading this! I gained some solace from having carved out my own private corner, so cozy just for me. I learned at the beginning of 2017 the internet was already too big. Still my goal was to be an attention whore, and to get that you /need/ to post your content on aggregator websites like reddit or Facebook (depending on your audience). Peeps aren’t going to type in jappie.me magically in the URL bar, and Google ain’t just gonna find you without some links pointing to you.

Getting a big name that last trough the ages was a goal of Jappie from 2017/2018, I don’t think this is no longer a goal of mine. Once I’m dead, well, I’m dead. You can remember me all you want, that ain’t gonna bring me back, and who wants to go back anyway?

Great compromise

Although my desire for being remembered has faded, I still want to get better at writing. Because I feel this still covers up flaws I have. For example, I’m quite bad at self promotion. I started seeing blogging as a sort of propaganda machine ², that would represent me so I didn’t have to. In real life³ social interactions, I often come across as doubtful on my skills, and I’ve trouble of thinking what to say in conversations. Furthermore, I don’t enjoy convincing that others I’m any good at anything, because that feels like bragging to me. However, as a programmer you pretty have to do this. For example during an interview, or when trying to get another assignment.

So rather then changing my personality, I realized I could get better at writing, The blog was a great compromise.

Ironically, trough writing I discovered I enjoy writing. I get a lot of pleasure out of being precise. Merely putting my thoughts onto paper is apparently a step up in precision and clarity. It’s quite shocking and enlightening at the same, to see how chaotic and rambly your own thoughts are once you start putting them in words on paper. After a mere day of writing a passage you may question who wrote that, because it looks so foreign to you. This is especially true with fleeting thoughts.

But I’ve also noticed that converse on topics I’ve written about is quite easy. This has been a huge benefit for me. No longer I’ve to think of what to say, but I can fully focus on how to say it. Especially during interviews where the other party has decided to read what you’ve written about. In those cases, conversation becomes a breeze! This truly is an unintended consequence, but I guess blogging has helped my career.

Frustration

Another use of blogging is letting out professional frustration in a constructive manner. For example the failing in haskell post was directly due to error handling encounters in professional code bases. Explaining this in painful detail.

This also happened with the stack dependency management post at the time. Both these posts have seen a lot of user interaction as well, people started debating about it on reddit or hackernews for example. So not only does this help me vent, it also gets my name out there! However I feel it’s not the best idea to become a divisive-andy. Because I don’t think it’s an explicit goal of me right now is to make this blog popular. After all this is my corner of the internet. I like the feeling of putting words out there that no-one reads. Making this blog popular would put pressure on me to write popular content. But I want to write abut what I want to write, not what’s popular. I could’ve written more about elm for example, because that elm on fire post was quite successful. But after making fire, my interest faded, and so did my motivation.

I did notice however that once I got a blog post out on something which frustrates me, the frustration goes away. It simply no longer bothers me, because as far as I’m aware this is the most impactful thing I can do to solve an issue. If publicly complaining about something, putting my name on the line, doesn’t help, what will?

Nostalgia

Interestingly I noticed another reason to blog lately. I enjoy reading back my old articles! Remember that time I immigrated to australia for a year to do programming there? Or doing a cute lil’ tool survey back in 2017 for example. At the time I thought I was using some awesome stuff. Now I know it’s a rather basic setup. My nixos based configuration is far more powerful and flexible for example.

I should make more of these slice of live posts. I’ve been only doing technical stuff the last couple of years, and to be honest, I like writing about my own live as well.

Interesting Challenge

It’s one thing to write a coherent story, it’s another thing to make that story interesting. Imagine shakespeare being writtin as a cooking book:

Start the thunder and lightening. Grab a first witch, a second witch and a third witch. Let the first witch say: When shall we tree meet again, in thunder lightening or rain.

It’s boring! Just like when talking about programming, we need to make it interesting. The solutions described in programing langauges are meant for machine consumption mostly, writing them down in natural langauge is a completly different excersize. You can be much more lax in natural langauge than in programming. A misspelled word? That’s fine, the reader will be a bit frustrated but it doesn’t break the meaning of the story. You can’t get away with that in programming.

However, in programming you don’t have the problem of keeping your audiences’ attention. Are you using a disgusting font on your blog? No-one is going to read that. Are you re-using the same vocab over and over? I’m gonna stop. You need to write interestingly, which is much harder than just doing something grammatically correct.

This is what most business writers don’t get. Dumping a bunch of jargon in your post is going to shoo away people. Using politically correct opinions is boring. You have to take a position to attract an audience. This is hard.

Conclusion

I write for self promotion, to vent, and for fun. I think writing is difficult, but I enjoy it. Writing something interesting is even harder, but I feel it’s well worth the exercise.

Do you feel writing is fun, or perhaps boring? Or do you also have a blog? Please let me know in the comments below!

I long had forgotten this was a serious goal of mine.↩︎
Very much like “Commentarii de Bello Gallico”, which more realistically should’ve been called the genocide of the Gallic people.↩︎
Offline interactions↩︎

Zurich hack 2022 Denotational Design

2022-09-26T19:00:00Z

This blog post and after action report is three months overdue, but I participated in Zurich hack 2022. Zurich hack is a voluntary hackaton organized in Rapperswil-Jona ¹, with as theme improving the Haskell ecosystem and socializing. Naturally I chose to work on the most research-y project I could find. Sandy was happy to oblige with his denotational design project. Here we build an “infinite” baseless chip design, with a homomorphism in natural numbers to proof correctness, more on this in the proofs and programs section.

Our presentation was surprisingly good considering we slapped it together in 30 minutes. However, I think we could’ve done a better job at explaining denotational design, and we could’ve elaborated more on why proving matters. I shall use this post to fill in these gaps. For starters the presentation can be seen here:

I helped presenting², however most of the implementation was done by Sandy and Nathan. I wish I could’ve done more, but my Agda isn’t good enough yet. I helped with cheering on their proving efforts and coming up with ideas for the design.

Denotational design

Let’s begin on what denotational design is. You could watch a video on this, but summarized in my own words:

We should decompose parts when possible.
Abstractions shouldn’t leak.
We should look for elegance.

The first point is “We should decompose parts when possible”. This means breaking up our design in such a way we can re-use parts into a larger whole. For example in Zurich hack, our first designs was a large record for multiplication that had everything baked into it. Then someone had the idea to split that record into a separate addition and multiplication records and re-express multiplication into addition. This allows us to work with the simpler problem of addition, before tackling multiplication. Which is what we eventually settled upon as well. I don’t think decomposition has been stated as an explicit goal of denotational design before, but it feels implied. Perhaps in Conals talk, “principled construction of correct implementation” can be interpreted as such.

The point about “Abstractions shouldn’t leak” is quite interesting. We wish to provided a simplified view of the world to the user through abstraction. In practice this means we should hide the implementation from the user. I once suggested for example to add a xor operation to our record to get rid of the carry bit in certain cases. After some discussion we settled on not doing this because xor isn’t really a thing you care about when thinking in terms of semirings³. In other words, when thinking in terms of multiplication and addition, you don’t want to care about the bit representation. For more examples of “abstractions shouldn’t leak” I recommend the book algebra driven design.

The final point is “We should look for elegance”. Which should serve as a compass upon iteration. Here again I’ve an example from just after Zurich hack: The overflow bit in our addition record bugged me. It kind off exposes the internals of addition. So I decided to delete it in favor of doing a full co-product instead. Doing this would break both multiplication and addition records, the proofs have to be redone, I’m not even sure if it’s possible. So there is definitely a cost. ⁴ However I like that design, it’s more elegant because this would make multiplication have a product type as input, and addition a sum type. Which would have some nice symmetry. This is something we didn’t drive home enough in the Zurich hack presentation. The proof we presented looked impressive, but this isn’t something you necessarily want. An elegant proof and design is what you want.

Proofs and programs

I pondered on proofs in software and how dependent types interplay. Is the design I dreamed up correct? How do you know this? We used a property called a homomorphism to prove correctness. In mortal words, our chip design was interpreted into natural numbers and we showed that addition and multiplication would be the same for our chip, as it is in natural numbers.

To start talking about proofs, we need a design and implementation to proof correctness for. In our case this was a chip design in Agda. This Adder⁵ is something we settled upon after several iterations of design, but I’m cutting that part out for brevity:⁶

record Adder {τ : Set} {size : ℕ} (μ : τ → Fin size) : Set where
  constructor adds
  field
    add : Fin 2 × τ × τ → τ × Fin 2 -- 1
    zeroA : τ -- 2
    proof-add -- 3
      : (mnp : Fin 2 × τ × τ)
      → toℕ (digitize (P.map μ id (add mnp))) 
      ≡ toℕ (addF'3 (P.map id (P.map μ μ) mnp)) 
open Adder

Here we’re saying, to define an adder you need 3 things. You need an add operation 1, you need a zero 2 ⁷, and you need a proof of addition 3. The proof of addition is the homomorphism from our chip to the natural numbers. Furthermore we don’t specify the input type, which is represented by τ. This is done because we want a baseless chip design. We’re fine with arbitrary inputs, As long as we can interpreted this, represented by μ. In μ the Fin size indicates a finite size, which we need because we want to map our design to the real world, which is finite.

Not shown from the record is that we’re able to make a bigger adder out of a smaller one trough composition of two adders. composition chips allows us to “grow” this size without needing to re-prove correctness. We only need to prove composition is correct. All of this also holds for our eventual multiplication design. which is build from an adder and composition of other multiplication chips. We shall see that proving correctness of composition isn’t easy. Once done, all compositions will provably correct however. ⁸

A concrete example of μ would be an interpertation into binary values:

If we put this into the Adder then τ = Bool. interpretBF in this case interprets our code as a boolean value in natural numbers. In other words the homomorphism. This says what value a true and a false are in natural numbers. Now can define how to add bits:


add2 : Adder interpretBF -- 1
add add2 (zero , false , false)     = false , zero -- 2
...
add add2 (suc zero , true , true)   = true  , suc zero
zeroA add2 = false -- 3

Here we first define the type of add2 at 1. This uses the previously defined interpretBF to set τ = Bool. Then we start giving an implementation for add at 2, which is a simple pattern match into values. finally we give an implementation of zeroA at 3.

You can do a quick correctness check before doing a full prove. For example, in multiplication we didn’t do the full on homorphism prove at first. It looked daunting, so we settled on making a unit test instead. If we map out all possible inputs to all possible outputs we got a crummy proof:

_ : (V.map -- 1
        (toℕ ∘ pairμ (pairμ interpretBF) ∘ uncurry (mult mul2x2)) $ 
        composeTheValues allBools2x2 allBools2x2
    )
        ≡ (0 ∷ 0 ∷ 0 ∷ 0 ∷  -- 2
           0 ∷ 1 ∷ 2 ∷ 3 ∷
           0 ∷ 2 ∷ 4 ∷ 6 ∷
           0 ∷ 3 ∷ 6 ∷ 9 ∷ [])
_ = refl

At 1 we put our chip design into the interpretation, and at 2 we expect a multiplication table as result. Is this test complete? No, this only works for binary values up to 9, we’ve not tested for trits or pentits or higher values. We did prove however the chip behaves like we expect for these values. If you’re building a chip company where you only need multiplication up to 9 in base 2 this is good enough. As far unit tests go this is incredibly thorough because we’re testing against all possible values in the chip design. The more common approach is to sample a couple values and call it a day.

Which lead to an alternative approach called property testing. Here you would generate two random inputs on one side, interpret it trough the homomorphism and then see if the multiplication in natural numbers is the same as the test. We didn’t do this because it’s sort of difficult to do in Agda.⁹ Now you need to figure out how to get your source of randomness. Also time was a serious constraint, and we had a more powerful and interesting technique, proving! A proof doesn’t have to be hard, for example consider the correctness prove for our add2 chip:

  proof-add add2 (zero , false , false) = refl
  ...
  proof-add add2 (suc zero , true  , true)  = refl

refl means, reflexivity. In other words, the statement is simple enough that Agda can just look at the definition to figure out what it means. In this case all we do is list out all possible input values, and tell agda to look at the definition. proof-add’s type signature ensures the implementation is correct. This is only possible because Agda is dependently typed. What we proved is that the homorphism is the same under composition for the addition.

This will work for the add2 chip. However the add2 chip is kindoff useless by itself since it can only add 2 bits. Our idea was to compose these adders into bigger adders so that any size can be represented. To do this we first define the type signature:

bigger-adder : {σ τ : Set} {σ-size τ-size : ℕ} {μ : σ → Fin σ-size} {ν : τ → Fin τ-size}
  → Adder μ -- 1
  → Adder ν -- 2
  → Adder (uncurry combine ∘ P.map μ ν) -- 3

Here we’re putting in a low adder 1, a high adder 2 which results into a combined adder 3. μ and ν are placeholders for different adders. This allows us to for example add trits to bits. The resulting adder 3 maps over both sides of the resulting tuple¹⁰ with the interpertation.

If you squint a little, the implementation looks like a circuit:

add (bigger-adder x y) -- 1
  (cin , (mhi , mlo) , (nhi , nlo)) -- 2
  = let -- 3
      (lo , cmid) = y .add $ cin , mlo , nlo 
      (hi , cout) = x .add $ cmid , mhi , nhi
    in ((hi , lo) , cout) -- 4

At 1 we’re copattern matching on bigger-adder so we can get the underlying μ and ν adders as x and y respectively. At 2 we’re getting the actual arguments into the bigger adder. The type of this is $ Fin _2 () ()$ this is where the carry comes into play as argument since an adder needs to be able to tell when it overflows ¹¹. The actual addition in 3 we pawn off to the underlying adders x and y, all we do is hook in the carry¹². In 4 we emit the results.

Once we were reasonably confident of our implementation, we want to prove this correctness. This is a bit more involved than proving the boolean interpretation:

proof-add (bigger-adder {σ-size = σ-size} {τ-size = τ-size} {μ = μ} {ν = ν} x y)
          (cin , (mhi , mlo) , (nhi , nlo))
  with y .add (cin , mlo , nlo) in y-eq
... | (lo , cmid) with x .add (cmid , mhi , nhi) in x-eq
... | (hi , cout) =
  let x-proof = proof-add x (cmid , mhi , nhi) -- 3
      y-proof = proof-add y (cin  , mlo , nlo)
      size = σ-size
  in begin
  begin
    toℕ (cast _ (combine cout (combine (μ hi) (μ lo)))) -- 1
  ≡⟨ toℕ-cast _ (combine cout (combine (μ hi) (μ lo))) ⟩
    toℕ (combine cout (combine (μ hi) (μ lo)))
  ≡⟨ toℕ-combine cout _ ⟩
    size * size * toℕ cout + toℕ (combine (μ hi) (μ lo))
  ≡⟨ cong (\ φ → size * size * toℕ cout + φ) (toℕ-combine (μ hi) (μ lo)) ⟩
    size * size * toℕ cout + (size * toℕ (μ hi) + toℕ (μ lo))
  ≡⟨ {! taneb !} ⟩
    toℕ (addF' cin (combine (μ mhi) (μ mlo))) + toℕ (combine (μ nhi) (μ nlo))
  ≡⟨ sym $ toℕ-addF' (addF' cin (combine (μ mhi) (μ mlo))) (combine (μ nhi) (μ nlo)) ⟩
    toℕ (addF' (addF' cin (combine (μ mhi) (μ mlo))) (combine (μ nhi) (μ nlo))) -- 2
  ∎

Note I drastically shortened this proof to make it fit ¹³. What do is making the first line (indicated by 1) be the same as the last line (indicated by 2) through steps with equational reasoning. Every step is small, and the process is almost fully mechanical pattern matching. A step is anything within ≡⟨ ⟩, which does some small syntax transformation. The ≡⟨ {! taneb !} ⟩ is a missing step, called a hole. In this case we request taneb¹⁴, to figure out what goes here. In 3, we’re summoning the proofs from the x and y adders to use in the proof later. We’re making a bigger proof out of smaller ones.

If this proof is incorrect, you’ll get a compile error.¹⁵ This is similar to property tests, although it doesn’t use randomness and shrinking, but rather the structure of the implementation through dependent types. This is a big step up in terms off correctness compared to property tests. No longer can you have stochastic issues like insufficient sampling, or biased distributions. Furthermore smaller proofs compose into larger ones (with the right design). We can see that for example with x-proof in the above block. Not just that, but every step between ≡⟨ ⟩ is a prove being re-used. Property tests however aren’t as composable as proofs. A value generator may be re-used, however care must be taken the sampling and bias doesn’t become unacceptable. Finally we’re able to prove on polymorphic type variables, which property tests can’t do. If you have software that /needs/ to be correct, I think this dependently typed prove approach is a very good option to consider. I also think Agda is an good choice for a language that supports that.

Parting words

Denotational design is an excellent topic of study if you’re struggling with questions like “how do I make my code be more pretty?”, or “how do I design nice and easy to understand libraries?”. Furthermore, even for commercial code bases we can have correctness proofs. This is a much more powerful technique than mere property tests, and puts all that hype around dependent types to work. We don’t need to rely on hand wavy laws asserted merely by stochastic approximations of proofs, we can do the real deal! Please reach out if you’re in a domain where correctness like this is important. I’d love to chat :).

Finally I especially want to thank to both Nathan and Sandy for giving useful feedback on my humble writings. I also wish thank all other volunteers who participated, I had a great time.

As the name implies. This place is 30 minutes or so driving from Zurich.↩︎
I’m on the left.↩︎
As the name implies. This place is 30 minutes or so driving from Zurich.↩︎
In a commercial setting we’d decide if it’s worth investing additional in this design. The one presented at Zurich hack works. But if this is intended to be used in a larger system, iterating upon the design may help, if the business can afford it.↩︎
Addition chip↩︎
In zurich hack we sortoff started out in a classroom with just random ideas. One was quite funny where we somehow ended up with a design that was equivalent to tallying the ones and zeros. But we went in all kinds of directions before settling on using a record. I guess that’s the point you’ve to just try a bunch of stuff and not put to much ego into it.↩︎
When I asked Nathan to review this post, he mentioned he still hasn’t figured out why we needed a zero. I’m not really sure either, so it’s quite likely this isn’t needed at all! This maybe an artifact of the time crunch at play.
```
interpretBF : Bool → Fin 2
interpretBF false = zero
interpretBF true  = suc zero
```
↩︎
Unfortunately during the hackaton we didn’t finish this for multiplication, but we did for addition.↩︎
For me that is, because rember, I’m quite new to this all.↩︎
P stands for product in this case. So it’s a bimap over a tuple (due to uncurry).↩︎
As I mentioned in the design section, I don’t believe this is necessary, but this is an open research question.↩︎
Someone may or may not have opened nandgame during zurich hack to show a schema off an adder when we were struggling with correctness.↩︎
The full proof can be seen in the github repository, although we made some additional changes to the project after the presentation as well.↩︎
Nathan, a magical ring solver, or flesh and blood person, whichever interpretation suits you better.↩︎
So what if you’re stuck on a proof? This either means you can’t think of a function, or it means the thing you’re trying to do is impossible. Here you’d need to think really hard if the thing you’re designing is correct. You could also try to discuss the issue with a friend or ask on the internet. Plenty of people eager to help.↩︎

Restoring mysql innodb on windows.

2022-06-16T22:30:00Z

Over the weekend a company had a power outage, causing corruption to the on premise hosted mysql innodb database. This means the company can’t do any work, so I had to fix this fast before they opened again on Monday. It’s not trivial because the system runs on windows, which means the online guides don’t quite work. Therefore I wrote these notes for future me.

We can boot the database with force recovery In my case I had to set it to 3, try as low as possible. It puts everything in read only but that’s good enough. The recovery is a process of exporting these read only databases to sql, creating an entire new instalation with a new datafolder, and reading these back into that. The linux instruction can be found here. This post is specifically about windows. We need to run from cmd not powershell, powershell doesn’t understand <, and I’ve doubts about it understanding >.

To get a working installation I set innod_force_recovery = 3 in mysql\bin\my.ini, which can be found in xampp. First we have to restart mysql so we can get the data out. Now we can dump the tables:

./mysqldump.exe -u root -pwhatever --skip-lock-tables -A > all.sql

And shut down the server.

./mysqladmin -u root -pwhatever shut

I just emptied the datafolder with explorer, by navigating to T:/xampp/mysql/data, and deleting everything in there. Now I ran:

./mysql_istall_db.exe --datadir="T:\xampp\mysql\data"

I set innod_force_recovery to a comment again and restart mysql from xampp, I also checked if it ran in process monitor.

set the root password correctly again:

mysql -u root
SET PASSWORD FOR 'root'@'localhost' = PASSWORD('whatever');
FLUSH PRIVILEGES;

restore:

mysql -u root -pwhatever < all.sql

Failing in Haskell

2022-02-27T00:00:00Z

Recently I encountered some dubious error handling code. Not only was it failing, it was failing WRONG ¹. This frustrates me because doing failing correctly in Haskell is quite easy, so why was it implemented wrongly? I believe no-one has addressed failing with an opinion. Plenty of people describe the various ways you can fail. But none give opinions on why certain ways are better than others. Therefore I shall share my failing expertise, and describe the correct way to fail in Haskell. In essence, this is an answer to Eric Kidd’s plea for consistency ². These are the properties we want from failure:

Preciseness, vague errors are bad.
Locality, we need to know where errors come from.
Recoverability, the program should be able to recover after an error.
Change ability, introduction of new error cases should be easy.

We want all these properties to make debugging easier. This allows you to solve complicated bugs within minutes. We structure the program so that it tells what goes wrong, where and why. This isn’t magic. It takes effort, but not magic at all.

I’ll describe how to achieve the above properties with some example code for both pure and side effectful code. However these principles apply to other languages as well ³. We’ll also go over some anti patterns and discuss their mediation.

Pure

Haskell programmers prefer so called ‘pure’ code. By which we mean immutable memory computations⁴. Therefore we start with the pure case. Ideally pure code fail management looks like this:

newtype LookupError = NotFound Text

lookup :: Map Text Double -> Text -> Either LookupError Double
lookup env argA = maybe (Left (NotFound argA)) Right $ Map.lookup argA env

data DivideErrors = DivideLookup LookupError
                  | DivisionByZero Double Double
                  | DivNegativeDivision Double Double
                  | DivNumberThreeIsBad Double Double

divide :: Map Text Double -> Text -> Text -> Either DivideErrors Double
divide env argA argB = do
    valA <- first DivideLookup $ lookup env argA
    valB <- first DivideLookup $ lookup env argB
    when (valB == 0) $ Left $ DivisionByZero valA valB
    when (valB < 0) $ Left $ DivNegativeDivision valA valB
    when (valA == 3.0 || valB == 3.0) $ Left $ DivNumberThreeIsBad valA valB
    pure $ valA / valB

Here we introduce the divide function. Which takes an environment map, looks up the values of said map, and then performs division after doing some checks. We introduce the environment as a convenient common “this can fail” lookup. But business may give us other in-variants such as DivNumberThreeIsBad, which also neatly fits in this “pattern”. Errors are emitted by using the Left constructor, which is the error branch according to Either’s Monad instance.

This code will tell you exactly what went wrong if something goes wrong. Locality in this case can be improved a little by adding a different constructor for each lookup call, but in this case I’d argue locality is close enough. The developer has the opportunity to recover from these errors, a simple pattern match would suffice. Furthermore if business demands yet another weird constraint, such as NumberFourIsBad, pattern matches at call sites will emit a -Wincomplete-patterns warning.

If we need to compose these errors in a larger program we can simply wrap previous errors in a bigger sumtype, consider the following function plusDiv which emulates (a + b) / c

data PlusErrors = PlusLookup LookupError
                | PlusNoZeroResults
                | PlusNumberThreeIsBad Double Double

plus :: Map Text Double -> Text -> Text -> Either PlusErrors Double
plus env argA argB = do
    valA <- first PlusLookup $ lookup env argA
    valB <- first PlusLookup $ lookup env argB
    when (valA == 3.0 || valB == 3.0) $
      Left $ PlusNumberThreeIsBad valA valB
    let res = valA + valB
    when (res == 0) $ Left PlusNoZeroResults
    pure $ valA / valB

data PlusDivErrors = PDPlusError PlusErrors
                   | PDDivErrors DivideErrors
                   | PDLookup LookupError

-- | (a + b) / c
plusDiv :: Map Text Double
        -> Text -> Text -> Text
        -> Either PlusDivErrors Double
plusDiv env argA argB argC = do
  res <- first PDPlusError $ plus env argA argB
  cres <- first PDLookup $ lookup env argC
  first PDDivErrors $
    divide (Map.fromList [("one", res), ("two", cres)]) "one" "two"

The higher ‘level’ function plusDiv absorbs all errors from other functions with extra constructors. This isn’t the most ergonomic approach because variables have to go through the environment map. In a real program you could factor out this function. For example with an innerPlus :: Double -> Double -> Either PlusErrors Double. This adds additional type safety as well, because the Double type is “smaller” then a Text type in both memory and cardinality. However I think for an example or early prototype it’s good. Once more this tells us exactly what part of the computation failed, if any. In this case, the first function is being used like lift: Transforming the function it wraps to run in the “higher level” Either environment. Some may even call this a natural transformation.

In certain cases you can use Data.Validation instead of Either. Which can collect more then one error. But usage of that is out of the scope of this blogpost.

`IO` and exceptions

It’s more difficult to recover from an exception than it is from a pure error value. As a rule of thumb you can use exceptions when you expect the program to stop. For example when you can’t find a critical resources from the database. Another consideration is the value of the error. If it’s important that an error be handled correctly, then exceptions should be avoided. However, this should only be done within a monad stack that has IO as base, because exceptions are part of the IO ‘contract’⁵. This contract extends to any transformer stack that has IO as base. For convenience however, I’ll write out the example in plain IO:

data DivideException where
    MkDivideException :: HasCallStack =>
        DivideErrors -> DivideException

deriving instance Exception DivideException

instance Show DivideException where
    show (MkDivideException errors) =
        renderExceptionWithCallstack errors "MkDivideException"

renderExceptionWithCallstack ::
        (HasCallStack, Show a) => a -> String -> String
renderExceptionWithCallstack errors valueConstructor = "(" <> valueConstructor <> " $ "
      <> show errors
      <> "/*"
      <> prettyCallStack callStack
      <> " */)"

throwDivide :: HasCallStack => Either DivideErrors a -> IO a
throwDivide = either (throwIO . MkDivideException) pure

myEnv :: Map Text Double
myEnv = Map.fromList [("zero", 0.0), ("one", -1.0),
                      ("two", 2.0), ("three", 3.0)]

main :: IO ()
main = do
    result1 <- throwDivide $ divide myEnv "one" "two"
    result2 <- throwDivide $ divide myEnv "one" "three" -- throws
    print (result1 , result2)

All this boilerplate attaches the callstack to our exception. We also put the entire pure error type directly into the exception. It composes. This is why pure error handling is preferable, but if you don’t have time to do this, exceptions like those described above are good too. This idea of attaching call stacks to your exceptions is explained further in this blogpost

If that’s to much work, the error call also has a stack trace, although its type allows vagueness unfortunately. Also make sure to attach it to IO, or the nullpointer anti pattern may occur. I think the errorIO function is a good idea instead of error because it avoids that nullpointer scenario entirely. Don’t try to catch errors, use exceptions if you need to catch.

MTL

I recently blogged about mtl, so I’ll briefly cover how to modify this code into mtl style as well:

throwDivide :: (HasCallStack, MonadIO m)  => ExceptT DivideFailures a -> m a
throwDivide meow = do
    res <- runExceptT meow
    case res of 
      Left err -> liftIO $ throwIO $ MkDivideException err
      Right res -> pure res

myDBFunc :: (MonadError DivideFailures m, MonadDB m) => m Double
myDBFunc = do
    myEnv <- getMyEnv
    divide myEnv "one" "two"

main :: IO ()
main = do 
    result1 <- runDB $ throwDivide $ myDBFunc 
    print result1

throwDivide works quite similarly as in the previous example but now it works with any transformer stack based on IO. This works because we pretend the ExceptT exists at the call site, which makes it come true. This is explained thoroughly in the previous blog post.

Anti patterns

Now I’ll cover several anti patterns I’ve seen and discuss what to do differently. Some of these patterns occur in the wild, for example aeson famously exposes a String for errors, which is problematic for a library. The next section explains why.

Text in left branch of Either

These examples contain text in the left branch:

y :: Either Text a
x :: Either String a

The problem is that we break the recoverability property. There is no way to make a closed pattern match on a string. Instead we should create a sumtype for possible errors:

data YErrors = YErrorOne
             | YErrorTwo Double

y :: Either YErrors a

This way client code can pattern match on all possible branches, and if the additional errors get introduced the compiler notifies the developer through -Wincomplete-patterns warning.

`throw`

throw allows throwing exceptions in pure code. This is very wrong because the code ends up behaving like a null pointer in Java, due to Haskell’s non strict evaluation. In other words, this breaks the locality of errors. Consider for example:

myInts :: [Int]
myInts = []

data EmptyException = MkEmptyException
    deriving (Exception, Show)

myHead :: [Int] -> Int
myHead [] = throw $ MkEmptyException
myHead (x : y) = x

main :: IO ()
main = do
   let x = myHead myInts
   print $ [0..5] <> myInts
   print x

It fails on that last line. Even though the error was at the x binding. Much better is to use throwIO, modifying the above example:

myInts :: [Int]
myInts = []

data EmptyException = MkEmptyException
    deriving Exception

myHead :: [Int] -> IO Int
myHead [] = throwIO $ MkEmptyException
myHead (x : y) = pure x

main :: IO ()
main = do
   x <- myHead myInts
   print $ [0..5] <> myInts
   print x

This will indeed fail on the first line. We lost purity, in this case Either could have been used as described above, which is even better. Alternatively we can use the HasCallStack trick, if we stick with exceptions.

Generic app exceptions

I’m talking about something like this:

data AppException = MkAppException Text
    deriving Exception

This is bad because it ends up being thrown at many places with no good way of recovering. Exceptions are already difficult to recover from, but if they’re re-used a lot, it becomes even more difficult. Now you’d have to pattern match on the Text to handle the correct error and hope no one re-uses that particular error text, ever! Furthermore the error may be vague, depending on what’s being put in the Text field.

It’s better to define a custom exception per situation. For example:

data DBUserNotFound = MkDBUserNotFound { userId :: UUID }
    deriving Exception

data AwsImgResourceNotFound = MkImgResourceNotFound {name :: Text, awsId :: UUID }
    deriving Exception

These are precise and type safe, we no longer can be vague because you have to provide a UUID. This forces the throw site to capture this UUID from the environment, making the error precise trough type safety.

However, you’ll likely already have generic app exception being called from different 400 places. Fortunately we can recover some of the locality property by rewriting the exception in a GADT and using the HasCallStack trick.

Squashing errors

This can occur when using bind >>= on Maybe, for example:

x :: Maybe Int
y :: Maybe Int

z :: Maybe Int
z = do
   x' <- x
   y' <- y
   pure $ x' + y'

Nothing won’t tell us why z failed. This is wrong because it breaks preciseness. instead we should restructure like so:

x :: Maybe Int
y :: Maybe Int

data ZFailures = NoX
               | NoY

z :: Either ZFailures Int
z = do
  x' <- maybe (Left NoX) Right $ x
  y' <- maybe (Left NoY) Right $ y
   pure $ x' + y'

This will tell you exactly what went wrong, while retaining most of the power of bind. It’s fine to use Maybe if there is a single error case, but often this isn’t the case.

Conclusion

The preference relation of failing is like this:

pure error handling > exceptions *with* stack traces > errorIO calls

Anything else is most likely wrong and should be avoided. Structure your programs so that it tells what goes wrong, where and why. I think most software would benefit from doing this, and these concepts are easy. Let me know if you disagree, or need help with this.

References

https://github.com/waddlaw/haskell-stack-trace-plugin
https://www.stackbuilders.com/blog/errors-and-exceptions-in-haskell/
https://wiki.haskell.org/Handling_errors_in_Haskell
http://www.randomhacks.net/2007/03/10/haskell-8-ways-to-report-errors/
https://maksbotan.github.io/posts/2021-01-20-callstacks.html
https://hackage.haskell.org/package/validation-1.1.2/docs/Data-Validation.html
https://www.parsonsmatt.org/2018/11/03/trouble_with_typed_errors.html

Wrong as in they were following some of the ani patterns listed below.↩︎
This is 15 years ago, ah well, better late then never I suppose.↩︎
When I was writing this I realized I do the exact same program structuring in Java or PHP. It is more work in Java or PHP, but possible and saves so much debugging time.↩︎
Accessing memory is for no particular reason considered pure, although one wonders if this could be changed to make managing memory bounds easier.↩︎
See the throw anti pattern ↩︎

Installing a NixOS desktop tracked with git

2022-01-30T16:20:00Z

A few years ago I wrote a post on installing NixOS on encrypted btrfs. I recently went trough that guide to install NixOS once more. It is good, but it has some issues:

btrfs: Which I no longer use due to performance concerns.
git: This requires some special attention, but composes really well with nix and nixos.
It doesn’t explain how to deal with secrets.

I’ll address these concerns here, since I just bricked an install due to git usage ¹. And I also wasted some time re-figuring out secrets.

Why use git if it introduces complexity? For one it serves as an excellent backup tool, furthermore it allows managing multiple deployments side by side trough branches. Finally having a log of changes can be useful when things break. Things will break. Such is the life of a tinkerer.

I imagine some people still want to use btrfs, so I’ll leave the old guide in place. However, I’ll copy over parts which were good in here for convenience. This updated guide still uses encrypted disks, I’ve had no problems with this at all, and I recommend disk encryption to all.

Getting started

Get yourself a NixOS live usb. I use the minimal ISO, because the graphical ISO slows booting and gives no advantage aside from being pretty. Use

cat minimal-nixos.iso > /dev/sdX

where X is the usb drive found by lsblk. X should be a letter, numbers indicate partitions, which we don’t want to cat upon because the ISO already contains a partitioning scheme.

Boot into it on the target machine. Become root with sudo -i.

Internet

Next step is to setup WIFI, you can skip this if you’re on Ethernet:

wpa_passphrase SSID PASS > /etc/wpa_supplicant.conf
systemctl restart wpa_supplicant

The first command creates a config for wpa_supplicant. The reader must fill in SSID and PASS of his target wifi network. The second command tells systemd to go restart wpa_supplicant and use the new config.

Ask google if you’re online:

curl google.com

Should return a 301 Moved. If it hangs or refuses the connection you likely have no internet. There is no point proceeding until you have internet access.

Partitioning

Now to setup the partitioning on the RIGHT device. Choose carefully. Use lsblk to figure out which device is RIGHT. You’ll know it’s the WRONG device if you lose data after partitioning. The RIGHT device will be called $dev hence forward.

There are no other partitioning tools than gdisk. Only heretics believe there are. Therefore we use gdisk:

gdisk $dev

Gdisk cheat sheet

Command	Effect
`p`	For printing, to see what’s going on.
`d`	For deletion, you should start out with deleting everything on `$dev`.
`n`	Is used for creating new partitions.
`w`	is used for writing once finished.

This table just describes the commands needed for the intended partitioning.

Intended partitioning

Number	type	size
1	ef00	+500M
2	8200	+$(SIZE_RAM+ a little)G
3	8300	(rest of disk)

The first partition will be boot, the second swap[^optional], the third will be everything else. We will encrypt everything else. With type ef00 we will use UEFI for booting. Don’t worry. nix will handle that, mostly. Done. Onwards! [^optional]: This one is optional but allows hibernation. Which is very convenient for laptops. It can also make your system more stable.

Encryption

We use cryptsetup for encryption. Make sure to select the right partition. We do not want to encrypt the boot partition because then we can’t boot. So if you followed above instructions it will be either 3 or p3 (depending on device type). We’ll call it 3.

cryptsetup luksFormat "$dev"3
cryptsetup open "$dev"3 nixenc

The first command does the actual formatting, the second one opens up the formatted disk. You’ll need to provide the right password in both cases. Choose one you can remember but is strong. Once decrypted the disk will be mapped to /dev/mapper/nixenc, note that we supplied that final part in the last command.

Formatting filesystems

Partitioning is a distinct step from setting up filesystems.

mkfs.vfat -n boot "$dev"1
mkswap "$dev"2
swapon "$dev"2
mkfs.ext4 -L root /dev/mapper/nixenc

The boot partition will be vfat because UEFI tells us to. The everything else partition will be ext4. Note that we point it at the mapped file, if the "$dev"3device were to be used directly we’d remove the encryption.

Mounting

Here we mount all partitions.

mount /dev/mapper/nixenc /mnt/
mkdir /mnt/boot
mount "$dev"1 /mnt/boot

This makes them detectable by the nix config generation script. Furthermore it allows the script to write the config on the proper disk.

Did I do everything right?

The second time I ran trough this post everything went quite quickly, so I became skeptical. To verify everything was sane I used the following commands:

mount | grep /mnt
ls /mnt

The first command is to check if the encrypted volume and boot is mounted at the right paths. The second one to verify the folders are created.

Configure nix part 1, in the live environment

We use a script to generate an intial nix configuration, which detects the hardware for us:

nixos-generate-config --root /mnt

This will give us a default config which we can customize now. You can also pick up ideas from my config, although its a mess. On a wireless laptop, it’s highly recommended to enable wpa_supplicant:

networking.wireless.enable = true

Furthermore we’re going to need the packages vim and git to modify the config after booting into the system:

  environment = {
    systemPackages = [
        pkgs.git
        pkgs.vim
    ];
  };

Make sure to set the channel to the same as in your tracked git configuration, or alternatively be ready to deal with upgrades. For example on a live ISO of 21.11 I downgraded to 21.05 with:

nix-channel --add https://nixos.org/channels/nixos-21.05 nixos
nix-channel --update

and then I changed the configuration.nix to:

 system = {
    stateVersion = "21.05";
  };

Dealing with channels is quite fragile, so I have these commands copied as comments in my configuration.nix.

Once configuration is done we can install nix:

nixos-install

Don’t worry, we can use nixos-rebuild switch to reconfigure nix whenever, once we’re booted into it. Hopefully we boot successfully:

reboot

Booting is hard, don’t worry if this goes wrong the first 10 30 times.

You may need to enable UEFI in your BIOS. It’s up to the reader to figure that part out ² ³. Alternatively one could setup grub. Good luck with that. You can’t read the rest of this post until you’ve booted. Go back if you haven’t booted, you messed up.

Configure nix part 2, with git

Once we’re booted into the installation, the absolute paths for the symlinks are different. For example root is no longer under /mnt, but under, well, root /. So login as root⁴ and clone your config project:

cd /
git clone https://github.com/jappeace/linux-config
chown jappie:users -R /linux-config

This puts the linux-config project on the /linux-config path. Some old time linux/unix users may puke in their mouths upon seeing the “standard” directories being ignored, but fuck them. We mustn’t forget to copy over the hardware generated config into our git project:

cp /etc/nixos/hardware-configuration.nix /linux-config/hardware/branch-name.nix

Where branch name the name is for the git branch you’ll use for this deployment. More on that in the branches section. Also we want to include this as a module in the configuration.nix

  imports = [ 
    ./hardware/branch-name.nix
  ];

The tracked configuration should be used by the system. What I usually do is login as my own user, and run the script setup-nixos.sh:

exit
cd /linux-config/scripts/
./setup-nixos.sh

Which sets up the symlink from the git tracked configuration to the standard location, in other words it does this:

ln -sf /linux-config/configuration.nix /etc/nixos/configuration.nix

This script also symlinks all relevant dotfiles from linux config into the home folder. You may want prefer home manager to symlinking dotfiles. But this works for me.

Multiple machines desktops

Branches are ideal for managing multiple machines, because it allows you to diverge oddities such as hardware specific configurations. For example I also have this crummy display switch script which is only relevant for the PC. On the laptop it has to be slightly different (if used at all).

This branch setup also allows merging back configuration changes from other machines. Which involves solving an ordinary git conflict. I recommend the reader to use merges rather then rebases, because that way git remembers how conflicts are resolved.

However I recently learned that a friend of mine handles this trough a module system. He then runs that with nixos-rebuild --flake .#machine-name. So rather then using branches he has an entrypoint per machine which he calls out directly with the flake. I find his setup interesting, and may move over to something like that in the future, although since flakes are still experimental, I’ll hold off.

Another alternative is multiple configuration.nix files in a repository and let the symlink decide which should be used for what machine. This would avoid any merge conflicts, although the versions between machines need to be similar. I think I prefer merge conflicts.

Secrets

I have three major secret sources. 1. ssh keys 2. gpg keys 3. The keepassxc database

The database in synchting, this gives me access to all services so that I can simply generate new gpg and ssh keys per deployment. Any other file manage service would do, but I like syncthing because it’s decentralized.

I go to https://localhost:8384 on some device that has the database, and the target device on the same address. I type over the device id into that screen to start syncing. after syncing completes I have access to the keepasscx database. Now I can generate new ssh keys and gpg keys, and login to services to update those. With that finished the installation is complete.

Aside from getting the keypass database up and running, it’s important to add your newly generated public key to the services you manage. For example this website is hosted on the nixos multi monolith. It be prudent to add the ssh key via a machine that already has access to it. Syncthing can also be used for this.

I had uuid for my disks, but I swapped the disks so the boot bricked.↩︎
press some f keys on boot, f11 or f2 maybe?↩︎
An issue I encountered was that rather then selecting EUFI boot, the bios did a traditional boot on the disk. So EUFI was correctly installed, I just had to select the same disk but with the EUFI label from the bios. Yup, all kinds of stuff can go wrong with booting.↩︎
Once rebooted you may be stuck at the display manager. Use Alt+f1 to switch to another TTY and login as root, then use passwd your-user-name to set an initial password for that user. Use Alt+f7 to go back to the display manager.↩︎

A brief intro to MTL

2021-08-10T00:05:00Z

Recently a blog post came out which I quite like, it describes how to use the concrete base transformers. It’s very thorough and gives a concrete example for using transformers. Although it looks quite low level and I think you’ll get more out of transformers by using full MTL.

That blogpost inspired me to write this, because I can’t find a succinct description on how to use MTL¹. I learned MTL by staring at reflex for days, if not weeks. Which is an uncomfortable learning process. To make MTL more accessible I’ll give a brief overview of this style². I’ll write down how MTL works from the ground up, so people can read how to use it rather then struggling with code for days like I did.

If you like video presentations, I also presented how to use MTL in a video format.

The type variable `m`

We start by introducing m. Which could’ve been named monad or x, but the community settled on using m for type variables with the Monad constraint, so I will use this too. Normally we use type variable’s in concrete types, for example Maybe a or [a]. However, Instead of having our type variable inside a concrete type, we can also flip it around:

moreMonad :: Monad m => m Int
moreMonad = return 5

This compiles because the Monad constraint on m gives us the return function ³. After you’re convinced this is a valid definition, let’s use it. What can we do with this moreMonad binding? Well, we can pattern match on it:

fiveTroughMaybe :: Int
fiveTroughMaybe = case moreMonad of
  Just x -> x
  Nothing -> 9

GHC will give moreMonad the type Maybe at this call site. GHC reasons backwards from the pattern match up to the case moreMonad of definition to figure out the type. In my head I describe this backwards reasoning process as: “pretending you have a Maybe which makes it becomes true”. So fiveTroughMayberesults in 5 because return is implemented as Just on the Maybe type’s Monad instance.

This is valid. You should convince yourself it’s valid. To convince yourself I’m not lying paste this code into GHCI before continuing, and gain some confidence, because yonder be dragons.

Continuing now with the same module we can also pattern match on Either:

fiveTroughEither :: Int
fiveTroughEither = case moreMonad of
  Right x -> x
  Left _y -> 9

Both fiveTroughMaybe and fiveTroughEither will result in 5. This is allowed in the same module because moreMonad will only get assigned the type at the call site. The compiler figures out the type of moreMonad by looking at usage per call site. This backwards ‘figuring out’ is normal for type variables.

(optional) mastery exercise

Can we always pattern match on every possible monad type like we just did with Just or Either? Can we always get the value out without being in the same monad? The answer is in the footnote. ⁴

Transformers as constraints on `m`

With that brief introduction, we can start applying this idea to the ‘“A Brief Intro to Monad Transformers” blogpost’ which inspired me to write this. In that blogpost, a newtype is constructed to hold the entire monad transformer stack like this:

newtype AppM a = AppM {
    runAppM :: ExceptT String (State (M.Map VariableName Int)) a
  }
  deriving newtype (Functor, Applicative, Monad, MonadError String,
                    MonadState (M.Map VariableName Int))

I call this AppM a concrete type because there is only one way to pattern match on it. We’re not allowed to pretend it’s a Maybe for example. This definition is used in the assignIndexToVariables function:

assignIndexToVariables :: AST VariableName -> Variables -> AppM (AST Int)

Instead of using the concrete type AppM, we could use MTL type classes to describe what is needed. These type classes will become constraints on m, similarly to how Monad was a constraint on m in the introduction. Which means we want to have MonadError String as replacement for ExceptT String ⁵, and MonadState (M.Map VariableName Int) as replacement for State (M.Map VariableName Int). Doing this will change the type signature of assignIndexToVariables as follows:

assignIndexToVariables ::
               MonadError String m
            => MonadState (M.Map VariableName Int) m
            => AST VariableName
            -> Variables
            -> m (AST Int)

So what’s the difference? The type signature is more verbose, although we no longer need the newtype. In trade for this verbosity, we can use the monad stack in any order at the call site. Both invocations of running this code are now allowed:

main :: IO ()
main = do
 ...
 print $ flip evalState mempty $ runExceptT $
    assignIndexToVariables ast vars
 print $ runExcept $ flip evalStateT mempty $
    assignIndexToVariables ast vars

This wasn’t possible in the original code. We’ve told the compiler that the order of a monad stack doesn’t matter. Which makes sense because consider two monad stacks:

at :: Char -> ExceptT String (State (M.Map VariableName Int)) Int
bt :: Int -> StateT (M.Map VariableName Int) (Except String) String

These describe the same capabilities, however the compiler says a and b should not compose. It’s impossible to write:

ct :: Char -> _ String
ct = at >=> bt

We don’t know what goes at _ for ct because at and bt have concrete types. We can write this composition however if these signatures are defined in MTL style:

am :: MonadError String m
  => MonadState (M.Map VariableName Int) m
  => Char -> m Int
bm :: MonadState (M.Map VariableName Int) m
  => MonadError String m
  => Int -> m String

cm :: MonadState (M.Map VariableName Int) m
    => MonadError String m
    => Char -> m String
cm = am >=> bm

Because constraints don’t specify an order on transformers, the MTL style function definition can compose. ⁶

This section described the core idea of MTL. In the following sections we’re going to extend MTL using type error driven development. This all sound like madness if you don’t try it out with a compiler. And I feel understanding errors is a large part of understanding MTL. The type errors are difficult to decipher. I’ve made an reference project so the reader can verify the truth of my claims.

Even though you may doubt me dear reader, let’s go deeper into the abyss.

(optional) mastery exercises

What does the function lift do? The answer is in the footnotes. ⁷
Is the order of a transformer stack always irrelevant? The answer is in the footnotes. ⁸
Say ct has this type signature:
```
  ct :: Char 
     -> ExceptT String (State (M.Map VariableName Int)) String
```
Call at and then bt from within ct such that it composes like the fish operator >=> would with help of liftWith. For additional background see this blogpost. The answer can be found in the reference project under the binding answer.

Lose and tight constraints

Say we want to use moreMonad from the introduction in assignIndexToVariables. I call moreMonad a tightly constrained binding. It’s only allowed to use what Monad typeclass provides. assignIndexToVariables on the other hand is less tightly constrained, since it has Monad by implication, and also everything in MonadError and MonadState. So let’s use it:

assignIndexToVariables ::
    MonadError String m =>
    MonadState (M.Map VariableName Int) m =>
    AST VariableName -> Variables -> m (AST Int)
assignIndexToVariables _ _ = do
   _z <- moreMonad 
   ...

This would just work, because both MonadError and MonadState imply Monad in their definitions. In mtl style, the functions with tighter constraints can be used in more situations without any refactoring.

Now we’re going to do the complete opposite. The most lax constraint possible is MonadIO, which gives access to arbitrary IO trough the liftIO function. I’m not casting judgement, I just want to show what happens. So let’s add a MacGyver ⁹ logging function as follows:

macGyverLog :: MonadIO m => String -> m ()
macGyverLog msg = liftIO $ putStrLn msg

The code is perfect. Y’know, it misses some things you’d expect from your regular logging library such as code positions, time stamping, etc. But that’s why it’s called macGyverLog and not kitchenSinkLog. And if we did some introspection we may find our code tends to look a lot more like macGyverLog then kitchenSinkLog, so let’s throw it into production:

assignIndexToVariables ::
    MonadError String m =>
    MonadState (M.Map VariableName Int) m =>
    AST VariableName -> Variables -> m (AST Int)
assignIndexToVariables _ _ = do
   macGyverLog "Starting reading more monad"
   _z <- moreMonad 
   macGyverLog "End reading more monad"
   ...

This will cause the following type error:

src/Lib.hs:31:5: error:
    • Could not deduce (MonadIO m) arising from a use of ‘macGyverLog’
      from the context: (MonadError String m,
                         MonadState (M.Map VariableName Int) m)
        bound by the type signature for:
                   assignIndexToVariables :: forall (m :: * -> *).
                                             (MonadError String m,
                                              MonadState (M.Map VariableName Int) m) =>
                                             AST VariableName -> Variables -> m (AST Int)
        at src/Lib.hs:(26,1)-(29,46)
      Possible fix:
        add (MonadIO m) to the context of
          the type signature for:
            assignIndexToVariables :: forall (m :: * -> *).
                                      (MonadError String m,
                                       MonadState (M.Map VariableName Int) m) =>
                                      AST VariableName -> Variables -> m (AST Int)

The possible fix read in the type error is correct, but we have to know what a context is, and a type signature to decipher that error message¹⁰. The compiler is saying in incomprohensible error speak that you need to add a constraint MonadIO m like so:

assignIndexToVariables ::
    MonadIO m =>
    MonadError String m =>
    MonadState (M.Map VariableName Int) m =>
    AST VariableName -> Variables -> m (AST Int)

Cryptic as it may be, the reason I’m writing about this isn’t that particular error message, it’s the next one:

src/Lib.hs:51:50: error:
    • No instance for (MonadIO Data.Functor.Identity.Identity)
        arising from a use of ‘assignIndexToVariables’
    • In the second argument of ‘($)’, namely
        ‘assignIndexToVariables ast vars’
      In the second argument of ‘($)’, namely
        ‘runExceptT $ assignIndexToVariables ast vars’
      In the second argument of ‘($)’, namely
        ‘flip evalState mempty
           $ runExceptT $ assignIndexToVariables ast vars’
   |
51 |     print $ flip evalState mempty $ runExceptT $ assignIndexToVariables ast vars
   |                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Here you may start thinking, WTF. Rightfully so. The reason it starts talking about identity is because evalState runs in identity. Look at the haddocks for evalState, then click on State. The base monad is identity. How do we fix this? First we replace Identity with a gap by invoking evalStateT instead of evalState (note the T). This will result in the following beauty :

src/Lib.hs:51:5: error:
    • Ambiguous type variable ‘m0’ arising from a use of ‘print’
      prevents the constraint ‘(Show
                                  (m0 (Either String (AST Int))))’ from being solved.
      Probable fix: use a type annotation to specify what ‘m0’ should be.
      These potential instances exist:
        instance (Show a, Show b) => Show (Either a b)
          -- Defined in ‘Data.Either’
        instance (Show k, Show a) => Show (M.Map k a)
          -- Defined in ‘Data.Map.Internal’
        instance Show a => Show (AST a) -- Defined at src/Lib.hs:21:13
        ...plus 18 others
        ...plus 9 instances involving out-of-scope types
        (use -fprint-potential-instances to see them all)
    • In a stmt of a 'do' block:
        print
          $ flip evalStateT mempty
              $ runExceptT $ assignIndexToVariables ast vars
      In the expression:
        do print
             $ flip evalStateT mempty
                 $ runExceptT $ assignIndexToVariables ast vars
           print (eitherFive, maybeFive)
      In the expression:
        let
          vars = S.fromList [...]
          ast
            = Node (Leaf "a") (Node (Leaf "b") (Node (Leaf "a") (Leaf "c")))
        in
          do print
               $ flip evalStateT mempty
                   $ runExceptT $ assignIndexToVariables ast vars
             print (eitherFive, maybeFive)
   |
51 |     print $ flip evalStateT mempty $ runExceptT $ assignIndexToVariables ast vars
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

src/Lib.hs:51:18: error:
    • Ambiguous type variable ‘m0’ arising from a use of ‘evalStateT’
      prevents the constraint ‘(Monad m0)’ from being solved.
      Probable fix: use a type annotation to specify what ‘m0’ should be.
      These potential instances exist:
        instance Monad (Either e) -- Defined in ‘Data.Either’
        instance Monad IO -- Defined in ‘GHC.Base’
        instance [safe] Monad m => Monad (ExceptT e m)
          -- Defined in ‘Control.Monad.Trans.Except’
        ...plus six others
        ...plus 16 instances involving out-of-scope types
        (use -fprint-potential-instances to see them all)
    • In the first argument of ‘flip’, namely ‘evalStateT’
      In the expression: flip evalStateT mempty
      In the second argument of ‘($)’, namely
        ‘flip evalStateT mempty
           $ runExceptT $ assignIndexToVariables ast vars’
   |
51 |     print $ flip evalStateT mempty $ runExceptT $ assignIndexToVariables ast vars
   |                  ^^^^^^^^^^

src/Lib.hs:51:51: error:
    • Ambiguous type variable ‘m0’ arising from a use of ‘assignIndexToVariables’
      prevents the constraint ‘(MonadIO m0)’ from being solved.
      Probable fix: use a type annotation to specify what ‘m0’ should be.
      These potential instances exist:
        instance [safe] MonadIO IO -- Defined in ‘Control.Monad.IO.Class’
        instance [safe] MonadIO m => MonadIO (ExceptT e m)
          -- Defined in ‘Control.Monad.Trans.Except’
        instance [safe] MonadIO m => MonadIO (StateT s m)
          -- Defined in ‘Control.Monad.Trans.State.Lazy’
        ...plus 11 instances involving out-of-scope types
        (use -fprint-potential-instances to see them all)
    • In the second argument of ‘($)’, namely
        ‘assignIndexToVariables ast vars’
      In the second argument of ‘($)’, namely
        ‘runExceptT $ assignIndexToVariables ast vars’
      In the second argument of ‘($)’, namely
        ‘flip evalStateT mempty
           $ runExceptT $ assignIndexToVariables ast vars’
   |
51 |     print $ flip evalStateT mempty $ runExceptT $ assignIndexToVariables ast vars
   |                                                   ^^^^^^^^^^^^^^^^^^^

All these m0 occur because we introduced the gap, GHC has no idea what the base monad is at this point. Which is what we want. Let’s solve it all in one change:

    print =<< flip evalStateT mempty
        (runExceptT $ assignIndexToVariables ast vars)

Note that we replaced the application of $ to a bind =<<. The base monad is now IO instead of Identity, which solves everything. It’s solved because IO, surprise, surprise, has an instance of MonadIO! Who would’ve thought liftIO could be id. The readers’ keen eye spots a pattern: We merely select instances with types that have code attached to them. It’s code generation based on type selection.

Reinterpreting IO or the MTL style

In most situations MonadIO is fine. However it doesn’t allow us to reinterpret effects that are using IO. An example of reinterpretation is a test where you would want to measure how often the mcGyverLog function is being called. This section will show you how to do that.

First we start by MTL-izing our IO based code. How can we rewrite mcGyverLog so that it doesn’t use IO explicetly? The function that needs IO is putStrLn. It’s type signature is String -> IO (). We want that IO to be an m, so we introduce a new typeclass for our not invented here log:

class (Monad m) => NotInventedHereLog m where
    nihLog :: String -> m ()

We already know what implementation m has if it’s an IO instance:

instance NotInventedHereLog IO where
    nihLog :: String -> IO ()
    nihLog = putStrLn

To make our previous example work we need to replace MonadIO in our function definition with NotInventedHereLog. Because we renamed the macGyverLog function to nihLog, we need to replace those calls with nihLog as well:

assignIndexToVariables2 ::
  NotInventedHereLog m =>
  MonadError String m =>
  MonadState (M.Map VariableName Int) m =>
  AST VariableName -> Variables -> m (AST Int)
assignIndexToVariables2 ast variables = forM ast $ \var -> do
    nihLog "start more monad"
    _z <- moreMonad
    ...

Running this will give us the following type error:

src/Lib.hs:54:52: error:
    • No instance for (NotInventedHereLog
                         (StateT (M.Map VariableName Int) (ExceptT [Char] IO)))
        arising from a use of ‘assignIndexToVariables2’
    • In the second argument of ‘($)’, namely
        ‘assignIndexToVariables2 ast vars’
      In the first argument of ‘runExceptT’, namely
        ‘(flip evalStateT mempty $ assignIndexToVariables2 ast vars)’
      In the second argument of ‘(=<<)’, namely
        ‘runExceptT
           (flip evalStateT mempty $ assignIndexToVariables2 ast vars)’
   |
54 |     print =<< runExceptT (flip evalStateT mempty $ assignIndexToVariables2 ast vars)

This looks scary, maybe this is the one, the one type error I can’t solve? After all, I’ve been waiting for that one perfect type error for over four years now. But no, this is known as the n²-instances problem ¹¹, which sounds very impressive. However for now we completely ignore that problem by providing an instance which solves this type error:

instance (NotInventedHereLog m) => NotInventedHereLog (StateT s m) where
  nihLog = lift . nihLog

This code says: If you’re a StateT and your base monad already has a NotInventedHereLog constraint, you also have NotInvnetedHereLog instance with help of lift. By providing this instance we’re generating lift calls over StateT for all occurrences of niLog.

Moving on we get the same error for ExceptT:

src/Lib.hs:54:52: error:
    • No instance for (NotInventedHereLog (ExceptT [Char] IO))
        arising from a use of ‘assignIndexToVariables2’
    • In the second argument of ‘($)’, namely
        ‘assignIndexToVariables2 ast vars’
      In the first argument of ‘runExceptT’, namely
        ‘(flip evalStateT mempty $ assignIndexToVariables2 ast vars)’
      In the second argument of ‘(=<<)’, namely
        ‘runExceptT
           (flip evalStateT mempty $ assignIndexToVariables2 ast vars)’
   |
54 |     print =<< runExceptT (flip evalStateT mempty $ assignIndexToVariables2 ast vars)

We will keep getting these errors for every unique transformer we use because of the n²-instances problem ¹². This blogpost isn’t about solving the n² problem, so I will ignore it. The solution to the type error however is pretty much the same:

instance (NotInventedHereLog m) => NotInventedHereLog (ExceptT e m) where
  nihLog = lift . nihLog

The example code will compile with these two additional instances. However we still don’t know how to reinterpret this purely, and we have introduced code duplication. This duplication can be removed with the default mechanism described in Alexis King her blogpost.

For the pure interpretation we need to introduce a newtype, which is used to attach an instance for NotInventedHereLog. This type’s instance will collect the logged messages. Turns out that the WriterT monad transformer does exactly what we want. So the pure code will look like this:

newtype NihLogT m a = MkNihLogT {
        runNihLog :: WriterT [String] m a
    } deriving (Functor, Applicative, Monad, MonadTrans, MonadWriter [String])

instance Monad m => NotInventedHereLog (NihLogT m) where
  nihLog msg = tell [msg]

The instance simply tell’s the message, which mappends it to the list. To run this code we use runNihLog:

    let pureCode :: (Either String (AST Int),  [String])
        pureCode = runWriter $ runNihLog $ runExceptT (flip evalStateT mempty $ assignIndexToVariables2 ast vars)
        (eitherAst, allLoggedMessages) = pureCode 
    print pureCode

Now allLoggedMessages will contain all messages emitted by nihLog. With this you can write property tests on assignIndexToVariables2. For example you could assert that an AST of size 20 should emit at least 40 log messages. Obviously this isn’t limited to tests or purity, you could also add a newtype that has a connection pool to send the messages to some database for example.

I’ll tap out here. This was supposed to be a short note on someone else’s blogpost, which spiraled into dumping all my knowledge here on MTL and extending it a bit as well. For example I didn’t even know about the defaults mechanism Alexis wrote about. I’ll tap out here, thanks for reading.

Let me know if you have any strong opinions on this style. Love it or hate it, I’d like to know! Or if you need any help using it. I’m interested in effect systems in general. Also let me know about your favorite effect system that I didn’t acknowledge.

Links

Inspirational blogpost
A functional example is available here
A video presentation on the exact same topic.
Full mtl style reinterpretation test example
And a library for mocking around mtl style, also gives more background.
If I didn’t manage to exhaust you, here is more background and alternatives.

The best I could find is this blogpost, which is more an experience report rather then an explanation of what’s going on. Ocharles could also be said to be talking about the subject, but it’s not an introduction, it’s an expert reference.↩︎
Aren’t transformers the same as mtl? No! Back in the stone ages, people weren’t sure what the right approach would be to doing this, and frankly people still aren’t sure. The base package is the transformers package. Although the mtl package is most popular, there is also mtl-tf. Polysemy also builds on top of transformers. I think the subject of best approach is a subject for debate. I’d like to point out as well that in technology, popularity means nothing. Cobol used to be more popular then C up till the 1990’s, and is practically dead now. JavaScript is also more popular then Haskell, is it therefore better?↩︎
Note that although this is valid haskell, in practice it’s nonsense to write anything with just a Monad constraint. I’m saying this because the constraint is too tight! You can’t do anything in just a monad. I’m only showing it here to raise a point. In this situation it would’ve made more sense to drop the entire monad sharade and assign 5 directly with type of Int.↩︎
No it’s not possible if a constructor isn’t exposed. Reflex uses this with Dynamic t to create a smart destructor. In reflex you can only use a value in a dynamic by putting an entire monadic action inside the dynamic, and then using dyn to get it out (for reflex-dom at least, sdl uses something different). I’m highlighting this because I feel this is a powerful idea.↩︎
Why MonadError and ExceptT don’t share their names? It’s explained here.↩︎
We’ve not solved the problem here of being able to write a Compose Monad instance at all. We just kind of worked around it by not talking about order. It’s truly an awful hack, which are the best.↩︎
It allows concrete monad transformer selection, For example : haskell x :: LogginT (ReaderT AppContext Maybe) String x = do callInLoggingT lift $ callInReaderT lift $ lift $ callInMaybe AppM from the original blogpost didn’t have to do this because the deriving mechanism generated the instances that in turn generate the lift calls.↩︎
No! It just happens to be irrelevant for most transformers in the transformers package. For example there exist the transformer ResourceT. I encountered a production bug due to wrong order of running the transformer stack. Only large queries crashed with an opaque message StatementAlreadyFinalized. The selectSource exposes the resource constraint. We also had a typeclass to embed that ReaderT backend into the constraints, since it’s just a regular reader after all (it’s not quite). We also had an instance that said order of this ResourceT and ReaderT backend didn’t matter, but it did. This instance caused the running of the query to leak outside of the runResourceT block. Removing the instance and being explicit about order solved the bug.↩︎
MacGyver is a TV character who was known to get out of precarious situations with absurdly little in terms of resources and a lot of improvisation, our logging function feels similar at this point. Another term that could’ve been used was Not Invented Here, but I liked MacGyver better.↩︎
The thing that baffles me is that GHC already manages to print your own code, they figure out the right constraint, why don’t they just give the example like I’m doing here with the message “try doing this:”. Instead they talk about constraints and contexts. Do they hate people using their language or something? I feel George Orwell’s guide to writing well also applies to error messages.↩︎
I’m linking to this blog as an explenation of the n^2 instances problem. I definitely don’t endorse using Overlappable as a solution for it. I’d prefer using Alexis method which is very copy paste-able.↩︎
I’m linking to this blog as an explenation of the n^2 instances problem. I definitely don’t endorse using Overlappable as a solution for it. I’d prefer using Alexis method which is very copy paste-able.↩︎

The Nix mutli-monolith machine (NMMM).

2021-07-31T15:05:00Z

I redid how my services are structured. Instead of running each project on a separate VM, they’re now all running on a dedicated Hetzner machine. This is what I call the nix multi monolith machine (hence forth called NMMM, pronounced like tasting something delicious prefixed with an N). There are some really big advantages to the NMMM approach.

Why use a NMMM?

Setting up NMMM has a couple of advantages:

the NMMM is cheap ¹
the NMMM is simple ²
the NMMM doesn’t need nixops while benefiting from nix ³
the NMMM spins out new services fast ⁴

Obviously there are also some disadvantages to this approach, which get answered in the discussion section. But for a typical startup situation, where money is tight and there is no product market fit yet, and time to market is important, I think the NMMM is the best.

What is NMMM

The core of the Nix mutli-monolith machine (NMMM) is a dedicated machine. A dedicated machine means no virtualization, in other word it’s bear metal.

The second major part of this configuration is nix. This means that every piece of software is described in nix files. Just like the configuration of this software. More on that in the nix config section.

The third major part is multi-monoliths. Meaning that you can have multiple, unrelated services running on the same machine. This is something different from micro services. Microservices communicate with each other at some point to provide a consistent frontend. The link calls this ‘loosly coupled’, or as I liked to call it: They chat with each other. Monoliths on the other hands should not communicate with each other and are isolated. They have independent frontends and backends. In other words, no chatting between monoliths. They just stand there silent and ominous. I feel this approach falls inline with the monlith first approach, but rather then a single monolith I can deploy many completely unrelated projects.

Nix config

I build this by relying on the module system. The main entrypoint is an ordinary nixos configuration file, other modules in general are also shaped like this even though the entry point depends on them. So we got a very versatile one trick pony!

Assuming / is the root of the project, the root configuration looks like this:

# /nix/hetzner/configuration.nix
{ config, pkgs, ... }:

{
  imports =
    [ 
      ./hardware-configuration.nix
      ../../videocut.org/nix/videocut.nix
      ../../massapp.org/nix/massapp.nix
      ../../raster.click/nixops/raster.nix
    ];
    ...

That has the same structure as the configuration.nix I use on my laptop. The difference this example has with my laptop is that I’m pulling in a bunch of additional modules aside from the hardware config trough the imports mechanism. imports tells nixos to also include those other configuration files. However before looking into those specific files, I need to explain how to run this entrypoint. I call this configuration from my /makefile:

deploy:
	nix-shell nix/nixpkgs-shell.nix --run "make deploy_"

ROOT_DIR:=$(shell dirname $(realpath $(firstword $(MAKEFILE_LIST))))

deploy_:
	NIXOS_CONFIG="$(ROOT_DIR)/nix/hetzner/configuration.nix" nixos-rebuild switch --target-host root@videocut.org --show-trace

make deploy will deploy if called form the root of the project /. It runs make deploy_ from a shell that sets the NIX_PATH to a pinned nixpkgs. This uses nixos-rebuild switch, just like on my laptop. However I specify the target host to be one of the domains hosted on the Hetzner machine. All domains domains lead to the Hetzner machine.

But how does the machine decide which HTTP request goes to what service? After all they all arrive on the same machine now, so something has to make a decision on this. Nginx can do that! this is a bit later in the entry point file:

  # /nix/hetzner/configuration.nix
  ...
  services.nginx = {
    virtualHosts =  pkgs.lib.foldr (x: prev: pkgs.lib.recursiveUpdate (import x) prev) {}
      [../../videocut.org/nix/vhost.nix
       ../../massapp.org/nix/vhost.nix
       ../../raster.click/nixops/vhost.nix
       ../../blog/vhost.nix
      ]
  }
  ...

We let the individual services decide how to configure the virtual hosts. Virtual hosts allow us to specify configurations per domain name. For example the massapp.org host looks like this:

  # /massapp.org/nix/vhost.nix
  lib = import ./lib.nix;
  sslBools = {
    forceSSL = true;
    enableACME = true; # E
  };
  base = locations:
    {
      inherit locations;
    } // sslBools; # C
  proxy = base {
    "/".proxyPass = "http://127.0.0.1:${
        toString lib.massapp_port # D
      }/"; 
  };
  redirect = { globalRedirect = "${lib.massappDomain}"; } // sslBools;
in {
  "www.${lib.massappDomain}" = redirect; # B
  "${lib.massappDomain}" = proxy; # A

The main thing we’re saying at A is that the massapp.org domain should point to the port defined at D. Furthermore in B we’re redirecting all www traffic to A. Which strips off www from www.masssapp.org resulting into massapp.org. For some reason people still yearn to type www. With this redirect we trash the peoples’ pointless dreams and desires. Finally in C we strip of the HTTPS, which the proxy will remake into an HTTP connection. Traffic at this point is internal, so SSL has served it’s purpose and we can safely strip it. In practice this means our application doesn’t have to deal with certificates or SSL. This leverages nixos built-in let’s encrypt support without even thinking about it in E ⁵.

Since we just bound all incoming domains to a unique port, we have to bind a program to that port as well. This program is our main application code, or the monolith. For example let’s look at the massapp module. Herein I register a systemd service which runs the main massapp executable. This is another file like configuration.nix (just like my laptop! The one trick pony):

# /massapp.org/nix/massapp.nix
{ config, pkgs, ... }:
let
    massapp = pkgs.callPackage ../webservice/default.nix { }; # A
in {
      ...
      systemd.services.massapp = # D
        {
        description = "Massapp webservice";
        serviceConfig = {
            Type = "simple";
            ExecStart = "${ massapp }/bin/massapp"; # B
        };
        wantedBy = [ "multi-user.target" ];
        requires = [ "postgresql.service" ];
        environment = {
            PORT = "${toString lib.massapp_port}"; # C
            ...
        };
    };
}

Within A I load the main binary of the service. We tell systemd about that program at B. Finally we make the program aware of the correct port in C by setting the environment variable PORT. Yesod has a configuration built in for that by default, and massapp is a Yesod application. This would work for any other application, you can even pass CLI arguments like this just by modifying the ExecStart. At D we give this systemd unit the name massapp, by doing this std out is logged in journalctl and tagged with the unit name. Nixos-rebuild will now also know if a service failed trough exit codes. For example if the service can’t find the database. Which we discuss how to setup in the database section.

Database integration

This is the database configuration, again in the entrypoint configuration.nix file:

  # /nix/hetzner/configuration.nix
  ...
  services.postgresql = {
    enable = true;
    # A
    authentication = pkgs.lib.mkOverride 10 ''
      local all all trust
      host all all ::1/128 trust
      host all all 0.0.0.0/0 md5
      host all all ::/0       md5
    '';
    settings = {  # B
      log_connections = "yes";
      log_statement = "all";
      log_disconnections = "yes";
    };

    # C
    initialScript = pkgs.writeText "backend-initScript" ''
      CREATE USER massapp_prod WITH PASSWORD 'someinitialpassword';
      CREATE DATABASE massapp_prod;
      ALTER USER massapp_prod WITH SUPERUSER;
      '';
   ...
   }

The initial scriptC is only run when the database is started for the first time, after that you need to manage users by hand. However I still add these users to script to keep track of them if I ever need to abandon this system. Like this at least the users will exist on the new machine. I give every monoliths’ user superuser access in C. This isn’t the best for security, but it’s really convenient. Besides if a hacker manages to get a shell for the system or a shell for the database it’s already to late for me anyway. I’d just scrap this system and start over elsewhere.

In B I set some additional logging options, which are just convenient when things break⁶. With statement logging I can still see what happened in the database if the application didn’t emit enough information. logging all statements slows down the database. However, my services are hardly taxed at the moment, they are taxed but no where near justifying not logging.

Localhost connections can authenticate without password in A. Outside connections are ignored, they’re not even asked for a password.

Discussion

This setup goes against advice from sysadmins, where they recommend you split up everything across VM’s as much as possible. I reject their hypothesis. I mean they argue for splitting services, but the tool used for splitting shouldn’t be dogmatic and always result in choosing a VM. Using multiprocessing, and tenanting for specific software packages is good enough. myron semacks at least gives some arguments on why he chooses to use a VM. However his reasons for choosing a VM’s revolve around windows related oddities. For example:

Think about what happens when you need to upgrade the OS, which typically means you make a new VM to replace the old one.

We can seamlessly upgrade nixos pretty much always. I’ve had configuration files change which caused errors when upgrading nixos stable versions, but these errors need to be solved before deploying. The systems I use with nixos themselves have always been stable. Or:

Think about what happens when there is a problem and the VM is down. (Bad Windows update, you fat fingered a network setting, etc)

How would I even fat finger a network setting? I have to modify a file, commit the file and deploy to do this. At this point you can’t even call it fat fingering. And even if you somehow screw up a network setting deliberately, you can reboot from an older generation and the problem is gone. The difference here is that this deployment is nix based, which allows me to manage the complexity of the monolith much more efficiently and reliably.

NMMM can also be used to impress potential clients by quickly creating new websites. What are the steps?

Get the new domain, for example newdomain.com,
copy over the massapp.org folder into newdomain.com,
hook it into the main config/nix/hetzner/configuration.nix,
setup the database,
redo branding
deploy.

That’s it. That’s two hours of work, maybe three if something goes wrong. Nothing is more impressive then have a functioning website, the next day. I guess aside from startups, consultancies should also look into this approach.

On several occasions I’ve mentioned that configuration.nix is just like on my laptop. Originally I even copied pasted the Postgres configuration from my laptop to this machine. It just works. This is one of the big benefits you get out of nix, it’s called the copy-paste monad.

If you’re having doubts about nixops and want to replace it, you may run into secret management issues. Fortunately there is an alternative project for secret management called agenix. Although I’m a bit skeptical over age, it seems to new to trust. You should do your diligence before using that in production.

Conclusion

I described this NMMM setup that’s working really well for me. Furthermore I think I gave compelling reasons for others to try this out. Perhaps I liberated some people from their virtual insanity. Let me know if I’ve inspired you to change your course of action, or you’ve some compelling reasons not to use this approach. Because how to structure services is a large decision, I find this all fascinating.

Using git for templates

2020-10-24T15:00:00Z

Over the past few years I’ve started using git as a template management tool¹. For example, I clone my haskell template project, edit the names, edit the readme end re-setup the remotes:

git remote add template git@github.com:jappeace/haskell-template-project.git
git remote set-url origin git@github.com:jappeace/new-project.git

This allows me to keep my downstream project up to date with the template. When I discover another tool, this can be merged into the downstream project for example. It allows evolving the template over time as I learn more about whatever programming language I’m using.

It also gives me default branding. Which makes it look more ‘professional’. Although I don’t like branding, I know this is necessary. Having it there by default makes it so that I don’t forget to do this.

When starting a project you also gain some free authoritativeness, because you have a longer commit history the project looks like it is more mature. Although this practice is a little deceptive, there is a core of truth, because the template has gotten a lot of attention. If you feel offended by this practice, ask yourself first, why are you trusting git histories? These can be rewritten anyway. This authoritativeness also gives me a small moral boost. It just feels better to have a project with well thought out tools being started and some history rather then the ‘normal’ stuff everyone starts with.

Alternatives

The alternatives are using tools that spit out templates. In the haskell world these are stack new, cabal init or summoner. These tools are simply a glorified sed on projects folders: They find and replace variables. There is no versioning, there is no personalization except what limited variables give.

Because these tools don’t take into account history, the first commit will be a large commit with a message like ‘Initial’. I find this problematic because it squashes all the work that went into creating the template. Information is lost, for no reason.

These tools don’t use git to track their templates. They could, but they don’t. Maybe they wanted to be VCS² agnostic? I’m speculating, but even if you wanted to be that, you could still use any version system to track the template and recreate the history in the VCS desired by the user. This at least gives the user their history of the template back. While also giving them the choice of VCS³

A template hierarchy

What I described in the intro can be taken a step further. A template of a template can be used to create another new template. For example, say you’re a fan of nix or make, so why not put a makefile and a nix pin in some template of templates project. These tools don’t care about which language you use, so their configuration can be shared among Python, Java and Haskell projects for example. Maybe for github it could even have a partial travis or github actions CI setup.

Now, say you’re starting a new Python project, but have no template yet for that programming language. You simply clone this template of templates and customize it to your own Pythonic needs. You can still pull in the upstream changes from your template of templates, for example to upgrade the nix pin. The big advantage of course is that a change in the template of templates can easily merged into all downstream projects with git.

Conclusion

I convinced the reader that using git is the superior way for template management. It’s simpler, and more flexible. I also convinced the authors of stack, cabal and summoner to drop their current way of managing templates and use git instead.

Or, more likely, no-one read this far.

This started out as having no better alternative, due to my nixos choice. But now I realise, this is the best alternative.↩︎
Version Control System↩︎
I wouldn’t bother with this choice unless someone is asking for it. I think git has won the VCS battle by a wide margin. Of course that’s easy for me the say as neither maintainer nor user. 25% SVN market share is a lot more then I expected (the absolute SVN repo count is still going up!).↩︎

Hacking atom

2020-08-16T21:12:00Z

On my twitch stream I had project where I made a CRT scan line monitor in atom:

The inspiration was the fallout pip-boy. I though it would be cool to have this be in your editor, maybe some of the gray beards are nostalgic for this?

What amazes me is that this is possible with just CSS. Atom has become my favorite editor after doing this project (although I hardly use it), and my love for CSS has grown even more!

Ghcid for multi package projects

2019-10-17T21:30:00Z

When I tried Ghcidfor a reflex project, it wouldn’t rebuild on file change. This is because reflex has a multi package project setup by default. Recently I found that it is possible to use Ghcid for multi package project builds. The trick is to create an executable that includes all the sources. For example in hpack style:

tests:
  unit:
    main:                Spec.hs
    source-dirs:
    - test
    # including extra source dirs allows Ghcid to watch
    - src 
    - ../common/src
    - ../frontend/src

Now Ghcid will watch all the sources. By targeting the test suite you can also immediately use this to run unit tests. If your other project use additional dependencies you may need to add them to that test as well.

However because I’m doing reflex, I had to compile JavaScript after Ghcid typed checked everything. I also had some unit tests to run. Luckily Ghcid supports both a run and a test flag. I simply used the run flag to run the tests and the test flag to start GHCJS compilation. Which brings us to this spell:

    ghcid -s "import Main" \
        -c "make update-cabal && cabal new-repl" \
        -T ":! cd .. && make after-native-build" \
        --run test:unit

The run flag can also be used to run your code if you don’t have a weird GHCJS requirement like me. Running like that gives fast feedback because it’s interpreted.

Conclusion

This setup is an order of magnitude faster for debugging in my case. If I have a compile error it now notifies me instantaneously, whereas before it would take a second or two. If I need to do a full build it takes about 10 seconds, whereas before it was somewhere around a minute or two.

Pinning nixops builds

2019-10-17T17:35:00Z

On one machine my Nixops builds and deploys, but on the other one it fails. Why? Isn’t nix supposed to deliver reproducible builds? Turns out nixops uses by default your system configurations’ channels rather then a pinned nix packages. Which is not why you’re using nix. You want a reproducible build, in other words, either both machines fail for the same reason or both succeed. We will discuss how to do this for nixops in this post trough a pin.

To pin a nixops deployment we create a shell[^alternative] from which we run nixops: [^alternative]: For my reflex project I simply made an alternative shell, because reflex-platform doesn’t make the shell easily modifiable. you can run a different shell by providing the filename to the nix-shell command: nix-shell nixops-shell.nix

let
   pkgs = import ./pin.nix;
in

  pkgs.stdenv.mkDerivation{
  name = "nixops-env";
  NIX_PATH="nixpkgs=${pkgs.path}";
  buildInputs = [ pkgs.nixops ];
  }

Where pin.nix looks like this:

let 
hostPkgs = import <nixpkgs> {};
pinnedPkgs = hostPkgs.fetchFromGitHub {
    owner = "NixOS";
    repo = "nixpkgs-channels";
    rev = "1601f559e89ba71091faa26888711d4dd24c2d4d";
    sha256 = "0iayyz9617mz6424spwbi9qvmcl8hiql42czxg8mi4ycq4p1k0dx";
};
in
import pinnedPkgs {
    config.allowUnfree = true; # took me too long to figure out
}

This is also possible for reflex-platform projects where we simply depend on the provided pin:

   pkgs = ((import ./reflex) { }).nixpkgs;

Conclusion

In this post we discussed how to pin nixops builds. There are other ways to do pinning, which is described here. I found it hard to get a working solution from that thread, therefore I recorded my own solution here.

Reflex server side html rendering

2019-08-05T19:05:00Z

Reflex is a single page app framework written in Haskell compiled to JavaScript. A major concern with reflex is the slow loading times, this can be mediated however by doing server side rendering¹. This blog post will discuss how to do that².

The main idea is that we can share most of our single page app code on both the frontend as well as the backend. The backend will create a static HTML page, whereas the frontend will be a JavaScript client. The browser can display this HTML content very fast without having to interpret any JavaScript. This is where all the speed gain comes from. ³ JavaScript developers call this technique ‘isomporhic’⁴ JavaScript. This blogpost will tell you how to do it with reflex. I found it surprisingly easy, and the results very impressive.

The renderStatic function does the HTML rendering. However to use it we need to jump trough some oddly-shaped, MTL flavoured, hoops⁵. Doing this is not much harder than writing a regular reflex app. Jumping trough these solves handling browser native primitives, such as XHR calls. This is done with help of prerender. Which is also the key to displaying authenticated content fast, but we have to rewire the the initial app state to do that. Fortunately this rewiring caused the initial state management to become more elegant then it was in the authentication post.

The source code is available in the reference project.

Strange MTL Hoops

We need to get rid of WidgetMonad and replace it with the underlying builders. We do this because the WidgetMonad has a nasty equality constraint on GhcjsDomSpace that prevents the use of renderStatic:

type MonadWidgetConstraints t m =
  ( DomBuilder t m
  , DomBuilderSpace m ~ GhcjsDomSpace -- <-- nasty constraint
  ...
  )

class MonadWidgetConstraints t m => MonadWidget t m
instance MonadWidgetConstraints t m => MonadWidget t m

A good strategy is replacing it with DomBuilder and add more constraints based on compiler requests. MTL monads push outwards. That is to say, the low level widgets determine the constraints of the higher level widgets.

Unfortunately this is not the only place for that equality constraint. The input and textarea widgets indirectly cause this constraint too because of their return types. For example in input:

data TextInput t
   = TextInput { 
		... 
		, _textInput_builderElement 
		:: InputElement EventResult GhcjsDomSpace t  -- <-- nasty fundep
		}

There is no direct equality constraint, but it gets asserted anyway because of functional dependencies. However I made an alternative to both these widgets in the Bulmex library. In which I just copied the original and removed the constraining records from the result types. They can be found here.

Prerender Parts

As mentioned before, we intent to share most code between the client and the server. In case of the client we use a JavaScript executable entry point and get a GHCJS DOM environment. In case of the server we use a servant endpoint as entry point and get a static DOM builder environment. When the GHCJS environment is accessed reflex needs to know what to do in the other environment and vice versa. In the browser for example we’ll happily do XHR calls, but when doing server side rendering we can simply ignore the request⁶. This use case between environments is handled by prerender:

prerender :: m a
            -> (PrerenderClientConstraint js t (Client m) => Client m a)
            -> m (Dynamic t a)

Depending on the entry point we use we get a different monad environment. On the server we get a StaticDomBuilder environment (m a), and in the JavaScript executable frontend we get a GHCJS DOM environment (Client m a). The first monad is the one being displayed on the server, the second one has access to the GHCJS DOM environment. The resulting value will be captured in a dynamic and can be used by the shared code.

Now we can enhance all servant-reflex calls with prerender calls, to make it do nothing in the static environment and do the XHR calls on the GHCJS dom environment. For example the postLogin function from the authentication blogpost now looks like:

postLogin :: (DomBuilder t m, Prerender js t m)
          => Dynamic t (Either Text.Text User)
          -> Event t ()
          -> m (Event t (ReqResult () (AuthCookies NoContent)))
postLogin dd yy = fmap switchDyn $ prerender (pure never) $ postLogin' dd yy

Here postLogin' is the generated endpoint from servant-reflex. In case of a server StaticDomBuilder environment we do nothing never (pure never)⁷, in case of a GHCJS DOM environment we do the postLogin' function. The fmap switchDyn part is to get rid of the dynamic, which is unnecessary because we already return an event. Similar code can be used for all generated endpoints⁸.

Backend servant endpoint

We need to add an endpoint to servant which will serve the statically rendered HTML. To add it to servant we make an authenticated entry point, which returns HTML as a ByteString. Because it’s authenticated we can choose what to serve. The authenticated app, if the user has the permission, or the public site, if the user doesn’t have that. It looks like this:

type Webservice = ...
      :<|> Auth '[Cookie, JWT] User :> (Get '[HTML] BS.ByteString)
      ...

In the previous blogpost we’d simply return a ‘401 unauthorized’ status code. Servant auth allows us to decide how to respond. Which we can see in the implementation:

renderHtmlEndpoint :: HeadSettings -> AuthResult User -> Handler BS.ByteString
renderHtmlEndpoint settings authRes = do
  fmap snd $ liftIO $ renderStatic $
    htmlWidget settings $ main $ IniState $ toMaybe authRes

The main function is the entry point for the reflex code. We wrap it around in htmlWidget, which has HeadSettings to load the JavaScript client as a simple URI. The decision of what to show is completely pushed into the reflex code, the only thing we do at server side is give it the IniState. Next we’ll discuss how the reflex app decides what to show in this environment.

Reflex App

So now we have an IniState for the app, we can read or write it to the DOM depending on environment. I wrote a function for that. On the app side this is done by writeReadDom:

writeReadDom ::
    ( FromJSON a
    , ToJSON a
    , Dom.DomBuilder t m
    , Dom.Prerender js t m
    ) =>
    Text.Text -> a -> m (Dynamic t a)

The first value is the HTML id, the second one is the app state as received by it’s respective endpoint. The result behaves roughly like this:

<!-- Static dom builder environment (server) --> 
	<script id="Text.Text" type="application/json">
		<?php encode a ?>
	</script>
<!-- GHCJS environment (browser) -->
	JSON.parse(document.getElementById("Text.Text"));

Of course the JavaScript will look completely different and the result is put in a dynamic. But the resulting behavior is the same, the server encodes the JSON and the client will decodes it giving us the app state from the server without additional requests. Under the hood prerender figures out if we need to read or write. All the client needs to do is provide said value, in case of the example app this is just a Maybe User. The decision of what to display is all integrated within the same reflex code like this:

main :: (AWidget js t m) => IniState -> m ()
main iniState = do
  iniDyn <- writeReadDom "iniState" iniState
  rec loginEvt <- elDynAttr "div" loginAttr $ loginWidget iniDyn
      loginAttr <- holdDyn (Map.empty) $ hidden <$ loginEvt
  void $ holdEvent () loginEvt authenticatedWidget

authenticatedWidget :: (AWidget js t m) => User -> m ()
authenticatedWidget = ...

We always get an IniState, but we make sure to first put it in writeReadDom. This ensures that the IniState is the actual IniState from the server and not the one we hardcoded in the executable to make the compiler happy⁹. This iniDyn then gets used for the loginWidget which returns a loginEvt. If the event fires it has a user, with which we can display the authenticatedWidget. This is done with help of holdEvent. Note that if no event is received, no authentication is done. loginAttr will however become hidden as soon as there is a loginEvt, hiding the login form. The login widget itself looks like this now:


loginWidget :: (AWidget js t m) => Dynamic t IniState -> m (Event t User)
loginWidget iniDyn = do
  pb <- getPostBuild
  formEvt <- loginForm
  pure $ leftmost [formEvt,
       noNothing $ updated $ userDyn,
       noNothing $ current userDyn <@ pb]
  where
    userDyn = unpackUser <$> iniDyn

Compared to the previous blogpost we got rid of autoLogin and use the IniState instead. The post build event is used to sample the current value of userDyn. This ensures a user event fires if there is one available. That mechanism also works for static rendering.

Conclusion

Loading times was one of reflex major weaknesses and this blog post discussed how to mostly remove it. It still takes time before the app becomes interactive, but at least now useful information can be shown to the user instead of a loading screen. I’m really happy with the end result, and I think anyone with a reflex app should seriously consider doing this.

References

Going from 8 seconds in my own app to about 0.5 seconds.↩︎
A prebaked solution is available in oblisk, one should consider using that.↩︎
The server doesn’t need to interpret JavaScript, Haskell can also target native as compile target. But even if it were running JavaScript you can still get speed gains by doing page caching.↩︎
So it’s not isomorpism categorical terms, but JavaScript terms.↩︎
I figured out all this stuff on stream ↩︎
Special thanks to lumie for pointing this out.↩︎
I could imagine an alternative design: For example with a micro service setup where you just let the app decide what to call. You’d still need to use a different client on the server side though because it’s native code. But this can be done for with servant-client. You define your micro services around the ‘servant’ based REST contracts and let the app pull it in.↩︎
It may also be possible to make a more generic function to do this, but I didn’t want to spend the time (I’ve been working on this for days).↩︎
I don’t know of a type safe way around this inital state problem. Note that the JavaScript loading by itself is also not type safe, a URI is used. With that in mind we can kind off consider this the ‘edge’ of the system.↩︎

Citrix XenCenter 7.6 notes

2019-04-18T00:00:00Z

A client had a Citrix environment running. To debug the deployment of a machine, I copied the Citrix environment locally in VirtualBox. Note that the server doesn’t run any machines inside citrix, but it does get the XenCenter operational. It appears that in nixos you can just setup a local xen server. The configuration described here will setup a Citrix XenServer that installs and runs, it just can’t run other VM’s. It says it misses HVM.

This post contains my notes on getting XenCenter operational in VirtualBox. It’s also a guide for people familiar with Linux, but not with Citrix.

Citrix XenServer

It’s actually CentOS with XenServer installed on it. The VM shell will drop you into a ncurses menu which has an option to go to a bash shell.

The branding says it’s a ‘smaller’ operating system, looks pretty complete to me though. I think they mean that they have kernel modules that may allow the guest VM’s to use hardware directly. To do this you have to modify the host as well as the guest OS. They both have to understand the virtualization situation.

If you want to do this properly you should seriously consider containers. Even though virtual machines are more secure and virtual machines are more portable¹, the trade off however is that virtual machines are really inneficient. Giving them direct access to hardware won’t change the nature of OS kernels. To hog all resources, which they have to do to manage them. A container setup will simply have a single kernel, yet many operating systems (user spaces).

Get XenServer

Go to citrix.com, you must get it from there. Don’t bother googling this. Google doesn’t find the download. You have to make an account to download it.

Then go for the citrix hypervisor in the downloads area. By the way: XenServer = citrix hypervisor I think the only thing they did to the XenServer is add branding.

Get client

The XenServer also contains the client. No need to download any of the other stuff. XenServer is one of the many products hosted by citrix.

You can run XenServer in virtual box

You need enough RAM though, just create a swap file if you run into trouble.

Make sure to bridge it on a device like WIFI. This allows one VM to access another VM. Also put promiscuous mode on allow all. I think this is why I couldn’t transfer the VHD.

Install it, go into the root shell and type ip addr to get the ip. We need this for the client.

Creating an extra storage repository:

For some reason the XenServer created a very small 8G repository, I just added another disk with VirtualBox and then ran this on the XenServer vm shell:

xe sr-create name-label=<Storage ID> shared=false device-config:device=<Path of the Storage device> type=lvm content-type=user

Actually I attempted to partition it, but I think the command already does that. If not see my other post where gdisk is explained.

Use the windows client

I couldn’t get the citrix Linux client to work on nixos. I attempted libvirt because it supports remote but it seemed fairly complicated. I ran the XenCenter client in the edge iso provided by MS. This VM was bridged too.

Once inside the VM, click on a VMDK to import it into XenCenter. I could find no way of doing this in the XenCenter program itself. Also make sure to have a server selected before importing it, otherwise you’ll get a null reference error on the storage screen. It apparently uses the selected one instead of the location from previous screen.

Log for notices

If you ever get a “generic error”, check the log:

    C:\Users\IEUser\AppData\Roaming\Citrix\XenCenter\logs

There is a ton of information in that. Pasting the relevant part into google will get you more progress than “generic error”.

VirtualBox VMDKs

Citrix XenServer does not appear to accept VirtualBox VMDKs. I got VHD to work locally, but for some reason it didn’t work at the client. If you want a NixOS VHD checkout this post

If you mix OS’es for some reason you’re right in using virtual machines↩︎

Nixos notes

2019-04-18T00:00:00Z

This is a post of things I wanted to do in nixos but isn’t described anywhere. I had to read source code to figure these things out. By explaining here what is going on I make things easier for other people.

Nix custom image

It’s possible to bypass virtual box and make a bunch of different image formats directly, I used two files image.nix:

{ config, ... }:

let
  pkgs = import ./pin.nix { };
in

{ 
  system.build.image = import <nixpkgs/nixos/lib/make-disk-image.nix> {
      name = "nixos-vmdk-${config.system.nixos.label}-${pkgs.stdenv.hostPlatform.system}";
      format = "vpc";
      inherit pkgs config;
      lib = pkgs.lib;
      partitionTableType = "legacy";
      diskSize = 11 * 1024;
  };
  .... # remaining config, same as configuration.nix
}

and the disk.nix:

{ nixos ? <nixpkgs/nixos>
, system ? builtins.currentSystem
}:

let
  machine-configuration = import ./image.nix;
  machine = import nixos {
    inherit system;
    configuration = machine-configuration;
  };
in 
machine.config.system.build.image

If you run nix-build disk.nix you’ll get a VHD with the configuration from image.nix which is just a nixos standard configuraiton.nix.

Nixos-rebuild remote

We can use nixos-rebuild to do in place updates of a running system remotely. If your deployment is a single VM this is significantly easier than using nixops. I ended up with this make file:

IP="192.168.0.39"
deploy: 
	NIXOS_CONFIG=$(shell pwd)"/image.nix" nixos-rebuild switch --target-host root@$(IP)

You may also want to set --build-host, because by default it will build on target-host.

Install nix on running nix

For some reason my boot disk gets corrupted after switching a couple of times. A solution is just to never reboot, however you can also just fix this while running the system. Switching doesn’t do this apparently. I know this because it didn’t boot. This doesn’t matter because you can install your currently running system!

Mount root as /mnt. Format your boot partition and mount it on /mnt/boot. Then run nixos-install. For example:

mount # list everything mounted
mount /dev/nvme0n1p2 /mnt
umount /boot
mkfs.vfat -n boot /dev/nvme0n1p1
mount /dev/nvme0n1p1 /boot
mount /dev/nvme0n1p1 /mnt/boot

Authentication in Reflex & Servant

2019-02-28T15:00:00Z

In the previous blog post we saw interaction with servant in reflex. Although that covers the basics, there are several more hurdles to overcome to get comfortable with Reflex. I think most of these are encountered by building a simple login system. So let’s build something like:

													
                   	   +------------------+
   +-----------+       |    .      -      |
   | +--------+|       | .    -  .. .     |
   | +--------+|       | . .-- - + m..-.  |
   | +--------+|    \  |   m# + .-..% -   |
   | +--------+|     \ |   + .+ #.+...+   |
   | login ----+------X|    .. -.- *      |
   +-----------+     / |    ..  ..-#+ +   |
                    /  |      .  - -. .   |
                   	   |         . .      |
					   |          -       |
					   |                  |
		     		   +------------------+
					        Awesome app

I’ve experienced that this is hard for the first time. With this blog post I hope that setting up authentication becomes easier, considering the following pain points:

‘Switching screens’ after login, requires recursive do.
Dealing with cookies yourself is pretty hard, many pesky (security) details.
Rendering widgets inside FRP constructs requires widgetHold or ‘dyn’.

I’m of course a world expert on login systems, so in production do everything exactly as I do. Trust me, I’m from the internet. But seriously If you see something dubious, do contact me. I’ll happily rectify mistakes, and put you on the page of honour (if you want).

I left some code out of this blog post for succinctness. But the full source is available on github.

API Endpoints

type ServiceAPI = PublicAPI :<|> Auth '[Cookie, JWT] User :> AuthAPI

First we need to split our API into two types. The PublicAPI and the AuthAPI. Once the user is logged in he gets access to the AuthAPI. We’ll secure this with a JWT cookie. If the user does not have a proper cookie, he’ll get a 401 unauthorized status code.

type PublicAPI = "api" :> "1.0" :> "login" :> ReqBody '[JSON] User
								:> Post '[JSON] (AuthCookies NoContent)

This is our entire public api, we only expose the login endpoint. The result contains no content, but cookies. Servant assumes all status codes are possible in every request. Therefore we don’t have to mention the 401 status code.

type AuthAPI =
           "api" :> "1.0" :> "me" :> Get '[JSON] User
      :<|> "api" :> "1.0" :> "users" :> Get '[JSON] [User]
      :<|> "api" :> "1.0" :> "message" :> ReqBody '[JSON] Message
						  :> Post '[JSON] [Message]

The AuthAPI is similar to the ServiceAPI from the previous blog post, which only contained the users and message endpoints. Now we’ve extended it with a getme endpoint. The getme endpoint is a hack to do auto login with cookies. It allows us to do a request on initial page load to see if we have the cookies or not. Technically we shouldn’t have to do this trough a request, but it works for version 0.1.

Next we’ll implement these types into handlers:

login :: ApiSettings -> User -> Handler (AuthCookies NoContent)
login settings user = if elem user users then do
	withCookies <- liftIO $
		acceptLogin cookies (jwtSettings settings) user
	pure $ maybe (clearSession cookies NoContent) (\x -> x NoContent)
		withCookies
  else throwAll err401 -- unauthorized
  where
    cookies = cookieSettings settings

We added a new login handler, which checks if the user exists within the users list. If the user is in the list, we use acceptLogin to create a JWT from the user. [A]cceptLogin returns maybe a function which applies the JWT cookie. In the success branch of maybe we apply this function to NoContent to get an AuthCookies NoContent. The Nothing branch also produces AuthCookies NoConent, but it sets the cookie values with clearSession resulting in nothing instead of a JWT.

The ApiSettings is just a data type with various configurations:

data ApiSettings = ApiSettings
  { cookieSettings :: CookieSettings
  , jwtSettings    :: JWTSettings
  , connection     :: Connection
  }

You can’t use connection like this in production, it needs to be a pool, because servant is fully concurrent. You’ll end up with data races if you use a plain connection.

For the cookieSettings I modified the defaults quite a bit:

cookieConf =
  defaultCookieSettings
    { cookieIsSecure = NotSecure
    , cookieMaxAge = Just $ secondsToDiffTime $ 60 * 60 * 24 * 365
    , cookieXsrfSetting = Just $
        def { xsrfCookieName = Text.encodeUtf8 cookieName
            , xsrfHeaderName = Text.encodeUtf8 headerName
            }

    }

Cookies are set to NotSecure to allow it to work on HTTP. This is required for local testing, and avoids confusion about why your cookies, and your entire login system, don’t work locally. You should simply disable HTTP in production anyway. There is no good reason for using plain HTTP on a live website, ever since let’s encrypt became a thing.

The max age is simply an auto sign out after a period, a year in this case. This is a bit more secure because we don’t trust someone’s login forever.

XSRF settings are set to use the names from the Common XSRF module. This ensures the requests from the frontend use the same names as servant-auth expects.

authenticatedServer :: ApiSettings -> AuthResult User -> Server AuthAPI
authenticatedServer settings (Authenticated user) =
    (pure user :<|> pure users :<|> messages (connection settings))
authenticatedServer _ _ = throwAll err401 -- unauthorized

The authenticatedServer handles the endpoints for the authenticated API. The only new one is getMe, which just returns the authenticated user. All authenticated endpoints now have access to the user who was authenticated. This user is decoded from the JWT by servant-auth.

We get an AuthResult from servant-auth-server to work with. If the user is authenticated, we give access to the API, if not we return a 401 status code. This is done manually, which means the 401 response is not mandatory.

server :: ApiSettings -> FilePath -> Server Webservice
server settings staticFolder =
	(login settings :<|> authenticatedServer settings)
	:<|> serveDirectoryFileServer staticFolder

The server function now has to split our API on authenticated and public parts. It’s similar to the previous blog post. We still serve the static folder for testing.

app :: ApiSettings -> FilePath -> Application
app settings staticFolder =
  serveWithContext webservice context $ server settings staticFolder
  where
    context = cookieSettings settings :. jwtSettings settings :. EmptyContext

We now serve with context, this is the servant-auth entry point for decoding of the JWT from the Cookie.

This is probably not the way you want to do login on the sever side for the following reasons:

We don’t handle passwords, period. (Authentication by trust is a thing?)
One shouldn’t use JWT’s for sessions

Solving these issues is out of the scope of this article[^scope]. [^scope]: My articles tend to snowball anyway, for example invented how to do XSRF specifically for this article. I Don’t want to cargo cult a bunch of XSRF vulnerable websites.

The client API is much simpler:

postLogin :: MonadWidget t m
          => Dynamic t (Either Text.Text User)
          -> Event t ()
          -> m (Event t (ReqResult () (AuthCookies NoContent)))
getUsers :: MonadWidget t m
          => Event t ()
          -> m (Event t (ReqResult () [User]))
getMe :: MonadWidget t m
          => Event t ()
          -> m (Event t (ReqResult () User))
postMessage :: MonadWidget t m
            => Dynamic t (Either Text.Text Message)
            -> Event t ()
            -> m (Event t (ReqResult () [Message]))
(postLogin :<|> (getMe :<|> getUsers :<|> postMessage)) = apiClients

The clients are still generated, the only thing I’ve done is spell out the type signatures.

Reflex

Time for some reflex. The reflex changes are rather small but dense:

loginWidget :: (MonadWidget t m) => m (Event t User)
loginWidget = do
  autoLoginEvt <- autoLogin
  formEvt <- loginForm
  pure $ leftmost [formEvt, autoLoginEvt]

This function performs either an auto login, or shows a login form. leftmost is a function that combines events, by using the value of whichever fires. If they fire both, the element that occurs first in the list is used. Hence the name leftmost.

Now let’s dive into it’s sub components, first autologin:

autoLogin :: (MonadWidget t m) => m (Event t User)
autoLogin = do
  pb <- getPostBuild
  withSuccess <$> getMe pb

We get the post build event, an event that fires after the widget is placed in the DOM. We use that to greedily to make the getMe request. If successful that is used as the resulting event. However the loginWidget makes a login form regardless of success:

userInput :: (MonadWidget t m) => m (Dynamic t User)
userInput = ...

loginForm :: (MonadWidget t m) => m (Event t User)
loginForm = do
  user <- userInput
  buttonEvt <- button "login"
  postResult <- postLogin (Right <$> user) buttonEvt
  void $ flash postResult $ text . Text.pack . show . reqFailure
  pure $ current user <@ withSuccess postResult

The userInput has remained the same as in the previous blog post. After the userInput form we create a button, which gives us a resulting buttonEvt event. This event only fires if the button is pressed. We use the buttonEvt to call postLogin. As input we use the dynamic user from the userInput form. This gives us a postResultEvt, an event that only fires on request completion. Remember postLogin is a function generated from our API type signature:

postLogin :: MonadWidget t m
          => Dynamic t (Either Text.Text User)
          -> Event t ()
          -> m (Event t (ReqResult () (AuthCookies NoContent)))

The ReqResult is a container for dealing with HTTP status codes, connection errors and decoding issues ¹. In case of failure the postResultEvt is ‘flashed’, or shown briefly for a couple of seconds. In case of success we tag the postResultEvt with the user and use that as resulting event.

Note that <@, is the same as <$, except it works on behaviors: The event gets the value of whatever the behavior is at the time of the event. A behavior is something that always has a value, but which can change at any moment. Whereas an event is something that happens at a point in time with some value. A mouse position is an example of behavior, whereas a mouse click is an event².

Next we move onto the function that ties everything together:

reflex :: MonadWidget t m => m ()
reflex = do
  rec loginEvt <- elDynAttr "div" loginAttr loginWidget
      loginAttr <- holdDyn (Map.empty) $ hidden <$ loginEvt
  void $ holdEvent () loginEvt authenticatedWidget

Once the login event happens this function will hide the loginWidget and put the authenticatedWidget on the DOM. loginAttr is used before it’s assigned however. This is impossible in a normal do block, but it is possible within a rec block. The rec keyword is from recursive do. It allows referencing of variables ‘higher’ in a do block. In our case we need to have loginAttr within the elDynAttr function.

I don’t want to go into how rec works³, but I do want to make clear why it’s needed. There is a reference cycle⁴. Look closely at loginAttr. It depends on loginEvt. Now look at how the loginEvt is made. It comes from a div that requires loginAttr. A cycle. I don’t know of any other way to solve this than using recursive do.

So the login widget resides in a div with dynamic attributes. These attributes are set on the next line, which starts out as an empty Map, no attributes. Once the login event happens, it becomes the hidden, which sets the style to display:none.

holdEvent is used to extract the user as a value from the event and render authenticatedWidget as a new part of the DOM. The holdEvent functions is a convenience function for widgetHold:

widgetHold :: (DomBuilder t m, MonadHold t m)
  => m a
  -> Event t (m a)
  -> m (Dynamic t a) 

holdEvent :: (Dom.DomBuilder t m, MonadHold t m)
  => b
  -> Event t a
  -> (a -> m b)
  -> m (Dynamic t b)
holdEvent val evt fun =
  Dom.widgetHold (pure val) $ fun <$> evt

widgetHold will show the first given widget, until the event happens which has a widget as value. Then the widget within that event is put onto the DOM instead of the original. It’s a bit like sequence ⁵. In any case it returns the widget value as a dynamic.

holdEvent however assumes we initially don’t want to render anything on the DOM. Then it asks you to provide an event with any value and finally a function that consumes the value to produce the widget. It will execute the function and display the resulting widget on the DOM instead of nothing.

The first argument of holdEvent is the default value. The second argument is the event which we want to hold. The final argument is the function we want to execute producing a widget. The function keeps returning the default value until the event fires for the first time, then it will keep on displaying the fired event.

Note that widgetHold is slow ⁶ because it modifies the DOM ⁷. It’s much better to use dynText and elDynAttr to modify the dom/layout. However, widgetHold is really convenient to get access to values within events. I also think that the parts inside a widgetHold function don’t get evaluated until the event occurs. This is really convenient for login. Now you don’t have to evaluate the bulk of your app on initial page load. widgetHold can postpone evaluating large parts of your app. Which makes that initial render much faster⁸.

Anyway as we can see from the type signature, in this case the b ~ (), and the a ~ User. Which leads us to authenticatedWidget:

authenticatedWidget :: MonadWidget t m => User -> m ()
authenticatedWidget user =
  el "div" $ do
    getUsersWidget
    sendMsgWidget user

This is the same as the app discussed in the previous blog post. Although now we use the logged in user to send messages.

XSRF

To make servant reflex work nicely with servant-auth we need to modify the requests a bit, servant reflex supports this with ClientOptions:

clientOpts :: ClientOptions
clientOpts = ClientOptions $ tweakReq
  where
    tweakReq r = do
      mayCookie <- findCookie cookieName
      return $ r & headerMod headerName .~ mayCookie -- forgive lenses
    headerMod d = xhrRequest_config . xhrRequestConfig_headers . at d

apiClients :: forall t m. (MonadWidget t m) => _
apiClients = clientWithOpts
	serviceAPI (Proxy @m) (Proxy @()) (constDyn url) clientOpts

The client options lives in the JSM monad and gives us an opportunity to modify the XHRRequest how we want. We make sure the names are the same by using the ones defined in the common module.

Conclusion

So there you have it. Authentication. Not the most exciting thing in the world, but once this is done you can start making something cool. I hope I helped you get trough this ordeal fast, and explain some of the finer reflex points. Now I hope to see many cool reflex projects popping up. PM me your cool projects.

References

With the release of reflex 0.5 we now have updated docs!

What I’m doing in production is a clear cache refres on any 4xx status code and decode/encoding errors. This automatically will fix any stale clients without any mental overhead. I just attach a monad to most api calls which does that.↩︎
For completeness: The userInput form returns a dynamic user, which is both a behavior as well as an event: the event fires whenever the behavior changes value.↩︎
This explains how it works, note the thesis on value recursion too.↩︎
I currently believe that in reflex the cycle can be broken because we start with the preposition that a dynamic and behavior always have a value. Even in this case we can just set it to empty, until the event fires, which is by definition after rendering. This also explains why you use a sampled value from a behavior in a rec block.↩︎
sequence doesn’t work because event isn’t foldable, and it will never be foldable because that breaks FRP semantics.↩︎
And dyn for that matter↩︎
Even though some reddit thread disagrees with me thinking the DOM is slow and saying it’s because of layout trashing. Heretics.↩︎
I still think reflex load times are too slow, especially on mobile. I know about the app option, but an app is just a dirty hack to get your crappy website to speed up. It’s much better to have everything as a website, it results in less maintenance and better UX (if you can pull it off). Nobody should have to install anything in 2019. But the mobile web will remain slow for good reasons and if I knew how I’d happily help speeding up GHCJS and reflex. I think for example that widgetHold and dyn may be a good candidates for code splitting entry points. But I believe you’d need to make the compiler aware of that somehow. I also believe you probably don’t need the entire Haskell runtime immediately, lazily loading exceptions would be good for example.↩︎

Lens into wrapped newtypes

2018-12-30T21:34:00Z

All newtypes are isomorphisms

– My mother

Control.Lens.Wrapped uses the isomorphism property to introduce a type class Wrapped. Let’s explore use cases, because after all, it doesn’t appear to do much at first glance. What’s the point of formalizing wrapping and unwrapping of types?

Instance boilerplate will be reduced in this blog post. In my use case this will include orphan instances. Furthermore, I believe that this technique will make using newtypes more attractive.

Newtype

Consider the following common code in a fullstack haskell webapp:

data Login = Login
	{ email :: Text
	, password :: Text
	}

Looks good? No of course not. This common code is shared between both client and server, therefore we should be pendantic about these record field. We wrap common occurrences such as Text in newtypes so we don’t accidentally mix up the fields. This would be a better representation:

newtype Email = Email { unEmail :: Text }
newtype Password = Password { unPassword :: Text }

data Login = Login
	{ email :: Email 
	, password :: Password
	}

Of course this extra safety comes at the cost of more boilerplate, but these few lines add a lot of safety: Mixing of email and password becomes less likely, from frontend input to the backend. This is analogous to having integration tests on user input fields, AJAX calls, HTTP endpoints, and database insertion. That’s a lot of safety for two extra lines, therefore we accept this trade and move on.

Database

Now we can put the fields of login directly into our user table schema for the database:

-- database
data UserT f = User 
	{ email :: C f Email
	, password :: C f Password
	}
	-- etc beam boilerplate...

Although this is what we want, it doesn’t compile, we need to tell beam how to get the right underlying type so it can produce the right queries and schema:

-- backend orphanage
instance HasSqlEqualityCheck PgExpressionSyntax Email

This instance allows us to use the ==. operator on beam expressions directly. We can now compare the column email with a client email. I don’t know why an instance is needed for this, but the compiler wanted it whenever I used that operator.

-- backend orphanage
instance HasSqlValueSyntax PgValueSyntax Email where
  sqlValueSyntax = sqlValueSyntax . unEmail

Here we use unEmail to the underlying type their sqlValuesyntax.

-- backend orphanage
instance FromBackendRow Postgres Email

I believe this tells beam we want to be able to use a postgres database on email. We don’t have to instantiate sqlite instances (beam can do multiple backends).

-- backend orphanage
instance FromField Email where
  fromField a b = Email <$> fromField a b

FromField is a type class from PgSimple. Here, we’re telling the compiler to just use the from field from the underlying type, and once it’s done we can wrap it back into an email.

-- backend orphanage
instance HasDefaultSqlDataTypeConstraints PgColumnSchemaSyntax Email

This is a constraint coming from beam migrate, if you want to have automatic schema generation, or be able to step trough various schemas you need this.

-- backend orphanage
instance HasDefaultSqlDataType PgDataTypeSyntax Email where
  defaultSqlDataType proxy = defaultSqlDataType $ unEmail <$> proxy

This is also needed for migrations. Here we’re removing the newtype from the proxy to tell it to use the underlying type. Note that a proxy isn’t holding any data, those functions will never be executed, in case of proxies we’re just interested in type.

All of this is repeated for Password to, and any other newtypes you want:

-- backend orphanage
instance HasSqlEqualityCheck PgExpressionSyntax Password
instance HasSqlValueSyntax PgValueSyntax Password where
  sqlValueSyntax = sqlValueSyntax . unPassword
instance FromBackendRow Postgres Password
instance FromField Password where
  fromField a b = Password <$> fromField a b
instance HasDefaultSqlDataTypeConstraints PgColumnSchemaSyntax Password
instance HasDefaultSqlDataType PgDataTypeSyntax Password where
  defaultSqlDataType proxy = defaultSqlDataType $ unPassword <$> proxy

I put these instances into an orphanage (dedicated file for orphan instances) because I want to put the newtypes directly into the database. However I don’t want our common code to be dependent on beam, that would mean the frontend JavaScript suddenly would pull in beam as a dependency for no reason. We’ll eliminate the need for these orphans later.

What did we gain? The ability to put these newtypes in the database, what did we lose? Well we now have a lot of extra boilerplate to content with.

Wrapped

Let’s kill the boilerplate!

-- common
newtype Email = Email { unEmail :: Text } deriving Generic
newtype Password = Password { unPassword :: Text } deriving Generic

instance Wrapped Email
instance Wrapped Password

If you can provide an Iso’, then you instantiate the Wrapped type class. If a newtype has derived generic we get an instance for free by just declaring it, and using the default. This is possible because generic knows about constructors.

-- backend orphanage
wrappedSqlValueSyntax  :: (Wrapped a, HasSqlValueSyntax b (Unwrapped a)) => a -> b
wrappedSqlValueSyntax  = sqlValueSyntax . view _Wrapped'

fromWrappedField :: (Wrapped a, FromField (Unwrapped a)) => FieldParser a
fromWrappedField a b = review _Wrapped' <$> fromField a b

wrappedDefaultSqlDataType :: (Wrapped a, HasDefaultSqlDataType b (Unwrapped a)) => Proxy a -> Bool -> b
wrappedDefaultSqlDataType proxy = defaultSqlDataType $ view _Wrapped' <$> proxy

These functions pull out the essence of wrapping. If a is wrapped, we can speak about it’s unwrapped form (which is why we need the type class). If a his unwrapped form for example implements FromField, we can make a FieldParser for it.

-- backend orphanage
instance HasSqlEqualityCheck PgExpressionSyntax Email
instance HasSqlValueSyntax PgValueSyntax Email where
  sqlValueSyntax = wrappedSqlValueSyntax
instance FromBackendRow Postgres Email
instance FromField Email where
  fromField = fromWrappedField
instance HasDefaultSqlDataTypeConstraints PgColumnSchemaSyntax Email
instance HasDefaultSqlDataType PgDataTypeSyntax Email where
  defaultSqlDataType = wrappedDefaultSqlDataType

instance HasSqlEqualityCheck PgExpressionSyntax Password
instance HasSqlValueSyntax PgValueSyntax Password where
  sqlValueSyntax = wrappedSqlValueSyntax
instance FromBackendRow Postgres Password
instance FromField Password where
  fromField = fromWrappedField
instance HasDefaultSqlDataTypeConstraints PgColumnSchemaSyntax Password
instance HasDefaultSqlDataType PgDataTypeSyntax Password where
  defaultSqlDataType = wrappedDefaultSqlDataType

The instances themselves do the same thing over and over, they wrap or unwrap types to get the underlying interesting value. Here this is obvious by having the instances of both Email and Password point to the same functions.

At first glance, this implementation does not look better. However we now can clearly see that the wrapping is indeed the same operation because the instances all point toward the same functions. The FromField instance for Password is implemented with fromWrappedField, so does the Email instance. This is possible because both Email and Password have instantiated the Wrapped instance.

This change is a lot better if you consider that there is no more logic being repeated here. Which means there are no more logic bugs in the repetition. The boilerplate is now in it’s purest form: Dumb repetition. By itself I wouldn’t consider the current state to be that bad anymore. However, these are still orphans, which can cause bad problems. We should kill all orphans!

A general instance

Can’t we make a generalized instance that does all of this wrapping for all newtypes? My first attempt was rather crazy looking back. I wanted to create the ultimate orphan. a polymorphic instance that was kept in check by constraints such as Wrapped and the fact underlying types would have these beam instances implemented. I attempted this but did got very far, I wasn’t very sure what was going on anymore with the type errors I got out of that. But after a bit of searching I realized that what I attempted to do was ridiculous and dangerous. Of course that wouldn’t work, now all previous and future Wrapped newtypes would have to be able to be fit into postgres or fail. This piece of code would break all existing libraries that would’ve had a Wrapped newtype. No this was an absurd idea.

Rather than solving the problem for all newtypes, I stepped back, and added yet another newtype:

-- wrapped.hs
newtype DBFieldWrap a = DBFieldWrap
  { _unField :: a
  } deriving (Generic, Show)

instance Wrapped a => Wrapped (DBFieldWrap a)
instance (Wrapped a, BeamBackend be,
          BackendFromField be (DBFieldWrap a),
          FromBackendRow be (Unwrapped a)
         ) =>
         FromBackendRow be (DBFieldWrap a)

That final instance shows the core idea. We add the wrapped restriction on a, which allows us to speak about the unwrapped form of a. The unwrapped type of Email would be Text. Beam has already made a FromField instance for Text, so we’re done.

instance ( IsSql92ExpressionSyntax be
         , Wrapped a
         , HasSqlEqualityCheck be (Unwrapped a)
         ) =>
         HasSqlEqualityCheck be (DBFieldWrap a)

instance (Wrapped a, FromField (Unwrapped a)) => FromField (DBFieldWrap a) where
  fromField a b = review (_Wrapped' . _Wrapped') <$> fromField a b

The review function just calls the constructor. We have to call two _Wrapped's with it because we need to put it in the Email or Password, and then we need to put it into the DBField.

instance (Wrapped a, HasSqlValueSyntax be (Unwrapped a)) =>
         HasSqlValueSyntax be (DBFieldWrap a) where
  sqlValueSyntax = sqlValueSyntax . view (_Wrapped' . _Wrapped')

The view function is just an alias for ^., a getter. We get the result of wrapping twice.

instance ( IsSql92ColumnSchemaSyntax be
         , Wrapped a
         , HasDefaultSqlDataTypeConstraints be (Unwrapped a)
         ) =>
         HasDefaultSqlDataTypeConstraints be (DBFieldWrap a)

instance ( IsSql92DataTypeSyntax be
         , Wrapped a
         , HasDefaultSqlDataType be (Unwrapped a)
         ) =>
         HasDefaultSqlDataType be (DBFieldWrap a) where
  defaultSqlDataType proxy =
    defaultSqlDataType $ view (_Wrapped' . _Wrapped') <$> proxy

Apparently you can fmap into a proxy to get the right type out. Even though a proxy has no data, it will change type.

This does exactly the same thing as the independent functions did in case of the orphanage, the only difference is that they wrap twice. We essentially tell the type checker to look for the beam instance two levels deeper. We tell it by using restrictions on the instances (the stuff before =>).

Now we can insert our newtypes directly into database without having to implement all those Beam instances:

-- database
data UserT f = User 
	{ email :: C f (DBFieldWrap Email)
	, password :: C f (DBFieldWrap Password)
	}
	-- etc beam boilerplate...

Adding an additional newtype for the database is now easy:

-- common
newtype DateOfBirth = DateOfBirth { unEmail :: Day } deriving Generic
instance Wrapped DateOfBirth 

data UserT f = User 
	{ email :: C f (DBFieldWrap Email)
	, password :: C f (DBFieldWrap Password)
	, dob :: C f (DBFieldWrap DateOfBirth)
	}

Note that this technique doesn’t just work for beam instances, one could do the same for Aeson, or any other library that requires many instances on newtypes. The Wrapped instance can be re-used.

Conclusion

So what did we gain?

We no longer have orphans!
Boilerplate has been reduced to just the wrapped instance.

What did we lose?

Unfortunately we need to unwrap and wrap at the call sites (beam queries).
To use this we need to depend on lens.
The beam schema is a little bit more verbose.

Because all those negative points are really small, I’m calling this a win. The sources can be found on github and hackage.

Fullstack Haskell: Reflex and Servant

2018-10-09T12:08:00Z

In the pragmatic haskell series, we saw how to setup a simple webserver with database. But at some point you still need a frontend. If it were 2005 you may have been able to get away with just blaze. But we are in 2018+, and JavaScript is a problem. In this blog post we will explore how to deal with JavaScript trough reflex and GHCJS. An alternative to consider is miso, which uses the elm architecture (or redux if you’re from JS), here is a comparison. Obviously I chose reflex.

Preparation

First we need to setup the dev environment. This time we’ll double down on nix because reflex does that too and fighting build tools is no fun. This has the advantage that the resulting code on github is reproducible. All need to be done is setup the file watch for which I wrote a make command:

make file-watch

This rebuilds both the Haskell back end and JavaScript front end incrementally.

There are two separate environments now, one is for the native Haskell target (x86), and the other is the JavaScript target. We can enter the shell environment for the native target with make enter and the JavaScript target with make enter-js. This is convenient for doing one of commands.

The biggest issue I had when setting this up was figuring out how to add extra dependencies not in the nix repo. I found out by reading the nix code that this can be done with the overrides flag. Another issue was tools for shells, such as hpack which generates cabal files from the package.yaml file. I really wanted to use that as I didn’t want to learn cabal, besides, hpack’s is much more succinct, it doesn’t require explicit module exports. there is a shellOverrides attribute for that.

    overrides = self: super: rec {
      beam-core = self.callPackage ./packages/beam-core.nix { };
	  ...
    };
	...
    shellToolOverrides = ghc: super: {
        inherit (ghc) hpack;
        fswatcher = pkgs.inotify-tools;
		...
    };

Back end

The backend is mostly the same as the result of the pragmatic haskell series. We moved the API endpoints that need to be accessed by client to the e common code, and added an additional endpoint for hosting the html. Normally we wouldn’t use Haskell to deliver static assets and use a specialized program such as nginx. Since this is for experimentation however we made an exception:

type Webservice = ServiceAPI 
      :<|> Raw -- JS entry point

webservice :: Proxy Webservice
webservice = Proxy

... 

server :: Connection -> Server Webservice
server conn=
  (pure users :<|> messages conn) :<|> serveDirectoryFileServer "dist-ghcjs/build/x86_64-linux/ghcjs-0.2.1/frontend-0.1.0.0/c/webservice/build/webservice/webservice.jsexe/"

The webservice type definition has the aditional Raw endpoint, which allows hosting of custom wai apps. The serveDirectoryFileServer is that custom wai app and just hosts the JavaScript output of the client.

Common code

This is where the shared code between client and server lives. We put the API definition in here. Since servant can create both servers and clients it’s a great library for this use case.

Any change in API will now cause the type checker to tell us where this is affected in both client and server. Type safety becomes amplified, making bugs more obvious and increasing developer productivity.

The actual content of this module isn’t that interesting, it’s just the API definition. Common code gets compiled within the JavaScript client. This means it’s public, one should not put any passwords or trade secrets in here.

Front end

I started with trying to get reflex to work with servant because it seemed the most uncertain. After this I intended to use servant-client for generating the client functions, here I ran into another hurdle as the latest servant wasn’t available. Apparently reflex is pinned to an old hackage repository, I attempted to upgrade but abandoned that endeavour as it required more nix modifications and I’d prefer to keep the same pin as upstream so I could get help when I needed it. Using the older servant, I hit a run time exception:

uncaught exception in Haskell main thread: ReferenceError: h$hsnet_getaddrinfo is not defined

This is because servant client uses a system call for networking which is unavailable in the browser sandbox. A bit of googling led me to servant reflex. Using this was hard because are no official haddocs since it hasn’t been released yet. Finding an example was of great help which led me to this client definition:

apiClients :: forall t m. (MonadWidget t m) => _
apiClients = client serviceAPI (Proxy @m) (Proxy @()) (constDyn url)
  where url :: BaseUrl
        url = BasePath "/"

This creates both functions for querying the serviceAPI from the common module. All this seems to do is getting that m in scope and applying it to the client with a proxy. The partial type signature was left from the example this way intentionally, because it’s formalized below anyway and it’s rather big.

getUsers :: MonadWidget t m
          => Event t ()
          -> m (Event t (ReqResult () [User]))
postMessage :: MonadWidget t m
            => Dynamic t (Either Text.Text Message)
            -> Event t ()
            -> m (Event t (ReqResult () [Message]))
(getUsers :<|> postMessage) = apiClients

This pulls out the functions from apiClients, we also get there final signature here. The entire file can be seen in the sources.

Reflex

After getting the API to function I started working on making an actual UI. Which is what this code does for the getUsers function:

reflex :: IO()
reflex =
  mainWidget $
    el "div" $ do
        -- babys steps, get users from memory
        intButton  <- button "Get Users"
        serverInts <- fmapMaybe reqSuccess <$> getUsers intButton
        display =<< holdDyn ([User "none" "none"]) serverInts

mainWidget is the root of reflex, we use el to specify HTML elements that surround other elements. the button functions creates a button (no surprise). This is within the monad widget, interaction between components is handled trough that monad.

On the next line we use the intButton immediately. If we look at the getUsers type signature we see that it requires an Event t () argument, this is satisfied by the button. In other words getUsers will triggered on the button event. The result is once more put in the monadwidget. Finally we map the result to assume success or Nothing, it will be just a list of users now. The holdDyn function is then used to give a default value to the resulting event in case of nothing, we always display either the default or the request result.

Markup with reflex

For the postMessage function I made a form with text inputs:

		-- Post a usermessage and display results
		input <- messageInput 
		sendMsg <- button "Send Message"
		messages <- fmapMaybe reqSuccess <$> postMessage (Right <$> input) sendMsg 
		resulting <- holdDyn
			([Message (User "none" "none") "ddd"]) -- what to show if nothing
			messages -- source of messages (if any)
        _ <- el "div" $
            simpleList resulting fancyMsg

messageInput is a function that returns a “dynamic message” (see below), the button is for sending of messages. To display we use a similar pattern however this time we’ll mark it up in HTML with fancy messages. We traverse over the dynamic list with the simpleList function, here I expected traverse to work.

  where
    fancyMsg :: (MonadWidget t m) => Dynamic t Message -> m (Element EventResult GhcjsDomSpace t)
    fancyMsg msg = elClass "div" "message" $ do
        _ <- elDynHtml' "h1" $ Text.pack . name . from <$> msg
        elDynHtml' "span" $ Text.pack . content <$> msg

Every message is put in a div element, for displaying dynamic content however we need to use elDynhtml' function, there is no way of getting a value out of dynamic, we can only show it to the user. This is a strong safety guarantee.

Reflex “react component”

Input fields can be combined together into larger components, which is showcased in the Message form:

messageInput :: (MonadWidget t m) => m (Dynamic t Message)
messageInput = do
    user <- userInput
    message <- labeledInput "message"
    pure $ (Message <$> user) <*> (Text.unpack <$> _textInput_value message)

userInput :: (MonadWidget t m) => m (Dynamic t User)
userInput = do
        username <- labeledInput "username"
        emailInput <- labeledInput "email"
        pure $ User . Text.unpack <$> _textInput_value username <*> (Text.unpack <$> _textInput_value emailInput)

labeledInput :: (MonadWidget t m) => Text.Text -> m (TextInput t)
labeledInput label = elClass "div" "field" $ do
    elClass "label" "label" $ text label
    elClass "div" "control" $ textInput (def & textInputConfig_attributes .~ constDyn (Text.pack "class" =: Text.pack "input"))

This is done with applicative fmap <$> and spaceship <*>. That last line sets some extra confiugrations can be set withLenses for textInput, which is another rabithole. They can simply be thought of as getters and setters for haskell, although more powerfull.

Note that these functions are analogue to a react component, and see the difference! They compose perfectly and will function independently.

Feel the power.

My only complaint is that the resulting JavaScript binary is huge, 8MB, 2MB after using the closure compiler.

Conclusion

I’m very pleased with reflex, now I don’t have to deal with JavaScript, I can prototype my API rapidly and I’m not restricted to an architecture. It is better than I expected, the core seems really well designed. The only downside is the large binary. None the less I’m willing to use this for a larger project.

Links

For convenience here is a list of used resources:

Sources

The project has become too big to share all files, as always there is the github link. I will however put all discussed code in complete form in here.

backend/src/Lib.hs

{-# LANGUAGE DataKinds #-}
{-# LANGUAGE TypeOperators #-}

{-# OPTIONS_GHC -Wno-missing-monadfail-instances #-}


module Lib
    ( webAppEntry
    ) where

import Servant
import Common
import Control.Monad.IO.Class(liftIO)
import Network.Wai(Application)
import Network.Wai.Handler.Warp(run)
import           Database.PostgreSQL.Simple   (Connection)
import qualified DB as DB
import           Database.Beam.Backend.SQL.BeamExtensions (runInsertReturningList)

import qualified Database.Beam                            as Beam
import qualified Database.Beam.Postgres                            as PgBeam
import Data.Text(pack, unpack)

type Webservice = ServiceAPI 
      :<|> Raw -- JS entry point

webservice :: Proxy Webservice
webservice = Proxy

users :: [User]
users =
  [ User "Isaac Newton"    "isaac@newton.co.uk"
  , User "Albert Einstein" "ae@mc2.org"
  ]

messages :: Connection -> Message -> Handler [Message]
messages conn message = do 
  fromDb <- liftIO $ 
    PgBeam.runBeamPostgres conn $ do
      let user = from message
      [foundUser] <- runInsertReturningList (DB._ausers DB.awesomeDB) $ 
          Beam.insertExpressions [DB.User 
            Beam.default_
            (Beam.val_ (pack $ name $ user ))
            (Beam.val_ (pack $ email $ user ))
        ]
      _ <- runInsertReturningList (DB._messages DB.awesomeDB) $ 
          Beam.insertExpressions 
            [DB.Message 
              Beam.default_ 
              (Beam.val_ (Beam.pk foundUser))
              (Beam.val_ (pack $ content message))
            ]
      Beam.runSelectReturningList $ Beam.select $ do 
        usr <- (Beam.all_ (DB._ausers DB.awesomeDB))
        msg <- Beam.oneToMany_ (DB._messages DB.awesomeDB) DB._from usr
        pure (msg, usr)
  pure $
    fmap (
      \(msg, usr) -> Message
        (User
          (unpack $ DB._name usr)
          (unpack $ DB._email usr))
        (unpack $ DB._content msg)
    ) fromDb


server :: Connection -> Server Webservice
server conn=
  (pure users :<|> messages conn) :<|> serveDirectoryFileServer "dist-ghcjs/build/x86_64-linux/ghcjs-0.2.1/frontend-0.1.0.0/c/webservice/build/webservice/webservice.jsexe/"

app :: Connection -> Application
app conn = serve webservice (server conn)

webAppEntry :: Connection -> IO ()
webAppEntry conn = run 6868 (app conn)

common/src/Common.hs

{-# LANGUAGE DataKinds #-}
{-# LANGUAGE TypeOperators #-}
{-# LANGUAGE DeriveGeneric #-}

module Common where

import GHC.Generics(Generic)
import Servant.API
import Data.Proxy
import Data.Aeson(ToJSON, FromJSON)

type ServiceAPI = "api" :> "1.0" :> "users" :> Get '[JSON] [User]
      :<|> "api" :> "1.0" :> "message" :> ReqBody '[JSON] Message :> Post '[JSON] [Message]

data Message = Message {
  from :: User,
  content :: String
} deriving (Eq, Show, Generic)

instance ToJSON Message
instance FromJSON Message

data User = User
  { name :: String
  , email :: String
  } deriving (Eq, Show, Generic)

instance ToJSON User
instance FromJSON User

serviceAPI :: Proxy ServiceAPI
serviceAPI = Proxy

frontend/src/Lib.hs

{-# LANGUAGE OverloadedStrings #-}
{-# OPTIONS_GHC -fprint-explicit-kinds -Wpartial-type-signatures #-}


module Lib
  ( reflex
  ) where
import Reflex
import Reflex.Dom
import qualified Data.Text as Text
import Control.Applicative ((<*>), (<$>))
import Common
import Servant.Reflex
import ServantClient

reflex :: IO()
reflex =
  mainWidget $
    el "div" $ do
        -- babys steps, get users from memory
        intButton  <- button "Get Users"
        serverInts <- fmapMaybe reqSuccess <$> getUsers intButton
        display =<< holdDyn ([User "none" "none"]) serverInts

        -- Post a usermessage and display results
        input <- messageInput 
        sendMsg <- button "Send Message"
        messages <- fmapMaybe reqSuccess <$> postMessage (Right <$> input) sendMsg 
        resulting <- holdDyn
            ([Message (User "none" "none") "ddd"]) -- what to show if nothing
            messages -- source of messages (if any)

        _ <- el "div" $
            simpleList resulting fancyMsg 
        pure ()
  where
    fancyMsg :: (MonadWidget t m) => Dynamic t Message -> m (Element EventResult GhcjsDomSpace t)
    fancyMsg msg = elClass "div" "message" $ do
        _ <- elDynHtml' "h1" $ Text.pack . name . from <$> msg
        elDynHtml' "span" $ Text.pack . content <$> msg

messageInput :: (MonadWidget t m) => m (Dynamic t Message)
messageInput = do
    user <- userInput
    message <- labeledInput "message"
    pure $ (Message <$> user) <*> (Text.unpack <$> _textInput_value message)

userInput :: (MonadWidget t m) => m (Dynamic t User)
userInput = do
        username <- labeledInput "username"
        emailInput <- labeledInput "email"
        pure $ User . Text.unpack <$> _textInput_value username <*> (Text.unpack <$> _textInput_value emailInput)

labeledInput :: (MonadWidget t m) => Text.Text -> m (TextInput t)
labeledInput label = elClass "div" "field" $ do
    elClass "label" "label" $ text label
    elClass "div" "control" $ textInput (def & textInputConfig_attributes .~ constDyn (Text.pack "class" =: Text.pack "input"))

frontend/src/ServantClient.hs

{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE RankNTypes #-}
{-# LANGUAGE TypeApplications #-}
{-# OPTIONS_GHC -Wno-partial-type-signatures #-}

{-# LANGUAGE NoMonomorphismRestriction          #-}
{-# LANGUAGE PartialTypeSignatures #-}
{-# LANGUAGE ScopedTypeVariables #-}
{-# LANGUAGE  TypeApplications #-}
{-# LANGUAGE TypeOperators #-}

-- | This modules purpose is just to generate the xhr clients.
--   there is some type magick going on generating these,
--   therefore the functions are isolated.
module ServantClient
  ( postMessage, getUsers 
  ) where

import Reflex
import Reflex.Dom
import qualified Data.Text as Text
import Servant.API
import Common
import Servant.Reflex
import Data.Proxy

-- | This intermediate definition is necisarry because the @m is similar for both clients,
--   they have the same wrapping monad however the containing type is different
--   (which is why we have the nomonomorphism restirction disabled)
apiClients :: forall t m. (MonadWidget t m) => _
apiClients = client serviceAPI (Proxy @m) (Proxy @()) (constDyn url)
  where url :: BaseUrl
        url = BasePath "/"

getUsers :: MonadWidget t m
          => Event t ()  -- ^ Trigger the XHR Request
          -> m (Event t (ReqResult () [User])) -- ^ Consume the answer
postMessage :: MonadWidget t m
            => Dynamic t (Either Text.Text Message)
            -> Event t ()
            -> m (Event t (ReqResult () [Message]))
(getUsers :<|> postMessage) = apiClients

Back to the Netherlands!

2018-09-23T18:21:00Z

After a year in foreign countries, I’ve returned home. Last week my working holiday visa expired, and unlike a friend of mine, I did not want to be deported in handcuffs.

I quite enjoyed my stay in Australia, the people are relax, life is good. I’ve drank lots of coffee and ate many a Kangaroo, but it’s good to be back and speak my own language. The people just “act normal”,

First I was a month in Indonesia, essentially looking for jobs in Australia (as you do), and hanging out with my then girlfriend. Then in Sydney I quickly found a job and place to live. Three interviews were pre-arranged, they all got converted into offers. The first one was stackchat, they gave me a pretty bad offer, I needed money most yet they came with equity. Then there was ayuda, I probably could’ve asked them lots more money as they were hardly strapped for cash. Openlearning offered most though, more then I asked for, which was nice. I took the highest.

Around new year the relationship with Jesiska was in crisis, and I went back to Indonesia, where it failed. Happy new year.

then, around February the company wanted to initiate sponsorship. However I had become pretty unhappy with the working arrangement. It looked like I was just being kept busy, while having no impact on anything. That working relationship also ended there.

At that point I seriously considered going back, This was my lowest point in Australia. I had nothing there except a little hobby project started with some friends. ex-Colleagues at Open Learning, I was crestfallen. Although they were being nice about my essentially destroyed life, in Netherlands we would rip on each other at this point, You could at least joke about it you know. I’m too facetious I hear.

Here I decided that rather than giving up so easily I’d fight for staying in Australia as a software engineer. I just had arranged better housing, now I bet I could also arrange better work! Easier said then done, I emailed so many companies, I even made a little movie to sell myself. Few replies but some did.

First there was Silicon Valley Chicken, they wanted to sell chicken trough an app with a game. Completely aimed at Gamers, and only order trough the app. I wasn’t too excited about that, they weren’t excited about me being on working holiday visa. After that I met Daisee at the functional programming meetup. I managed to get an interview. Turned out it was a Haskell job. I was super excited about that. Then Trunk wanted to deliver version control to designers. But I kind off neglected them in favor of the Haskell opportunity.

This was role reversal, now the friends had a pretty crappy job compared to doing Haskell. Especially considering the company had taken up Cosmos db which required running windows. They were both open source advocates, requiring to run this made them.. Unhappy.

But I was happy. For a couple of months. Daisee kept hiring people and we grew. The more time passed the more responsibility I had to give away, until I was focused upon the one project. Which then got canned. Not that it bothered me as I was planning for the next stage in my life.

Going to the Philippines! Except, everyone I spoke to was pretty negative about it, and the offer from Daisee wasn’t as nice as I expected. Back home I am, not that it’s bad, just sudden. I’ll be staying here for some time, doing remote work for Daisee as a company! Meanwhile Daisee is working on a work visa for Australia. This seems much better than the Philippines, which felt more like a golden prison.

This is all seems pretty random, oh well, such is life. I had fun at least.

NixOS on encrypted btrfs

2018-08-19T13:02:00Z

Nixos is heroin for tinkerers. Paradise can be tinkered together and be freely shared among peers because it’s fully reproducible! Jappie wanted more, he wanted a secure disk and a BTRFS. There used to be no guides for this, now there is.

The bullet was bitten, BTRFS was made to work on a LUKS encrypted disk. This isn’t hard, with care and precision. To help a reader we document the journey towards BTRFS. Commands compiled and included.

Getting started

Get yourself a NixOS live usb. I use the minimal ISO, because the graphical ISO slows booting and gives no advantage aside from being pretty. Use

cat minimal-nixos.iso > /dev/sdX

where X is the usb drive found by lsblk. X should be a letter, numbers indicate partitions, which we don’t want to cat upon. Boot into it on the target machine.

Networking

Next step is to setup WIFI, you can skip this if you’re on Ethernet:

wpa_passphrase SSID PASS > /etc/wpa_supplicant.conf
systemctl restart wpa_supplicant

Ask google if you’re online: curl google.com should return a 301 redirect:

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

There is no point proceeding until you have networking.

Partitioning

There are no other partitioning tools than gdisk. Only heretics believe there are. Therefore we use gdisk:

gdisk $dev

Gdisk cheat sheet

Command	Effect
`p`	For printing, to see what’s going on.
`d`	For deletion, you should start out with deleting everything on `$dev`.
`n`	Is used for creating new partitions.
`w`	is used for writing once finished.

This table just describes the commands needed for the intended partitioning.

Intended partitioning

Number	type	size
1	ef00	+500M
2	8200	+$(SIZE_RAM+ a little)G
3	8300	(rest of disk)

Encryption

cryptsetup luksFormat "$dev"3
cryptsetup open "$dev"3 nixenc

Formatting filesystems

Partitioning is a distinct step from setting up filesystems.

mkfs.vfat -n boot "$dev"1
mkswap "$dev"2
swapon "$dev"2
mkfs.btrfs -L root /dev/mapper/nixenc

The boot partition will be vfat because UEFI tells us to. The everything else partition will be btrfs, because why are you following this guide if not? Note that we point it at the mapped file, if the "$dev"3device were to be used directly we’d remove the encryption.

Moutning and subvolumes

Wouldn’t it be nice to have subvolumes on your BTRFS? This is not cargo culted at all.

mount -t btrfs /dev/mapper/nixenc /mnt/
btrfs subvol create /mnt/nixos
umount /mnt
mount -t btrfs -o subvol=nixos /dev/mapper/nixenc /mnt

First we create a nixos subvolume below the root subvolume, eg the nixos operating system will not be installed in the root, but one node below the root, allowing potentially more operating systems to be installed on the same partition. Reverse explaining cargo culting behavior, this maybe a good idea.

btrfs subvol create /mnt/home

Setting up a subvolume for home allows btrfs based backups. I used too also make subvolumes for tmp¹ and var, but I don’t see the merit in that.

mkdir /mnt/boot
mount "$dev"1 /mnt/boot

Here we mount the boot partition. Just to make it detectable by the nix config generation script.

Did I do everything right?

The second time I ran trough this post everything went quite quickly, so I became skeptical. To verify everything was sane I used the following commands:

mount | grep /mnt
ls /mnt

The first command is to check if the encrypted volume and boot is mounted at the right paths. The second one to verify the folders are created, which are subvolumes. The subvolume command creates a folder so if it exists we presume it worked. But if you’re really unsure you can use btrfs subvol list /mnt/.

Configure nix

We can use hardware detection to figure out how to setup nix on this setup:

nixos-generate-config --root /mnt

Done.

Now the user needs to write his own nix config, or copy mine or cherry pick whatever they need (recommend). On a wireless laptop, it’s highly recommended to enable wpa_supplicant:

networking.wireless.enable = true

Once configuration is done we can install nix:

nixos-install

Don’t worry, we can use nixos-rebuild switch to reconfigure nix whenever once we’re booted into it. Hopefully we boot successfully:

reboot

Booting is hard, don’t worry if this goes wrong the first 10 30 times.

You may need to enable UEFI in your BIOS. It’s up to the reader to figure that part out. (press some f keys on boot, f11 maybe?). Alternatively one could setup grub. Good luck with that.

You can’t read the rest of this post until you’ve booted, go back if you haven’t, you messed up.

Final steps

Once rebooted you may be stuck at the display manager. Use Alt+f1 to switch to another TTY and login as root, then use passwd your-user-name to set an initial password. Use Alt+f7 to go back to the display manager.

I personally haven’t moved all my configuration into nix yet (it’s a big project), but I wrote a script that symlinks all dotfiles, and hardlinks the configuration.nix to my linux-config project.

If you want a subvolume for /tmp, make sure to chmod it to 777 Otherwise various applications get upset. Pulse audio for example doesn’t work well if it can’t write into /tmp.↩︎

Pragmatic Haskell III: Beam Postgres DB

2018-08-05T17:30:00Z

Note, I no longer recommend using beam for business as ORM. Please use persistent instead. Beam is far to complicated for it’s use case. Migrations can be run with postgresql-migration for example, using the suggested migrations from persistent. I’d only recommend using beam for hobby projects or as a case study for finally tagless.

No need to read a book to use Haskell! This post will get you going with a serious web application while only sticking to the concepts that are encountered. This is a Haskell safari with as end goal a working webapp with database.

Web applications need to store data. In the previous blog post we did this in a file for simplicity. Now we will use something more appropriate: A relational database. The beam library is used for this because it is closest to the “ORM” way of thinking: Model a schema, generate SQL to query that schema, and have migrations to move between different versions of that schema. Migrations are left for another post for simplicity.

For the inpatient: Resulting source

Preparation

Unfortunately this post requires us to do quite a bit of devops to get started. We need to:

Install Postgres
Create a user
Create db
Populate structure

Installing Postgres is out of the scope of this post. We sidestep using migrations for with the data_model.sql file (see sources). Use the following commands to prepare the database:

sudo -u postgres createuser -s $USER
dropdb awesome_db
createdb awesome_db
psql -f ./data_model.sql -d awesome_db

Congratulations, devops was survived. Note that using this sql file is not idiomatic to beam. The schema should be managed by beam, but getting migrations to function is currently hard (it’s a work in progress).

Creating structure

The beam library models our desired structure at type level. This is done in a separate file called DB.hs. It can be seen in the sources. With help of this code beam can inspect the definitions, and it also provides type safety for the beam sql domain specific language. In other words if the migrations work there would be a path towards bringing the database up to date with the code base, or a compile error. This is very pleasant because we get a thight feedback loop. Now let us carefully inspect that file to understand it.

Language extensions

{-# LANGUAGE DeriveGeneric         #-}
{-# LANGUAGE FlexibleInstances     #-}
{-# LANGUAGE MultiParamTypeClasses #-}
{-# LANGUAGE OverloadedStrings     #-}
{-# LANGUAGE StandaloneDeriving    #-}
{-# LANGUAGE TypeFamilies          #-}
{-# LANGUAGE DuplicateRecordFields #-}

This looks daunting however these extensions are individually quite simple. Generally they either make nicer API’s or an easier language to use. We will go over each of them.

StandaloneDeriving

StandaloneDeriving allows us to use the derive mechanism outside of a data declaration, for example:

deriving instance Show Message

This mechanism allows deriving (automatic code generation) to be used more flexibly. In this case we want to do this because the MessageT f type constructor to derive (as f is unknown), but MessageT Identity is known so we can derive that. The GHC manual lists more possible reasons to derive like this instead of the standard method.

TypeFamilies

Type families allow us to declare data inside an instance. The Table class requires TypeFamilies to instantiate because it needs a data called primary key in it’s instance:

instance Table UserT where
    data PrimaryKey UserT f = UserId (Columnar f Int) deriving Generic
    primaryKey = UserId . _id

This example is a prime reason to use type families: It allows beam to assume the primary key exist for all tables.

The Haskell wiki goes more in depth on type families

FlexibleInstances

If we don’t enable FlexibleInstances we get the following error:

/home/jappie/projects/haskell/awesome-project-name/src/DB.hs:36:10: error:
    • Illegal instance declaration for ‘Beamable (PrimaryKey MessageT)’
        (All instance types must be of the form (T a1 ... an)
         where a1 ... an are *distinct type variables*,
         and each type variable appears at most once in the instance head.
         Use FlexibleInstances if you want to disable this.)
    • In the instance declaration for ‘Beamable (PrimaryKey MessageT)’
   |
36 | instance Beamable (PrimaryKey MessageT)
   |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Without Flexible instances parenthesis aren’t allowed. We know the parenthesis are the problem because the following line does not get an error:

instance Beamable MessageT

MultiParamTypeClasses

If MultiParamTypeClasses is disabled an error appears:

/home/jappie/projects/haskell/awesome-project-name/src/DB.hs:65:10: error:
    • Illegal instance declaration for ‘Database be AwesomeDb’
        (Only one type can be given in an instance head.
         Use MultiParamTypeClasses if you want to allow more, or zero.)
    • In the instance declaration for ‘Database be AwesomeDb’
   |
65 | instance Database be AwesomeDb
   |          ^^^^^^^^^^^^^^^^^^^^^

Because we use two parameters for this instance (be and AwesomeDb). By default Haskell only allows one.

DeriveGeneric

DeriveGeneric was discussed in a previous blog post. In short: Generic allows for introspection of data structures using the fact any data structure can be modeled in a regular (generic) pattern.

OverloadedStrings

OverloadedStrings is probably the most common language extension. It converts string automatically, for example String -> ByteString. In our case it’s only used for connection string:

connectionString :: BS.ByteString
connectionString = "dbname=awesome_db"

In this case it inserts automatically a function String -> ByteString. Using this extension avoids tedious conversions.

DuplicateRecordFields

DuplicateRecordFields allows creation of records with the same name. For example both user and messages have an _id record. Because they have the same name, type annotations are used to determine which function is called, for example in:

instance Table UserT where
    data PrimaryKey UserT f = UserId (Columnar f Int) deriving Generic
    primaryKey = UserId . (_id :: UserT f -> C f Int)

Imports

import qualified Data.ByteString                  as BS

ByteString is required for the type signature of connection string.

import qualified Data.Text                      as Text

Text is required for the column datatype.

import           Database.Beam

We import all of beam, for convenience. This entire module is a client of beam, there is no need for explicit imports.

User table

We start with defining the structure of our user table.

data UserT f = User
                { _id     :: C f Int
                , _name   :: C f Text.Text
                , _email  :: C f Text.Text
                }
                  deriving Generic

What are these C and f’s doing here? The C is an abbreviation for Columnar, which is a type that requires two other types to complete. In this case C is given an f, and a second argument with the actual type of the column. f is not defined, instead it’s also an argument of UserT, therefore UserT is of kind * -> *. What we know however is that this f is the same for all columns in UserT.

Now this f can be thought of as a ‘gap’ that can be filled up with anything. This gap allows the beam library to inspect the structure we have defined, and therefore create a schema out of it and hold the data of a row. The hackage page goes deeper into the definition.

type User = UserT Identity

This defines a type alias. A userT with Identity is simply a user. In this case we are filling the f with Identity, a container that exposes on function runIdentity which removes the container and does nothing with the content. This is usefull for the case where we want to have the user as a result from the database.

deriving instance Show (PrimaryKey UserT Identity)
deriving instance Show User

Implement the show function automatically for the identity case and for the primary key which will be introduced below.

instance Table UserT where
    data PrimaryKey UserT f = UserId (Columnar f Int) deriving Generic
    primaryKey = UserId . (_id :: UserT f -> C f Int)
type UserId = PrimaryKey UserT Identity -- For convenience

Here we make UserT an instance of a Table. To do this we must specify a PrimaryKey, of which we define the type in the data line. The next line tells which function must be used to access the primary key. In other words we implement the primaryKey function here by saying _id must be used for it. We use a type annotation to indicate which _id function is used. Leaving that out results in an ambigious type error.

instance Beamable UserT
instance Beamable (PrimaryKey UserT)

Beamable provides several ‘introspection routines’. We require it to create a Database out of the table. The database will be described below.

Message table

data MessageT f = Message
                { _id        :: C f Int
                , _from      :: PrimaryKey UserT f
                , _content   :: C f Text.Text
                }
                  deriving Generic
type Message = MessageT Identity
deriving instance Show (PrimaryKey MessageT Identity)
deriving instance Show Message

instance Table MessageT where
    data PrimaryKey MessageT f = MessageId (Columnar f Int) deriving Generic
    primaryKey = MessageId . (_id :: MessageT f -> C f Int)
type MessageId = PrimaryKey MessageT Identity -- For convenience

instance Beamable MessageT
instance Beamable (PrimaryKey MessageT)

The boiler plate is similar to that of User, the only new concept is the from field, which points at the user table with the primary key. Beam can make joins on this with the DSL.

Database

data AwesomeDb f = AwesomeDb
                      { _users    :: f (TableEntity UserT)
                      , _messages :: f (TableEntity MessageT) }
                        deriving Generic

This type defines the entire database. Again it provides a ‘hole’ with the f.

connectionString :: BS.ByteString
connectionString = "dbname=awesome_db"

The connectionString is required to connect to the database. One probably doesn’t want to hard code it, but for this guide hard coding is good enough.

instance Database be AwesomeDb

Here we create a beam database out of the AwesomeDB type. the be hole is reserved for a back end, which we don’t specify.

awesomeDB :: DatabaseSettings be AwesomeDb
awesomeDB = defaultDbSettings

The implementation of AwesomeDB just uses the default database settings. All structural information is already provided at type level.

Using structure

Now we have a database structure defined we can use it in Lib.hs. We have already seen most of this source file in the previous blog post, the new version can be seen in the sources.

The functionality is still the same except now we’re using a database as backend rather than a file. Just like with the previous post, the example shows both how to insert, as well as retrieve data. Let’s inspect the new changes.

messages :: Connection -> Message -> Handler [Message]
messages conn message = do 
  messages <- liftIO $

Messages will hold the resulting messages we’re to querying from the database. liftIO allows functions within the IO context (eg, interact with the world).

    PgBeam.runBeamPostgres conn $ do

Run the beam Monad with help of a connection. In other words, everything within this do block is a query for the database, and we’re explicitly using postgres to solve ambiguity.

      let user = from message

Retrieve the user from message for convenience.

      [user] <- runInsertReturningList (DB._users DB.awesomeDB) $ Beam.insertExpressions [DB.User{
            DB._userId = Beam.default_,
            DB._name = Beam.val_ (pack $ name $ user ),
            DB._email = Beam.val_ (pack $ email $ user )
        }]

the runInsertReturningList function call is quite complex. The first argument defines in which table we’re using, we want to insert something into the users table. The second argument is a list of expressions. To get the expressions we use the Beam.insertExpressions function. This is how we insert an item, in this case we only want to insert one user. We use the User constructor defined earlier in the DB.hs module to obtain a user. The fields are populated with values or a special default value.

Note that although this function is complex, if we do anything wrong we get a type error. Our code will not compile unless we do it right. This is one of the strengths of beam.

      _ <- runInsertReturningList (DB._messages DB.awesomeDB) $ Beam.insertExpressions $ [DB.Message{
            DB._messageId = Beam.default_,
            DB._from = Beam.val_ (Beam.pk user),
            DB._content = Beam.val_ (pack $ content message)
        }]

These lines insert the message into the db, linking it up with the newly inserted user trough the pk.

      Beam.runSelectReturningList $ Beam.select $ do 
        usr <- (Beam.all_ (DB._users DB.awesomeDB))
        msg <- Beam.oneToMany_ (DB._messages DB.awesomeDB) DB._from usr
        pure (msg, usr)

This query gets the resulting messages and their respective users joined together.

  pure $
    fmap (
      \(msg, usr) -> Message
        (User
          (unpack $ DB._name usr)
          (unpack $ DB._email usr))
        (unpack $ DB._content msg)
    ) messages

here we convert the database user and database message, to the ‘API’ user and ‘API’ messages. The reason we need to do this is because our database data structure does not implement toJSON. Also the database structure has extra information such as the primary key which we may want to hide from API clients.

Execute!

To run the program we use:

   stack build
   stack exec webservice

To test it a simple curl request was made:

   curl --header "Content-Type: application/json" -v --data '{"from":{"email":"d","name":"xyz"}, "content": "does it word?"}' http://127.0.0.1:6868/message/

We can inspect the database with postgres

   psql "dbname=awesome_db"
   \dt
   select * from messages;

Conclusion

We have looked at the beam library in this post and it’s interaction with postgres. Although the example is simple, there is quite a bit of boilerplate involved, but once setup it provides a complete type safe DSL to the database. With the database and web server in place nothing is stopping the reader from making his next major project in Haskell! We hereby conclude our Haskell safari successfully.

Complete sources

The complete sources can be found on github, and below.

Db.hs

{-# LANGUAGE DeriveGeneric         #-}
{-# LANGUAGE FlexibleInstances     #-}
{-# LANGUAGE MultiParamTypeClasses #-}
{-# LANGUAGE OverloadedStrings     #-}
{-# LANGUAGE StandaloneDeriving    #-}
{-# LANGUAGE TypeFamilies          #-}
{-# LANGUAGE DuplicateRecordFields #-}

-- | db structure and source of truth
module DB where
import qualified Data.ByteString                as BS
import qualified Data.Text                      as Text
import           Database.Beam


data UserT f = User
                { _id :: C f Int
                , _name   :: C f Text.Text
                , _email  :: C f Text.Text
                }
                  deriving Generic
type User = UserT Identity
deriving instance Show UserId
deriving instance Show User

instance Table UserT where
    data PrimaryKey UserT f = UserId (Columnar f Int) deriving Generic
    primaryKey = UserId . (_id :: UserT f -> C f Int)
type UserId = PrimaryKey UserT Identity -- For convenience

instance Beamable UserT
instance Beamable (PrimaryKey UserT)

  
data MessageT f = Message
                { _id :: C f Int
                , _from      :: PrimaryKey UserT f
                , _content   :: C f Text.Text
                }
                  deriving Generic
type Message = MessageT Identity
deriving instance Show (PrimaryKey MessageT Identity)
deriving instance Show Message

instance Table MessageT where
    data PrimaryKey MessageT f = MessageId (Columnar f Int) deriving Generic
    primaryKey = MessageId . (_id :: MessageT f -> C f Int)
type MessageId = PrimaryKey MessageT Identity -- For convenience

instance Beamable MessageT
instance Beamable (PrimaryKey MessageT)


data AwesomeDb f = AwesomeDb
                      { _ausers    :: f (TableEntity UserT)
                      , _messages :: f (TableEntity MessageT) }
                        deriving Generic

connectionString :: BS.ByteString
connectionString = "dbname=awesome_db"

instance Database be AwesomeDb

awesomeDB :: DatabaseSettings be AwesomeDb
awesomeDB = defaultDbSettings

Lib.hs

{-# LANGUAGE DataKinds #-}
{-# LANGUAGE TypeOperators #-}
{-# LANGUAGE DeriveGeneric #-}

module Lib
    ( webAppEntry
    ) where

import Servant
import Control.Monad.IO.Class(liftIO)
import Data.ByteString.Lazy as LBS (writeFile, readFile) 
import Data.Aeson(ToJSON, FromJSON, encode, decode)
import GHC.Generics(Generic)
import Network.Wai(Application)
import Network.Wai.Handler.Warp(run)
import           Database.PostgreSQL.Simple   (Connection)
import qualified DB as DB
import           Database.Beam.Backend.SQL.BeamExtensions (runInsertReturningList)

import qualified Database.Beam                            as Beam
import qualified Database.Beam.Postgres                            as PgBeam
import Data.Text(pack, unpack)

type UserAPI = "users" :> Get '[JSON] [User]
      :<|> "message" :> ReqBody '[JSON] Message :> Post '[JSON] [Message]

data Message = Message {
  from :: User,
  content :: String
} deriving (Eq, Show, Generic)

instance ToJSON Message
instance FromJSON Message

data User = User
  { name :: String
  , email :: String
  } deriving (Eq, Show, Generic)

instance ToJSON User
instance FromJSON User

users :: [User]
users =
  [ User "Isaac Newton"    "isaac@newton.co.uk"
  , User "Albert Einstein" "ae@mc2.org"
  ]

messages :: Connection -> Message -> Handler [Message]
messages conn message = do 
  messages <- liftIO $ 
    PgBeam.runBeamPostgres conn $ do
      let user = from message
      [user] <- runInsertReturningList (DB._ausers DB.awesomeDB) $ 
          Beam.insertExpressions [DB.User 
            Beam.default_
            (Beam.val_ (pack $ name $ user ))
            (Beam.val_ (pack $ email $ user ))
        ]
      _ <- runInsertReturningList (DB._messages DB.awesomeDB) $ 
          Beam.insertExpressions 
            [DB.Message 
              Beam.default_ 
              (Beam.val_ (Beam.pk user))
              (Beam.val_ (pack $ content message))
            ]
      Beam.runSelectReturningList $ Beam.select $ do 
        usr <- (Beam.all_ (DB._ausers DB.awesomeDB))
        msg <- Beam.oneToMany_ (DB._messages DB.awesomeDB) DB._from usr
        pure (msg, usr)
  pure $
    fmap (
      \(msg, usr) -> Message
        (User
          (unpack $ DB._name usr)
          (unpack $ DB._email usr))
        (unpack $ DB._content msg)
    ) messages


server :: Connection -> Server UserAPI
server conn= (pure users) :<|> (messages conn)

userAPI :: Proxy UserAPI
userAPI = Proxy

app :: Connection -> Application
app conn = serve userAPI (server conn)

webAppEntry :: Connection -> IO ()
webAppEntry conn = do
  run 6868 (app conn)

data_model.sql

DROP TABLE ausers cascade;
DROP TABLE messages cascade;
CREATE TABLE ausers (
    id serial NOT NULL PRIMARY KEY,
    "name" varchar NOT NULL,
    email varchar NULL
);

CREATE TABLE messages (
    id serial NOT NULL PRIMARY KEY,
    from__id int REFERENCES ausers(id),
    content varchar NULL
);

Pragmatic Haskell II: IO Webservant

2018-06-27T22:00:00Z

Most Haskell language guides will leave IO until later. This guide is different, this guide is about using Haskell. Our focus is different: We build first, then learn trough delight.

The previous blog post explained how to get going with a simple minimalist servant web server. In this blog post the simple web server will get an extra REST endpoint that can do IO actions. This is an important part of pragmatic Haskell programming. Without IO our program can do nothing. Programmers are not theorists, therefore we need IO.

Preparation

To keep things simple, the code assumes a file exists. Create one with an empty JSON array in the project root:

echo "[]" > messages.txt

Bytestrings are a convenient way of opening files and putting the results into aeson, the JSON library. Dealing with bytestrings requires another dependency:

dependencies:
- base >= 4.7 && < 5
- servant-server # http server
- aeson # json
- wai # web application (interface)
- warp # web application implementation
- bytestring

A lot of code

These are the ~~magic spells~~ changes which add an endpoint, explained in detail below:

{-# LANGUAGE DataKinds #-}
{-# LANGUAGE TypeOperators #-}
{-# LANGUAGE DeriveGeneric #-}

module Lib
    ( webAppEntry
    ) where

import Servant
import Control.Monad.IO.Class(liftIO)
import Data.ByteString.Lazy as LBS (writeFile, readFile) 
import Data.Aeson(ToJSON, FromJSON, encode, decode)
import GHC.Generics(Generic)
import Network.Wai(Application)
import Network.Wai.Handler.Warp(run)
import Data.Maybe (fromMaybe)

type UserAPI = "users" :> Get '[JSON] [User]
      :<|> "message" :> ReqBody '[JSON] Message :> Post '[JSON] [Message]

data Message = Message {
  from :: User,
  content :: String
} deriving (Eq, Show, Generic)

instance ToJSON Message
instance FromJSON Message

data User = User
  { name :: String
  , email :: String
  } deriving (Eq, Show, Generic)

instance ToJSON User
instance FromJSON User

users :: [User]
users =
  [ User "Isaac Newton"    "isaac@newton.co.uk"
  , User "Albert Einstein" "ae@mc2.org"
  ]
messageFile :: FilePath
messageFile = "messages.txt"

messages :: Message -> Handler [Message]
messages message = do 
  result <- liftIO $ LBS.readFile messageFile
  case decode result of
    Nothing -> pure []
    Just x -> do
      let contents = x ++ [message]
      liftIO $ LBS.writeFile messageFile (encode contents)
      return contents

server :: Server UserAPI
server = (pure users) :<|> messages

userAPI :: Proxy UserAPI
userAPI = Proxy

app :: Application
app = serve userAPI server

webAppEntry :: IO ()
webAppEntry = run 6868 app

This will setup another endpoint for messages. The new endpoint will accept a post request under “/message”. It will write the message to a file and then it will return the contents of that file.

Line by line inspection

type UserAPI = "users" :> Get '[JSON] [User]
      :<|> "message" :> ReqBody '[JSON] Message :> Post '[JSON] [Message]

The :<|> operator is used to add an extra endpoint. It combines the two endpoints into one. In these lines, we are constructing something akin to a jump table. The decleration of this operator is surprisingly simple:

data a :<|> b = a :<|> b

Left sign of equality is used for type, right side for data construction. Skimming over this, like true Haskellers we will ignore the inner workings .

The extra end point “message” is similar in structure to the existing “user” endpoint. If read like a sentence “message” is a POST only endpoint, which accepts a Message JSON body, and it returns a list of Messages in JSON.

Message data

data Message = Message {
  from :: User,
  content :: String
} deriving (Eq, Show, Generic)

instance ToJSON Message
instance FromJSON Message

This is Message, apparently it’s from a User and has some content String. Aside from using another data type inside an existing data type, no new concepts are introduced.

File path

messageFile :: FilePath

The type FilePath is just an alias for a String: type FilePath = String. In other words we can use them interchangeably. FilePath acts as documentation.

messageFile = "messages.txt"

This is where we define what file name is used.

Message handler

messages :: Message -> Handler [Message]

We define a Handler (which is an servant api endpoint). It requires a message, then it will return a list of messages within a Handler.

Do notation

messages message = do

The do keyword allows us to code with do notation. This allows us to do assignments with <- (not assignment, but close enough). This only works when the result container is a Monad. How monads work is a mystery, but usage is simple: Use do notation.

LiftIO

  result <- liftIO $ LBS.readFile messageFile

Here we read the message file as a lazy bytestring and put it into the result. One may wonder why we are using liftIO, if it’s deleted we get this:

/home/jappie/projects/haskell/awesome-project-name/src/Lib.hs:46:13: error:
    • Couldnt match type ‘IO’ with ‘Handler’
      Expected type: Handler Data.ByteString.Lazy.Internal.ByteString
        Actual type: IO Data.ByteString.Lazy.Internal.ByteString
    • In a stmt of a 'do' block: result <- LBS.readFile messageFile
      In the expression:
        do result <- LBS.readFile messageFile
           case decode result of
             Nothing -> pure []
             Just x -> do ...
      In an equation for ‘messages’:
          messages message
            = do result <- LBS.readFile messageFile
                 case decode result of
                   Nothing -> pure ...
                   Just x -> ...
   |
46 |   result <- LBS.readFile messageFile
   |             ^^^^^^^^^^^^^^^^^^^^^^^^

The LBS.readfile function has a return type of IO. However the return type of messages is Handler. Therefore the compiler says that it expects Handler, but the actual type is IO. Handler implements the MonadIO typeclass however, which allows IO by calling the liftIO function. The liftIO function simply tells the Handler container to execute some function within the IO container.

The dollar sign can be replaced with an open parentheses, which is closed at the end of the line. This is equivalent for example:

liftIO (LBS.readFile messageFile)

As said before <- is (basically) used for assignment in do notation, using <- is a good way to get rid of a monad container. result now contains the contents of messageFile, the IO is being evaluated and removed by the <- operator. In Haskell a lot of wrapping and unwrapping is done.

Case .. of

  case decode result of

Here we’re decoding the contents of result. decoding JSON may not succeed and therefore the library authors of aeson made decode return a maybe container:

decode :: FromJSON a => ByteString -> Maybe a

In this signature, a can be anything as long as it implements FromJSON. We fulfill this condition with generic and instance FromJSON Message. We give as bytestring the result to decode, in return it gives us Maybe a. The compiler deduces that a in this case is [Message].

The return value will have content if decoding succeeded (Just), or it won’t if it fails (Nothing). To get rid of the container we pattern match it. This can be thought of as a switch case statement in other languages.

Nothing

    Nothing -> pure []

In this case decoding fails. An empty list is returned to the client. We still must wrap this list in a Handler, pure is used for that. Note that pure == return. These functions both exist for historical reasons.

Just a

    Just x -> do

In this case there is success, the result is taken and put into x, after which another do block starts.

Let

      let contents = x ++ [message]

Unlike the <- operator, let does not do any unwrapping. We can see what happens if we replace the let binding by contents <- x ++ [message], the errors are:

/home/jappie/projects/haskell/awesome-project-name/src/Lib.hs:50:19: error:
    • Couldnt match type ‘[]’ with ‘Handler’
      Expected type: Handler Message
        Actual type: [Message]
    • In a stmt of a 'do' block: contents <- x ++ [message]
      In the expression:
        do contents <- x ++ [message]
           liftIO $ LBS.writeFile messageFile (encode contents)
           return contents
      In a case alternative:
          Just x
            -> do contents <- x ++ [message]
                  liftIO $ LBS.writeFile messageFile (encode contents)
                  return contents
   |
50 |       contents <- x ++ [message]
   |                   ^^^^^^^^^^^^^^

/home/jappie/projects/haskell/awesome-project-name/src/Lib.hs:52:7: error:
    • Couldnt match type ‘Message’ with ‘[Message]’
      Expected type: Handler [Message]
        Actual type: Handler Message
    • In a stmt of a 'do' block: return contents
      In the expression:
        do contents <- x ++ [message]
           liftIO $ LBS.writeFile messageFile (encode contents)
           return contents
      In a case alternative:
          Just x
            -> do contents <- x ++ [message]
                  liftIO $ LBS.writeFile messageFile (encode contents)
                  return contents
   |
52 |       return contents
   |       ^^^^^^^^^^^^^^^

In the first error the compiler says that a list is not a Handler container. Which we expect because the return type of this function is Handler [Message].

The second error assumes contents is of the correct type, since <- unwraps contents would be of type Message. The return type does not fit in this case either, we would get Handler Message instead of Handler [Message].

Write

      liftIO $ LBS.writeFile messageFile (encode contents)

We encode the contents, then we write it to the message file in an IO effect. Type safety ensures encoding always succeeds.

Return

      return contents

Finally we wrap the contents in a Handler type.

Adding the handler to the routes map

server :: Server UserAPI
server = (pure users) :<|> messages

This is the implementation of the UserAPI type described before. We use the same operator to also add the messages handler into the server. We don’t need to put messages in a container with pure because the functions’ return type is already a Handler.

Something worth pointing out is that this construction is in order, if we pull it out of order we would get a type level. Changing the line into:

server =  messages :<|> (pure users)

Will cause an error:

/home/jappie/projects/haskell/awesome-project-name/src/Lib.hs:55:11: error:
    • Could not match type ‘[User]’ with ‘Handler [Message]’
      Expected type: Server UserAPI
        Actual type: (Message -> Handler [Message])
                     :<|> (Message -> [User])
    • In the expression: messages :<|> (pure users)
      In an equation for ‘server’: server = messages :<|> (pure users)
   |
55 | server =  messages :<|> (pure users)
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^

Execute it!

curl --header "Content-Type: application/json" \ 
  --request POST \
  --data '{"from":{"email":"d","name":"xyz"}, "content": "does it word?"}' \
  http://localhost:6868/message

Worked on this machine…

Perhaps that’s why the theorists avoid IO 🤔.

In conclusion

Without going into much theory, we dealt with IO. For example we saw that haskell is about dealing with containers, to put certain functions in IO rather than the return type, one uses liftIO. do notation was also encountered, which makes working with monads easier. Now we can affect the world with our programs trough IO!

The complete code can be found here. In the future we shall attach this simple web server to a database.

Pragmatic Haskell: Simple servant web server

2018-06-25T15:25:00Z

There are many guides available for learning Haskell. Setting up a something simple like a web server isn’t so straight forward. Perhaps choosing one of the 14 libraries is a bit much.

This guide will give opinionated web server start. This guide assumes no experience with Haskell, and will get you up to speed with a (REST) web server called Servant. Servant is a good choice as it can describe both a server and client API. In the future this guide may be used as a foundation to create something more meaningful than just a very basic REST API, this will provide a good starting point however. Basic UNIX (command line) skills are assumed.

From nothing, start with build tools

Install stack:

curl -sSL https://get.haskellstack.org/ | sh

Only attempt shortly to install it trough a package manager. There are other Haskell build tools, they will be more difficult in use. There is also the possibility for fully reproducible builds at a system level (nix). Which is out of the scope of this guide.

Now setup a new project:

stack new awesome-project-name 
cd awesome-project-name

Hello world with stack

Appreciate what happens when this is build:

stack build && stack exec awesome-project-name-exe

This should build successfully and output someFunc. Open up src/Lib.hs with one’s favorite editor. This contains a few lines, created by stack:

module Lib
    ( someFunc
    ) where

someFunc :: IO ()
someFunc = putStrLn "someFunc"

This is where the someFunc output came from when the program was ran. Change it to something a bit more appropriate, and rename the function too:

module Lib
    ( webAppEntry
    ) where

webAppEntry :: IO ()
webAppEntry = putStrLn "This is the beginning of my greetings to world"

Does it compile?

stack build && stack exec awesome-project-name-exe

/home/jappie/projects/haskell/awesome-project-name/app/Main.hs:6:8: error: Variable not in scope: someFunc :: IO ()
  \|
6 \| main = someFunc
  \|        ^^^^^^^^

It does not compile. There is an app folder where by default all the executable reside (which is where the error occurs), and a src folder where the library code lives (the modified file is in there). One can future proving themselves by putting as much code in the library as is reasonable.

Fix the error in app/Main.hs:

module Main where

import Lib

main :: IO ()
main = webAppEntry

It builds! Functions can be renamed, simple compile errors can be solved, and strings can be changed. Progress!

Servant: Your first dependencies

For the impatient, there is a minimal example already available by the library author. This guide will explain how to get there step by step. In ./package.yaml, on line 22 there is a dependencies key, add servant-server, aeson, wai and warp to it like this:

dependencies:
- base >= 4.7 && < 5
- servant-server
- aeson
- wai
- warp

It may seem strange to immediately add four new dependencies, however this is because Haskell libraries are setup to be flexible. Even small projects grow quickly to have into the twenties of dependencies. Code reuse is not a myth.

servant-server is the servant web server. aeson is for JSON parsing and producing. wai is a web application interface and warp uses wai to implement a web application (it binds to the port).

Ensure that that this is done at the root of the yaml file (no indentation). Stack provides a way of specifying dependencies of either the executable or library. If its done on line 22, the root of the yaml file, it will be a dependency for everything in the project.

A minimal servant

A good start is going to servants’ Hackage page, which linked to a tutorial. Servant does API definition at type level.

If it’s unknown to the reader what a type is, think of it as describing the shape of a function. Functions of different shapes don’t fit together, and won’t compile. What servant allows us to do is define this shape for a REST API. To gain a deeper understanding of this a concrete example will be inspected line by line. First all lines are listed for a minimal servant (Lib.hs) server:

{-# LANGUAGE DataKinds #-}
{-# LANGUAGE TypeOperators #-}
{-# LANGUAGE DeriveGeneric #-}

module Lib
    ( webAppEntry
    ) where

import Servant(serve, Proxy(..), Server, JSON, Get, (:>))
import Data.Aeson(ToJSON)
import GHC.Generics(Generic)
import Network.Wai(Application)
import Network.Wai.Handler.Warp(run)

type UserAPI = "users" :> Get '[JSON] [User]

data User = User
  { name :: String
  , email :: String
  } deriving (Eq, Show, Generic)

instance ToJSON User

users :: [User]
users =
  [ User "Isaac Newton"    "isaac@newton.co.uk"
  , User "Albert Einstein" "ae@mc2.org"
  ]

server :: Server UserAPI
server = return users

userAPI :: Proxy UserAPI
userAPI = Proxy

app :: Application
app = serve userAPI server

webAppEntry :: IO ()
webAppEntry = run 6868 app

Language extensions

The first three lines are languages extensions, Haskell behaves different for this module according to these. data kinds Can be temporary deleted to see what happens:

/home/jappie/projects/haskell/awesome-project-name/src/Lib.hs:14:16: error:
    Illegal type: ‘"users"’ Perhaps you intended to use DataKinds
   |
14 | type UserAPI = "users" :> Get '[JSON] [User]
   |                ^^^^^^^

/home/jappie/projects/haskell/awesome-project-name/src/Lib.hs:14:31: error:
    Illegal type: ‘'[JSON]’ Perhaps you intended to use DataKinds
   |
14 | type UserAPI = "users" :> Get '[JSON] [User]
   |                               ^^^^^^^

Data kinds is needed to insert data into a type. A string being data in this case, it is unclear what '[JSON] is, probably also something data. Temporary breaking a program to see what GHC will say is an effective way of learning more about Haskell.

If TypeOperators is disabled, GHC says it doesn’t like :> in the UserAPI line. Apparently :> is a type operator. Apparently types can have operators.

If DeriveGeneric is disabled, GHC says it needs to derive generic in the data definition of User. Generic is required for serialization (in our case JSON conversion).

Modules

module Lib
    ( webAppEntry
    ) where

import Servant(serve, Proxy(..), Server, JSON, Get, (:>))
import Data.Aeson(ToJSON)
import GHC.Generics(Generic)
import Network.Wai(Application)
import Network.Wai.Handler.Warp(run)

Moving onward, there is the module definition that stack generated, modules are just namespaces, or similar to python modules. Nothing really special about those. Then there are many imports which pull functions into the module namespace.

Type level REST API

type UserAPI = "users" :> Get '[JSON] [User]

This line defines the UserAPI type, which will serve as the REST endpoint. The image at the beginning of the post was about this line. Perhaps reading it as a sentence will give us some insight, without worrying about how it fits together: It’s a Get request, mounted below /user, returning something JSON and of shape/type User. Conveniently what a User is will be discussed in the next section.

Domain data

data User = User
  { name :: String
  , email :: String
  } deriving (Eq, Show, Generic)

instance ToJSON User

User is just a data structure consisting of two strings: Email and name. This declaration method is called record syntax. This data structure derives Show, Eq and Generic. Deriving means that GHC will generate function implementations for this data structure. If one calls show on a User, it will know what to do (show is toString in Haskell). instance ToJSON User allows the User to be converted to JSON (implementation is provided by generic).

Functions

Done with data, time for code!

users :: [User]

Specifies a function that will always return a list of Users. There are no arguments to this function. It can be assumed the list is always the same. This is how immutable constants are specified.

users =
  [ User "Isaac Newton"    "isaac@newton.co.uk"
  , User "Albert Einstein" "ae@mc2.org"
  ]

This is the implementation of the before defined function. There are apparently two users in this list, one Isaac, and another Einstein. Note that positional arguments are used to create the Users.

Servant server

server :: Server UserAPI

server :: Server UserAPI says that there is something called a Server which has a UserAPI. A UserAPI is known, it is defined above. A Server is defined in servant. The type signature is rather complicated: type Server api = ServerT api Handler, looking at the definition of ServerT introduces a lot of complexity: type ServerT api (m :: * -> *) :: *.

There are some clues that can be derived (such as that m), but it’s not that important to make something work. Therefore this guide ignores it. Note that ignoring scary looking things is an important Haskell technique. If one is interested, help can be found here, just in case ❤.

server = return users

The implementation is very simple however. The reader should be cautious, to think that return is a keyword. It’s a function. What both return does is wrap a value into a container. For example an element can be wrapped in a list: return 2 == [2]. That’s all one needs to know for now (the interested reader may look at monads).

Proxy

userAPI :: Proxy UserAPI
userAPI = Proxy

This is just some type level magic. Library author needed type information for a function, but they didn’t need a value. Proxy does that. It’s useful if you store data at type level, for example with the datakinds language extension, which was seen earlier.

Application

app :: Application
app = serve userAPI server

This combines the proxy and server. A serve function takes a Proxi API, Server API and returns an application. If type Application is inspected one can appreciate what serve does for us better:

type Application = Request -> (Response -> IO ResponseReceived) -> IO ResponseReceived

The arrows indicate function arguments. An application receives a request, then a callback which expects a Response to produce an IO action which gives the result ResponseReceived. However to return this function must also return a type ResponseReceived wrapped in IO. It may be the case that the only way to obtain this response received is to call that callback. The freedom to do whatever one wants is meanwhile granted with the IO return type. To compile that ResponseReceived has to be obtained however.

Running it!

webAppEntry :: IO ()
webAppEntry = run 6868 app

Our initial function! Rather than saying hello world the app is ran on port 6868 (best port). Now build and run it in one terminal, and in another curl it:

stack build && stack exec awesome-project-name-exe &
curl localhost:6868/users

> [{"email":"isaac@newton.co.uk","name":"Isaac Newton"},{"email":"ae@mc2.org","name":"Albert Einstein"}]

In conclusion

A lot of concepts have been treated within this blog post while also moving towards something productive. The reader can now start a new project and add arbitrary dependencies. He knows what language extensions are and how to see them in use. Type level magic has been encountered, and wisely was ignored. In future this post will build on top of this work to extend the API and do something with something within the handlers. However this post has grown to big already.

The complete code can be found here.

Elm on fire! Shaders in elm

2018-06-23T12:00:00Z

Shaders have long been on the list of possible subject to study for Jappie. The potential of both creating beautiful art as well as doing parallel processing seem incredible valuable capabilities to have. This post comments on the effort of porting a JavaScript WebGL fire to an elm implementation. Elm was chosen as target language because it is opinionated, easy and type safe. In this post we explore how to get started in elm with shaders, and then move on trying to port the fire project, finally performance is increased as much as possible.

In the beginning there was nothing.

There are some example shader setups for elm. The `crate’ was copied over resulting into having a fully 3d crate! This is not exactly the output desired, a crate is not a fire (obviously), but now there is a skeleton for the elm architecture and some example shaders to play with.

From here on there are two possible paths to continue, one can try and completely understand what the shaders do and how they work, or one can just copy over the shader code from the JavaScript project and see if we can make that work. Although initial work was started on the former approach, the latter approach won out because the topic of ‘shader’ is just too large. There is a lot of math involved. Although this is an exercise of exploration and learning, trying to understand it all is a massive scope creep.

Unbreak rendering

After copying over the shader logic from the fire project, everything broke. This was not surprising as the crate project was 3d, whereas the fire project was 2d. Luckily Elm has strongly typed input for the shaders. Therefore solving these mismatches was relatively easy. We could just follow the compile errors. After that, the example program was essentially gutted, only the basic architecture and API calls were left in tact. Elm forces this architecture upon us, there is no choice in this. The result of this effort is shown below.

It does not look like much of anything, however, this is counted as progress. Not having a blank screen is good. The next thing to do was fixing the colors. This happened by porting the hue code, there was no elm implementation for this particular kind of Hue representation. Because a white background and the hue produces light blue, we added a black background which mixes into an orange. Now we had the right color, however transparency was also broken. Transparency was quite interesting because my initial fix involved changing the shader. However the the right (API) option was eventually found that solved this issue. With all of this in place we get a single circle with the right color!

Baby steps. Graphics take time.

Random spheres

This is not impressive at all. However, in life one may find that arity changes everything. A single dot on it’s own is just a single dot, but if we randomly place it all over the screen we get something nice to look at (live here):

Your browser does not support the video tag.

Aside from random creation this doesn’t bring us much closer to the goal of fire. However some more work was done on it because Jappie thought it looked beautiful. Performance was increased by converting a particle immediately into it’s WebGL representation.

Movement

To do movement we dropped some changes from the random sphere case. The idea of not doing an update loop at all was temporary put aside, because using an update loop is closer to the JavaScript original. Keeping it would make porting easier.

Doing movement is simply adding velocity times time to position every frame. That’s it. The simplex noise part of the JavaScript code was also implemented for variation in movement.

It turns out however that the result is somewhat unimpressive. Yes it looks like fire, but after about 20 seconds the memory is full, garbage collection kicks in and the program grinds to an halt. Here is an example (live here, may grind your computer to a halt):

Your browser does not support the video tag.

Speed

The problem is that aside from creating particles and sending them to a GPU, All existing particles must every cycle be updated with the new location. We may observer however that the path of the particles after creation is entirely deterministic. Why don’t we let the shaders do this? The idea being that we create particles with an initial position, timestamp and velocity. Then let the shaders calculate the position for whatever the current timestamp is.

When trying to implement this, it was found out that the elm gl API was used in a inefficient way. The realization came that using an entity per quad doesn’t allow us to share the uniform across all quads. These entities are analogues to WebGL ‘programs’

The architecture was redesigned to take into account multiple particles per elm entity. Rather than tracking lists of entities, lists of tuples of vertices are now being tracked: List (Vertex, Vertex, Vertex). It would’ve been preferable to use Mesh Vertex as type, but this type does not support appending in the elm shader API.

This approach seemed to work much better, in fact this is probably how one should use this API. It was possible to render 500 particles now and the computer didn’t lock up (at all):

live here Your browser does not support the video tag.

It’s still not very good, as the original JavaScript implementation was able to do up to 3000 particles per seconds quite comfortably (with a much better frame rate)… There is not a lot of things done in the this implementation on the CPU side, and still the CPU intense JavaScript implementation is faster. Perhaps this is just a limit of using elm.

More speed?

After thinking about the problem for some time another idea came to mind. To increase speed, the amount of information send to the GL pipeline can be reduced. Every frame sends this Mesh collection to the GPU trough a buffer, if this buffer can be decreased in size, speed would increase. It would also lighten the load on garbage collection, as less objects need to be created. The suspicion is that elm is slow just because of garbage collection. We can do this rather trivially by representing each particle as a single vertex, with a position and size. Then we just use a shader to reconstruct the vertices into quads (squares). The vertex shader would move the vertex first, then another (unkown) shader would do reconstruction, then the fragment shader would do drawing. Easy as π.

Stack overflow suggests that we can use a geometry shader for this. Unfortunately the elm GL API doesn’t support this, it only has a slot for vertex, and fragment shader in the entity function. Jappie briefly got excited about adding this shader type to the elm API, however he discovered that WebGL doesn’t support this type of shader at all. From this point it’s unclear how to increase speed. Changing WebGL itself is borderline impossible (it would take at least years).

In conclusion

Upon discovery of the WebGL API for elm Jappie was quite excited about using that. However after using it, and finding the rather large performance difference the excitement has been tempered. Still a lot was learned from doing this project, elm is a good entry point for graphics development, type-safety makes the complexity quite manageable. In fact the idea for using geometry shaders would not had been realized at all in a faster language.

However in future a faster language will be used. Not being able to get everything from a machine is quite frustrating. This therefore will exclude any use of WebGL. A major drive for using WebGL in the first place was to share the result online, however this reasoning is quite flawed in that a video of the result can just be made. After all we don’t even include the live results here because of concern that it will freeze the readers’ computer.

To all who are interested in graphics elm is recommended to start with. Especially if they are already familiar with elm or the react/redux architecture. Type safety on shader level is really nice, especially when you do things wrong structurally and the compiler exactly tells you where you need to repair things (as happened during this project with the entities). The price one pays is execution speed in return for development speed.

Fun with stack: Haskell dependency management

2018-05-13T16:00:00Z

Working at Daisee, Jappie uses a lot of Haskell programming. Although Haskell is obviously as amazing as the stereotype asserts, the tooling can be a bit challenging. In this blogpost we explore these challenges.

One’s understanding start with the fact that there is not one unified Haskell package manager. Well there is, but only experts use cabal directly. The issue is that often dependencies don’t quite match from the Hackage repository, which cabal uses as repository. Loose version bounds within cabal configurations, cause one developer to develop against version 1.1 and another against version 1.2.

This is were stack stepped in. Stack guarantees that at some point in time all the provided packages in their repository will build together. They do this trough package curation and providing reproducible builds at least for Haskell packages. This is hosted on their alternative to Hackage, called Stackage.

Because of curation and being more restrictive, the stackage repository doesn’t host as much software as hackage does. In certain cases we may still need to configure stack in such a way that it pulls in a package or two from hackage.

System dependencies

Haskell is not an operating system. There is no Haskell Linux kernel or Haskell std lib. Therefore many Haskell libraries are dependent upon C code libraries. Stack does not include interaction with these system libraries as part of it’s reproducible build contract.

The user experiences this once stack starts requesting ‘devel’ packages, for example: Stack will request ‘ssl’ headers, and recommends the user to install a devel system package from his distribution package manager. It is up to the user to find the name of this system package for his respective distribution, in case of fedora it’s: “openssl-devel”, and in case of Ubuntu it’s “libssl-dev”. But gentoo wouldn’t even have devel packages, all packages in case of gentoo are devel packages.

Enter nix

Nix is also a package manager like stack and cabal are, however, it’s not focused on any specific language. Nix is about reproducible builds, eg guaranteeing some build on a developer machine is exactly the same as on the user machine.

A developer may for example have had the ssl library installed globally, but forgot to mark it as a dependency in his software package. A build tool would happily build that on his machine, but now when a user downloads this code and tries to build the package it will fail. Nix solves this.

In principle nix is completely independent from the Haskell ecosystem. Build around this package manager is even an entire distribution. But because it provides system dependencies and haskell dependency management, and does so in a reproducible manner at system level, it has become very popular among the correctness loving Haskell programers.

However it may be pointed out that although what nix does is impressive compared to stack it’s harder to use. The langauge nix langauge is peculiar, for example it has both a ;, and being reduced to a single expression. It has no type system but does full correctness checking before exucting. But it has a massive repository of examples.

Stack and nix

Stack provides the ability to specify system dependencies trough nix. Which is very useful as you now can let any developer regardless of Linux distribution, get on board with your stack project, without having to find out which specific system packages need to be installed. Is this combination the solution? Easy to use and being fully reproducible? Alas no this setup also has issues.

It has an opt in for the isolated builds nix provides, so stack would first look if a package is specified in the nix configuration, if not it would fall back on system libraries. This is rather strange, because you enable the plugin for reproducible builds, but the default behavior is to look for system libraries (eg be not reproducible). There is an opt out for this behavior (pure:true), but it seems more sensible to make this an opt in.

The reason why this behavior is undesired is because if one programmer starts using http-client-openssl, and happens to have openssl-devel installed on his system, then his colleague try to build the project, his build will fail. If pure:true was enabled the first developer would’ve noticed the issue and added it into the nix configuration.

However the way stack tells the user about the missing dependency is also peculiar. In case of the pure configuration, if stack can’t find the library, it will ask the user to install the dependency on his system. Which is of course pointless because one needs to add it to the nix configuration Although the error message was intended to be helpful, in this particular case it ends up sending the user in the wrong direction.

Git and stack

Stack’s power lies in that it makes it easy to do the common things. For example creating a library for github is trivial. Stack helps by allowing users to specify a ‘github’ field in the package.yaml file, which will generate an url to both the source and the issue tracker on github.

name:                voicebase
version:             0.1.0.0
license:             BSD3
author:              "Jappie Klooster"
maintainer:          "jappie.klooster@daisee.com"
copyright:           "Daisee"
github:              blah/blah

Now although github is extremely popular in opensource it’s of course not the only choice, One may put it on a different host, such as bitbucket, or host their own source code with gitlab.

There is no doubt that stack supports specifying an alternative, however it is unclear how to replace the github field properly with alternatives. The assumption was that the gihtub field was just a shorthand to fill in some other fields. It is hard to find what it does because it’s an odly specific question, but it turns out that hpack is used to parse the package.yaml file. Perhaps the git field is used to replace github? It is unclear. It’s also ironic in that we can immediatly observe the difference strong types make in documentation ability, this yaml file is not strongly typed.

Commit dependencies

In certain situations one may need to change a library. The typical way of doing this is to make a github fork, and then specify in the dependency manager, that a specific (git) url needs to be used on a specific commit hash. For example: If one tries to get the 56e833 commit from the voicebase library, you should record the hash and url somewhere.

If we google for the query stack.yaml git dependency, you just need to know that you configure this in stack.yaml, which is completely unrelated to package.yaml google points you to an outdated doc page which fails to work. A helping stackoverflow gives the reason why it doesn’t work, the syntax recently has changed and Google hasn’t updated yet. One can hardly blame stacks maintainers for this of course. But it still impacts a users’ experience.

Conclusion

Dependency management in Haskell is complicated. Even if one is able to become productive in the langauge, any of the problems described here could still make it difficult enough for them to give up on the system they want to build. Learning about functors and applicative is fun, learning what specific syntax to use to make stack pull in a git repository is not.

Reddit poster plugin for pelican

2018-05-07T09:00:00Z

Yesterday the Reddit Poster plugin for pelican was finished. This is an initial step towards providing Reddit integration with pelican.

What this plugin does is look a predefined list of subreddits names in an article, then it posts the article to all those subreddits. Aside from the subreddits in the list, there is also a ‘collection’ subreddit where all articles posted too.

Another goal in the future may be the ability to fetch comments from reddit at compile time and display them below posts. However the decision was reached to reduce the scope of this project to just posting first, as there was not enough time available to make fetching work. It is hoped however that many people now will start reacting to these posts so that it may be useful to implement this ability too. That will however be done in a separate plugin.

This project has been lingering in Jappie’s TODO list for years now. It is a great relief to finally make some progress in it. To advertise this website now the only thing that needs to be done is specifying the list of subs in the relevant article.

Starting at Daisee

2018-04-11T20:30:00Z

Recently quite a few changes have happened within Jappies’ life. First of all, he is no longer at OpenLearning. This relationship ended at the end of February. March was used for job searching. To help with this a proper portfolio was constructed. Last week there were three interviews with Daisee, and Saturday Jappie accepted the offer.

Bio

For this new job Jappie was asked to write a short bio, which he thought would be a nice addition to this blog post as it explains his current situation quite well:

Jappie joins Daisee as Software Engineer, he is quite excited about working with Haskell and practical AI.

In his previous role Jappie was a Full stack developer at OpenLearning. Building various systems, from spam detection to remote code execution within containers. Before that he just graduated from his MSc in AI from Utrecht University, he chose to travel rather than taking up an offer for a PhD position
because he didn’t felt much for a career in academia. The decision for doing a masters in AI was made during his time in China where he did an internship for his bachelor degree in Software Engineering.

Arriving here on a `working holiday’, Jappie likes it here and hopes he can stay longer. The weather is better, the meetups are interesting, and it’s quite satisfying to build up a life starting from almost nothing. Jappie enjoys building things, from programming project to virtual kingdoms in computer games.

Expectations

Daisee and Jappie found each other at the functional programming meetup. They announced it as doing work with AI, and Jappie happens to be interested in both functional programming as well as AI.

During the interview it was explained that initial tasks however were of a more practical nature. Such as devops and implementing a better UI. so there isn’t much AI work to begin with. However Jappie expects that after a while there would be more tasks in either of that.

First day

After finishing his first day Jappie looks back with satisfaction. The CTO takes a much more active role than at his previous company, which is pleasant because now it feels like there is direction. His colleagues seem nice too, all day people were coming over to introduce themselves to Jappie. His desk neighbor is also nice and very knowledge-able. Apparently a hardened Haskell veteran and a fellow Gentoo user/victim (rip Gentoo).

Then there was the coffee machine. At his previous job the CEO didn’t like coffee, so didn’t invest in it. In the new job however there was a full blown espresso machine. This machine is both a blessing and a curse, it’s a blessing in that now one can drink all he desires, but a curse in that addiction will follow. Not to speak about the impact it has upon ones’ sleep cycle.

Thinking about the introductions, some of his colleagues had very impressive track records indeed. For example, one of the founders apparently led a merger between two big companies. At the time it Jappie didn’t quite realize how difficult doing such a thing is. Jappie thinks it’s a good thing he’s surrounded by veterans. He can learn a lot from these people, just by being around them.

Flask, docker and the backend

2018-03-11T00:00:00Z

Jappie Klooster is working with friends on a react native app. It was attempted to do this completely without server-side with help of Firebase. This post describes the thought process behind not using Firebase for everything, and setting up a custom backend instead.

The first major issue, for this use case specifically, is the type of database provided within Firebase. It’s a NoSQL database that promises to sync between all devices. They probably mean syncing lazily, it’s implied in the docs, but there is no definitive statement around that. NoSQL is the idea that data doesn’t need relationships. For the core business logic of our app, we need relationships between data, and we need a lot of them. Technically one can represent these within NoSQL, but now you need to solve problems which most SQL databases have already done for you.

Another big disadvantage of Firebase is that it’s proprietary. Once a large chunk of the business logic, and data resides inside the Firebase ecosystem, it’s hard to get out. This means it’s relatively easy for the owner of the proprietary system, Google, to crank up the price. Neither Jappie nor his collaborators wanted to

Language choice

The primary question imposed for deciding which language to use was “How fast do you want it?”. Fast. Therefore a host of options was eliminated, such as Haskell and Rust. In fact because we wanted a pan-team familiar choice, only two real contenders were left. NodeJS and Python. The choice of Python was made because it is a much more simple language than JavaScript. Another advantage of choosing either for these languages would be that our editors are already setup to handle them.

Docker

Many members of the team stressed the importance of containerization. Jappie, being the only one that had worked intimately with this technology was quite happy with this development. What was rather surprising was the suggestion that the team ought to run these containers by hand on rented VPS’es. Is docker not supposed to make orchestration more easy? Do there not exist many tools that build on top of docker to provide variances in orchestration? Such as docker-compose, docker swarm, appengine and Kubernetes? All these questions entered into Jappie’s mind, but he felt arguing in this case would be difficult wihtout an example. Once the team would start working with the containers they would see for themselves how easy orchestration is. The primary building block was in place anyway (docker), moving to another orchestration methodology will be easy.

It was here that Jappie decided to pick up this task. He knew about docker from previous experience already, and his other major task had just finished. He started with a little hello world in python served through flask. The he put this in a docker container by writing a Dockerfile, and finally he wrote a little docker-compose script to attach an idle Postgres container which would be need later. All this was done in the space of about an hour or two. Setting up docker isn’t difficult. Then it was time for the next step, connecting flask to Postgres.

Flask

Flask is a micro framework. It gives some Rest support, and that’s about it. They sometimes provide some glue scripts that connect some projects to other projects, but there is no main monolithic architecture behind it. In stark contrast to Django, which has already done most things for you.

This micro framework setup, gives the project a big advantage in that they can have independent versions, and gives the capability to swap out some projects for others quite easily. The big disadvantage, as Jappie encountered, is that its fairly challenging to get into. One may even say it’s hard to get started. The reason for this is that gluing the ORM to the framework has to be done by hand. This is of course because people may want to choose a NoSQL storage mechanism rather than an SQL. Jappie wanted to have an ORM to handle migrations and as a method of describing the model in.

The thing that ended up working for Jappie was this bootstrapping script which was about 5 years old and of course outdated. But Jappie managed to get it running, by googling errors, and stripping features. This should make it easier for future projects to start with flask in a docker environment. A pull request was made to merge it back into the original project.

Deploying

Once the docker-compose setup was completed considerations went out how to deploy this live. Being able to setup a live instance early has the advantage that the team can test the entire (rather complex) infrastructure. AppEngine was chosen as method of deployment as it could save in costs. AppEngine essentially allows client code to be shared across Kubernetes clusters (one assumes), making it cheaper for small apps to use as the overhead of running Kubernetes itself does not need to be taken in account.

Since Jappie had never worked with AppEngine before he chose to do the hello world tutorial first. Note that he had to choose the flexible environment as the standard one only provides python 2.7, which at the time of writing has a bit over two years left until deprecated. Using python 2.7 is therefore a really bad choice. Even though google says python 2.7 is the standard, Jappie urges his peers to take this claim with suspicion. Which the python wiki backs up:

Python 2.x is legacy, Python 3.x is the present and future of the language

After the hello world was successfully put online (in python 3 with the flexible environment), it was time to try setting up the entire system. For this a managed Postgres needed to be setup on Google Cloud and connected to AppEngine.

Google however seems to favor more popular technologies, and the primary docs assume you want to use MySQL. Postgres was the preferred choice for the team however. It is more standard compliant and has better data types. An Uber post however claims there are some interesting advantages to MySQL if you manage the architecture yourself. This is done by google however in this use case, besides because an ORM is used, switching database technology is relatively easy.

The search engine doesn’t show the docs for Postgres with flask in AppEngine. These docs were found in the example project. The source file had a link to using postgres with AppEngine. To test if the Postgres instance was running this guide was used. Jappie was uncertain why a cloud proxy was recommended this much, in his memory he hadn’t set this up last time with a Django project. However with that in place one can create a specialized user for manipulating the database (rather than the root Postgres account).

Going trough those tutorials the useful commands were collected into a makefile. With this in place he had no longer need to refer to them and could just cat the makefile to see how to deploy for example:

deploy:
    SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://USER:PASWORD@/DB?host=localhost python ./manage.py db upgrade || (echo "make sure the proxy is running" && false)
    gcloud app deploy

describe:
    gcloud sql instances describe DB_NAME

proxy:
    cloud_sql_proxy -instances=INSTANCE_PATH=tcp:5432

connect:
    echo "warning, requires you to setup the proxy in another terminal"
      # psql "host=127.0.0.1 sslmode=disable user=postgres"
    psql "host=127.0.0.1 sslmode=disable user=USER dbname=DB password=PASSWORD"

Now any of the team could deploy with make deploy. Note that the credentials where retracted out of security reasons.

Conclusion

There is a lot involved to setting up a backend. It took about two days going from knowing almost nothing about flask and AppEngine, to having a hello world. Not even taking into consideration all the previous knowledge Jappie already had acquired with docker, Postgres and python itself. Note that at this point we haven’t even made anything of interest. This post was about setting up infrastructure… It is to hope that this story may help some other hackers, so that they can save a few days.

New new website style changes

2018-03-08T00:00:00Z

Due to circumstances in his life, Jappie Klooster decided it’s time for some more website changes! These are all intended to simply reduce the amount of time it costs maintain this website. This should allow Jappie Klooster to update it more regularly as the friction for updates is decreased.

Brutalist style

Jappie Klooster became unhappy with the style, although it entertains him endlessly, it may give the wrong idea to a third party. To untrained eyes it may look like he attempted to make it look good (which wasn’t the intention at all). So rather than trying to make it beautiful, Jappie decided to go in full brutalism mode. In essence all style except the most practical ones will be stripped. For reference:

old

As the reader see the old style was rather dark themed. It used the linux kernel compilation program as a reference, as discussed in another post.

new

In the new style Jappie just started deleting html and CSS untill he came accross this, it’s not as brutalist as initially promised, but looks a lot more attractive and was a lot more simple to accomplish.

Jappie intentionally put the image of the new style in here, even though it’s the current style. It is assumed that this style is not permanent either.

Getting rid of org mode

The believe is still held that org-mode is a superior standard compared to markdown. It can do so many things, such as inline code execution and pipe the result to the document itself, as having full reference support. These are just some of the many great features of org-mode. However org-mode has issues. The fact that it’s entirely confined to Emacs for HTML rendering is a massive problem. Emacs needs to be booted for each article just to do the HTML rendering. There is no easy way Jappie Klooster knows of to keep this process active as a daemon somehow. Technically Emacs can do this, but it is meant to do communication with XORG, not to act as a script execution platform.

In the future Jappie Klooster may do investigation to make this work fast, however for now the easiest way of increasing building times for this website is to just use the markdown export feature of org-mode. The reason he chose org-mode in the first place as a viable alternative is that it doesn’t try to lock you into it. He now executes this option, because there is simply no time to investigate the preferred route.

Plan prediction

2017-12-21T12:00:00Z

For the raster project a main selling point will be the automatic prediction of future scheduling. There are two major schools of thoughts to go with that I know of. Constraint satisfaction solving and data driven approaches (use statistics).

First of all the most straight forward approach is to use something like OptaPlanner. In this approach constraints are laid upon the problem, such as 2 cooks need to at least work on Saturday, but preferably 3. This single constraint has two different part, the hard minimum, and the ‘soft’ preference. With these kind of rules in place, the OptaPlanner can use heuristics to figure out who to plan when. You could add as many constraints as you want, such as have less preference for when employees asked free or let students not work during the day.

This is a very precise approach and almost surely will work, as other systems have demostrated already. However the issue with this approach is the actual specification of rules. This is quite difficult to do, so for a fact I can’t push this onto the company owners. What I could do is create some kind off helping GUI around it. Or simply use generic predefined rules and if owners want something more sophisticated they should contact me. On top of that the OptaPlanner system I know of is written in Java, which is kind off problematic since I now need to develop some micro-service architecture around it.

Here is started considering the second approach. The idea I had is that I use the already entered data by the owner to create a statistical model of how he likes to make his plannings. Then we use this model to make future predictions. This requires zero configuration by the owner, and also doesn’t have the language interaction problem as it’s just some calculations. What I imagine is just adding a predict button to the current roster screen that uses the statistical model to magically create a roster.

Now the method I thought of is simple, it’s an adaptation of q-learning, which uses a lookup table for getting action probabilities, if successful probability is increased, otherwise decreased. I’m doing something more simple, here is the gist of it: We count per day how many job types are planned in on average over n weeks. Then we plan in that amount of job types, for remaining fractions we flip a coin. If someone has asked free we remove him from the potential pool of workers. If there are not enough workers for a job type we just throw a warning for that date and let the owner figure it out.

The beauty of this technique is that it is simple, but it takes a lot of work out of planning in people. If this is well received there is tons of room for extension. For example naive Bayesian networks can be used to take into account ‘experience’. If we add a reservation datasource somehow prediction could become even more accurate by using it as another influencing factor. Whilst the basic interface remains the same, press the ‘predict’ button and magically you get a roster.

In conclusion I will attempt a statistical approach rather than OptaPlanner, because it’s more simple. In the future I can still choose to experiment with OptaPlanner. It would be much more interesting to use that, once I know how to register demand accurately, for example the amount of reservations for an evening would be a good indicator for restaurants.

Jakartian Christmas

2017-12-19T21:00:00Z

Unfortunately Jesiska couldn’t make it to Australia in time for Christmas. Therefore I decided basically last week to go to Jakarta instead! I personally don’t care much about Christmas, but I thought the office would be closed during this time. (Many companies close in the Netherlands for sure during this time), and I was just very eager to see Jesiska again.

I’ve been neglecting this blog a little bit lately, I’ve just been busy, well actually work is a huge time sink.. it sucks. Then I’ve also been trying to increase my income with various project I will discuss next. Although I’ve been neglecting the most important one which is the financial one. It just seems so boring every time I think about it. I need to do this soon because my bonus interest of my savings account will stop at the new year. This holiday conveniently overlaps with that.

The raster project core is up and running. Basically anyone can sign up to it and take advantage of it for free! Because I haven’t setup payment yet! Because many payment providers ask tax details! Which is difficult to do in Australia!.. I need to go back to the Netherlands sometime to finalize this part. Meanwhile I’m letting my father test it so I can get some feedback. He was genuinely quite excited about the system. This is quite a help because there are just details I don’t think about, such as making sure a week always begins on Monday (rather than it just being whatever day your on), or making the grid clickable so it’s quicker to plan in. etc.

In the weekend I was hacking with some colleagues on a little app idea we had. This is totally different than the raster project, and I get to work together with some other people. Which is nice cause most of my projects tend to be alone undertakings. Not sure if this going to become anything, although it is ambitious, but we’re keeping it secret for now. What we will do is try and share technical ideas and contribute to the open source community.

I’ve also been considering applying for permanent residency. I looked into that and apparently to get enough points I need to take an English test. Seems a bit ridicules considering that I’ve done my masters in English, but that’s not the worst part. The English test would cost 300$, and there is no guarantee that I get an high enough score (although I can’t imagine why I wouldn’t, except for my dyslexia).

What I want most of all is too see Jesiska again. This situation sucks even with all this stuff happening, it’s pointless without Jesiska.

Jappie lives with kangaroos

2017-11-06T00:00:00Z

I’ve been living in Australia for well over a month now and I really enjoy it! The first two weeks were quite a wild ride, looking for my first job and finding a permanent place to stay.

I had 3 interviews appointments prepared before arriving. I finished them in the first week and managed to get 3 offers. I just choice the one that looked most interesting and also happened to pay the most!

Finding a house was a whole other level of insanity. I think I’ve been in contact with a scammer, that offered an apartment in the middle of the business district for 900$ per month (900$ per week would be more realistic). All I had to do is deposit some money in a bank account and he would surely sent the keys. In another situation the rent agent wanted me to deposit money into his account for the deposit without signing a contract! What are these people thinking? Rental prices are trough the roof too, I’m paying 330$ per week right now.. That’s about the same amount what I’d pay in the Netherlands per month!

Things seem to have settled more down now. I have this pattern of a morning training, going to work, go to home and put food in the microwave which I prepared over the weekend.

I quite enjoy the job too, although it’s a lot less programming than what I expected. A lot of project management is involved too, and QA is super slow.

Meanwhile I’ve been working on a little side project to try and increase my income. It’s http://raster.click, a roster system aimed at restaurants. This is basically re-doing an idea I had when I was younger, but this time I set it up as a service rather than selling the source. This allows me to keep working on it if it becomes adopted by a reasonable amount of people.

Finally I’ve been trying to come up with a way to invest all this money I’m earning. The banks are much nicer in Australia because you get actual interest. However It’s too low in my opinion. I’ve had several ideas ranging from making an automated trader to start mining crypto currencies.

Sadly Jesiska hasn’t arrived yet. I’m not even sure if she will before Christmas. It seems to take ages and while I enjoy living like this, it’s not the same without Jesiska. I actually wish I would’ve stayed longer in Indonesia at this point. Oh well we’ll see what happens next.

Work work work, ya ya ya

2017-10-02T00:00:00Z

Last Wednesday I started my first job at Open Learning. It has been interesting, I know for a fact I contributed almost nothing to the company up to this point. This is to be expected according to reddit since it’s the first week. I did ask a lot of questions since I was able to start programming for real yesterday.

The first today I just spend on getting familiar with the platform and the last day I dabbled around a little in react (hello world!). The reason I have for saying I did little contributions is because I just took up more time of other employees asking questions then producing much. Even the UX work hasn’t been integrated yet because the designer was busy.

What I didn’t expect however was having a good time over there. I enjoy talking with the various people and the tasks aren’t that bad. There is also a pantry, free fruit and some tea I take advantage of. For lunch we usually go out eating but I’m considering taking lunch as I wanna safe some money especially this first month.

I noticed a lot of discussions where about food, actually the people at the office seem genuinely really interested in food. Which is good because I like trying new foods. Unfortunately Australia doesn’t seem to have much of local cuisine, but they’ll happily import dishes from neighbouring countries (which are delicious).

What’s even better is that they allow you to work from home. At this point I can’t really do that yet because I’m a newby and ask a lot of questions, but at some point I also maybe able to do that. Once per week having no travel times would be very good!

Aside from asking to many questions (in my experience) the only other big issue I have is leaving times. I’m not sure how they decide upon that. Its always kind off awkward for me to leave the place, but on the other hand I also don’t want to do (free) overwork. I think I’m gonna ask about these things because they maybe an issue later.

The place is nice, the things they’re doing could definitely help society and I enjoy the work (not so much the UX testing part, but it was necessary).

What do you think about jakarta

2017-09-28T00:00:00Z

Now I’ve finally arrived in Australia, its time to reflect upon my expectations I had for Jakarta. Actually.. While being in Jakarta I got asked this question quite frequently: “What do you think about jakarta?” and I just stood there with no answer. This post is an attempt to answer that question, and at the same time I address the expectations post I made before.

For Indonesia I’m expecting the population to be better at English then the local Chinese where in China.

People definitely where better at English than in China, at least in my experience. I had several taxi drivers that could speak some English and even the locals know or try some words. In China only students ever tried to speak English, other people just didn’t want to or couldn’t. I suspect this is in part due to cultural difference. However for China the great Chinese firewall and systematic cultural isolation (that is in part by design) helps to marginalize exposure to English for sure.

In China learning English is something which parents are willing to pay good money for, but Indonesians are willing to go much further with international schools. All day to day conversation happens in English in these schools and the entire education system of Singapore is blatantly copied. This turns out in little children walking around malls speaking in an almost perfect British accent with each other. This was such a weird experience to hear perfect British by a 10 year old’s with Bahasa (Indonesian) all around you.

The schools are a little weird, apparently there coincide 3 different major systems, Indonesian, Chinese and Singapore(/American). However once I thought about this a little I realized we basically have the same in the Netherlands, for example the ‘free schools’ or ‘open schools’.

My grandfather actually told me many Indonesians would be capable of speaking Dutch. I seriously doubt that

I was totally right by the way about nobody being able to speak Dutch, I only ever met one person who said something in Dutch to me and it was done in a way where the words didn’t make much sense. They said ‘nice to meet you’ in dutch, without introducing themselves first.

The weather will be hot in Indonesia, at least in Jakarta and there will be lots of rain.

The weather wasn’t that hot actually. Most of the time I was quite comfortable in the Jakarta heat. There was also practically no rain, although one time there was a thunderstorm, which killed the electricity at the place I slept. This caused me to sleep in the heat which is a terrible experience. After this I realized that people in the slums sleep trough this every day. I suddenly felt very bad for those poor souls living there. They probably never get a good night rest.

Surprises

What really surprised me though was the food. If you’re a fan of strong flavoured foods Jakarta is a great place to go! Another thing I didn’t expect was the insane amount of malls. They just casually build three malls next to each other, and then fill it up with schools, fitness clubs and travel agencies, aside from the stuff you’d expect such as shops and restaurants. A reason for the popularity of malls may be the security that keeps out street sellers and the AC which makes walking around there much more pleasant.

Another really nice thing was the thousand island trip. So this is basically a boat ride of about an hour to one of the islands on the coast of Jakarta. However it’s really similar to Aruba for about a tenth of the cost to get there.

Okay so here we’re kind off getting out of Jakarta, but what I really really recommend is the night safari. You’re being taken around the zoo in a little train, although it goes a little to fast to experience well, and can’t take pictures for sure. However it also includes a theme park and taking pictures with baby animals (need to pay a little for the animals but it includes the opportunity).

What’s really strange to see is that some people, have as job, guiding traffic but they’re not employed by government. They just sort off start guiding traffic on their own and then live of the tips. A similar kind of people do the same for parking, they’re just living parking meters, but they also help with getting in and out and making sure you park straight. As a consequence all cars are parked really nicely.

Finally another peculiar thing is that there are so many people self employed, it’s ridiculous if you think about it. Little shops everywhere, the traffic people, parking people etc. All self employed. I think this is a major difference from the developed world, where you generally start working for a boss because it just pays that much better. But with those low wages, it’s just that more attractive to start for yourself.

So what do I think about Jakarta?

I think that Jakarta is like an uncut gem. It already has some really attractive attributes, but until issues such as the crazy traffic and drinkable tap water are addressed it isn’t as good as it could be. Still it was an amazing experience to be there and I’d definitely come back for the food alone.

Beginning of eternal summer

2017-08-16T16:00:00Z

Right now I’m in the airport waiting to go to Jakarta. It’s really weird what I’m doing I think, few people actually travel alone by plane, but in my case I’m just sort of emigrating and finding a job in a foreign country.

I want to try and do an eternal summer. What this entails is basically following the pattern of migration birds, once winter come you move south, where it’s warmer. In my case, when dutch winter comes, I’m going all the way south to Australia, where winter just is about to end and summer is about to begin and when Australian winter comes I go back again. This probably won’t work if I get a good job in Sydney. However getting a programmer job right now looks like a big hurdle.

I listened to freakonomics that told me most of the population is over educated, which means that people with master degrees are competing for entry level positions, and people with bachelors are again competing for even more lower level positions. IE, it’s crowded at the top, which pushes competition downwards. This seriously disturbed me, because I started doing this too. For example, I started replying to junior positions because it looked like nobody reacted to my other letters. I must have applied to over 50 positions right now. Being a foreigner also doesn’t help I think. At this point I would love to do the low level trainee-ship (Java 101) they offered me in the Netherlands except then in Sydney. I still hope this can be done, like at this point the economy is growing again right? People need more programmers right? Nope, I think the job market tends to lag behind economic numbers.

However when I arrive in Indonesia I’ll be doing a holiday first. Not only that I’m expecting to work a little on the web shop system I’m building, and maybe even do some applying. Of course just in the evenings, when I’m not doing cool traveling stuff.

I’m also thinking about tracking the stuff I’m doing in Indonesia holiday a little on the blog in a journal kind off style, I know nobody is reading this but I like writing this. Directly encoding images etc, would also be nice.

Maybe I’ll also add my Chinese journeys to the blog. I at least have the journal of my second voyage in the bag. which I don’t have yet digitally. In the passed however I usually started full of enthusiasm with recording but after a while I got bored of it. Perhaps I should take it more casually and just spend a limited amount of time on it. Actually that maybe a good idea because then I’ll only record interesting stuff. And it help with motivation because I’ll think “Oh these 20 minutes are over quickly”.

Journey to Oceania

2017-08-07T22:00:00Z

Wednesday 2017-08-16 I’ll leave towards Oceania. First I will go for a month to Indonesia, having a little holiday and meeting my almost girlfriend. Then I will move on to Australia to do a “working holiday”, where I hope to do my profession, as a software engineer or work in AI (think either data mining or machine learning).

This is the second major journey I undertake, which in my case means being away for more than three months. This time I’m expecting to stay away for at least six months, depending on if I can get good work and a visa. Perhaps I’ll write in a later blog post about the first journey I undertook to China, where I had a great time and which motivated me to leave again. In this blog post however I’ll record my expectations for this journey, so that one day I can look back and see what a fool I was.

For Indonesia I’m expecting the population to be better at English then the local Chinese where in China. A Chinese man can say with quite some confidence he’s not going to encounter many foreigners in his life, in fact I had people tell me this when I was there, Which is in stark contrast to Dutch mentality where you expect to encounter many foreigners and thus you’d have to learn at least English and either German or French (all three also is common). I also expect Indonesians to be willing to learn more foreign tongues, because they don’t have a common tongue such as the Chinese have with Mandarin (in fact there are thousand of languages in Indonesia), which means they’re expecting to meet people that don’t know their native language, and thus English will be used as lingua franca. especially on Bali island everyone will know English because of tourism and also in Jakarta, but to a lesser extend.

My grandfather actually told me many Indonesians would be capable of speaking Dutch. I seriously doubt that, and my (almost) girlfriend has confirmed this is only true for older Indonesians. My Indonesian girlfriend has also warned me life is slower in Indonesia than in China (which was already pretty laid back). By this I think she means that people aren’t too strict with appointments and that we can expect to wait for stuff to happen.

The weather will be hot in Indonesia, at least in Jakarta and there will be lots of rain. I know this because we talk often about the weather and I’m pretty sure it rains there more. Another environmental thing is that there will be lots of traffic in Jakarta, and especially many scooters and motorbikes. Probably similarly to the amount of Guilin, actually the climate maybe also similar to Guilin summer times.

Australia is a whole other beast of course. Everyone will speak English there and I’ll probably communicate quite easily. My grandfather thought I’d have some problems with Australian English, but I doubt that. I mean perhaps I have to get used to it for a week, but that pales in comparison to the efforts I had to take for learning the heavy accents of mainland Chinese people.

I do expect Australians to be incredibly polite and somewhat more laid back than Dutch people, however still ‘faster life’ than in China and Indonesia. By politeness I mean, trying to keep a kind off distance from strangers and embezzling your words so they sound ‘better’. This politeness of course breaks down between friends.

I’m expecting Sydney to be dominated by cars. Like cars to the extreme. Whereas in the Netherlands we use bicycles, you can get anywhere by car in Sydney. I’m expecting Sydney to be about the size of Xiamen. Although much less dense (because of the cars needing their space). Sydney will probably have about the same weather as the Netherlands, laying a lot farther south than Indonesia, although with the seasons mirrored of course.

My friends also told me Australia is a very deadly place, where everything tries to kill you. I think that’s highly exaggerated, mostly because people tend to put as much distance between themselves and nature as possible, especially in cities.

It’s a little short but there is it, a quick overview of my expectations for these two countries. Probably full of stereo types, but this is why I do the journey.

Travel is fatal to prejudice, bigotry, and narrow-mindedness, and many of our people need it sorely on these accounts. Broad, wholesome, charitable views of men and things cannot be acquired by vegetating in one little corner of the earth all one’s lifetime. – Mark Twain

My thesis

2017-07-07T00:00:00Z

I’m done. The master in AI is complete. I finished the thesis about adding `Jungian personality in a chatbot’. In this blog post I will give a summery and some comments from a programmers’ perspective that just don’t fit in a thesis. For example, the implementation are not at all important, but I like talking about using Java in a functional programming style and the pains this caused.

The source is available here, for anyone interested. The full text is available here (and thesis source). The presentation I did can be found here (and it’s source).

I start by discussing what the thesis was about in a summery (for those who don’t want to read about 80 pages). after that I’ll talk about the implementation details (I just want to lay my egg). Finally I will give an overview of the experience, including the defence (for those who didn’t do a master, it maybe interesting),

What did I do?

I build a very fancy chatbot called Gaia, which is not based upon ALICE. The core is a business rule engine called Drools cite:vcimbora2015usability. Once the scenario described by YAML files are read, everything chatbot related happens in Drools. This means you can make it as smart as you want, for example you can make a rule that detects when a user is starting to repeat himself (and deal with that), or use timers that are native to drools to create a bot that acts impatiently. Pattern matching also happens inside drools, which means your not stuck to the regex system I provided. Unlike the ALICE chatbot where you have to use the ALICE patterns (which are less power full than regular expressions).

That was made to just to get started with personality. The issue with the ALICE chatbot was that it already predefined a response before any deliberation could happen. What I did was simply separating the `figure out what user said’ step from the `this needs to happen next’ steps. To make this separation more clear lets look at the syntax of both systems. AIML, which is the core of the ALICE chatbot looks like source block code:aiml. A diagram representation of this block can be seen in figure fig:dep:aimlcats.

AIML syntax for a reply

<aiml>
    <categroy>
        <pattern>
            Hello
        </pattern>
        <template>
            Hi
        </template>
    </categroy>
    <category>
        <pattern>
            How are you
        </pattern>
        <template>
            Not doing too well today.
        </template>
    </category>
    <category>
        <pattern>
            How * you
        </pattern>
        <template>
            <srai>How are you</srai>
        </template>
    </category>
</aiml>

Now let’s look at the new representation I came up with in source blocks code:yaml:greeting, code:yaml:how, code:yaml:not and code:yaml:connections. Of which I made a diagram in figure fig:dep:aimlsyms.

Gaia yaml greeting.yml file

literals:
  - Hi
  - Hello

Gaia yaml status_inquery.yml file

literals:
  - How are you?
patterns:
  - How * you
  - How are you *

Gaia yaml reply_bad.yml file

literals:
  - Not doing too well today.
regexes:
  - doing (*.) badly

Gaia yaml _connections.yml file

from:
 - greeting
to:
 - symbol: greeting
 - symbol: status_inquery
---
from:
 - status_inquery
to:
 - symbol: reply_bad
 - restricted_to: patient

The new representation looks quite verbose too, but it does much more modelling than the original. Additionally, we separated the concerns of legal connections, from pattern matching. Modelling the scenario now almost entirely happens in the _connections file, and while doing that you don’t have to deal with pattern matching. Using the file names as identifies guarantees uniqueness.

We can model complicated examples such as in source block code:yaml:complex. This can be seen as a diagram in figure fig:dep:filedconns. Trying to put such an example in AIML is basically impossible. First of all the concept of actors doesn’t exist, secondly categories can’t model the availability of choice. There are if statements, but that’s making the decision in place. Aside from the fact you shouldn’t do conditionals in xml structurally.

Connections grouped into a file

from:
 - greeting
to:
 - symbol: greeting
 - symbol: ask_reason_here
   restricted_to: doctor
---
from:
 - ask_reason_here
to:
 - restricted_to: patient
   symbol: need_medicine
 - restricted_to: patient
   symbol: broken_arms
 - restricted_to: patient
   symbol: feel_sick
---
from:
 - need_medicine
 - greeting
to:
 - restricted_to: doctor         
   symbol: why_need
 - symbol: status_inquery

The personality stuff

With the availability of choice in place, I could do the personality stuff. Jung’s theory is used for personality to decide what the algorithm should use, this is also the core theory of for example MBTI. Jung said that each function has an attitude, either introversion or extroversion. Introversion deals with the inside world, memories and ideas. Extroversion deals with the outside world, which can be seen. An overview of the function can be seen here: 𝒥 = {T_e, T_i, F_e, F_i, S_e, S_i, N_e, N_i} Each of these does something different, for the entire description I refer to the thesis or this source cite:hall1973primer.

What we wanted is that these functions would plan ahead in cooperation with each other. This would be personality as a process rather than value based cite:campos_mabs2009, this was a requirement by my teacher. To do this we introduced the dialogue tree data structure: u = (a, s) D = (u, [D])

Description of symbols
/	<>
u	Utterance
a	Actor
s	Symbol
D	Dialogue Tree

Where u is an utterance, a an actor, s a symbol and D the dialogue tree (see table tab:dialoguetree). With this data structure we can plan ahead, each node is an utterance made that can have multiple possible responses (see figure fig:dialoguetree). What we then pass this dialogue tree trough the functions either growing or sorting on preference. Each function in the personality can do modification, but the order of execution determines their `strength’.

We assumed that Jung meant that action generation was done by irrational functions, and preference ordering by rationale. What we did was giving all these functions the same type signature and then putting them into an order. This looked with the Haskell notation like the following: $$ \left (\overset{next}{B \to D \to (B, D)}\right ) \to B \to D \overset{f_a}{\to} (B, D) $$ The next argument allows us to encode a sequence of functions, however this was problematic because I was asked to make operation in between functions available to the drools rule engine cite:droolsdocs,vcimbora2015usability. We ended up with a hybrid approach where the functions were stored in a list and drools parsed them, but they could also be composed. Actually if I could change anything of the thesis it would be this part, it’s kind-off messy right now, but I simply didn’t have any more time left to figure this out properly.

How this looked in java can be seen in figure fig:jungjavaclass. The core is the enumeration of Jungian Functions, they all have the same type signature with JungFuncArgs as argument and result. These arguments can be modified by the functions and they can use apply next to apply the next function in the sequence to the arguments. This is only part of the story, not telling about how drools rules deal with the functions in order, but they are simply functions with as input JungFuncArgs and as output. Which means they are endomorphisms. I was tempted to put that in the title, because it sounds impressive, but then I realized it’s just a minor part of my thesis, and I think that part is messy.

Steering

To steer dialogue two major methods are used. Feeling functions use perlocutionary values as directions, which is based upon speech act theory cite:shoham2008multiagent, and as an example can be seen in source block yaml:values. The numbers used per perlocutionary value can differ per agent, their names can be attached to connections, see source block yaml:values:connections.

Values in believes.yml

values:
  enthusiasm: 8
  polite: 5

Value example connections _connection.yaml

from:
 - greeting
to:
 - symbol: greeting 
   values:
   - Polite
 - symbol: status
   restricted_to: patient
   values:
   - Polite
   - Enthusiasm

Thinking functions go primarily towards goals and can be seen in source block yaml:goals. What we do is marking that we want certain symbols to be uttered by a particular actor. In the example the patient want the doctor to utter “Have some painkillers”. Goals are entirely encoded in the believes.

Goals in believes.yml

goals:
  - actor: doctor
    scene: diagnoses
    symbol: have_painkillers
  - actor: patient
    scene: information_gathering
    symbol: back_pain

To encode the personality we simply specify which Jungian functions we want in what order, see source block yaml:personality. In the thesis we specifically used MBTI cite:website.mbtitypedynamics as a guide line, but the PPSDQ cite:kier1997new,king1999score and SL-TDI cite:arnau2000reliability can also be represented like this. Although some work needs to be done to add scalar values they require.

Personality in believes.yml

# ENFP
personality: [Ne, Fi, Te, Si]

Finally we need to specify all actors, in case a connection didn’t specify which actors are available, and we need to specify which actor the agent is. We need to do this because we model both sides of the conversation, so actors need to be specified explicitly, an example can be seen in source block yaml:actors.

Actors in believes.yml

self: patient
actors:
  - patient
  - doctor

With all of this in place the varied personalities can go over different modeled paths. Which is sort of what my thesis was about I guess. We did not specified values (unless you count perlocutionary values and goals), and the personality process will figure out what paths to take.

Crazy programming stuff

Ok ok, so now we have some context we can go to some of the more interesting parts (to me at least). I wasn’t allowed to go into the details of the programming techniques I applied, but boy did I do some interesting things.

To bring you in the mood let’s sketch the environment, I’ve been doing a lot of Scala, some Haskell and Rust before I started working on the thesis. The Salve game was written in Java, so guess what style I used for this typical Object Oriented programming language? Pure Functional! By this I mean that aside from local scope mutations, the entire structure was immutable. Take for example source block java:immutable. We need to make the collections private because Java collections are mutable. There is no need for the name and scene attributes to become private because they are already immutable, so they will never change. We made hash_value private, even though it’s immutable, because code shouldn’t depend on that. This is a core principal of the code base, make everything immutable even though Java doesn’t really cooperate with that.

Immutable example

@Immutable
public class Symbol {
    public final String name; // filename
    public final Scene scene;

    private final List<String> literals;
    private final Set<TemplateAttribute> requiredTemplateVars;

    private final int hash_value;
    ...
}

Ironically enough I undo this with the builder pattern in the unit tests. The issue is that immutability in Java is quite verbose to do, and I wanted a nice api to setup my the current dialogue on which I wanted to test the functions.

I also wanted to have a good api for modeling the scenario from java code in the unit tests, and especially for this one I think I’ve succeeded (see figure java:testapi). We either connect up with any actor, or a restricted actor, however as you may see the result of these functions both go trough the same method connect. We do this by using an Either type, which allows us to treat the same information kind off similarly for a while, and eventually on the right place we treat the cases separately. It’s kind off a delayed if statement. We can see the expansion of the if statement in figure java:test:either, this happens with help of the fold method, which receives a lambda per either path. Of course there are other ways to do this¹, but at the time of writing, I thought this was a really neat construct, because it’s precise and terse. I’m not sure if it’s a good or bad practice, but I think it looks interesting.

API usage of creating in memory scenario

public class MockBelievesFactory {
    ...
    public static final String hellos = "hellos";
    public static final String whyhere = "whyhere";
    public static final String maybeimsick = "maybeimsick";
    public static final String ilikevistingyou = "likevisitingyou";

    public static final String needmedicine= "needmedicine";
    public static final String imthedoctor= "imthedoctor";

    public final Believes createTestBelieves(){
        connect(hellos,
            any(whyhere, "Angry"),
            any(hellos, "Happy"),
            any(needmedicine, "Persuading", "Scary")
        );
        connect(whyhere,
            restricted(needmedicine, actor_patient, "Enlightening"),
            restricted(imthedoctor, actor_doctor, "Angry"),
            restricted(maybeimsick, actor_patient, "Angry"),
            restricted(ilikevistingyou, actor_patient, "Happy")
        );
        ...
    }
    ...
}

API implementation with either types

public class MockBelievesFactory {
    @SafeVarargs
    public final void connect(
        String one,
        Either<
            Pair<String, PerlocutionaryValueSet>, 
            Triplet<String, Actor, PerlocutionaryValueSet>
        >... values
    ){
        Set<Connection> connections = createConnections(values);
        setconnect(one, connections);
    }

    @SafeVarargs
    public final Set<Connection> createConnections(
        Either<
            Pair<String, PerlocutionaryValueSet>, 
            Triplet<String, Actor, PerlocutionaryValueSet>
        >... values
    ){
        return Arrays.asList(values).stream().map(tupple ->
            tupple.fold(
                pair ->
                createConnection(pair.getValue0(), actor_any, pair.getValue1()),

                tripple ->
                createConnection(tripple.getValue0(), tripple.getValue1(), 
                  tripple.getValue2())
            )
        ).collect(Collectors.toSet());
    }
}

Fancy tree traversal

In many ways this structure was the core of deliberation. The Jungian functions needed to make modifications to this structure, but I wanted it to be immutable.

To modify an immutable tree we need to pass a function down to the node where we want to do the modification and then apply it, once this is done we can go back up the tree with the new modified tree as leaf passing as a result the new tree. The function that does this is withPrefferdIfAtHeight in source block java:dialoguetree. In this example we make heavy use of continuations to make a really terse tree traversal (at least for java). The copyWithAboveLeftMostLeaf and copyWithStartAtUntilLeaf are the main clients of this function, however they just fill in the continuations.

Tree recursion with continuations

@Immutable
public class DialogueTree {
    public final Utterance utterance;
    public final Connection connectionUsed;
    private final List<DialogueTree> options;
    ...
    /**
     * If we have a preffered, execute withPrefferd on it, If we don't have,
     * execute ifNoPrefferedWithThis on the current object.
     */
    private DialogueTree mapPreffered(
        Function<DialogueTree, DialogueTree> withPreffered,
        Function<DialogueTree, DialogueTree> ifNoPreferedWithThis
    ){
        final Optional<DialogueTree> prefferedOption = getOptions().findFirst();
        return prefferedOption.map(preffered -> {
            final List<DialogueTree> options =
                getOptions().collect(Collectors.toList());
            options.set(0, withPreffered.apply(preffered)); // 0 being preffered
            return replaceOptions(options);

        }).orElse(// there is no first option
            ifNoPreferedWithThis.apply(this)
        );
    }

    /**
     * Generalization of 'copyWithStartAtUntilLeaf' and
     * 'copyWithAboveLeftMostLeaf'
     *
     * You could very easily traverse the tree with this if you attach whenNot
     * into the called function of the argument dialogueTree.
     *
     * Whenat will always be exeucted on the leaf.
     */
    private DialogueTree withPrefferdIfAtHeight(
        int height,
        Function<DialogueTree, DialogueTree> whenNot,
        Function<DialogueTree, DialogueTree> whenAt
    ){
        if(thisIsAtHeight(height)){ // in practice equal, but we just don't want stackoverflows
            // note return
            return whenAt.apply(this);
        }
        // we execute whenNot on preffered, because if we were at height the
        // previous condition woudl've been true
        // however if there is no prefered we are at leaf level.
        return mapPreffered(whenNot, whenAt);
    }

    /** go down until height, then keep applying function until leaf */
    public DialogueTree copyWithStartAtUntilLeaf(
        int height, 
        Function<DialogueTree, DialogueTree> function
    ){
        if(height < leaf_height){
            return this;
        }
        return withPrefferdIfAtHeight(
            height,
            tree -> tree.copyWithStartAtUntilLeaf(height,function),
            tree -> {
                final DialogueTree result = function.apply(tree);
                return result.mapPreffered(
                    prefferd -> prefferd.copyWithStartAtUntilLeaf(
                        height, function),
                    Function.identity()
                );
            }
        );
    }

    /** A more generalized form that can opperate on any height */
    public DialogueTree copyWithAboveLeftMostLeaf(
        int height, 
        Function<DialogueTree, DialogueTree> function
    ){
        return withPrefferdIfAtHeight(
            height,
            tree -> tree.copyWithAboveLeftMostLeaf(height,function),
            function
        );
    }
    ...
}

The tree traversal is extensively tested upon correctness by the unit tests aimed at the Jungian functions. This helped me a lot with coming with this design in the first place, because the unit tests would tell me if I did something different. I thought this example was interesting because of the use in continuations, I’ve never really done tree traversal like this aside from studying Haskell. I did find it really difficult to think of appropriate names for the continuation functions because they’re so abstract. At this point I also started to wonder, are these kind off levels of abstractions even useful? I mean dialogue tree traversal became in my case really easy , I would say yes. This only happened after I implemented all the Jungian Functions and did a refactor round with the unit tests in place came I up with this design. I would imagine most code bases not really wanting to go this far.

Graph duality

This piece of code lingers on the point of madness..

Chatbot works modularized pattern matching called scenes. When a scene is active we only match upon patterns of symbols in that scene, if there are no such patterns we look at the connections going out to neighbouring scenes and match upon the patterns of the symbols leading to those.

To do this we have two pattern databases, the first one for within the scene and the second going out of the scene. The entire code that construct these databases can be seen in source block java:patternprocessing. We can see the first database be constructed in createSceneContained function. It just groups patterns based up their sybmols’ scenes. The patterns then point to their respective symbol with help of PatternSymbol structure that is setup in the flatten function.

The second database is much more difficult. We need to go trough all the PatternSymbols and see if they came from any connections that transit scene, this is what the filter function does in the stream. To figure out in what scene to put this pattern symbol we create a different kind of connection database. This connection database has all the connections point in the opposite direction, we call this the dual. This idea just use a dual came from my geometric algorithm course, where they significantly reduced the complexity of an algorithm by converting points in lines and vice versa. The dual in this case does something similar, because if you call it twice you end up with the same structure.

The final step in both cases is constructing the hash map, this is used in a various places, therefore it was moved to the functions class.

Constructing pattern databases, core functions

public class PatternProcessing {
    public static PatternDatabase createSceneContained(
        Map<Symbol, Set<Pattern>> from
    ){
        return new PatternDatabase(
            Functions.streamToHashMapSet(
                flatten(from),
                key -> key.symbol.scene,
                Function.identity()
            )
        );
    }

    public static PatternDatabase createSceneNextTo(
        Map<Symbol, Set<Pattern>> from, 
        ConnectionDatabase db
    ){
        ConnectionDatabase dual = db.createDual();
        return new PatternDatabase(
            Functions.streamToHashMapSet(
                flatten(from)
                .flatMap(patternSymbol ->
                    dual.getConnections(patternSymbol.symbol)
                        .filter(connection ->
                            !connection.to.scene.equals(
                                patternSymbol.symbol.scene
                            )
                        )
                        .map(connection -> 
                            new Pair<>(connection.to.scene, patternSymbol)
                        )
                ),
                Pair::getValue0,
                Pair::getValue1
            )
        );
    }

    public static Stream<PatternSymbol> flatten(Map<Symbol, Set<Pattern>> from){
        return from.entrySet()
            .stream()
            .flatMap(entry ->
                entry.getValue().stream().map(
                    pattern -> new PatternSymbol(pattern, entry.getKey())
                )
            );
    }
}

I really wanted to show the dual idea somewhere because I know this is a hard problem to solve, but it didn’t take a lot of effort because of the dual idea. Not sure how readable this is though, this is a problem I have more often with functional programming.. How do you know what is a good or bad pattern? I guess I just need more experience or talk with other people about this.

Lazy hashing

The only reason I’m discussing this is because I worked with Java (Scala does the hashing stuff for you in case classes), and in some situations you did not want to calculate the hash code eagerly because the model object contained a collection (which could be a lot of work). I modified this stack overflow to work for hashing resulting in the code seen in source block java:lazyhash. So what happens is as soon as the hashCode function is called we calculate it, and then replace the supplier hash with a new lambda that just returns the result. Note that this will never change because the model object is immutable.

The lambda replaces itself with the result

@Immutable
public class Utterance {
    public final Informative informative;
    public final Instant when; // immutable
    public final CapturedMatchDB capturedDB;
    public final PerlocutionaryValueSet perlocutionaryValues;

    private Supplier<Integer> lazyHashValue;

    public Utterance(Informative informative, PerlocutionaryValueSet perlocutionaryValues, CapturedMatchDB capturedDB) {
        this.informative = informative;
        this.capturedDB = capturedDB;
        this.perlocutionaryValues = perlocutionaryValues;
        this.when = Instant.now();

        lazyHashValue = () -> {
            // since the class is immutable and we don't deal with collections,
            // we can calulate this now, if it every is required...
            final int hash_code =
                311 * informative.hashCode() -
                193 * this.perlocutionaryValues.hashCode() +
                701 * capturedDB.hashCode();
            lazyHashValue = () -> hash_code;
            return hash_code;
        };
    }

    @Override
    public int hashCode(){
        return lazyHashValue.get();
    }
    ...
}

The experience

I specifically asked my teacher for getting a ‘practical’ assignment because I’m good at that. When he mentioned personality research I also opted into that, because I already knew a fair bit about MBTI. Finally, the personality as a process bit was all my teachers’ suggestion, but I really liked that idea.

Doing research

When I started doing the thesis I was mostly on my own, my guiding teacher had left for Australia for 6 weeks, and I just started with what I think had to be done. I never had done before any research of this kind of scale so I just used common sense to decide what to do. However I made sure to keep my teacher up to date with weekly updates trough email.

The researching part consisted of several parts. First of all, the personality research with which I started, this was just ploughing trough papers on my own. Then came analyzing the chatbot, this was quite fun because it was just reverse engineering some poorly written code, which is challenging but also rewarding (I always get the idea I learn to know the author better by studying his code). Finally I needed to develop a theory of Jung and Dialogue, this was done mostly with the Haskell notation and giving my own interpretations of the Jungian functions. Then I also developed a way of combining them.

When my teacher came back I was mostly done with all of that. So he had a lot to catch-up with because I was writing my thesis while doing research. Even though I was thoroughly working for just six hours per day, he complemented me and said I had done a lot of work. I continued working just six hours per day.

Implementation

Once I was finished writing what I wanted to do in a functional design I started with the implementation. I quickly decided to not use the ALICE bot. It was poorly written, with for example many global mutable variables, frantic use of public mutable attributes and all the things you shouldn’t do.

In the thesis I justify moving to the new system by saying that AIML doesn’t offer the capability of providing choice, which is a much better reason that what made me look for alternatives in the first place. The first strike AIML got was by just being based upon XML, most programmers will know XML sucks (usually, there are good cases for XML). The reason I was pushed initially started looking for an alternative was because I didn’t like the jury rigged combination of drools and AIML.

What I did was a quick implementation of how I envisioned the chatbot that could co-exist beside XML. I showed this to my programming guiding teacher after about two weeks of hacking, and he recommended me to just dump the old implementation and go with whatever I was making. He also pushed me to use drools much more intensively rather than java, which resulted in some good changes such as the pattern matching code becoming a drools rule, and some changes I like less such as the personality order being in both a list and the next function.

Presentation

I had way to much time to prepare for the preparation. Partly because one of the faculty members got a disease. But also because the primary guiding teacher was a very busy man and I had a long thesis.

I think I practiced the entire thing about 10 times in total. In the beginning I would often change the presentation after practicing but the presentation would become more final after each run. Each time I would be over time by a margin, however on presentation day itself I somehow managed to get exactly the right time. The difference probably is in stress level.

I don’t remember giving the presentation, I know I stood there, said words, but I have barely any memories from the event. This usually happens to me when giving presentations, luckily my father filmed the entire thing. I was a little disappointed with the grade, but not too much, the criticism that I didn’t add much theory was fair. However I think I couldn’t do this much better because I just don’t know how to develop theoretic foundations. This is partly because of my software engineering background, whereas the second judge was mostly from a mathematical background.

The questions I got were really quite though, firstly I was asked to describe precisely if a thinking function would be first in order, how would it still get to influence the result. The answer was that it just could inspect the result, because we have a two pass architecture, going deeper first. However because this is a very detailed question it took me a while to figure out what he meant by that. They also asked me about if the division between rational and irrational as action generation and sorting was a design decision, and yes it was. Then another question was about, can we extract the ‘communicate!’ game information from the GUI and encode it into the new game, what personality would the actor have in that game? I would say yes upon extraction (even automatic) but I didn’t know what personality because I didn’t study those dialogues (in fact I barely studied the communicate game). Finally a question was asked in which cases this would help doctors, I replied with the more emotional situations because it would be important to treat someone right under these cases.

Conclusion

I would say that I liked doing my thesis as a whole. In fact I would say I enjoyed the entire master, but I do know that I’m not an academic, my initial hasty assumption of “head in the clouds” was quite correct. The very notion of just always trying to do stuff in new approaches bothers me, I would rather just solve real world problem, with old approaches if they work, and do new approaches only when the use case demands it.

Then there is also the issue of neglecting to publish source code, I just think that is terrible for science as it creates a lot of double work and since I personally prefer digging around in source code I’m almost certain I work badly in academia. By which I mean it would just make me unhappy, I don’t get excited by writing large bodies of text to the point of perfection. I just want to get out what is on my mind, release early release often rather than peer reviewed based academic releases. I choose the bazaar cite:raymond1999cathedral.

bibliographystyle:unsrt bibliography:./files/2017/refs.bib

For example: let the any function also return the triplet but setting it to any actor↩︎

Website changes

2017-06-30T00:00:00Z

I recently made a lot of changes to this website, they aren’t big changes but the devil is in the details. I made these mostly in support of writing about my thesis as a blog post. I for example moved to SVG images because I recently fell in love with Inkscape, however I wanted to ensure that compression was enabled for these because XML is so verbose. Which took me on a small NGINX adventure. Then I did various styling changes in light of my job hunt. I still think this theme is correct but I made it more ‘tight’. These were just tweaking with CSS, except for the desire to typeset like latex, which I’ll discuss in the end.

Org export

I wanted to have source code highlighting with Pygments. Org mode export didn’t do any coloring for me, however I came upon this question. which sort of answered it for me, except for telling how to load the lisp files which I figured out by studying the GitHub. Then I modified the answer to include that so others don’t have magically know that. There were standard style sheets for syntax highlighting available for pygments, I chose my favorite: monokai.

In the org references I wanted to have numbered citations rather than names. This article was invaluable to help with that. The issue I had with that code is that I called it org-ref-unsrt-html-processor, but I was using a custom variant called org-ref-unsrt-pelican-html-processor, because of the Pygments extension. Renaming that correctly made it work and cost me about 4 hours of my life. So you can just dump that code in you config.el file, but be aware of giving it the right export name.

The final thing I did was making SVG files export to img tags rather than object tags, because object tags just work very poorly. (Once I published it I couldn’t get it to work at all). What I did was overwriting the image formatting function in elisp as can be seen in source block elisp:fixsvg. The only thing I did was remove the check upon SVG files, instead it’ll use the default path which is using an img tag. I placed this after the ox-html require so it got overwritten by all following code (including the deriving stuff.. apparently).

Elisp image override

(require 'ox-html)
(defun org-html--format-image (source attributes info)
  "Fix org's implementation, no check for svg files (object is deprecated at 
    this point and not trusted by most browsers)"
    (org-html-close-tag
     "img"
     (org-html--make-attribute-string
      (org-combine-plists
       (list :src source
             :alt (if (string-match-p "^ltxpng/" source)
                      (org-html-encode-plain-text
                       (org-find-text-property-in-string 'org-latex-src source))
                    (file-name-nondirectory source)))
       attributes))
     info))

I’m still very happy to work in org mode, in fact the evaluation of source blocks makes it a great combination with plant UML, which I then can export to SVG images. This makes it much easier to describe ideas (being a visual thinker), and allows version control upon UML.

NGINX

I enabled gzip compression for a bunch of different file extensions, (note this is handled by your web server, so its a practically free speedup for your website). but I know for a fact the SVG images profited a lot form that. This article explains how to do it for NGINX.

I also tested the speed and it told me to enable caching of the SVG files, this had me confused for a little while but it meant you should tell the browser clients that the resource will stay the same for a long time (reducing load on your server and since networking is slow giving a speedup). This serverfault explains it for NGINX, I had another post but couldn’t find it back, that said to put it on a year however (365d). Since I’m not planning to change anything except the PDF’s my NGINX config now looks like source block nginx:caching.

Cache NGINX rules

# browser caching 
location ~*  \.(jpg|jpeg|png|gif|ico|css|js|svg)$ { 
        expires 365d; 
        add_header Pragma public;
        add_header Cache-Control "public";
} 

# pdf also cache, shorter because we want to update our resume frequently 
location ~*  \.(pdf)$ { 
        expires 30d; 
        add_header Pragma public;
        add_header Cache-Control "public";
}

Typeset like latex

If you’ve ever worked with latex files you may have noticed how the text is aligned. Normally (MS Word for example) this is in ragged mode, where one side is a straight line down and the other side is ‘ragged’. Latex type-sets with full justification but doesn’t varied spacing between the words. I wanted that on this website. Btw, I’m fully aware that only a select few people will even notice this, the important point is that I will notice this.

I basically went trough this entire site, I recommend doing the same if you manage your own CSS for your website. However there were two important sections, the one that describes ragged, and the one about hypenation. They basically recommend against trying to emulate latex. But me, being the arrogant little bastard I am, was like “This is my blog, don’t tell me what to do” and started experimenting. The result of which you can see in source block src:sasstypeset.

Sass of typeset experimentation

p
    text-align: justify
    hyphens: auto
    line-height: 1.3
    \+p
        text-indent: 1.375em

// display inline otherwise hypenation doesn't work (normaly inline-block)
.entry-content a
    display: inline

The key to having proper justification is text-align: justify and hypens: auto, do note that for auto hyphens you need to set your website language, they actually have made dictionaries per language, of where to place hyphens. Another issue that I had was that certain lines would have massive spacing in the words, after staring at that for several hours I noticed this only happened before inline links. I put those on display inline-block, which apparently means: no line breaks. Once I put that on display inline it looked practically perfect.

There is some other stuff in that sass, such as doing text indent for succeeding paragraphs (rather than margins), which I think looks the ‘nicest’, but that’s just personal preference. There are various other techniques, for which I refer to the expert website.

Endorsement

2017-06-28T00:00:00Z

A post I’ll periodicly update (no promises) of links I’ve found that are interesting, funny whatever. I guess its a ~~hall~~ wall of fame of sorts except it has zero prestige. (well maybe some day it will). It also serves as a kindoff reminder of stuff I used to like.

Beauty

A work of art

https://aphyr.com/posts/342-typing-the-technical-interview I’m envious if the writing style in that blog post. It weaves together some extremely advanced type level programming with a funny narrative. I hope that I could produce this kind of writing in the future too.

Wallpapers

http://www.erikjohanssonphoto.com/ I used several images as wallpapers from this guy, at the time I couldn’t pay him so I guess I repay him by giving him some realestate on my website.

Funny

Suicide linux

I remember a script that would execute rm -rf / if you’d type a command incorrectly, just to teach you to be precise. It was sucide linux.

I’m not running that.

Interesting

Suckless

It’s software that sucks slightly less. http://suckless.org/

Amongst other things they have a irc client that works with fifo queues on the filesystem. It’s beautifull.

Tsoding

A streamer who makes a show of programming. With the slogan ‘coding for fun’ he traverses the programming landscape from haskell to assembly and from markov chains to grub drivers.

CGPGrey

A youtuber, although I’m linking to his website. He has some interesting videos (of which I know him), but his podcasts are quite good too.

Brady haron

Also a youtber, his website. His numberphile channel is fun.

How computers learn

A back to basics overview of machine learning: https://youtu.be/T1O3ikmTEdA?t=644

Even though I knew most stuff he talked about in the beginning, the talk later becomes more interesting since I didn’t follow machine learning that much.

Tech blogs

At some point I wanted to read interesting soemthing interesting, I found these who look all very interesting:

As I grow older I prune this list more. Technology in isolation is boring.

Blogs

These people write well. I like reading about the day to day issues in random company or struggles in life.

Stratechery

This guy is some kindoff analist I encountered on reddit (probably the /r/investing sub). He has some really interesting takes on tech companies and I learned a lot about buisness in general from just reading his blog. I especially liked his aggregration theory, makes me wonder if I could make one of those.. (probably not). https://stratechery.com/

danluu

I found one of his articles on reddit, and he has an interesting perspective (as moslty a hardware person). I was reading the article normalization of defiance which talks about how strange, shady or broken practices are born, and what one could do about them. Then I went to the index and it turns out there is a huge treasure trove of interesting articles on that blog.

https://danluu.com/

Philosophize this

I really enjoyed this podcast. He goes trough all great philosphors in chronoloigical order, which as a side effect helps you understand history better. It also made me think about my own life.

xoreaxeaxeax

Everything in this github profile page is ~~insane~~ beautifull. This youtube explains the sandsifter thing in quite detail.

Reading

Books in public domain

I’ve been reading the ‘classics’, of which many are in public domain. No need to spend precious money on something which ought to be free. Project gutenberg offers public domain ebooks for free:

https://www.gutenberg.org/

I do recommend donating to them though. They’re doing important work.

Programming for personal growth

A really well written article about how simple satisfication (with programming in this case) may make you more confident and happy in life: https://medium.com/the-polymath-project/programming-for-personal-growth-64052e407894

Writing well by george orwell

http://www.orwell.ru/library/essays/politics/english/e_polit

Any one interested in writing demands a read of that. It describes why texts are interesting or not. It suggests for example that each sentence should anwser these questions:

What am I trying to say?
What words will express it?
What image or idiom will make it clearer?
Is this image fresh enough to have an effect?
Could I put it more shortly?
Have I said anything that is avoidably ugly?

It also explains why we see this horrible accedemic style such as: “If a new spirit is to be infused into this old country, there is one thorny and contentious reform which must be tackled, and that is the humanization and galvanization…”

Paul graham essays

These are both inspiring as interesting. I’ve found myself identifying with much he wrote about. Especially now I’ve decided to start my own startup, it’s really motivating, although I know he writes those mostly to just get more candidates to chose from. I like to believe he also does it to genuinly help people. Getting a job is bad for your brainz.

Tools

Vps comparison sites

For if you just need a cheap computing device somewhere on the internet. I wanted one that had high CPU but didn’t care about the rest, this one found it: http://vps-list.cryto.net/index.php?action=list

Haskell blogposts

Comonads for builders.

Anime

I watch way to much anime. Sometimes I encounter little gems, recorded here.

Puella Magi Madoka Magica

A beautifull juxtposition of childishness and despair. I don’t want to say anything because I’ll spoil it but give it a try, it’s so worth it.

Code geass

This was one of my first gateway anime drugs. It’s an excellent classic keeping you on the edge of the seat all the way. All characters have good motivations and are believeable.

Soul eater

I especially liked the first season, it drags on for a bit after that. But the theme around crazy-ness being the enemy is really appealing to me.

Making money with foss

2017-05-22T00:00:00Z

In this blog post I will discuss how to make money with open source software. Why do you care? Making money of this model is hard, yet you got to pay the bills. Even though the free software variant is consumer friendly. (note that these consumers maybe businesses too). This is a big problem. Lot’s of projects are maintained in people’s unpaid free time, next to their day jobs. So they don’t really get the love they deserve. Can we fix this?

I would like to do this myself full time. Lest I became financially independent, I couldn’t. So in here I will list the options I found trough researching the subject. As both an overview to myself and to the interested reader.

Google results

In my naive approach I searched for “Making money with foss”. The first site that came up was info world. This site was full of warning flags, for example having a click bait title and using a pagnation. After skimming the article it wasn’t about making money of an open source project, but potential reasons for open sourcing (part of) your code. It was aimed at mid to high level management, non technical texts are of no interest to me, mainly because they usually have an agenda.

A second article linked to another page with a better list, but I couldn’t find it. The third article was just a summery of a video and wasn’t very focused. Okay so my search terms were bad. Time to be more specific.

I searched for “foss business models”, you know you did something right when scholarly articles start being presented. There was an entire Wikipedia page on the subject. I guess more people are really interested in this. So does this blog post die in the cradle? I say no, I can still give my opinions on that page. Another thing is that discussing this page on my blog will force me to closely look at the possibilities. Besides the fact that there is a (quite big) wiki on it only shows how much people care about this subject.

The Business models

The core problem open source software faces is not having a monopoly on distribution that copyright provides. Therefore we can’t sell it directly: Anyone can start distributing the software without asking anything for it, without running a loss because distribution is (practically) free for software on the internet. Giving up this monopoly is a necessary requirement for open source, called the second and thirth freedoms. This is also done with the BSD like licenses, so we can consider it a part of open source. Therefore we need to find a way of making money by working around this restriction.

Sell stuff besides software

One of the primary strategies is selling stuff besides the software. Red Hat does this trough providing support contracts for example. This can be extended by providing certificates of expertise (which Red Hat also does).

I don’t like this approach as it gives the incentive to make bad software. If your software is hard to use, then only you can sell support. You should make your software easy to use.

Another strategy is selling of merchandise, such as fan T-shirts or coffee cups. Which the wikimedia foundation does. Selling merchandise is quite clever because its much more easy for someone to buy ‘something’, rather than just donating. Besides its free advertisement for your project, and your users now have something to identify themselves with. So I think if you have an opensource project you should seriously consider doing this.

Finally a programmer can rent himself out to a project to add features or do bug fixes. These include Bountysource or kickstarter, where Bountysource is usually fix an issue first and then get paid whereas kickstarter works on promises. In essence you’re offering up your own time to work on a project. In both cases there is of course the persistent issue that its hard to decide how much time this will take. With kickstarter however you can estimate it better because you’re probably starting your own bigger project, so the overhead of estimation is smaller. With Bountysource each bounty requires its own estimate and these are often smaller, think of about \$100, its hard to decide to even bother with that. Another issue with Bountysource is that you have to switch from project to project often, which requires a lot of getting familiar with the code base overhead. The model of Bountysource is less then perfect to live off. Although for existing developers of a project its a great motivation to work a little more. But don’t expect strangers to ‘join in’, just because of the bounties unless they’re highly valued. In fact you can see that few people ‘live’ off Bountysource just by the overview of bounty hunters. Only three people at the time of writing got over \$2000 in the last 90 days and only 9 over \$1000.

In the strategy of renting yourself out is going to interested parties yourself and offer to write software. I’ve attempted this at times with for example offertex and schijt je rijk. The issue that always gets me is negotiating about price. And of course once the work is finished, you need to find more work. Therefore you won’t get better overtime. Imitating red hat with support contracts could do that, but starting with that on your own is hard. Although I had other factors working against me, living in a remote village didn’t help. So you mileage may vary.

Donations

Another powerful trick of trying to become self sustained on software is using donations. The big issue with this is of course getting people to donate in the first place. A famous example of a donation scheme as that of mozzilla firefox 1.0 release.

Recently another option has popped up in the form of patreon. The idea is pretty simple, subscribe to a creator and every time he puts something out (or every month) donate a small amount decided by you. Some developers have embraced this idea, although I haven’t seen anyone that earns self sustaining amounts. It does provide a reliable income stream and is aimed at individuals, and since there are other creators who have managed to get to a sustainable level. We can expect this to happen eventually for software developers too.

The big advantage patreon offers over Bountysource is that, rather than having to think about how much time creating a feature costs as with Bountysource, you can just continue improving the project how you think it should be done. Of course some trust in the developer is necessary for that, Bountysource doesn’t have that problem, since payment is done afterwards and I assume regulated by Bountysource. However apparently Bountysource offers a similar service for projects.

Another approach to this is to start a foundation. Blender is developed like this, and if your project has any kind of traction, do consider this. Trusting a foundation is much more easy because they’re regulated by law.

A more recent iteration of donations is using twitch. Essentially you’re just explaining what you’re doing on a live stream and then people can ask questions while you’re programming. Twitch handles the payment processing for you.

A trick often overlooked by most developers is advertising. In principle free software is not against the idea of advertising. However a problem with this is that anyone can take your software, remove the advertisements, and redistribute the add free version. You can prevent this from happening by offering two versions, one with adds and one without and then ask your users to support the project by downloading the one with adds.

Addblock plus famously white listed adds as a way of generating revenue. It may be considered hypocritical, but remember that as free software anyone can fork it and remove this feature. Not that you have to since there are already alternatives. Note that although Adblock Plus probably made quite a good load of money with their white listing program, it will probably kill the project eventually. Because now they’re essentially running an extortion racket, becoming nothing more than parasites and providing lower quality user experience. I imagine the people who go out of their way to install an ad blocker are not the kind of people who are satisfied with an ever laxer white list.

Crypto Currency

There is an opportunity in crypto too. Although this immediately sounds fishy. I’d say that taking a couple cycles per minute to do some mining isn’t that bad. I mean how is it any different from stealing a users attention with advertisements? It’s probably even better because the users’ time is more valuable than machine cycles. No wonder that google cracked down on this. Crypto is a substitute for advertising, Google is essentially a massive ad company. But this has the same problem as advertisment in that forks will occur which would take it out.

License tricks

The final category for making money with open source is license trickery. So this comes from the idea that, if you are the sole copyright holder, you can put the software under various licenses. Note that it is a big if to be the sole copyright holder, you need consent that any contributor is handing over their copyright. And as we will see in the re-license case, this can be abused. Which increases contributing barrier significantly. But in return as project owner you get a lot more possibilities for making money.

Dual licenses are a practice where you offer one open license, and another business aimed license which promises more support than the open license or removes restrictions (such as forcing open source). This is where AGPLV3 shines, original authors can offer large organizations an alternative license, however downstream receivers of the code under AGPL cannot do this. Mongodb is an example that does this and they can only do this by asking contributors to hand over copyright. As a contributor you should be wary of doing this, copy left doesn’t work if you hand over copyright. If you work under BSD or MIT kind of licenses it doesn’t matter (and I assume you already came to terms with this).

However, this does allow a company to thrive upon open source. A company is still required to hoard in the business deals with other companies, and to collect the copyright assignments from contributors. It may be taken over similarly as Oracle did to Sun. But the AGPL based code was already under license, so the community can step in and take over development, as happened with Illumos (Solaris fork) after Oracle went on its rampage. Although a recent article by Ben Thompson cast some doubt on the long term efficacy of this strategy.

This has however a darker side in potential license trolling. Which Oracle (who else) did for example with a database. Changing from BSD to AGPLv3, which in case of Debian required around 100 other dependent packages to change to AGPLv3 too. Which of course is not going to happen. Oracle probably did this to force users of that database to take a commercial license instead, taking foss projects (such as Debian) as collateral damage.

Proprietary extensions

Proprietary extensions involve releasing an open source core and add (usually business centred) proprietary extensions. The Wikipedia page lists several example but the one I’m personally familiar with is the IntelliJ project. Which is opensource, has a proprietary paid extension, and also a cla (which doesn’t hand over copyright but does a similar thing, licenses a right to copy).

IntelliJ is kind off open source, but many developers want to pay for things such as CSS or JavaScript support. Although I’d say any text editor can do that, such as spacemacs. where IntelliJ shines is Java and Scala. As far as I can see are the ‘supported’ features, just bells and whistles. However they maybe valuable for a professional developer, or to a software house to which the license fees are nothing compared to developer time.

Because IntelliJ is open source it allowed Google to create android studio. This is great for the IntelliJ team because now there is another party that is dependent on their core of which they hold all copyright. Google may help developing the IntelliJ Java core, just to get it to work for android developers. This is the thing most companies are after with open source, free programming manpower.

Delayed open sourcing

This is the thing John Carmack famously did with ID tech. After some time selling the games he would release the source of the games. Which he did for Doom, Quacke even Doom 3. This in turn led the games to be developed upon for a long time after their release. For example ioquake3, still actively develops the quake engine, driving sales of the quake game itself because the assets aren’t freely available.

So he open sourced partly dated tech, because he was confident he could do better now. And he increased sales of already released games, because people wanted to hack on it. I think this is pretty genius, gaining some extra juice out of old projects. The really nice thing is that the lifespan of these games has tremendously increased. Creating a core of die-hard followers. Doom has been ported to essentially everything. It’s a mystery to me why not more companies are doing this.

Re-license

If you are the sole copyright holder, you can stop distributing under the open source license and re-license it. Originally I didn’t want to include this option because you’re no longer doing foss at this point. But if your project is dying, because you can’t give it enough love. This maybe totally valid. It’s your project after all, if you can make a valid business out of it by stop doing opensource I’d give you nothing but praise.

Having said that, this also opens up the opportunity to hate upon Oracle. So lets hate upon Oracle.

So if we Google: Why oracle is horrible, we can get some dumb quora answers. These are just not the point. This reddit thread, sums it up nicely. What is really dog kicking evil were the Solaris issues, which is discussed in this this video. A little further in the video its is explained how it happened. So what happened is that Oracle obtained all copyright from various authors by buying SUN which required initially handing over copyright, Open Solaris was closed by Oracle with a re-license. This was only possible because Sun asked contributors to fork over copyright. What we can learn from this is that if you contribute to free software and care about it, never hand over copyright. I’m happy to say however that a fork of Solaris occurred called Illumos that seems to still be active.

In conclusion

Because we are interested in making money, this post will took us all over the place. On the one hand we have the greedy businesses, and on the other side the diligent developer. Licenses were never discussed in university, which is interesting because this is the method for making money with software. I think having discussed the overview and shown some concrete examples was a good exercise. I was not aware at all for example of the AGPLv3 practices which are interesting (without passing moral judgment).

Tool survey

2017-04-01T00:00:00Z

Some time ago I made a blog post about thesis writing tips. However while writing that a large part started to be about text editing tools and version control. To keep the thesis writing tips post more focused I postponed writing about that. This post treats my tools of choice.

All people will only ever care about a niche of tools. The reason is that in our world, we have too many choice to form an opinion about each of them. Tell me for example about your favorite pick axe or transport ship, you probably don’t have an opinion about these tools. Which just shows how more complex our society became compared to for example the stone age, where everyone had a favorite type of stone.

Note that isn’t enough to form an opinion about a tool to care about it, you actually have to have used it for a while, and then switch to another quite different tool to start caring. Only once you switch you will form a deeper ‘bond’ with the new one or old one. I think I switch quite often. Which brings us to another reason for this post, I want to compare my current tools of choice with those of future me.

Operating System

This is where things will get interesting. The main theme of my selection has always been tweakability. I want to be able to change things I don’t like. This is seen most clearly from my OS choice: Gentoo/Linux, this is a source based rolling release distribution. Most distributions work with pre-compiled software. However software provides often at compile time, flags that enable or disable features. Gentoo standardizes this and allows you to specify per software package which options you want or don’t want.

The reason I chose Gentoo initially is more petty, a class mate of my was using it, and I didn’t even understand what “source based” or “rolling release” meant. I wanted to be at least at the same level as my classmates, so in my stubbornness I spent about 2 weeks installing it on my own the first time, with help of the handbook. It took so long because I didn’t know anything really about the shell and unix, Gentoo really forces you to learn.

Now I stick with Gentoo because it allows me to do customizations, while updates won’t break these. Gentoo’s packagemanager is very careful in preserving configuration files. Unlike Ubuntu, which upgraded me out of my custom alternatively installed desktop environment. Gentoo has to be more careful, because of its inherent configurable-ness. Whereas Ubuntu provides ease of use, which goes hand in hand with, knowing it better than the user.

Control everything with keyboard

During my software engineering classes I got introduced to the idea that the mouse is pretty slow to work with. The key difference between mouse usage and keyboard only usage is that mouse usage often involves searching the right place to click, wheareas using the keyboard is just doing what you want to do. By systematically making the mouse less important, working with the computer becomes more like playing an instrument. I use several programs that minimize mouse usage:

i3-wm or its wayland successor sway (as soon as wayland takes off, it’ll probably be some years later) This is the window manager that fills the screen with active window by default, or splits it if there are multiple windows open. It also has work spaces (basically groups of windows under a global tabbing system).
vim. This is a text editor with “modes”. normal mode is navigation, and insert mode is for typing text. If you work a lot with text its definitely worth learning this. But beware, expect the first two weeks or so of using this program to have lower productivity. (and basically no productivity the first day ;) )
Spacemacs. This is an evil combination of Emacs and vim. I use this for software development and writing larger texts. whereas I use vim for quick and dirty edits (besides vim is available on every Unix system whereas Emacs and especially Spacemacs is not). Oh I also run this in daemon mode so that I can use i3 to move windows rather than the pretty bad buffer navigation of Emacs.
Ranger. This is a file browser that works with vim like bindings, I also have a Spacemacs plugin that does the same inside Spacemacs. Actually if you combine i3 + vim + ranger you have a really solid IDE.

I haven’t found a good way to eliminate mouse usage from the browsing experience. There are plugins that give vim like navigation, but I never really found them compelling to use. This maybe just personal taste, but I think it also has to do that browsing, is usually quite similar to searching.

Write everything down

I’ve slowly over the years come to realize how unreliable my memory is and how liberating it is to write stuff down. No need to think about something which you’ve already written down. I have a little idea project in which I write down random ideas I have for programs or systems, even political once. The reason for doing this is that the idea can then exit my mind without the fear of forgetting it, often I will later extend it or realize its unfeasible or impractical.

To this project I also added a planning file which contains some stuff I probably should work on. This really helps focusing and narrowing down a direction I want to go in. They also provide a dialogue between passed me, current me, and eventually future me. It was a classmate who brought me up on this idea, but CGPGrey pointed out the dialogue aspect. The dialogue aspect also holds true for ideas, but ideas are generally not plans yet. Plans have some form of commitment too it, you just need to find the time.

Version control

I mentioned “projects” before, by this I mean I have the files under version control. I use git for version control.

Wait, I’ll clarify the concept of version control first. Think of it as Ctrl+Z on steroids managed by a dedicated program. It can track changes over multiple files and directories, and with each change you want to save you can add a message. It can also send these changes to other machines or ‘locations’. There are actually a bunch of options for this svn, cvs, bazaar and many more. At this point its safe to say however that git crushed all competition, the reason for this is that the architecture of git compared to other systems is dead simple. Also github helped a lot.

However I don’t just use git for just programming projects, no. I use also it for important configuration files, my todo list, my idea project and of course my thesis. Basically most information I put into the computer that can be represented as plain text without to much trouble, I put in git. The reason for this is that it has a much nicer history than for example dropbox, and provides advanced merge mechanisms. Besides with git I don’t need to have to trust some external service, since its decentralized.

Actually git has slowly become a sort of diary of mine. For example when I delete items from the planning or ideas they’re not truly gone, just stored in a previous commit (=ctrl+z log entry), with a message why the change happened. I think future me, or historians if I ever become that important, can track most of my life at this point with help of git. In the back of my mind I’ve actually developed an opinion that git usage is a basic computer skill and should be thought on schools, since it could make collaborative work on documents much easier. Although I know this won’t happen in the near future, since it would also require the rejection of binary formats for text processing all together. Which is something harder to ask than replacing with an open format, with which entire foundations have trouble pushing upon people and organizations.

Discovery

The source of finding new tools is an important thing to discuss. I think I had three major sources over the years:

School, mandatory content
School, classmates
Reddit

The first two sources have essentially become closed since I started writing the thesis and no longer physically are at the school. However I’m expecting a fourth source in the form of colleagues once I start working.

I think mandatory school content is generally the worst quality. First of all it lags behind the inovation curve. Secondly most people already know how to use that since its mandatory so you’ll have most competition on that area if you’re an expert. Finally it doesn’t really give you much of opinions on the matter at all. Giving dumb reasons for learning the tool “you have to”, or “this is industry standard” rather than giving a comparison between alternatives. However I did find that the school of applied science (Windesheim), usually was even further behind than the University of Utrecht. But maybe it was because I was following a master course rather than a bachelor. Note that I’m not really talking about the theory, but the tools with which you can put theory into practice.

Classmates are a much better source, in fact one of the best I think because they can tell you their opinion, and opinions are so important. Even if you disagree with them, it forces you to defend your position and reconsider, which makes your choice stronger. Or if you fail you may flip and gain a new experience and form new opinions.

I have mixed feelings about reddit, its a major source of distraction, but it has a lot of gems (not even hidden, I mean reddit has search functionality!). I often subscribe to sub-reddits of tools I’m interested in using to learn more about them and to lurk on those valuable opinions. But the hivemind, seriously fuck that politically correct piece of shit… ^{^{I love you}}. Our relation is complicated.

Paranoia (security)

I encountered the abysmal state of computer security first in Windesheim. I won’t talk about the state of computer security because I don’t care. What they did on Windesheim, was: They just taught is to hack so that we would know how to defend. I hated it. You may say, oh isn’t hacking super exciting? No its not. Its not really constructive, you just need to think in terms of possibilities and try out everything. Rather than programming, where you just try out something that comes to mind and then later review to see how to improve (or stop once satisfied). Then this gets mixed with the idea that paranoia is good. I know from my experience that under certain circumstances I have paranoid tendencies, I don’t need to promote that in my life. Finally its never enough. Being secure is always a moving target unless you can formally proof you’re secured, which you can’t because that’s as hard as programming.

Since that class, which is now almost 3 to 4 years ago, I had several concerns in the back of my mind. My weak set of just 3 passwords used rationally. If one got cracked I would lose a third of my online accounts. If my email account got cracked I would lose everything. And the fact my identity could easily be faked. So I addressed these concerns recently (about 3 months ago).

To start with the most important issue, passwords. For this I use something called a ‘password manager’ which called Keepass, actually I use the qt based one (works better with Linux). I use this one because its opensource, and has a GUI and an android client. The reason for using this is complicated and explained very well here but basically its more safe than memorizing several and it also happens to be more convenient. I use syncthing to share the database with my phone and my server, I don’t wanna lose that. I just have a huge password, I didn’t want to do the keyfile, because I’m afraid to lose that.

I recommend everyone to use a password manager. Its easier to use, it can autofill passwords in your browser and you just have to remember the master password. It allows you to have much less “stuff” in your mind and its more secure. I even put WIFI passwords in there. Network manager also allows viewing of WIFI passwords, at least in plasma, but since I’m almost always in i3 its just easier to have in Keepass. Besides now I have the WIFI passwords on my phone too.

For authentication (making it difficult to steal my identity), I went trough the effort of setting up pgp for email and git. For email I use the Thunderbird extension enigmail. All my git commits and emails are signed now. To configure signing in git I followed this. This stackoverflow question explains how to sign automatically.

Linus Torvalds believes auto signing is unnecessary. I disagree, it makes it much harder to steal my identity, without me needing to think about it.

For email I know its a good thing to sign, spoofing email is incredibly easy and by signing it, spoofing my email becomes difficult. Assuming the recipient checks it, which most people don’t, but if nobody sets this up nobody will ever start using it. So I setup email signing more as a matter of principle I guess. Oh and btw, if everyone actually did this, the life of spammers would become significantly harder, since spoofing is no longer an option. So shame on you if you don’t do this.

I also setup a VPN on the server with openvpn, mainly in case I want to go to china again, I’ll just have that around. It allows you to visit google there. The VPN client is basically a use flag for network manager.

Using a VPN by default won’t help you with security much. Maybe if you use not encrypted WIFI, but even then if you visit https websites the connection is encrypted anyway. Using a service for it is especially dumb, because now that service provider has always access to your packets. If you want to do a VPN you need to setup your own VPS, and then setup your own VPS. I actually host this website and the VPN on the same VPS at Galaxy host plus. with the cheapest offering VPS OpenVZ Hosting - OVZ - 2016 - LC1.

Other random stuff

I don’t know what to say about most of these.

Hostblock for disabling reddit, youtube and facebook while working on stuff. This is a huge productivity boost
Firefox for browsing. I think google has enough opurtinity to spy upon me. They won’t get my browser.
gnome terminal as terminal emulator, wait I have a reason for this. It’ll rewrap text on screenresizing. Konsole won’t do that, but konsole renders firacode litagures correctly.
You complete me to give both Spacemacs and vim better contextual awareness for various languages
org mode to write any document in.
Fbreader for reading ebooks.
qpdf for pdf files.

Comparing tax in different countries

2017-03-02T00:00:00Z

Soon I will be graduating from my masters in AI. However since I was planning to go abroad I wanted to see how much worse or better taxation laws are compared to my own country (The Netherlands).

For starters we will compare with the eastern neighbours (Germany). Scale 3 has been removed because its not relevant for me:

NL	from	till	%
1	€ 0	€ 19’982	36,55%
2	€ 19’982	€ 67’072	40,80%
4	€ 67’072	€ ∞	52,00%

source

DE	from	till	%
1	€ 0	€ 7’664	0%
2	€ 7’665	€ 52’153	15%
3	€ 52’154	€ 250’000	42%
4	€ 250’001	€ ∞	45%

source. Just glancing at it and you see a rather big difference. However its even worse with Australia:

AU	from	till	%
1	€ 0	€ 13’250	0%
2	€ 13’250	€ 26’934	9.7%
3	€ 26’934	€ 63’347	22.8%
3	€ 63’347	€ 130’995	30.1%
4	€ 130’995	€ ∞	45%

source (Converted to euros)

So ok, say I earn 3500 euro’s per month, which is an entirely reasonable assumption since I saw multiple government vaccancies with my skill set (or lower) for around that price and I asked a recruiter too who said around that (actually he low-balled me with 3’000-3’500, but I guess that’s his job).

So the amount per year is: 3500 × 12 = 42^′000,

The tax I would pay in the Netherlands then would be: 19^′982 × 0.3655 + (42^′000 − 19^′982) × 0.408 = 16^′286 or 16^′286/42^′000 = 39% goes to taxes from my total income.

For Germany however (42^′000 − 7^′664) × 0.15 = 5^′150 or 5^′150/42^′000 = 12%. That’s quite a dramatic difference. On top of that the VAT is 2% lower in Germany so you’ll have 2% more purchasing power.

So now to calculate for Australia.

(26^′934 − 13^′250) × 0.097 + (42^′000 − 26^′934) × 0.228 = 4^′762 or 4^′762/42^′000 = 11% in taxes, Australia also has a lower VAT (just 10%). And the amount of money going to ‘pensions’ is a lot lower in Australia, rather than 25% (hard to find a right source on this) of the Netherlands only 9% goes to it, which is by the way redeemable as soon as you leave (source). Which is perfect, I just go visit family for a week and get a fat paycheck, because I didn’t want to do pensions anyway in my twenties, I have other costs to worry about.

Optimus time!

2017-01-29T00:00:00Z

Using gentoo is a bliss most of the time. The package manager portage is one of the most advanced managers that I’ve ever used. Last week I encountered how much better it is than for example apt, when my old laptops power supply burned out and I had to resort to a xubuntu machine. Apt has no capability for example to run multiple install processes at the same time, whereas portage just goes with that. Apt also has no propper versioning system, which makes python 2 and python 3 seperate packages, but portage can use slots for that. This has the advantage that the search tool results become a lot less cluttered.

However using gentoo has also its disadvantages. It can demand a lot of time once you start tweaking with it. Yesterday I spent about 2 hours figuring out why my graphics card was slow. 3 hours trying to use the card in a dedicated manner ( rather than hot switching which the optimus technology does, promises to do, but doesn’t do). Which failed because documentation is often to technical and its hard to debug, because they sort of expect you know how to debug. I ended up with the galium driver which upon kvm from vmware?? That didn’t sound right, and indeed this was the software render.

Then I tried using the bumlbebee and primus driver from the main portage tree and non masked primus variant instead of the bumblebee overlays ones. This maked primus run work again. (it was borke for about one year, and I used optirun -b primus instead which worked). using the mainline gentoo tree also revealed that apperanlty primus requires nvidia to have a use flag enabled. Its the thing I don’t like about useflags, how poorly they are documentated I think the idea is that normally portage will tell you what use flags you need, but not for edge cases (such as using an overlay).

So at this time I was being hope full again because I got some results, therefore I did one last ditch attempt to try and use a dedicated one. I rebooted and X11 ended up in a black screen with a nonworking keyboard. Thanks X for taking con troll over everything (wayland still worked).

If your keyboard fails you’re in big trouble, X is sort of optional to fix things, but without a keyboard you can’t even access the backup tty’s linux comes with (alt+{f1,f2,…}).

So now I had to make a new live usb, because this new laptop has no cd player, and I only had live CD’s. With the live [usb|cd] the plan is to fix your system so you can boot at least into a tty. So in my situation it just involved disabling the display manager (which disables the keyboard). In case of systemd you just remove the symlink `/etc/systemd/system/display-manager.service`. So after I did taht I spend over 4 hours figuring out what was wrong with the display manager. Apparently this new bumbleed clashes with X11, I used the `/var/log/emerge.log` to figure this out (I kindoff forgot all the stuff I changed and for some reason journalctl recorded nothing of sddm. Also deleting the xorg.conf was a bad idea. it’ll also produce a black screen. However at this point I had about 5 different configuration of xorg so I just coppied the working one in there. I also found out that if you specify the wrong name in auto login of sddm you will also get a black screen but with active keyboard so you can just login to another tty to do more debugging.

So this time the great optimus technlogy (which I really start to hate), has taking 9 hours. This is on top of the first time when I learned about bumblebee in my early gentoo/steam experience (I guess another 8 hours at least). And that one time optimus suddenly broke (perhaps 4 hours untill I found the optirun -b PRIME trick, prime is required btw otherwise steam won’t accept it).

I don’t know, I’m salty at both optimus and gentoo to a lesser extend. I mean all these problems were just configuration problems I would’ve encountered too on other distro’s I guess. But gentoo makes it so easy to go out of your way and play with the configurations.

Perhaps next time I should just go with an AMD laptop, I hear they are making tons of progress with their opensource drivers.

Thesis writing tips

2017-01-14T00:00:00Z

So I recently started writing my master thesis and looking at some of my fellow master students their work I realized that perhaps many people have trouble with managing references, quality control and keeping motivated.

In this post I’ll explain how I managed to do it, most of these ideas are either directly stolen from my classmates, ripped from reddit and a few are my own “invention”, but others may already have thought of it.

So perhaps the best way to see this post is as a aggragetion of techniques and ~~tools~~ (the tools part will be moved to another post) I used to waste less time. So my advice to you is to skim this article and look for any ideas that seem interesting and try them out.

Reference management

Reference management seems to be a thing which can be quite tricky. It is extremely tedious and dull work, and therefore I obviously don’t do it by hand. I use bibtex as core platform, I haven’t compared the available options but since it integrated with org-mode I was happy with it.

To find papers I mainly use Google scholar trough the university proxy. However sometimes papers aren’t available for my university, so I’m forced to use http://sci-hub.io/ (if that link gets blocked in the future, try the wikipedia entry, wikipedia is impartial and will therefore provide a proper link (unlike Google)).

Once I got access to the paper I will skim trough it to see if there is relevant information. If so I will make a copy to a paper folder and use Google scholar to generate the bibtex entry for me, there is a citation option at the bottom of the search result, click it and then bibtex. That little feature of Google scholar has already saved me probably several hours of making bibtex entries. The copied paper will have the same name as the bibtex entry key, this is important if you want to recheck your reference later again. Doing this will also lower the psychological barrier of re-checking something since you won’t have to go trough the entire obtain paper process again.

Note that you shouldn’t track it in your version control, because first of all you don’t expect it to change, and second of all if you push it to a public repository you’re breaking the law on a whole other level. (distributing rather than just consuming, consuming is more easily ignored by law enforcement).

Books

I haven’t completely solved books. It seems that getting books is a lot more tougher than papers, probably because most scientist feel that papers should be easy obtainable but don’t necessarily feel for books the same way (many would have written one or contributed to one). I as a programmer feel very deeply about the necessity for freedom of information and therefore will share some possible sources. It will be up to the reader to decide if they should be using such sources. However do note that these sources aren’t always successful and you should prefer obtaining papers since sci-hub works practically always.

library genesis should be your first source of books, it has a crappy interface and some mirrors may not work but if you get a result you usually also get a book.

Then if that fails you can try and use IRC, which is described very thoroughly here. But in short for archiving purposes, go to undernet, join bookz (/join #bookz) and then search for your book title (@search). Then if you get a result you need to save that file, which isn’t a book just the query result, open that and then if it contains your book use the code to download it.

Finally there is a desperate method I used one time. It takes time but it can be worth the effort. There is this concept called digital borrowing, where HTML drm is used to ensure only n copies get borrowed at a time. An example of such a website is the open library. The system is kind off dumb since making copies of digital stuff is practically free. However I do like the fact that some publishers fell for it. Anyway if it has your book you can just “borrow” it and make screen shots of each page. You just bypassed a bunch of HTML based drm and encryption with a button on your keyboard. Yes its pretty dumb. (not that I disliked the open library initiative or the internet archive, in fact I love there effort of tricking the publishers. There work may save some books from the copyright abyss, where literature gets lost forever because laws prevent copying before the last copy gets lost).

Time management

I have a pretty strict schedule by working 6 hours per day 5 days in the week. I won’t do any work on the thesis in the weekend. I only do 6 hours because personally I feel like those last two hours are usually wasted, only when I don’t notice the time pass by I will go overtime. I start every working day at 9 in the morning to about 3-ish. The days where I meet my teacher are of course the least productive. Currently this is on Wednesday, breaking up the week nicely.

I do have to say that when I start there is no distraction. I will block websites such as Facebook, reddit and YouTube. 6 hours is a short time and the personal contract I have with my self is that this time will be used productively.

Also note that annexing the weekend to try and finish the thesis a few weeks earlier is probably a bad idea. I mean focusing for extended amount of times on just one subject can only be done so long. The best I ever did was a commit streak on a hobby project on github for 3 weeks, and I had a lot of fun in making that. This probably won’t be true for all the work in your thesis. You may get burned out quickly if you try and do this. (Although I personally think it is quite a lot of fun to work on it.

TODO management

Tracking what stuff you have to do is rather quite important. What I’ve found most effective is a latex package called todonotes. it allows you to insert todo items in the document itself, which will be shown as long as your document class options are drafting, but as soon as you set it to final they will be removed from the final document.

This integrates good with org, although misses any sort of highlighting. Org also doesn’t understand line breaks in todo items. Which makes it rather difficult to stay below 80 chars per line for todo items (this breaks my heart).

I use the general todo items for feedback from the teacher and also for my own thoughts. Things I think I should do end up in a todo item, this allows the teacher to see what goes on in my mind and also if I understood his feedback correctly.

Then with these notes I create some specialized commands:

\usepackage[obeyFinal, colorinlistoftodos]{todonotes}
\newcommand{\drafting}{\todo[noline, color=gray]{Working draft}}
\newcommand{\toReview}{\todo[noline, color=yellow]{To review}}
\newcommand{\newlycleared}{
  \todo[noline, backgroundcolor=white, bordercolor=red]{Newly cleared}
}
\newcommand{\cleared}{\todo[noline, color=white]{Cleared}}

Drafting for paragraphs that are incomplete and should be finished. toReview for paragraphs that have been finished but should be checked again at a later time for spell/grammar reasons. newlycleared for the items that have been cleared but haven’t been seen by the teacher (he’s the guy giving me the grade, so gotta keep him on my side). Finally cleared for the items that should be done.

Usually I build up a bunch of toReview items throughout the week and then go on a clearing spree the day before meeting the teacher. This also helps me reminding what I wrote about.

Note that the idea of using todonotes I basically stole directly from one of my classmates. But this significantly made the quality of my documents better. Since I can manage my attention much more focused. Besides looking at the todo list and looking at the document have now become the same thing.

In short

This got way longer than I thought it would. I still even haven’t talked about editing tools, version control, org mode itself, and that awesome UML library called plantuml which integrates excellently with org. I will discuss these things in a later post. I hope these tips can be use full to you.

Deployment and todo's

2016-12-27T12:00:00Z

Yesterday I deployed the website, to my terror it wasn’t responsive on mobile devices (adapt to screensize). So with this post I fixed that, by replacing the basic font-size specifications from small to 1vw (everything else was em). The rest of the site already used percentages. The site needs to be responsive because its ironic to have a CGA based theme being responsive.

So to deploy I found a vps partner in Galaxy Host Plus. They offer a vps package for about 2 euro 50 in the month I get 512MB of ram, 50GB of disk space 500GB of bandwidth and an IP address. This is a pretty good deal I think, took me about 2 hours to find (you can get a lot more expensive for less good specs, it should become cheaper every year). The domain name I had already registered at Starhosting. Its 11 euro per year. So to make this site (self) sustainable I need to make at least 11/12 + 2.5 = 3.42 euro per month. Although this is not completely fair since I’ll be using the vps for other stuff as well, such as Syncthing and private git (not for source code but just personal stuff, such as undeveloped ideas).

To setup this website it took about 2 days to figure out both pelican and styling the theme. I could have done this a lot faster with wordpress or something but it wouldn’t have made me as happy as this theme and going full static website mode. Full static website mode scales a lot better than wordpress and is cheaper in maintenance (less band with, less memory used because no PHP or Apache). Besides, I already knew how to CSS, HTML, and more pure JS than would be considered healthy (yes I’ve done some native JS development, no libraries). Actually now I think about it I would’ve probably gotten lost in wordpress and for me it wouldn’t have been faster since I really wanted this theme.

Lets talk about why I wanted this theme. First of all it fits me, I’m the kind of person who doesn’t care much for designs. Secondly I suspect its more memorable then a standard fashionable website with bootstrap. Thirdly I made it from the ground up (not the html structure, I sort off borrowed that, since I didn’t know how to pelican), which means its quite easy for me to change it. Fourthly I think there is a select group of people who like this theme, and those people probably also like what I have to say. Therefore the theme is practical.

Todo list

+Scale mobile+ This is done now.
HTTPS With lets encrypt around I probably should do this to please the security
Add advertisements This is until my Patreon finally takes off
Comment system. I have an idea for this, basically to make a reddit bot to auto create a thread for me and then link this that per post in this website with javascript. Shouldn’t be to hard, but probably a weekend of work (If anyone wants to help please contact me)

Website Launch

2016-12-25T12:04:00Z

So this is the post where I’m launching this website. I finished the css mostly as a beautifull CGA theme. In this post I’ll discuss my decision making process.

Why make a site?

As an extension of my youtube account I will use this website, mostly for subjects I can’t make videos for. Either because of lack of time or because a video format just isn’t suited for it (for example when talking about a subject involving a bunch of shell commands, this site can be a place to put those commands). It can also be for themes that need to be well referenced (controversial opinions for example).

The main purpose of this is to make money. I want to adopt a digital nomad like lifestyle (or something similar), and maintaining a website seems like a good addition. Currently I’m running this at a loss of course. Therefore I included advertisements on release and setup a patreon page. I need to make this work as soon as possible. However I also hate advertisements, therefore I make a promise to delete them as soon as patreon can cover the costs of running the website. By which I mean hosting costs (vps in my case) and domain name costs.

Site setup

I use pelican to create this site. It is a script for parsing text based content formats and generate html from them, so that you don’t have to redefine the menu every blog post for example. So what happens it that you write your content in markdown, then you run the script which generates the static html for you. I tried setting up org, but had some problems so I’ll use markdown first, I think I can mix formats anyway.

You can specify the structure of the html in the theme and of course the css. Note that even if the html is static you can still embed JavaScript, so you’re not committed to just static on each page. This is why I chose this tool. No commitments for the future and a relative simple setup. I’m thinking of later embedding a comment system (probably reddit), and macgyvering together a vote system, but this has to wait.

The theme

The CGA theme was an intentional choice. My target audience is mainly people with interests in technical things, and I’m assuming that these kind of people will like such a theme either out of nostalgia or pragmatism.

I got the inspiration from the Gentoo website they launched I think 2 years ago as an April fools joke, here is a screenshot:

(btw, if anyone has the original html implementation please contact me, I want to re-host it). I really loved the idea of just hosting a website like that. I had started working on a theme, using the kernel config as a style guide:

However I’ve been remarkably busy since that time (grad school, China internship etc), so I hadn’t come around finishing it. Untill now since I started writing my thesis, I’m realizing that soon I need to work, and I don’t really want to work for someone else.

I like making CSS designs based of images, and I’m one of those oldtimers that doesn’t use anything fancy like less.js or bootstrap. So you can see here the result of my work. I may add some more art work in the future but I’m intending to use this theme for quite a while.

Which brings us to another advantage of this theme, it will age very well. Unlike the newest fashion websites where people put huge images at the beginning of their posts and where you have to scroll endlessly to read something is this kind of theme tried and tested. I see the modern website industry becoming more like the fashion industry where practicality has to make way for beauty. I refuse to partake in such an “arms race”, redesigning my website every year or having to choose a picture every post seems just like a chore to me. If I want to add pictures to a post I’ll do it because I want to, not because its fashionable.

No CDN’s FB links or google analytics

External requests to other domains significantly decrease page loading times. Don’t believe me? Well you can measure it yourself with your browser, it has an build-in network monitor (right click -> inspect element -> network, now press ctr+f5 and see what takes the longest to load, you can see most things go in parallel, but you should pay special attention to the things that go in sequence). Many page load speed improvement website actually recommend a CDN, which is stupid because laying the http connection to a CDN is much slower then just using the existing connection.

But CDN’s aren’t meant to increase pageload, they’re meant to decrease serverload by moving parts of the static content delivery from the dynamic server to dedicated servers, freeing up server time and thus decreasing server load. But my entire website is static so why would I bother with that? server load is not going to be my bottleneck any time soon.

When I browse the internet I use umatrix to block things such as google analytics and whatever the facebook requests do. Mainly out of privacy concerns but also because this speeds every website significantly, especially if you have a poor internet connection. Besides I don’t imagine my target audience uses facebook that much, correct me if I’m wrong.

Update: I was totally wrong about CDN’s (at least for fonts)

I did some testing, and it turns out there are significant speed improvements for using a CDN. Ok you win this one, fonts.google.com. But I’m still not going to add those like buttons.

Please do not that if you test this stuff that you have to press ctrl+f5 to tell the browser not to use a cache.

See the difference in results below:

Testing myself (from the Netherlands)

Testing with a test service configured from new york:

From hong kong I got a ridiculous increase, from 3 seconds down to about 40ms. I don’t have a screenshot of that though (and to lazy to revert).

Second post

2016-12-24T19:51:00Z

Yeah, I needed two actually to make sure everything looks good.

So I guess I can type lots of information here.

I wonder how long it will continue..

Quite long acutually.

Well, that’s fine. I’ll just think off more spaced out words!

Damn I need to type so much before it will crop.

Heheheh, this is cropped.

Ok I guess this happens on word count. Or character, doesn’t really matter, at least not vertical space. Which I wondered and why I typed so much.

First post

2016-12-24T19:43:00Z

So I’m trying to setup this website and to do this I need some filler text. Some people would make prommises about putting something up every week or so. But I don’t wanna do that because I hate chores, so this may as well be a huge waste of time.

Why setup a site?

Because I’m oppinionated people.

You have some spell error over hre

Use the comment section to help fix any spell errors

There is no comment section

Exactly. We must conclude that there are no spelling errors eiher.

Spotify - Web Player: Music for everyone

Hatter: Native Haskell mobile apps

We’re All Mad Here: Wrangling Probabilities

Announcement: HSMin - more AI-friendly Haskell code!

Announcement: unwitch

Announcement: memory / ram fork

Why fork?

Haskell 💜 Vibes

Trust

Job

Conclusion

Announcement: Esqueleto postgis v4

Death💀 to type classes

sources

Private repository flake input

Full commercial example

Stacked against us

By the skin of our teeth

Winning

Firmware lemons 🍋

Experiment

Sorting

Mutex nonsense

Fixing concurrency

Results: From lemons to lemonade

Analyzing Haskell stability

Cabal exact printing

Announcement: Updated Esqueleto text-search & PostGIS bindings

Lessons from creating a vacation rental.

Follow Up on the Follow-Up

BONUS Announcement mysql pure unfork

Announcing mysql pure fork

Follow up release rodeo

MySQL Persistent Support for Haskell on Windows

Install ghcup

Build manually with msys2

Cabal

Conclusion

Update: Cabal edit

The Release Rodeo

About my startup: raster.click

Restoring msyql backup system on windows.

The peculiar event sourced deadlock

Deadlock

Now what?

Closing thoughts

Resources

Summerhouse Paradis Aruba

Summerhouse paradis on aruba vlog: Colors, mold and a hole

How do you even build a pool? Summerhouse paradis vlog III, 2022-12-18

Why do I still write this blog?

Great compromise

Frustration

Nostalgia

Interesting Challenge

Conclusion

Zurich hack 2022 Denotational Design

Denotational design

Proofs and programs

Parting words

Restoring mysql innodb on windows.

Failing in Haskell

Pure

IO and exceptions

MTL

Anti patterns

Text in left branch of Either

throw

Generic app exceptions

Squashing errors

Conclusion

References

Installing a NixOS desktop tracked with git

Getting started

Internet

Partitioning

Gdisk cheat sheet

Intended partitioning

Encryption

Formatting filesystems

`IO` and exceptions

`throw`

The type variable `m`

Transformers as constraints on `m`