Jappiehttps://jappieklooster.nl/2024-02-25T15:01:00+01:00Announcement: Updated Esqueleto text-search & PostGIS bindings2024-02-25T15:01:00+01:002024-02-25T15:01:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2024-02-25:/announcement-updated-esqueleto-text-search-postgis-bindings.html<p>I’ve updated the esqueleto bindings for
<a href="https://hackage.haskell.org/package/esqueleto-textsearch">esqueleto-textsearch</a> to include a <a href="https://hackage.haskell.org/package/esqueleto-textsearch#tutorial">tutorial</a> and
<a href="https://hackage.haskell.org/package/esqueleto-textsearch-1.1.4/docs/Database-Esqueleto-TextSearch.html">documentation</a> so it no longer requires
guesswork.
Furthermore I’ve also created new <a href="https://hackage.haskell.org/package/esqueleto-postgis">esqueleto bindings for PostGIS</a>.</p>
<p><img alt="esqueleto" src="/images/2024/esqueleto.webp"></p>
<p><a href="https://hackage.haskell.org/package/esqueleto">Esqueleto</a>
is a more advanced query library that builds on top of the <a href="https://hackage.haskell.org/package/persistent">persistent</a> <span class="caps">ORM</span><sup id="fnref-object-relational"><a class="footnote-ref" href="#fn-object-relational">3</a></sup>.
<a href="https://rachbelaid.com/postgres-full-text-search-is-good-enough/">Postgres text search</a> brings …</p><p>I’ve updated the esqueleto bindings for
<a href="https://hackage.haskell.org/package/esqueleto-textsearch">esqueleto-textsearch</a> to include a <a href="https://hackage.haskell.org/package/esqueleto-textsearch#tutorial">tutorial</a> and
<a href="https://hackage.haskell.org/package/esqueleto-textsearch-1.1.4/docs/Database-Esqueleto-TextSearch.html">documentation</a> so it no longer requires
guesswork.
Furthermore I’ve also created new <a href="https://hackage.haskell.org/package/esqueleto-postgis">esqueleto bindings for PostGIS</a>.</p>
<p><img alt="esqueleto" src="/images/2024/esqueleto.webp"></p>
<p><a href="https://hackage.haskell.org/package/esqueleto">Esqueleto</a>
is a more advanced query library that builds on top of the <a href="https://hackage.haskell.org/package/persistent">persistent</a> <span class="caps">ORM</span><sup id="fnref-object-relational"><a class="footnote-ref" href="#fn-object-relational">3</a></sup>.
<a href="https://rachbelaid.com/postgres-full-text-search-is-good-enough/">Postgres text search</a> brings <a href="https://en.wikipedia.org/wiki/Elasticsearch">Elasticsearch</a> like functionality to Postgres.
<a href="https://postgis.net/">PostGIS</a> is a spatial database extension for Postgres.
This allows querying on longitude and latitude for example, or any geometry.
As part of postgres it allows mixing normal relational data with space and geometry,
which makes space itself relational!</p>
<p>Roughly four years ago I had drafted out an implementation for the admin pages for some company.
This was part of my “trial” 2 days, where you just work as a “technical interview”. <sup id="fnref-great-for-me"><a class="footnote-ref" href="#fn-great-for-me">1</a></sup>
Ironically the admin pages ended up having far better search than the main app,
unfortunately, implementing proper text-search to the main app was never prioritized.
A colleague of mine had ported that to the <a href="https://hackage.haskell.org/package/esqueleto-textsearch">current library</a>.
Recently I needed this again.
However, I completely had forgotten how it worked and there was essentially <a href="https://hackage.haskell.org/package/esqueleto-textsearch-1.0.0.3/docs/Database-Esqueleto-TextSearch-Language.html">no documentation</a>. <sup id="fnref-hackage-upload"><a class="footnote-ref" href="#fn-hackage-upload">2</a></sup>
Since I was somehow the maintainer of this package I decided to just fix this for all Haskellers.</p>
<p>PostGIS has a different motivation.
My current contract is about finding stuff.
Now I could do this with some geometry library in pure haskell.
However, considering most logic is already in the database,
I decided it was worth a try to see if PostGIS would be a good fit.
There was already some existing art done, such as <a href="https://hackage.haskell.org/package/wkt-geom-0.0.12/docs/Data-Ewkb.html#v:parseHexByteString">reading</a> from the database,
but there was no off-the-shelf, functional writing.
I initially handcrafted some haphazard persistent instances to see if a full solution was feasible.
It worked, and one interesting outcome is that I could combine this PostGIS implementation with text-search!
For example, we’ve named locations on a map,
and objects within the database have some point on the map,
we can now search for some location on the map and find the objects through normal full-text search.
The database does all the thinking.
Because this worked so well, I got motivated to do the full bindings for PostGIS,
and prevent the issue I just had with text-search.</p>
<p>I’m slowly transforming into a database engineer 😅.
Anyway, everyone should try these libraries out!
You’re in space after all, and you lose stuff all the time, go get searching in space!
Let me know if you’ve any comments or suggestions.
Furthermore, let me know if you need help with database woes, I’m becoming quite good at this.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-great-for-me">
<p>I perform way better in such a situation than a take-home test, because it’s incredibly hard for me to get motivated to do throw away work, but here I did something useful. <a class="footnote-backref" href="#fnref-great-for-me" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-hackage-upload">
<p>Note that the hackage upload was also done by me, because my dear colleague had ported it to a library which wasn’t even published! It was just some github repository. After a couple years I decided to just upload it, right before I jumped ship to another company. <a class="footnote-backref" href="#fnref-hackage-upload" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-object-relational">
<p>Object relational mapping <a class="footnote-backref" href="#fnref-object-relational" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>Lessons from creating a vacation rental.2024-01-02T19:00:00+01:002024-01-02T19:00:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2024-01-02:/lessons-from-creating-a-vacation-rental.html<p><a href="https://www.airbnb.nl/h/jappie-aruba">Villa Katalina in aruba~ is online</a>.</p>
<p><img alt="pic of the house" src="/images/2024/katalina.jpeg"></p>
<p>You can book <a href="https://www.airbnb.nl/h/jappie-aruba">this wonderful rental now!</a>
This is the rebranded <a href="https://jappieklooster.nl/summerhouse-paradis-aruba.html">summerhouse Paradis</a>.
It’s a villa that can host up to 8 guests, located in Aruba.
I think this rental project is slowly turning out to be successful,
but it wasn’t easy …</p><p><a href="https://www.airbnb.nl/h/jappie-aruba">Villa Katalina in aruba~ is online</a>.</p>
<p><img alt="pic of the house" src="/images/2024/katalina.jpeg"></p>
<p>You can book <a href="https://www.airbnb.nl/h/jappie-aruba">this wonderful rental now!</a>
This is the rebranded <a href="https://jappieklooster.nl/summerhouse-paradis-aruba.html">summerhouse Paradis</a>.
It’s a villa that can host up to 8 guests, located in Aruba.
I think this rental project is slowly turning out to be successful,
but it wasn’t easy.
I went into this project knowing full well there would be trouble,
and boy, there was trouble.
I’ll list the issues and lessons learned from doing this for future me.
I may wish to do this again; it was a fun ride.
If you’re interested in doing something similar, I think this blog post
is useful.</p>
<p>The prices for real estate
were really good in Aruba compared to the Netherlands at the time of buying;
I got about double of what I could expect in the Netherlands.
Since my parents already have an
<a href="https://www.casaaruba.info/">investment property</a> in Aruba, it seemed
relatively easy to set up, not to say it was easy.
Another reason for doing this was the low interest rates,
which looked like attractive leverage.
Furthermore, I already had experience with running an Airbnb in my own home,
so I felt somewhat confident I could make this work.
Because my own house is not in a touristic area at all, but Aruba is.
I’d be lying if I said this only done for financial reasons.
Ego played a large part in deciding to do this as well.
For one, not being completely dependent on a single source of income
allows me to take more professional risks and become more productive as a result.
The other reason is bragging rights, not that I wish to brag,
but the only way to be humble is to have something to brag about <a href="https://www.youtube.com/watch?v=S-huj6EL3A4">and not do that</a>.
This is so hard.<sup id="fnref-ego"><a class="footnote-ref" href="#fn-ego">1</a></sup> 😬</p>
<p>I considered many other properties before settling on this one;
at first, I wanted to build a new place.
This made sense to me since labor costs are low in Aruba.
However, my father strongly urged against that, citing many experiences
of even the smallest jobs going wrong.
You have to watch contractors like a hawk if you want to do this.
I experienced such issues as well with some contractors while doing the pool house.
For example, the fuse switch in the pool house couldn’t switch off because
the plastic door was jammed too close to the switches, giving no space to move.
If you’re considering building on Aruba,
keep in mind you have to be there, or have someone trustworthy there,
and watch the contractors.</p>
<p><img alt="jappie digs the pool" src="/images/2024/jappie-dig.jpg"></p>
<p>The amount of corruption or incompetence<sup id="fnref-incompetence"><a class="footnote-ref" href="#fn-incompetence">2</a></sup> on Aruba is rather staggering.
For example,
the notary <a href="https://www.johnsonnotary.com/">Johnson and Johnson</a>,
took ages to close when I was buying the place.
The closing date was at the end of July;
I assumed it would be done before that time,
but we closed in October.
I was familiar with Aruba’s culture of taking things slowly,
but I didn’t expect things to take this long!
What I could have done differently in this case is contact the notary
more often for progress updates.
I think they just forgot about this deal or something.
I was even told this was exceptionally slow.
Because the closing took so long, I essentially missed out on opportunity costs.
I could have opened the Airbnb two months earlier, in March instead of May,
which means I missed all that income.</p>
<p>Another example of incompetence I’m currently facing is with the power company <a href="https://www.elmar.aw/">Elmar</a>,
and <a href="https://mistergreenaruba.com/">Mr. Green</a> solar services.
The solar panels were installed on 15 August, and I was promised they
could at least be turned on from that point,
subtracting power generation from my usage.
It’s now January, and this still hasn’t happened.
Even after that, the excess is still given away
to Elmar for free, after some “mythical” inspection which
has to take at least 6 months.
This delay is costing me the full power bill, which is 250 euros a month.
I have no idea why it takes this long; it’s not a big island!
Having learned from my Johnson experience, I’m contacting Mr. Green weekly.</p>
<p>What also caught me off guard was the fluctuation in exchange rates,
which caused a lot of stress.
The euro fell sharply against the dollar while I was waiting
for the deal to close,
but the listing price was denominated in dollars,
and I had euros saved up!
This made the property effectively 10% more expensive.
Considering the price was 180,000 dollars, this cost me 18,000.
I must admit, I shed a tear over this.
In the future, I should put the closing money
in an account of the same currency as soon as I sign.</p>
<p><img alt="course of euro/dollar" src="/images/2023/course.png"></p>
<p>However, what I’m really happy with is not going into hyper debt.
My father, who financed half of this investment property<sup id="fnref-interest"><a class="footnote-ref" href="#fn-interest">3</a></sup> with a mortgage-like loan,
recommended I look at more expensive properties.
So, I was looking at properties around 400k at first,
however, I decided to go with a cheaper option instead, at 180k,
even though the lender would have agreed to the more expensive option.
I’m really happy I did this because during the renovation,
I decided that a pool would drastically increase the
appeal, allowing me to charge much more rent <sup id="fnref-more"><a class="footnote-ref" href="#fn-more">4</a></sup>,
but this would require another 30k investment.
This was still possible because the monthly payments weren’t as high yet.
The biggest advantage of these more expensive properties
was their proximity to the beach.
However, I think this is somewhat overvalued,
because public transit is poor in Aruba,
so unless you’re right on the beach,
tourists are going to have to rent a car anyway.
In this regard, my more budget-friendly stay offers a lot of value to tourists.</p>
<p><img alt="pool" src="/images/2024/skilled.jpeg"></p>
<p>After handover, when doing some renovations,
many people will recommend extending the scope of a project significantly.
This can come from friends and family.
I had discussions about adding more walls for additional bedrooms, or installing doors,
or various other changes, but most of these wouldn’t be very profitable.
However, they would extend the timeline, so I refused most of these requests.
The first guest <a href="https://www.airbnb.nl/progress/reviews/details/921014041531665037">recommended</a>
me to put a fence between the pool
and the patio, to ensure the kids could play safely outside.
This was a minor change overall,
and could happen with other guests staying there.
This change solved a real issue that this guest, and by extension other guests,
were concerned about.
None of my family or friends had suggested such a change,
as nobody had stayed at the place with kids.
I kept focus on solving all these problems guests highlighted in reviews,
when they were reasonable, such as
“we miss wine glasses”, “we need trash cans in the bathrooms”,
“we don’t have oven-ready pots and pans, but there is an oven”.
Some requests were impractical, like “move the pool behind the house”, and I ignored those.
Addressing these issues usually cost less than 100 euros, and even the fence was done for less than 1000 euros.
Compared to the amount I spent on the pool and the house itself,
this was minor and a fixed costs.
However, after addressing these issues after the first few visits, the complaints
stopped and people began giving me consistent 5-star reviews,
even though the price had increased significantly in December.
I learned that the difference between a good stay and a great stay
lies in many small details, which are hard to anticipate.
Guests will share their problems if you ask.
Recommendations from family and friends during renovations need to be
carefully evaluated, as they’re not always beneficial.</p>
<p><img alt="the fence is shown behind the pool" src="/images/2024/fence.jpeg"></p>
<p>I learned that people who are good at one job
may underperform if you let them do a slightly different job.
For example, the guy who built the
wall (Cesar) and the tiling around the pool
did some great jobs, but then I let him install a shower,
and he inadvertently installed the faucet upside down.
It’s still in this awkward position.
Aruban contractors often will say they can do something,
even though it may be their first time doing such a job.
One has to be careful about this.
I like this particular contractor, however, because he can’t speak English
and I find the non-verbal communication rather amusing.
So, I’d hire him again, but not for showers!</p>
<p><img alt="shower flipped" src="/images/2024/douche.jpeg"></p>
<p>Knowing skilled people
is something that is difficult for newcomers in Aruba.
I got incredibly lucky in that regard, considering my
parents had already sifted through so many contractors for their <a href="https://www.casaaruba.info/">Casa Aruba</a>
rental property.
For example, I was able to hire a <a href="https://www.facebook.com/steigerhoutaruba/">local carpenter</a>
to redo the entire kitchen for a third of the price I would have
had to pay if I had used a European kitchen supplier.
The kitchen looks better, too!
The entire house now has this scaffolding wood
style, giving it a robust and cozy ambiance.</p>
<p><img alt="second kitchen pick" src="/images/2024/kitchen-2.jpeg"></p>
<p>I was also blessed with the support of my parents.
Besides financing, they supported me with the paintwork,
and there was a lot of it.
Without their help, I probably would have spent another 8k on that
and ended up with a worse result.</p>
<p>I’m not sure if it was financially worth it.
Intuitively, I think it was, but I still need to run the numbers more thoroughly once the first year is finished.
My preliminary observations suggest I need
to make more than 1500 euros a month to turn a profit,
and more than 3000 euros a month to cover mortgage payments as well.
I have a strict payback schedule,
because I didn’t like the high interest rate from the private market,
but this will be completed in 6 years.
It’s difficult to measure if it’s “worth it,”
because the first few months of this project were naturally the worst
I could have had.
I had no reviews so I had to compensate by lowering the price.
Furthermore, things like solar panels and my custom
<a href="https://github.com/jappeace/rentals/">rental booking</a>
site are still not operational<sup id="fnref-link"><a class="footnote-ref" href="#fn-link">5</a></sup>.
The solar panels should reduce the electric bill from roughly
250 euros to close to 0, and the rental booking website eliminates
the need for Airbnb, which takes 17% cut of bookings.</p>
<p><img alt="bed" src="/images/2024/bed.jpeg"></p>
<p>Ironically, despite all the setbacks, I may do this again.
Seeing the house come to life, reading all the reviews from people,
and experiencing their appreciation is fantastic.
However, I’d like to confirm first if it’s financially worth it.
After it has been open for a year,
I plan to create an private overview of this investment,
detailing how much it brought in and how much I spent on it.
Furthermore, I currently simply do not have the cash to set up another
one.
I feel I’m quite good at this.
However, if you happen to run a rental and have ideas for improvement,
please let me know in the comments!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-ego">
<p>Of course, I did tell people about the BnB I bought.
One could even say this post is a giant brag, but hey, I’m not forcing you to read this,
my dear reader. <a class="footnote-backref" href="#fnref-ego" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-incompetence">
<p>I say incompetence because, according to <a href="https://en.wikipedia.org/wiki/Hanlon%27s_razor">Hanlon’s razor</a>, it’s usually just that. <a class="footnote-backref" href="#fnref-incompetence" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-interest">
<p>He got a 5% interest rate while other mortages were going around for 1.5%.
So it’s not like I got a handout. <a class="footnote-backref" href="#fnref-interest" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn-more">
<p>Say 50% more. <a class="footnote-backref" href="#fnref-more" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn-link">
<p>If it is operational this <a href="https://rental.jappie.me">link</a> should work. <a class="footnote-backref" href="#fnref-link" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
</ol>
</div>Follow Up on the Follow-Up2023-11-11T00:00:00+01:002023-11-11T00:00:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2023-11-11:/follow-up-on-the-follow-up.html<style>
img[src="/images/2023/follow-up.png"]{
height: 20em;
}
</style>
<p><img alt="Reflective" src="/images/2023/follow-up.png"></p>
<p>This is a follow-up on the <a href="https://jappieklooster.nl/follow-up-release-rodeo.html">follow up</a> of <a href="https://jappieklooster.nl/the-release-rodeo.html">release rodeo</a>.
Okay, I should stop doing these,
however, I thought the situation was a bit too ironic not to record.
On my final week working for that company <sup id="fnref-unrelated-reasons"><a class="footnote-ref" href="#fn-unrelated-reasons">1</a></sup>,
the product manager …</p><style>
img[src="/images/2023/follow-up.png"]{
height: 20em;
}
</style>
<p><img alt="Reflective" src="/images/2023/follow-up.png"></p>
<p>This is a follow-up on the <a href="https://jappieklooster.nl/follow-up-release-rodeo.html">follow up</a> of <a href="https://jappieklooster.nl/the-release-rodeo.html">release rodeo</a>.
Okay, I should stop doing these,
however, I thought the situation was a bit too ironic not to record.
On my final week working for that company <sup id="fnref-unrelated-reasons"><a class="footnote-ref" href="#fn-unrelated-reasons">1</a></sup>,
the product manager involved found my original blog post!</p>
<p>A colleague and I were discussing some performance degradation,
and I brought up the <a href="https://jappieklooster.nl/the-peculiar-event-sourced-deadlock.html">event sourcing</a> post,
because this company also used event sourcing and it was maybe related.
However the <span class="caps">PM</span> apparently followed that link,
thought it interesting, looked around more, and found the <a href="https://jappieklooster.nl/the-release-rodeo.html">release rodeo</a> post!</p>
<p>He got rather upset.
His issue was that I didn’t approach him directly.
I think that’s fair,
I feel I would do this under normal circumstances,
however due to history,<sup id="fnref-ayden"><a class="footnote-ref" href="#fn-ayden">3</a></sup>
it was difficult for me to take that step.
I did solve the underlying process problem to him. <sup id="fnref-perhaps-antisocial"><a class="footnote-ref" href="#fn-perhaps-antisocial">4</a></sup>
There may not even have been a need
to single out specific roles in the original blog post,
I could’ve chatted in more general
terms and get the story across.
I think in the future I won’t do that anymore.
It’s just distracting to single out people (even anonymously).
Because it wasn’t a post about venting,
although I admit, there was a fair bit of venting.
It was about identifying and solving the underlying issue,
despite being frustrated!
Reading it back now I don’t think I said anything wrong.
Aside from using chatgpt to make it sound like it’s not me 😅</p>
<p>Anyway this is the last post in this unexpected “series”.
I learned that I should be careful with what I write,
these pages are read by more people than I thought!
Please leave a like, subscribe and share your job-rants in the comments below.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-unrelated-reasons">
<p>I decided to leave due to unrelated reasons.
It’s a good place to work,
however I had some tax issues forcing me to quit. <a class="footnote-backref" href="#fnref-unrelated-reasons" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-seriously-why-reading">
<p>I guess the days of nobody reading these are really over…
Back in 2017 nobody <a href="https://jappieklooster.nl/why-do-i-still-write-this-blog.html">read</a> these! <a class="footnote-backref" href="#fnref-seriously-why-reading" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-ayden">
<p>At some point the company had hired another manager whose only skill was office politics.
Which caused me a lot of grief.
This other manager brought the product manager from the original blog post on board
and for some reason I had an association in my brain, perhaps unfairly. <a class="footnote-backref" href="#fnref-ayden" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn-perhaps-antisocial">
<p>Perhaps in a not too social way, but what do you expect from someone who obsesses over computers for most of their life! <a class="footnote-backref" href="#fnref-perhaps-antisocial" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
</ol>
</div>BONUS Announcement mysql pure unfork2023-08-14T17:37:00+02:002023-08-14T17:37:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2023-08-14:/bonus-announcement-mysql-pure-unfork.html<p>Good news! I’ve come to an agreement with the maintainer of
<a href="https://hackage.haskell.org/package/mysql-haskell">mysql-haskell</a>, <a href="https://github.com/winterland1989">winterland</a>.
I’ll become a co-maintainer.</p>
<p>What this means in practice is that I’ll deprecate <a href="https://hackage.haskell.org/package/mysql-pure">mysql-pure</a>
and merge the changes back into <a href="https://hackage.haskell.org/package/mysql-haskell">mysql-haskell</a>.
This will make upgrades far more convenient for users.
That’s you!
It …</p><p>Good news! I’ve come to an agreement with the maintainer of
<a href="https://hackage.haskell.org/package/mysql-haskell">mysql-haskell</a>, <a href="https://github.com/winterland1989">winterland</a>.
I’ll become a co-maintainer.</p>
<p>What this means in practice is that I’ll deprecate <a href="https://hackage.haskell.org/package/mysql-pure">mysql-pure</a>
and merge the changes back into <a href="https://hackage.haskell.org/package/mysql-haskell">mysql-haskell</a>.
This will make upgrades far more convenient for users.
That’s you!
It isn’t a big deal for me either way, as I simply want the project to build reliably.
There’s no other agenda than that,
so convenience for users is paramount.
<a href="https://hackage.haskell.org/package/persistent-mysql-pure">persistent-mysql-pure</a> will, from now on, depend on mysql-haskell again.
This fork is still necessary for the persistent users out there,
because my <a href="https://discourse.haskell.org/t/ann-mysql-pure-fork-of-mysql-haskell/7297">announcement</a>
and <a href="https://github.com/naushadh/persistent-mysql-haskell/issues/2">issue</a> failed to reach that maintainer.</p>
<p>When winterland becomes active again in the future,
as they intend to,
I hope to hand off maintainership back to them.</p>Announcing mysql pure fork2023-08-13T13:37:00+02:002023-08-13T13:37:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2023-08-13:/announcing-mysql-pure-fork.html<p>I’ve forked <a href="https://hackage.haskell.org/package/mysql-haskell">mysql-haskell</a> and <a href="https://hackage.haskell.org/package/persistent-mysql-haskell">persistent-mysql-haskell</a>
into <a href="https://hackage.haskell.org/package/mysql-pure">mysql-pure</a>
and <a href="https://hackage.haskell.org/package/persistent-mysql-pure">persistent-mysql-pure</a>.
The original packages were no longer maintained and frequently caused me issues
during <span class="caps">GHC</span> upgrades,
so I decided to take over maintainership.
For example, bounds were outdated, or I needed minor patches found on
<a href="https://github.com/chordify/persistent/tree/persistent-mysql-haskell-9.2">obscure branches</a>
in <a href="https://github.com/naushadh/word24/tree/ci">unrelated repositories</a>.
I …</p><p>I’ve forked <a href="https://hackage.haskell.org/package/mysql-haskell">mysql-haskell</a> and <a href="https://hackage.haskell.org/package/persistent-mysql-haskell">persistent-mysql-haskell</a>
into <a href="https://hackage.haskell.org/package/mysql-pure">mysql-pure</a>
and <a href="https://hackage.haskell.org/package/persistent-mysql-pure">persistent-mysql-pure</a>.
The original packages were no longer maintained and frequently caused me issues
during <span class="caps">GHC</span> upgrades,
so I decided to take over maintainership.
For example, bounds were outdated, or I needed minor patches found on
<a href="https://github.com/chordify/persistent/tree/persistent-mysql-haskell-9.2">obscure branches</a>
in <a href="https://github.com/naushadh/word24/tree/ci">unrelated repositories</a>.
I’ve set up various <span class="caps">CI</span> automations.
Specifically, I’ve integrated automated version
bumping from <a href="https://github.com/nomeata/haskell-bounds-bump-action">nomeata</a> and
incorporated <a href="https://github.com/jappeace/haskell-nightly">haskell nightly</a> builds<sup id="fnref-issue"><a class="footnote-ref" href="#fn-issue">1</a></sup>.</p>
<p>I forked because I didn’t want to go through the <a href="https://wiki.haskell.org/Taking_over_a_package">process</a> of
obtaining the Hackage namespace,
which I found discouraging.
Furthermore, mysql-pure merges several dependencies
of mysql-haskell into one package.
<a href="https://hackage.haskell.org/package/word24">word24</a>,
<a href="https://hackage.haskell.org/package/binary-parsers">binary-parsers</a>,
and <a href="https://hackage.haskell.org/package/wire-streams">wirestreams</a>
are all now part of this one package.
This presents a somewhat different perspective on package organization,
which might warrant a new package <em>anyway</em>.
However, practically no third-party packages depend on the packages
I merged,
So having them separated results in more work nobody appears to benefit from.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-issue">
<p>This instantly found an issue in <span class="caps">GHC</span> on Windows for the linker: https://gitlab.haskell.org/ghc/ghc/-/issues/23835. <a class="footnote-backref" href="#fnref-issue" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Follow up release rodeo2023-07-17T00:00:00+02:002023-07-17T00:00:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2023-07-17:/follow-up-release-rodeo.html<p>This is a follow-up on the <a href="https://jappieklooster.nl/the-release-rodeo.html">release rodeo</a>.</p>
<p>After I wrote that blog post, the <span class="caps">CTO</span> found it quickly by coincidence.
He was quite okay with everything I had written.
However, he took issue with that I called his critique “scolding”.
I suppose he didn’t exactly scold me.
I …</p><p>This is a follow-up on the <a href="https://jappieklooster.nl/the-release-rodeo.html">release rodeo</a>.</p>
<p>After I wrote that blog post, the <span class="caps">CTO</span> found it quickly by coincidence.
He was quite okay with everything I had written.
However, he took issue with that I called his critique “scolding”.
I suppose he didn’t exactly scold me.
I described it that way because it aligns with my personality:
I’m an honest person,
I have high standards, and I take criticism extremely seriously.
Even when someone says, “You’re kind of doing this badly sometimes,”
I interpret it as “You suck, Jappie, you need to do better”.<sup id="fnref-psyche"><a class="footnote-ref" href="#fn-psyche">2</a></sup>
This doesn’t mean I’m unkind to others,
however I put myself to high standards.
He also warned me that if the <span class="caps">PM</span> were ever to stumble upon this post, they might get quite upset.
I am aware of this possibility, which is why I didn’t advertise this article at all.
It seems rather unlikely to me he’d go read a tech blog anyway.
The point wasn’t to create more conflict,
but instead to analyze the situation and prevent it. <sup id="fnref-venting"><a class="footnote-ref" href="#fn-venting">1</a></sup></p>
<p>A few days later, another engineer found themselves tangled in the release flag situation.
At this point, I realized it was crucial to inform everyone about how these
release flags should work.
Therefore, I set out to describe the current situation and propose some recommendations.
After outlining the recommendations, I realized the points of contention weren’t numerous.
The process worked in principle (aside from some peculiar naming conventions),
and I believe that most of the involved parties were simply uninformed.
I created a GitHub <span class="caps">RFC</span><sup id="fnref-rfc"><a class="footnote-ref" href="#fn-rfc">3</a></sup> and tagged the relevant parties on it.
This enticed them to read it and become informed and give comments if necessary.
As an open discussion, this also helped get clarification on points I had missed.
Since then, we’ve had no more release issues.
It appears that writing down the process and informing everyone resolved the issue.</p>
<p>A lesson learned from this experience is that anger can be rather productive.
If there’s something you can do about the situation,
it’s productive to think about why you’re angry and devise a plan of action.
In this case, I indeed solved a rather annoying but important issue by taking a relatively simple action.
Anyone could’ve done this.</p>
<p>Note that this post has a <a href="https://jappieklooster.nl/follow-up-on-the-follow-up.html">follow up</a>.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-venting">
<p>And also a little bit of venting! <a class="footnote-backref" href="#fnref-venting" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-psyche">
<p>From this, we can also derive the reason for my bluntness when angry. For a mind that is harsh on itself by default, being nice requires extra effort. <a class="footnote-backref" href="#fnref-psyche" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-rfc">
<p>Request for change, eg an invitation to discuss a change in process or system. <a class="footnote-backref" href="#fnref-rfc" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>MySQL Persistent Support for Haskell on Windows2023-07-01T15:44:00+02:002023-08-13T14:09:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2023-07-01:/mysql-persistent-support-for-haskell-on-windows.html<p>Using Haskell on Windows can be very useful in a <a href="https://www.wampserver.com/en/"><span class="caps">WAMP</span></a> like situation,
where the main legacy codebase is stuck at <span class="caps">PHP</span> 5.6<sup id="fnref-forever"><a class="footnote-ref" href="#fn-forever">1</a></sup> .
However, any new pages can be written with Haskell,
which is much easier to upgrade since the compiler will inform you about most changes.
New …</p><p>Using Haskell on Windows can be very useful in a <a href="https://www.wampserver.com/en/"><span class="caps">WAMP</span></a> like situation,
where the main legacy codebase is stuck at <span class="caps">PHP</span> 5.6<sup id="fnref-forever"><a class="footnote-ref" href="#fn-forever">1</a></sup> .
However, any new pages can be written with Haskell,
which is much easier to upgrade since the compiler will inform you about most changes.
New pages can, for example, be written in <a href="https://www.yesodweb.com/">Yesod</a>,
and then simply linked to, or posted to with normal <span class="caps">HTML</span>/<span class="caps">HTTP</span>.
The database can be used as an <span class="caps">API</span> to interact with <span class="caps">PHP</span> state.
Although I’ve not set up a full Yesod web server on Windows (yet),
these steps include the hardest part, the database connection.
Feel free to <a href="mailto:hi@jappie.me">contact me</a> if you want help with that. </p>
<p>This is a small instructional blog post,
primarily for my own reference,
on how to get MySQL support for programs on Windows.
These steps are for <em>native</em> Windows support. There’s no <span class="caps">WSL</span> involvement.
Specifically,
we are obtaining MySQL-persistent support with the help
of the pure Haskell <a href="https://hackage.haskell.org/package/persistent-mysql-haskell">MySQL bindings</a>.
Because the bindings are in pure Haskell, they’re easily portable.
Haskell abstracts most operating system oddities away.</p>
<p>There is support for cross-compilation on Windows somewhere
available <a href="https://github.com/input-output-hk/nix-hs-hello-windows">here</a>.
However, at the time of writing,
it has not been implemented in <a href="https://github.com/NixOS/nixpkgs/issues/36200">mainstream nixpkgs</a>.
Therefore, future Jappie,
it would be better to abandon your Nix aspirations and follow these steps.</p>
<h1 id="install-ghcup">Install ghcup</h1>
<p><a href="https://www.haskell.org/cabal/">Cabal</a> is a package manager for Haskell only.
It assumes that native bindings exist.
We’ll provide these using <a href="https://www.msys2.org/">msys2</a>,
which offers a shell and system isolation.
On top of that, we utilize <a href="https://wiki.archlinux.org/title/pacman">pacman</a>.
With this combination, we can create a Linux-like environment to generate <em>native</em> Windows executables!
We use ghcup to manage the versions of these programs.
The name comes from ghc, the main compiler. this is also installed. <sup id="fnref-nix-dfference"><a class="footnote-ref" href="#fn-nix-dfference">2</a></sup></p>
<p>Follow the instructions in the install link: <a href="https://www.haskell.org/ghcup/#ghcup-instructions-win">https://www.haskell.org/ghcup/#ghcup-instructions-win</a></p>
<p>Earlier versions suggested using <a href="https://chocolatey.org/">Chocolatey</a>,
but don’t use Chocolatey.
It is akin to <a href="https://manpages.ubuntu.com/manpages/xenial/man8/apt.8.html">apt</a>
and can cause significant problems with Haskell packages.
If you need system package management and Haskell package management,
use <a href="https://nixos.org/">Nix</a>.
It’s the only package manager that seems to work reliably.
However nix has no native Windows support.</p>
<p>Answer the other questions:</p>
<div class="highlight"><pre><span></span> default path "C:\" -- 1
cabal to C:\cabal -- 2
HLS N -- 3
stack N -- 4
msys2 Y -- 5
</pre></div>
<ol>
<li>The path used for ghcup isn’t important.</li>
<li>Again, it doesn’t matter. Placing everything at the top-level C makes it easy to find.</li>
<li>I don’t use Windows for editing. I mounted the project folder as a virtual box folder.
On Linux, I use Emacs to maintain changes,
and I run the Windows based cabal command in this mounted folder.</li>
<li>We don’t use stack as it adds another layer of complexity on top of Cabal.</li>
<li>This is necessary to build the required system dependencies with relative ease. </li>
</ol>
<p>Keep in mind that you need to reopen the terminal for the new programs,
such as <code>cabal</code> and <code>ghcup</code>, to register on the <code>$PATH</code>.</p>
<h1 id="build-manually-with-msys2">Build manually with msys2</h1>
<p>Next, you need to open an msys2 terminal (which is different from a PowerShell).
To install git, use pacman in a mingw64 terminal, which can be found in <code>C:\ghcup\msys64</code>:</p>
<div class="highlight"><pre><span></span>pacman -S git
</pre></div>
<p>There</p>
<p>are two spells, and I’m not sure which one made it work:</p>
<div class="highlight"><pre><span></span>cabal user-config -a "extra-prog-path: %HOME%\.ghcup\bin, %HOME%\AppData\Roaming\cabal\bin, C:\\ghcup\msys64, C:\\ghcup\msys64\mingw64\bin, C:\\ghcup\msys64\user\bin" -a "extra-include-dirs: C:\\ghcup\msys64\mingw64\include" -a "extra-lib-dirs: C:\\ghcup\msys64\mingw64\lib" -f init
</pre></div>
<p>I also did this:</p>
<div class="highlight"><pre><span></span> <span class="o">$</span><span class="nt">Env</span><span class="p">:</span><span class="nd">Path</span> <span class="o">+=</span> <span class="s2">";C:\ghcup\msys64\mingw64\bin"</span>
<span class="o">$</span><span class="nt">Env</span><span class="p">:</span><span class="nd">Path</span> <span class="o">+=</span> <span class="s2">";C:\ghcup\msys64\usr\bin"</span>
</pre></div>
<p>After these steps, it worked.
Now cabal can find git, and therefore download our patched versions.
Fret not, I’m working on getting these changes into Hackage directly.
It just takes time to do all the politics involved and
give people the opportunity to reply to my requests.
However, even if I manage to get the changes upstream,
there is a big chance you’d need one native dependency or another anyway,
which this setup will give you.</p>
<h1 id="cabal">Cabal</h1>
<p>You can generate an intial configuration with <code>cabal init</code>,
then add <code>persistent-mysql-haskell</code> dependency to your build-depends:
<code>packagename.cabal</code> file:</p>
<div class="highlight"><pre><span></span> build-depends:
, base >=4.9.1.0 && <5
, persistent-mysql-pure
</pre></div>
<p>In combination with the <code>cabal.project</code> file, cabal should be using
git instead of hackage to find that package.
Test it out with with <code>cabal build</code>.</p>
<h1 id="conclusion">Conclusion</h1>
<p>This should set you up with Windows development for Haskell
in more complicated setups than just some puzzles.
You’re probably better off getting an Ubuntu <span class="caps">VM</span> if you just
want to build something <em>new</em> in Haskell.
However, for legacy integrations, this will work.
If not, please leave a comment below, or <a href="mailto:hi@jappie.me">contact me</a>.</p>
<h1 id="update-cabal-edit">Update: Cabal edit</h1>
<p>The cabal section used to contain below text, however since I’ve
forked <a href="https://jappieklooster.nl/announcing-mysql-pure-fork.html">mysql-pure</a>. this is no longer necessary.</p>
<p>You need to instruct Cabal to use the correct MySQL packages <code>project.cabal</code>:</p>
<div class="highlight"><pre><span></span>packages: .
allow-newer: wire-streams:bytestring, binary-parsers:bytestring
source-repository-package
type: git
location: https://github.com/chordify/persistent
tag: cf735587e590369e62168dacc2c3c2411493ae6d
subdir: persistent-mysql-haskell
source-repository-package
type:git
location: https://github.com/jappeace/mysql-haskell
tag: 8f95f0a4749b888eba96173378acbede3955ab60
source-repository-package
type:git
location: https://github.com/naushadh/word24
tag: 1cc234d53923c270e888fdeac868c34306c43c70
source-repository-package
type:git
location: https://github.com/naushadh/word24
tag: 1cc234d53923c270e888fdeac868c34306c43c70
</pre></div>
<p>You can generate an intial configuration with <code>cabal init</code>,
then add <code>persistent-mysql-haskell</code> dependency to your build-depends:
<code>packagename.cabal</code> file:</p>
<div class="highlight"><pre><span></span> build-depends:
, base >=4.9.1.0 && <5
, persistent-mysql-haskell
</pre></div>
<p>In combination with the <code>cabal.project</code> file, cabal should be using
git instead of hackage to find that package.
Test it out with with <code>cabal build</code>.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-forever">
<p>Forever, because upgrading is too painful and introduces to many runtime bugs. <a class="footnote-backref" href="#fnref-forever" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-nix-dfference">
<p>To be clear, the major difference with nix is that nix in principle leverages cabal and ghc, but nix manages the versions of these, but also solves the native part in a reproducible manner. This means I can write a script for a build and it mostly will keep on working forever, unless source mirrors disappear for example.  <a class="footnote-backref" href="#fnref-nix-dfference" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>The Release Rodeo2023-05-31T20:14:00+02:002023-08-19T12:38:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2023-05-31:/the-release-rodeo.html<style>
img[src="/images/2023/release-rodeo.png"]{
height: 20em;
}
</style>
<p><img alt="Release rodeo~" src="/images/2023/release-rodeo.png"></p>
<p>In the beginning, there was nothing.
We merged completed features after a month or so of development,
leading to large PRs and often unexpected implementations.
Then, feature flags were introduced.
This allowed the merging of code in chunks,
enabling <span class="caps">QA</span> to test …</p><style>
img[src="/images/2023/release-rodeo.png"]{
height: 20em;
}
</style>
<p><img alt="Release rodeo~" src="/images/2023/release-rodeo.png"></p>
<p>In the beginning, there was nothing.
We merged completed features after a month or so of development,
leading to large PRs and often unexpected implementations.
Then, feature flags were introduced.
This allowed the merging of code in chunks,
enabling <span class="caps">QA</span> to test thoroughly.
The people in charge could now also decide <em>when</em> to release.
Because this also facilitated more, smaller PRs<sup id="fnref-PR"><a class="footnote-ref" href="#fn-PR">1</a></sup>,
it also made the reviewing process much easier.
Everything was awesome in paradise,
but not all was as it seemed…</p>
<p>The flag count grew,
and salespeople started to realize they could enable flags for certain customers.
This went well, in the beginning.
Except that these features were still in development.
So, on one occasion, a client success person panicked, after
a fancy file uploader behind a feature flag disappeared.
This happened because it was merged into another feature flag<sup id="fnref-problem"><a class="footnote-ref" href="#fn-problem">2</a></sup>.
Now certain customers no longer had access to the file uploader
to which they had grown accustomed,
breaking their entire workflow.</p>
<p>Note that in this setting, a customer is a large corporation with thousands of employees.
So if one of them becomes unhappy,
the startup could lose a lot.
Not just money but also reputation, which translates into future sales,
potentially threatening the survival of the startup.
Up until now, this hasn’t happened;
all customers who’ve signed up have always been kept happy.</p>
<p>Anyway, long story short, since the feature was being used in production,
I strongly urged its release.
This was discussed internally,
and everyone agreed on releasing this file uploader feature.
<em>Everyone</em>. However, some requests were made, for example, there were redundant form elements.
The manager suggested also deleting the old file uploading button.
This was a good idea,
because with this change the old flow would be quite broken;
we had already removed error messages.
The manager in charge was tagged and ignored the <span class="caps">PR</span> for several days,
after which we decided to merge.
We had already decided to release this feature,
so his approval wasn’t necessary according to our customs.</p>
<p>Fast forward to the next week,
and our dear client success person was once again in a panic.
It turns out that one of our older customers was still using that confusing button.
So we had to put it back.
I warned them that this old flow was broken,
lacked a lot of elements and would cause confusion.
They just wanted it back however, so I complied.
This was a rather minor change,
however the manager in charge of releases,
did what any good leader would do and blamed the developers.
<em>Even though this was his own suggestion</em>.</p>
<p>I wouldn’t be writing an angry blogpost if it was the only incident.
There is more.
He was right about the confusion that followed from leaving the button.
<span class="caps">QA</span> got confused, another dev got confused, I’m surprised more customers weren’t complaining about this.
Although they likely just ended up feeling stupid and didn’t complain, which is <em>worse</em>.
It seems impossible to delete that darn button, however.</p>
<p>Moving forward a month, a large client was preparing for a demo.
I started working on a “detection” feature which we needed to impress said client.
We had a meeting to figure out what we could get done before the demo.
From this we learned we had to modify the Elm apps.
These Elm apps represent <span class="caps">HTML</span> tables that open up a modal with the specific data comprising those tables.
It’s like eight different tables, all looking slightly different,
all piped into the same Elm app with flags specifying how it should look.
We’ve had production bugs with these apps.
They just stop functioning and you’ve no idea why.
Furthermore, one of the best features of Elm,
its debugger, is unusable because they overlap,
since there are several Elm apps displayed on the page.
As you can tell, I’m not too happy having to modify these.
It’s a lot of pointless complexity.</p>
<p>Anyway, the demo went well,
as they skirted around those Elm apps and didn’t open the modals,
which show the wrong information our new feature detects.
Now, 2 weeks later, the manager decides it’s a good idea to release this feature.
I learned this through an announcement in #general.
But I wasn’t finished yet, since I was away for 1 week on vacation.
I’m kind of shocked.
If clients see those numbers don’t align with that new “detection” feature and our Elm apps,
they’re going to freak out.
Because they’re going to think our platform messed up their data somehow.
So I reply to his announcement, because I know he’s well aware of this.</p>
<blockquote>
<p>I’m not sure if I’d release the “feature” because “our Elm tables” are kind of incoherent right now. But if you’re happy with it 🤷♂️</p>
</blockquote>
<p>Here I redacted the names of the specific features,
replacing them by “feature” and “our Elm tables”.
Now, is this blunt? Yes, but there is <em>a lot of money on the line</em>.
The same manager who blamed the developers for the missing button incident
was also responsible for this release.
Naturally, he employed his remarkable management skills
and decided to throw a tantrum, persuading the <span class="caps">CTO</span> to scold me.
Indeed, I was reprimanded and praised simultaneously.
It was beneficial that we detected this issue early,
but I was told that my communication was too blunt and insufficiently precise.
I believe there’s truth in this.
I could’ve been calmer, but maintaining composure is challenging in a stressful situation.
Part of this blog post is about circumventing such situations in the future.
The manager could’ve performed better too.
For instance, he could’ve inquired in #tech before releasing,
“Are there any serious issues remaining with feature X, Y, or Z?”
Just like one would do in a team.
Alternatively, he could’ve engaged with my response in #general to figure out what was happening. </p>
<p>I was angry when starting to write this.
And while I wanted to blame a particular individual,
I won’t.
This is the wrong mindset. <sup id="fnref-dharma-samsara"><a class="footnote-ref" href="#fn-dharma-samsara">4</a></sup>
Rather than blaming specific individuals, I prefer to think in terms of systems.
How is our process so broken and causing stress?
Well, no one has cared about thinking about how all of this should work internally, at least.
This is so bad we don’t even have a proper definition of what it means to release
a feature:</p>
<ol>
<li>We delete the feature flag and old code.</li>
<li>We enable the feature flag for all customers as a setting.</li>
</ol>
<p>Point <code>1.</code> should happen eventually,
but after reading the external communication process, I think we should first do <code>2.</code>,
wait for some time, and then delete the feature flag.
It’d be nice if we put a limit on that as well, so developers can safely delete at some point.
Even though there appears to be documentation on what clients need to be informed,
internally we have no clue who to tell what or get involved.
We don’t have a grace period either,
allowing people to comment on a release.
For example, the customer success people can just say if a button is missing in this grace period,
or a developer implementing it can just say that hey, we’re showing incoherent stuff.
There is no guidance on how big a release should be,
are smaller ones better or bigger ones.
It seems like the customer success and marketing people prefer big releases.
So they can use it as a marketing moment,
and have an easier time demoing more features in one go.
But I suspect for engineers, smaller ones are preferred,
as this allows them not to be overwhelmed if issues arise.</p>
<p>How would you release?
I’ve always learned to release early and release often.
But ever since the introduction of feature flags,
we’ve just not been releasing,
until it became a problem.
In typical startup style sledgehammer fashion,
the problem was solved; release a bunch of stuff,
and for the unfinished parts we peeled them off, putting them behind different flags.
But does the path towards releasing this software have to be so stressful?
I feel we’re doing it wrong.
The lack of some kind of formalized strategy causes some issues here, I think.
We’re just doing stuff, inconsistently, and coordination revolves around people being shocked and in a panic.
This clearly isn’t working.</p>
<p>Documenting what I believe should happen,
without naming specific individuals or organizations, <sup id="fnref-could-be-anyone"><a class="footnote-ref" href="#fn-could-be-anyone">3</a></sup>
has at least alleviated my anger.
I’m not sure if the procedures I’ve outlined constitute a good process,
hence I welcome comments,
but I believe it’s a step forward from doing things haphazardly.
Getting a timeline for a release at the very least would be beneficial.
Therefore, with my anger tempered and a sense of progress, I deem this a successful blog post 💪.
Please share your thoughts in the comments.</p>
<p>Note that this post has a <a href="https://jappieklooster.nl/follow-up-release-rodeo.html">follow up</a>.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-PR">
<p>Pull requests, the primary mechanism in which developers align their changes.
It opens up a moment for questions or comments. <a class="footnote-backref" href="#fnref-PR" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-problem">
<p>This itself was also problematic.
The issue was that designers began wanting to make changes (as is their job),
but developers started asking questions about how to implement these changes,
and realized it was easier to just merge features. <a class="footnote-backref" href="#fnref-problem" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-could-be-anyone">
<p>Could be anyone really, since I do work freelance :) <a class="footnote-backref" href="#fnref-could-be-anyone" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn-dharma-samsara">
<p>Growing deaf to dharma and being trapped in samsara. <a class="footnote-backref" href="#fnref-dharma-samsara" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
</ol>
</div>About my startup: raster.click2023-02-25T20:40:00+01:002023-02-25T20:40:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2023-02-25:/about-my-startup-rasterclick.html<style>
img[alt="raster logo"]{
width:40%;
margin-left: 30%;
}
</style>
<p><img alt="raster logo" src="./images/2023/raster-logo.svg"></p>
<p>As a founder of <a href="https://raster.click/">raster.click</a>,
I was passionate about providing rostering systems for restaurants
that were easy to use and affordable.
I worked hard to build a functional product that our users loved,
and I was proud to have three paying …</p><style>
img[alt="raster logo"]{
width:40%;
margin-left: 30%;
}
</style>
<p><img alt="raster logo" src="./images/2023/raster-logo.svg"></p>
<p>As a founder of <a href="https://raster.click/">raster.click</a>,
I was passionate about providing rostering systems for restaurants
that were easy to use and affordable.
I worked hard to build a functional product that our users loved,
and I was proud to have three paying customers (eg restaurants)
and around 90 active users (employees).</p>
<p>Despite initial success,
I realized that growth was not fast enough to become economically viable.
I was unable to attract enough paying customers to cover costs.
As a self-bootstrapped startup,
I did not have access to significant funding,
which made it challenging to scale the company and invest in marketing efforts.
I also struggled with balancing time between
product development and customer acquisition,
which slowed growth even further.</p>
<p>In 2019, I put in a lot of effort and remained dedicated to Raster.click.
However, I eventually had to make the tough decision to discontinue
my work and return to contract work.
It was a heartbreaking decision,
as I knew my product was loved by my users.
They continued to use it until the <span class="caps">COVID</span>-19 pandemic hit
and practically all usage stopped.
The restaurant industry was severely affected by the pandemic,
rendering rostering completely useless during that period.</p>
<p>Although I still have an appreciation for the tech stack I chose - reflex + servant + beam -
I’ve come to realize that a cool tech stack alone doesn’t matter to users.
While it may result in faster product development,
it ultimately doesn’t help if I can’t gather user requirements or attract new users.
In hindsight,
I could have been more productive by using yesod and relying on browser defaults
instead of creating a single page app.
However, the decision to end my startup was not based on the tech stack.
Despite rolling out new features,
I simply wasn’t gaining enough customers to make it a viable business.</p>
<p>While I am proud of what I accomplished with Raster.click,
I learned valuable lessons about the challenges of building
and growing a startup without significant funding.
Such as:</p>
<ul>
<li>The importance of having a clear and viable business model:
It’s essential to have a solid plan for how your startup will generate revenue and grow sustainably.</li>
<li>The need for effective marketing and customer acquisition:
Even if you have a great product,
you need to be able to effectively reach and convince potential customers to use it.</li>
<li>The importance of balancing product development and customer acquisition:
While it’s important to create a great product,
you also need to be able to attract and retain customers.</li>
<li>The value of having access to funding:
Without significant funding,
it can be challenging to scale your startup and compete effectively in the market.
Not to mention the stress caused by seeing your life savings slowly drop to zero.</li>
</ul>
<p>I will take these lessons with me as I move forward with future endeavors.</p>
<p>Although I stopped working on this project,
it’s still <a href="https://raster.click/">operational</a>.
It costs very little money for me to keep this operational.
That’s because it runs next to this site, <a href="https://videocut.org/">videocut</a>,
and <a href="https://massapp.org/">massapp</a> on the
<a href="https://jappie.me/the-nix-mutli-monolith-machine-nmmm.html">nmm approach</a>.
Yes, it’s sad I stopped working on this project,
but if anyone is interested in using it,
<a href="https://raster.click/">it’s right there</a>.</p>Restoring msyql backup system on windows.2023-01-17T17:00:00+01:002023-01-17T17:00:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2023-01-17:/restoring-msyql-backup-system-on-windows.html<p>This is a quick after action report on restoring backups.
This is mostly for future me to know what I did.</p>
<p>The backup scripts on the main server stopped working.
The <span class="caps">IT</span> person at that company
had already figured out this was because
of the database not booting on the …</p><p>This is a quick after action report on restoring backups.
This is mostly for future me to know what I did.</p>
<p>The backup scripts on the main server stopped working.
The <span class="caps">IT</span> person at that company
had already figured out this was because
of the database not booting on the machines which were
supposed to receive the backups,
but figuring out the underlying cause was beyond his capability.
So I was called in.
Yes. I’m slowly turning into a database administrator
at <a href="https://jappie.me/restoring-mysql-innodb-on-windows.html">this</a> <a href="https://jappie.me/the-peculiar-event-sourced-deadlock.html">rate</a>.</p>
<p>So xampp<sup id="fnref-why-xampp"><a class="footnote-ref" href="#fn-why-xampp">1</a></sup> told us to look at the logs, those said:</p>
<div class="highlight"><pre><span></span><span class="n">error</span><span class="o">:</span> <span class="n">trying</span> <span class="n">to</span> <span class="n">access</span> <span class="n">page</span> <span class="n">number</span> <span class="mi">123</span> <span class="k">in</span> <span class="n">space</span> <span class="mi">0</span>
</pre></div>
<p>Stack overflow <a href="https://stackoverflow.com/questions/38245974/mysql-server-crashes-innodb-outside-the-tablespace-bounds">said</a>
that we need to recover from backup if <code>innodb_force_recovery</code>
doesn’t work.
It doesn’t say how.
Remember, we can’t even start the database!
What I did instead was deleting the <code>innodb0</code> file, and the <code>innodb.log</code> files.
Remember, I actually don’t care about the data,
it’s just a backup machine.
This made the database boot again,
making it ready to receive backups.</p>
<p>One more peculiarity occurred (of course).
The script would timeout saying it’s a connection error.
However this error happened every time I executed the script,
and I’d assume that at an office the connection is pretty
reliable.
This made me think it was just mislabeling the error.
Once again <a href="https://stackoverflow.com/a/24555191">stack overflow</a> agreed.
This database has lots of blob data,
so setting <code>max_allowed_packet</code> to <code>64M</code> indeed fixed the connection error.</p>
<p>I’m starting to think I should move this entire
stack over to postgres,
at least that doesn’t randomly start failing,
or has mislabeled errors.
But I know that’s next to impossible with this clients’
resources.
So for now, Jappie is here for all your database concerns.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-why-xampp">
<p>Why are they using xampp? Legacy choices.
Remember I don’t get to chose their tech stack,
that has been decided more then a decade ago.
I’m just keeping it operational. <a class="footnote-backref" href="#fnref-why-xampp" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>The peculiar event sourced deadlock2023-01-15T22:30:00+01:002023-01-15T22:30:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2023-01-15:/the-peculiar-event-sourced-deadlock.html<p><img alt="THE UNDEAD LOCK OF DETH" src="images/2023/postgresql-deadlock.png"></p>
<p>One thing that always surprises me is how casually
serious problems are phrased by business people
in their blissful ignorance.
“Hey why am I seeing the down for maintenance screen?”
“Oh try it now, the pack uploading has finished”,
Said the <span class="caps">QA</span> engineer to the product manager.
Once I saw …</p><p><img alt="THE UNDEAD LOCK OF DETH" src="images/2023/postgresql-deadlock.png"></p>
<p>One thing that always surprises me is how casually
serious problems are phrased by business people
in their blissful ignorance.
“Hey why am I seeing the down for maintenance screen?”
“Oh try it now, the pack uploading has finished”,
Said the <span class="caps">QA</span> engineer to the product manager.
Once I saw this on slack, I grew really suspicious
and started asking questions.
After all, isn’t it a bit odd we’re seeing a down for maintenance screen
in one part of the system,
simply because another part is being used?</p>
<p>Initially we thought this was caused by high <span class="caps">CPU</span> usage.
The graphs showed high <span class="caps">CPU</span> load while processing packs,
so maybe the rest of the system was being deprioritized somehow.
Before assuming that was the cause however,
I decided to reproduce the issue first.
Here I noticed I could for example load the risk index
easily (a read operation),
but connecting a risk to a pack (a write operation), would hang forever.
This made me suspect that the issue wasn’t
<span class="caps">CPU</span> usage at all,
so I asked Postgres to list <a href="https://wiki.postgresql.org/wiki/Lock_Monitoring">it’s locks</a>.
Which showed several locks in progress.
This lead me to the event source system.
The event source system is at the core of all our business logic.
In essence, it provides a ledger of all important business write
activities that can happen.
<a href="https://www.youtube.com/watch?v=8JKjvY4etTY">This is useful</a>
for auditing purposes for example.</p>
<p>Welcome to an after action report of a complicated
system level bug.
It took me a week to find a satisfying
solution.
To start I need to sketch context.
I’ll only use raw <span class="caps">SQL</span> because
this entire story is related to the database and how we use it for event sourcing.
So consider the tables of an event source system:</p>
<div class="highlight"><pre><span></span><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">event</span> <span class="p">(</span>
<span class="n">id</span> <span class="nb">serial</span> <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">payload</span> <span class="n">jsonb</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="k">type</span> <span class="nb">character</span> <span class="nb">varying</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">created</span> <span class="k">timestamp</span> <span class="k">with</span> <span class="n">time</span> <span class="k">zone</span> <span class="k">NOT</span> <span class="k">NULL</span>
<span class="p">);</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">event_last_applied</span> <span class="p">(</span>
<span class="n">id</span> <span class="nb">serial</span> <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">event_id</span> <span class="nb">bigint</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">REFERENCES</span> <span class="n">event</span> <span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="p">);</span>
</pre></div>
<p>In here the <code>type</code> and <code>payload</code> fields
contains the information to (re)apply that event.
The <code>type</code> will indicate what business logic or
queries to execute, and the <code>payload</code> holds
information for that logic.
As we’ll see later,
these queries will involve modifying other normal
tables within a transaction.
This application of events, or re-application
through business logic or queries is called projecting.
A <code>type</code> can for example be <code>create-user</code>
and the <code>payload</code> would contain the data required for creating said user,
for example <code>{email:'hi@jappie.me'}</code>.
The <code>id</code> provides a unique global ordering,
and the <code>created</code> field contains a timestamp of when the event
was created, which is used for database administration purposes.
Finally, the <code>event_last_applied</code> table is used to indicate
whichever event was last applied, so the system
can figure out if additional events need to be
re-projected from the <code>event</code> table.</p>
<p>Inserting an event
works by projecting an event to normal
Postgres tables in a transaction.
Once this operation is not rejected by foreign keys,
type errors or program exceptions,
the event gets recorded in the ledger,
also known as the <code>event</code> table. For example:</p>
<div class="highlight"><pre><span></span><span class="k">begin</span><span class="p">;</span>
<span class="cm">/* left out projection code, insert user into tables here,</span>
<span class="cm">or do other projection stuff, as dictated by the event type*/</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">event</span> <span class="p">(</span><span class="n">payload</span><span class="p">,</span> <span class="k">type</span><span class="p">,</span> <span class="n">created</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="s1">'{"email":"hi@jappie.me"}'</span><span class="p">,</span> <span class="s1">'create-user'</span><span class="p">,</span> <span class="n">now</span><span class="p">());</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">event_last_applied</span> <span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="n">event_id</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="mi">1</span><span class="p">,</span> <span class="k">max</span><span class="p">(</span><span class="n">id</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">event</span>
<span class="k">ON</span> <span class="n">CONFLICT</span> <span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="k">DO</span> <span class="k">UPDATE</span> <span class="k">SET</span>
<span class="n">event_id</span> <span class="o">=</span> <span class="n">lastval</span><span class="p">();</span>
<span class="k">commit</span><span class="p">;</span>
</pre></div>
<p>If the projection fails the entire event
gets rejected,
which means all changes within the
transaction get rolled back by Postgres.
This applies relational guarantees,
to a non-relational system trough a transaction.
We also weave this transaction trough business logic
code,
so that in case of an exception,
we rollback.
Quite an elegant solution, which
<a href="https://garba.org/posts/2016/event-sourcing/#materialised-view-pattern">I did</a>
<a href="https://www.ahri.net/2019/07/practical-event-driven-and-sourced-programs-in-haskell/">not invent</a>.</p>
<p>On system boot we figure out if we need to reproject
or not,
the query is rather simple:</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span> <span class="k">type</span><span class="p">,</span> <span class="n">payload</span> <span class="k">FROM</span> <span class="n">event</span>
<span class="k">WHERE</span>
<span class="n">id</span> <span class="o">></span> <span class="p">(</span>
<span class="k">SELECT</span> <span class="n">event_id</span> <span class="k">FROM</span> <span class="n">event_last_applied</span>
<span class="k">WHERE</span> <span class="n">id</span> <span class="o">=</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">ORDER</span> <span class="k">BY</span>
<span class="n">id</span> <span class="k">ASC</span><span class="p">;</span>
</pre></div>
<p>which returns something like this, telling the system what to do:</p>
<div class="highlight"><pre><span></span> type | payload
-------------+---------------------------
create-user | {"email": "hi@jappie.me"}
</pre></div>
<p>With that, we can reproject, also known as replaying history.
Replaying history involves truncating all tables
that are event sourced.
And then truncating the <code>event_last_applied</code>
table,
which in this case just removes the one row.
Then the system will notice it needs to replay
events on boot for example.
This is a rather dangerous operation,
because if any event fails,
you may have potentially lost data.
A lot of things can go wrong with a large history,
foreign keys, exceptions, serialization mismatches,
events out of order etc.
Transactions can help here as well, and make this re-projection safe.</p>
<h2 id="deadlock">Deadlock</h2>
<p>There is one more important piece of context:
An event maybe composed with other events
into larger transactions.
For example,
if we create a user,
we may also assign him to a company
within the same transaction.
In <span class="caps">SQL</span> that looks like this:</p>
<div class="highlight"><pre><span></span><span class="k">BEGIN</span><span class="p">;</span>
<span class="cm">/* left out projection code, insert user into tables here */</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">event</span> <span class="p">(</span><span class="n">payload</span><span class="p">,</span> <span class="k">type</span><span class="p">,</span> <span class="n">created</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span>
<span class="cm">/* whatever event source data*/</span>
<span class="s1">'{"email":"hi@jappie.me"}'</span><span class="p">,</span> <span class="s1">'create-user'</span><span class="p">,</span> <span class="n">now</span><span class="p">());</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">event_last_applied</span> <span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="n">event_id</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="mi">1</span><span class="p">,</span> <span class="k">max</span><span class="p">(</span><span class="n">id</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">event</span>
<span class="k">ON</span> <span class="n">CONFLICT</span> <span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="k">DO</span> <span class="k">UPDATE</span> <span class="k">SET</span>
<span class="n">event_id</span> <span class="o">=</span> <span class="n">lastval</span><span class="p">();</span>
<span class="cm">/* left out projection code, connect user to company */</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">event</span> <span class="p">(</span><span class="n">payload</span><span class="p">,</span> <span class="k">type</span><span class="p">,</span> <span class="n">created</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span>
<span class="cm">/* whatever event source data*/</span>
<span class="s1">'{"company-id":2, "user-id": 1}'</span><span class="p">,</span> <span class="s1">'connect-company'</span><span class="p">,</span> <span class="n">now</span><span class="p">());</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">event_last_applied</span> <span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="n">event_id</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="mi">1</span><span class="p">,</span> <span class="k">max</span><span class="p">(</span><span class="n">id</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">event</span>
<span class="k">ON</span> <span class="n">CONFLICT</span> <span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="k">DO</span> <span class="k">UPDATE</span> <span class="k">SET</span>
<span class="n">event_id</span> <span class="o">=</span> <span class="n">lastval</span><span class="p">();</span>
<span class="k">COMMIT</span><span class="p">;</span>
</pre></div>
<p>Transactions form proper <a href="https://hackage.haskell.org/package/base-4.17.0.0/docs/Data-Monoid.html">monoids</a>,
and they can grow arbitrarily large.
This is good because even for large chuncks
of business logic we always gaurantee our
event log remains in a valid state.
We’d expect our re-projections to always work,
because only correct ones get recorded.
Where does this go wrong then?</p>
<p>The issue is concurrency,
consider connection <code>A</code> and <code>B</code>:</p>
<ol>
<li><code>A</code> opens a transaction and inserts a user,
but has to do other projections and event insertions as well</li>
<li><code>B</code> opens a transaction and wants to insert an event,
<code>B</code> has to wait until <code>A</code> completes.
This is because <code>A</code> made an update to the <code>event_last_applied</code> on row number <code>1</code>,
as part of the insert event logic.
This row is locked until <code>A</code> completes, so <code>B</code> has to wait.</li>
<li><code>A</code> completes and releases the lock on row <code>1</code>.</li>
<li><code>B</code> can now complete as well.</li>
</ol>
<p>This is not a deadlock as long as <code>A</code> completes.
<code>B</code> can wait a long time
because our transactions can grow arbitrarily large.
For example when we’re inserting millions of rows of data,
taking up half an hour.
Which is far beyond the <span class="caps">HTTP</span> session length of 30 seconds,
or whatever length a user finds acceptable.
This was indeed the production bug encountered at <a href="https://supercede.com/">supercede</a>.
One user was doing pack ingestion,
which involves reading millions of excell file rows,
and the rest of the system became unusable because of that.</p>
<h2 id="now-what">Now what?</h2>
<p>At first I started with the most obvious solution.
I re-grouped how event sourcing took place.
I put the event sourcing code at the end of the
transaction in pack ingestion,
so that the event source table remained available
for other transactions up till that point.
Because event sourcing is only a small part
of normal transactions,
this created a small locking window.
Thus this worked!
However it only worked for this transaction with
pack ingestation,
I didn’t know if there were any other transactions
like this in our code base.
Furthermore, I had to bypass parts of the event
sourcing interface to make this work.
For example, I had to project events by hand,
and insert events by hand,
rather then using the internal library.
I decided this was a bad precedence to set.
I was afraid other engineers would copy this approach
when it wasn’t necessary.
So I went looking for other solutions.</p>
<p>Another idea is that instead of doing the large transaction,
we could split it up into smaller ones.
Allowing other events to clear while this bigger one
was in progress.
I didn’t like this either.
For one this code was old, tried and tested,
making a rather large modification like splitting
the transaction could introduce many unintended bugs.
For example when cleanup doesn’t happen correctly on failure.
I thought this was likely because this transaction was large,
and covered many tables.
Also our normal tools such as types and integration tests
wouldn’t help a lot with guaranteeing cleanup.
So this would become difficult to maintain fast.
Which is problematic for a piece of code which is the “money maker”,
and needs to change often.
Furthermore I had a much more simple but thorough solution in mind.</p>
<p>I decided to redesign the event source tables.
Naturally my colleagues exclaimed shouts of joy when
I decided to modify an even older system.
The event source system described above is almost as old as
supercede.
But I believed it was easier to modify,
and more importantly,
easier to test for correctness.
Furthermore this would also solve the problem for other,
possibly unknown, or future, large transactions.
This change would keep our code easy to maintain
and solve a bug.
The new schema looks almost identical to the old one:</p>
<div class="highlight"><pre><span></span><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">event</span> <span class="p">(</span>
<span class="n">id</span> <span class="nb">serial</span> <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">payload</span> <span class="n">jsonb</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="k">type</span> <span class="nb">character</span> <span class="nb">varying</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">created</span> <span class="k">timestamp</span> <span class="k">with</span> <span class="n">time</span> <span class="k">zone</span> <span class="k">NOT</span> <span class="k">NULL</span>
<span class="p">);</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">event_applied</span> <span class="p">(</span>
<span class="n">id</span> <span class="nb">serial</span> <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">event_id</span> <span class="nb">bigint</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">REFERENCES</span> <span class="n">event</span> <span class="p">(</span><span class="n">id</span><span class="p">),</span>
<span class="n">created</span> <span class="k">timestamp</span> <span class="k">with</span> <span class="n">time</span> <span class="k">zone</span> <span class="k">NOT</span> <span class="k">NULL</span>
<span class="p">);</span>
</pre></div>
<p>The big difference is that we renamed
<code>event_last_applied</code> to <code>event_applied</code>
and added a created field.
With this change,
inserting events is also quite similar to the initial system:</p>
<div class="highlight"><pre><span></span><span class="k">BEGIN</span><span class="p">;</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">event</span> <span class="p">(</span><span class="n">payload</span><span class="p">,</span> <span class="k">type</span><span class="p">,</span> <span class="n">created</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="s1">'{"email":"hi@jappie.me"}'</span><span class="p">,</span> <span class="s1">'create-user'</span><span class="p">,</span> <span class="n">now</span><span class="p">());</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">event_applied</span> <span class="p">(</span><span class="n">event_id</span><span class="p">,</span> <span class="n">created</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="n">last_value</span><span class="p">,</span> <span class="n">now</span><span class="p">()</span> <span class="k">FROM</span> <span class="n">event_id_seq</span><span class="p">;</span>
<span class="k">COMMIT</span><span class="p">;</span>
</pre></div>
<p>The big difference is that instead of modifying always
row number 1 to be the latest <span class="caps">ID</span>, we insert a new row into
<code>event_applied</code> with the latest id.
This avoids locking of row number 1.
For
re-projection we truncate the <code>event_applied</code>
table, allowing the code to rerun all those events.
The big difference is in figuring out which events
haven’t been applied yet:</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span> <span class="k">type</span><span class="p">,</span> <span class="n">payload</span> <span class="k">FROM</span> <span class="n">event</span> <span class="k">AS</span> <span class="n">e</span>
<span class="k">WHERE</span>
<span class="k">NOT</span> <span class="k">EXISTS</span> <span class="p">(</span>
<span class="k">SELECT</span> <span class="mi">1</span> <span class="k">FROM</span> <span class="n">event_applied</span>
<span class="k">WHERE</span> <span class="n">event_id</span> <span class="o">=</span> <span class="n">e</span><span class="p">.</span><span class="n">id</span><span class="p">)</span>
<span class="k">ORDER</span> <span class="k">BY</span>
<span class="n">id</span> <span class="k">ASC</span><span class="p">;</span>
</pre></div>
<p>We compare the event table to the <code>event_applied</code> table,
and return any events that don’t exist in that.
We’re still ordering by id to ensure the correct order.
<span id="is-this-correct">Is this correct?</span> Let’s consider concurrency once more
with connection <code>A</code> and <code>B</code>:</p>
<ol>
<li><code>A</code> opens a transaction and inserts a user, but has to do other event source queries as well.</li>
<li><code>B</code> opens a transaction does it’s projection work and wants to insert an event,
<code>B</code> creates a new row in the <code>even_applied</code> table and completes.
There is no need to wait since there is no single row lock.
So <code>B</code> finishes.</li>
<li><code>A</code> finishes it’s other event sourcing completes.</li>
</ol>
<p>This doesn’t deadlock.
However it’s not completely correct in that <code>A</code> get’s id 1.
and <code>B</code> get’s id 2,
but <code>A</code><span class="quo">‘</span>s transaction finishes after <code>B</code> by inserting another event with id 3.
So on reprojection one of <code>A</code><span class="quo">‘</span>s events get’s applied before <code>B</code>.
But in the initial projection, all of <code>A</code><span class="quo">‘</span>s event happened <em>after</em> <code>B</code>.
So the first event of <code>A</code> is out of order.
This <em>may</em> cause issues.
This problem was also present in the original implementation,
since an id is acquired before the lock waiting happens.
I think a solution would be to group the events by transaction id,
and then order by last created event.
In this case all events created before <code>B</code> in <code>A</code><span class="quo">‘</span>s transaction
would be pushed behind it by an event happening after <code>B</code> finishes.
If we do that, the event table gets an extra field:</p>
<div class="highlight"><pre><span></span><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">event</span> <span class="p">(</span>
<span class="n">id</span> <span class="nb">serial</span> <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">payload</span> <span class="n">jsonb</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="k">type</span> <span class="nb">character</span> <span class="nb">varying</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">created</span> <span class="k">timestamp</span> <span class="k">with</span> <span class="n">time</span> <span class="k">zone</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">transaction_id</span> <span class="nb">bigint</span> <span class="k">NOT</span> <span class="k">NULL</span>
<span class="p">);</span>
</pre></div>
<p>Our insert function retrieves the transaction id with <a href="https://www.postgresql.org/docs/9.0/functions-info.html#FUNCTIONS-TXID-SNAPSHOT"><code>txid_current</code></a>:</p>
<div class="highlight"><pre><span></span><span class="k">BEGIN</span><span class="p">;</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">event</span> <span class="p">(</span><span class="n">payload</span><span class="p">,</span> <span class="k">type</span><span class="p">,</span> <span class="n">created</span><span class="p">,</span> <span class="n">transaction_id</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="s1">'{"email":"hi@jappie.me"}'</span>
<span class="p">,</span> <span class="s1">'create-user'</span>
<span class="p">,</span> <span class="n">now</span><span class="p">()</span>
<span class="p">,</span> <span class="n">txid_current</span><span class="p">());</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">event_applied</span> <span class="p">(</span><span class="n">event_id</span><span class="p">,</span> <span class="n">created</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="n">last_value</span><span class="p">,</span> <span class="n">now</span><span class="p">()</span> <span class="k">FROM</span> <span class="n">event_id_seq</span><span class="p">;</span>
<span class="k">COMMIT</span><span class="p">;</span>
</pre></div>
<p>And our unnaplied events query now groups:</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span>
<span class="n">array_agg</span><span class="p">(</span><span class="k">type</span><span class="p">)</span> <span class="k">AS</span> <span class="n">types</span><span class="p">,</span>
<span class="n">array_agg</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span> <span class="k">AS</span> <span class="n">payloads</span>
<span class="k">FROM</span> <span class="n">event</span> <span class="k">AS</span> <span class="n">e</span>
<span class="k">WHERE</span> <span class="k">NOT</span> <span class="k">EXISTS</span> <span class="p">(</span>
<span class="k">SELECT</span> <span class="mi">1</span> <span class="k">FROM</span> <span class="n">event_applied</span> <span class="k">WHERE</span> <span class="n">event_id</span> <span class="o">=</span> <span class="n">e</span><span class="p">.</span><span class="n">id</span>
<span class="p">)</span>
<span class="k">GROUP</span> <span class="k">BY</span> <span class="n">transaction_id</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="k">max</span><span class="p">(</span><span class="n">id</span><span class="p">)</span> <span class="k">ASC</span><span class="p">;</span>
</pre></div>
<p>If we run that unnaplied events query on an event table like this:</p>
<div class="highlight"><pre><span></span>id | payload | type | created | transaction_id
---+-----------------------+-----------------+------------+----------------
6 | {email: hi@jappie.me} | delete-user | 2023-01-15 | 77958
7 | {email: hi@jappie.me} | create-user | 2023-01-15 | 77959
8 | {company-id: 2} | delete-company | 2023-01-15 | 77958
</pre></div>
<p>We’d get a result like:</p>
<div class="highlight"><pre><span></span><span class="x"> types | payloads</span>
<span class="x">-------------------------------+-----------------------------------------</span>
<span class="x"> {create-user} | </span><span class="cp">{{</span><span class="nv">email</span><span class="o">:</span> <span class="s1">'hi@jappie.me'</span><span class="cp">}}</span><span class="x"></span>
<span class="x"> {delete-user,delete-company} | </span><span class="cp">{{</span><span class="nv">email</span><span class="o">:</span> <span class="s1">'hi@jappie.me'</span><span class="o">},{</span><span class="nv">company-id</span><span class="o">:</span> <span class="m">2</span><span class="cp">}}</span><span class="x"></span>
</pre></div>
<p>Which is what we want.
Even though the create user event happened
while the delete user event was happening,
the delete user event was part of a larger transaction.
So the create user even should come first when re-projecting.
This allows
arbitrary sized transactions to
project alongside each-other and provides better
ordering guarantees then the original implementation.</p>
<h2 id="closing-thoughts">Closing thoughts</h2>
<p>Phew, that was a lot.
I didn’t think this would become such a large post.
Designing an event source system
on Postgres transactions is rather hard.
All I wanted to do is clear my thoughts on the matter,
but that grouping issue is another bug I just found
by writing about this 😅.</p>
<p>I think the biggest lesson I’ve (re)learned from
the deadlock bug itself is to make sure you
reproduce an issue first before diving into solutions.
Even nasty business threatening system level bugs like these can
sometimes be solved with some minor modifications to the system.
If we had skipped this small step of reproducing the issue,
we may have focused on the <span class="caps">CPU</span> observation and moved
pack ingestation to a separate machine,
which would’ve taken weeks to implement
and not solve anything.</p>
<p>Furthermore, it’s humbling to see that even after having
used relational databases for more then a decade,
I still can learn new things about them.
For example Postgres’ auto increment sidesteps the
transaction, which was quite shocking to me.
A rather important detail to keep in mind when reasoning
about these systems.</p>
<p>I made a <a href="https://github.com/jappeace/MAHDB">github repository</a>
for playing around with the queries more easily.
I hope you enjoyed this article, please leave
a comment if you have any questions or suggestions below.</p>
<h2 id="resources">Resources</h2>
<ul>
<li>The <a href="https://github.com/jappeace/MAHDB">code</a> in this <a href="https://github.com/jappeace/MAHDB">blogpost</a></li>
<li>Postgres <a href="https://wiki.postgresql.org/wiki/Lock_Monitoring">Lock monitoring</a></li>
<li><a href="https://garba.org/posts/2016/event-sourcing/#materialised-view-pattern">Blogs</a> on <a href="https://www.ahri.net/2019/07/practical-event-driven-and-sourced-programs-in-haskell/">event sourcing</a></li>
<li>Presentation on <a href="https://www.youtube.com/watch?v=8JKjvY4etTY">event sourcing</a></li>
</ul>Summerhouse Paradis Aruba2023-01-12T19:00:00+01:002023-01-12T19:00:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2023-01-12:/summerhouse-paradis-aruba.html<style>
.video{
width: 100%;
height: 20em;
}
</style>
<p>This post tracks the progress of a rental property I bought on Aruba.
The goal is to rent it out short term to tourists.
This video gives a rough overview of the current status:</p>
<iframe class="video" src="https://www.youtube.com/embed/V4p4mF6N4pc" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
<h2 id="summerhouse-paradis-on-aruba-vlog-colors-mold-and-a-hole">Summerhouse paradis on aruba vlog: Colors, mold and a hole</h2>
<p>A …</p><style>
.video{
width: 100%;
height: 20em;
}
</style>
<p>This post tracks the progress of a rental property I bought on Aruba.
The goal is to rent it out short term to tourists.
This video gives a rough overview of the current status:</p>
<iframe class="video" src="https://www.youtube.com/embed/V4p4mF6N4pc" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
<h2 id="summerhouse-paradis-on-aruba-vlog-colors-mold-and-a-hole">Summerhouse paradis on aruba vlog: Colors, mold and a hole</h2>
<p>A week later and we’re starting to work on the pool.</p>
<iframe class="video" src="https://www.youtube.com/embed/cyyV7WXNnEE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
<h2 id="how-do-you-even-build-a-pool-summerhouse-paradis-vlog-iii-2022-12-18">How do you even build a pool? Summerhouse paradis vlog <span class="caps">III</span>, 2022-12-18</h2>
<p>Once again, a week later, my final week on Aruba,
and the pool is almost concrete.</p>
<iframe class="video" src="https://www.youtube.com/embed/ck1uogmzXjE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>Why do I still write this blog?2023-01-11T17:10:00+01:002023-01-11T17:10:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2023-01-11:/why-do-i-still-write-this-blog.html<p>My motivation for blogging has changed over time.
I’d like to revisit my initial reason,
and discuss how this changed while maintaining the blog.
After all I keep posting articles on here,
something has to keep me going right?</p>
<p>We can read my original <a href="https://jappie.me/website-launch.html#why-make-a-site">motivation</a>.
The initial goal was …</p><p>My motivation for blogging has changed over time.
I’d like to revisit my initial reason,
and discuss how this changed while maintaining the blog.
After all I keep posting articles on here,
something has to keep me going right?</p>
<p>We can read my original <a href="https://jappie.me/website-launch.html#why-make-a-site">motivation</a>.
The initial goal was to make money apparently<sup id="fnref-forgotten"><a class="footnote-ref" href="#fn-forgotten">1</a></sup>,
through ads or Patreon.
T-That didn’t work out so well.
I now realize it’s quite hard to make money with blogging,
you’ve to be really good!
I’m not, and that’s okay.
Furthermore from ads you need tons of visitors to get even the smallest
amount of money.
And for patreon to work you need to make something for a niche audience,
and captivate them,
I think at the time I didn’t really know what that would entail.
Similar to patreon would be to use <a href="https://stratechery.com/2021/sovereign-writers-and-substack/">substack</a>,
however that didn’t exist at the time.</p>
<p>Another romantic desire
not listed in <a href="https://jappie.me/website-launch.html#why-make-a-site">that post</a>
was being remembered after my death.
For example consider
“Commentarii de Bello Gallico” (Commentaries on the Gallic War).
People still read that 2000 years after Ceasars’ death.
I was fascinated by this idea,
to have your words echo trough the ages.</p>
<p>To achieve this I knew my writing skill had to improve.
I hoped that trough blogging I’d get better.
I knew my first posts wouldn’t be interesting or engaging,
but I could learn from each post.
So once I got this going, I placed google analytics on it,
and soon realized no-one was reading this blog.
I published something on the internet and no-one was reading this!
I gained some solace from having carved out my own private corner,
so cozy just for me.
I learned at the beginning of 2017 the internet was already too big.
Still my goal was to be an attention whore,
and to get that you /need/ to post your content on aggregator
websites like reddit or Facebook (depending on your audience).
Peeps aren’t going to type in <code>jappie.me</code> magically in the <span class="caps">URL</span>
bar, and Google ain’t just gonna find you without some links pointing to you.</p>
<p>Getting a big name that last trough the ages
was a goal of Jappie from 2017/2018,
I don’t think this is no longer a goal of mine.
Once I’m dead, well, I’m dead.
You can remember me all you want, that ain’t gonna bring me back,
and who wants to go back anyway?</p>
<h2 id="great-compromise">Great compromise</h2>
<p>Although my desire for being remembered has faded,
I still want to get better at writing.
Because I feel this still covers up flaws I have.
For example, I’m quite bad at self promotion.
I started seeing blogging as a sort of propaganda machine <sup id="fnref-gallic-wars"><a class="footnote-ref" href="#fn-gallic-wars">3</a></sup>,
that would represent me so I didn’t have to.
In real life<sup id="fnref-non-internet"><a class="footnote-ref" href="#fn-non-internet">2</a></sup> social interactions,
I often come across as doubtful on my skills,
and I’ve trouble of thinking what to say in conversations.
Furthermore, I don’t enjoy convincing that
others I’m any good at anything,
because that feels like bragging to me.
However, as a programmer you pretty have to do this.
For example during an interview,
or when trying to get another assignment.</p>
<blockquote>
<p>So rather then changing my personality,
I realized I could get better at writing,
The blog was a great compromise.</p>
</blockquote>
<p>Ironically, trough writing I discovered I enjoy writing.
I get a lot of pleasure out of being precise.
Merely putting my thoughts onto
paper is apparently a step up in precision and clarity.
It’s quite shocking and enlightening at the same,
to see how chaotic and rambly
your own thoughts are once you start putting
them in words on paper.
After a mere day of writing a passage
you may question who wrote that,
because it looks so foreign to you.
This is especially true with fleeting thoughts.</p>
<p>But I’ve also noticed that converse on topics
I’ve written about is quite easy.
This has been a huge benefit for me.
No longer I’ve to think of what to say,
but I can fully focus on how to say it.
Especially during interviews where the other party
has decided to read what you’ve written about.
In those cases, conversation becomes a breeze!
This truly is an unintended consequence,
but I guess blogging has helped my career.</p>
<h2 id="frustration">Frustration</h2>
<p>Another use of blogging is letting out professional frustration
in a constructive manner.
For example the <a href="https://jappie.me/failing-in-haskell.html">failing in haskell</a>
post was directly due to error handling encounters in professional code bases.
Explaining this in painful detail.</p>
<p>This also happened with the stack <a href="https://jappie.me/fun-with-stack-haskell-dependency-management.html">dependency management</a>
post at the time.
Both these posts have seen a lot of user interaction as well,
people started debating about it on reddit or hackernews for example.
So not only does this help me vent,
it also gets my name out there!
However I feel it’s not the best idea to become a divisive-andy.
Because I don’t think it’s an explicit goal of me right now is to
make this blog popular.
After all this is <em>my</em> corner of the internet.
I like the feeling of putting words out there that no-one reads.
Making this blog popular would put pressure on me to write popular
content.
But I want to write abut what <em>I</em> want to write, not what’s popular.
I could’ve written more about elm for example,
because that <a href="https://jappie.me/elm-on-fire-shaders-in-elm.html">elm on fire</a>
post was quite successful.
But after making fire, my interest faded, and so did my motivation.</p>
<p>I did notice however that once I got a blog post out on something
which frustrates me, the frustration goes away.
It simply no longer bothers me,
because as far as I’m aware this is the most impactful thing
I can do to solve an issue.
If publicly complaining about something,
putting my name on the line, doesn’t help,
what will?</p>
<h2 id="nostalgia">Nostalgia</h2>
<p>Interestingly I noticed another reason to blog lately.
I enjoy reading back my old articles!
Remember that time I immigrated <a href="https://jappie.me/jappie-lives-with-kangaroos.html">to australia</a>
for a year to do programming there?
Or doing a cute lil’ <a href="https://jappie.me/tool-survey.html">tool survey</a>
back in 2017 for example.
At the time I thought I was using some awesome stuff.
Now I know it’s a rather basic setup.
My nixos <a href="https://jappie.me/the-nix-mutli-monolith-machine-nmmm.html">based configuration</a>
is far more powerful and flexible for example.</p>
<p>I should make more of these slice of live posts.
I’ve been only doing technical stuff the last couple of years,
and to be honest, I like writing about my own live as well.</p>
<h2 id="interesting-challenge">Interesting Challenge</h2>
<p>It’s one thing to write a coherent story,
it’s another thing to make that story interesting.
Imagine shakespeare being writtin as a cooking book:</p>
<blockquote>
<p>Start the thunder and lightening.
Grab a first witch, a second witch and a third witch.
Let the first witch say: When shall we tree meet again, in thunder lightening or rain.</p>
</blockquote>
<p>It’s boring!
Just like when talking about programming,
we need to make it interesting.
The solutions described in programing langauges are meant for machine
consumption mostly,
writing them down in natural langauge is a completly different excersize.
You can be much more lax in natural langauge than in programming.
A misspelled word?
That’s fine, the reader will be a bit frustrated
but it doesn’t break the meaning of the story.
You can’t get away with that in programming.</p>
<p>However, in programming you don’t have the problem of keeping your
audiences’ attention.
Are you using a disgusting font on your blog?
No-one is going to read that.
Are you re-using the same vocab over and over?
I’m gonna stop.
You need to write interestingly, which is much
harder than just doing something grammatically correct.</p>
<p>This is what most business writers don’t get.
Dumping a bunch of jargon in your post is going to shoo away people.
Using politically correct opinions is boring.
You have to take a position to attract an audience.
This is hard.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I write for self promotion, to vent, and for fun.
I think writing is difficult, but I enjoy it.
Writing something interesting is even harder,
but I feel it’s well worth the exercise.</p>
<p>Do you feel writing is fun, or perhaps boring?
Or do you also have a blog?
Please let me know in the comments below!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-forgotten">
<p>I long had forgotten this was a serious goal of mine. <a class="footnote-backref" href="#fnref-forgotten" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-non-internet">
<p>Offline interactions <a class="footnote-backref" href="#fnref-non-internet" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-gallic-wars">
<p>Very much like “Commentarii de Bello Gallico”, which more realistically should’ve been called the genocide of the Gallic people. <a class="footnote-backref" href="#fnref-gallic-wars" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>Zurich hack 2022 Denotational Design2022-09-26T19:00:00+02:002022-09-26T19:00:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2022-09-26:/zurich-hack-2022-denotational-design.html<style>
img[alt="zurich hack logo, uwu"]{
width:40%;
margin-left: 30%;
}
</style>
<p><img alt="zurich hack logo, uwu" src="images/2022/zurich-hack.svg"></p>
<p>This blog post and after action report is three months overdue,
but I participated in <a href="https://zfoh.ch/zurihac2022/">Zurich hack 2022</a>.
Zurich hack is a voluntary hackaton organized in
<a href="https://www.myswitzerland.com/en/destinations/rapperswil-jona/">Rapperswil-Jona</a> <sup id="fnref-name"><a class="footnote-ref" href="#fn-name">3</a></sup>,
with as theme improving the Haskell ecosystem and socializing.
Naturally I chose …</p><style>
img[alt="zurich hack logo, uwu"]{
width:40%;
margin-left: 30%;
}
</style>
<p><img alt="zurich hack logo, uwu" src="images/2022/zurich-hack.svg"></p>
<p>This blog post and after action report is three months overdue,
but I participated in <a href="https://zfoh.ch/zurihac2022/">Zurich hack 2022</a>.
Zurich hack is a voluntary hackaton organized in
<a href="https://www.myswitzerland.com/en/destinations/rapperswil-jona/">Rapperswil-Jona</a> <sup id="fnref-name"><a class="footnote-ref" href="#fn-name">3</a></sup>,
with as theme improving the Haskell ecosystem and socializing.
Naturally I chose to work on the most research-y project I could find.
Sandy was happy to oblige with his <a href="https://zfoh.ch/zurihac2022/projects.html#denotational-design">denotational design</a>
project.
Here we build an “infinite” baseless chip design,
with a <a href="https://en.wikipedia.org/wiki/Homomorphism">homomorphism</a>
in natural numbers to proof correctness,
more on this in the proofs and programs section.</p>
<p>Our presentation was surprisingly good considering we slapped
it together in 30 minutes.
However,
I think we could’ve done a better job at explaining
denotational design,
and we could’ve elaborated more on why proving matters.
I shall use this post to fill in these gaps.
For starters the presentation can be seen here:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/fCT0uVCe53Q?start=682" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>I helped presenting<sup id="fnref-i'm-on-left"><a class="footnote-ref" href="#fn-i'm-on-left">1</a></sup>, however most of the implementation
was done by Sandy and Nathan.
I wish I could’ve done more, but my Agda isn’t good enough yet.
I helped with cheering on their proving efforts and coming up
with ideas for the design.</p>
<h2 id="denotational-design">Denotational design</h2>
<p>Let’s begin on what denotational design is.
You could watch a <a href="https://youtu.be/bmKYiUOEo2A?t=871">video on this</a>,
but summarized in my own words:</p>
<ul>
<li>We should decompose parts when possible.</li>
<li>Abstractions shouldn’t leak.</li>
<li>We should look for elegance.</li>
</ul>
<p>The first point
is “We should decompose parts when possible”.
This means breaking up our design in such a way
we can re-use parts into a larger whole.
For example in Zurich hack,
our first designs was a large record for multiplication
that had everything baked into it.
Then someone had the idea to split that record into
a separate addition and multiplication records
and re-express multiplication into addition.
This allows us to work with the simpler problem
of addition, before tackling multiplication.
Which is what we eventually settled upon as well.
I don’t think decomposition has been stated as an explicit
goal of denotational design before,
but it feels implied.
Perhaps in <a href="https://youtu.be/bmKYiUOEo2A?t=871">Conals talk</a>,
“principled construction of correct implementation”
can be interpreted as such.</p>
<p>The point about “Abstractions shouldn’t leak” is quite interesting.
We wish to provided a simplified view of the world to the user
through abstraction.
In practice this means we should hide the implementation from the user.
I once suggested for example to add a xor
operation to our record to get rid of the carry bit in certain cases.
After some discussion we settled on not doing this because
xor isn’t really a thing you care about when thinking
in terms of semirings<sup id="fnref2-name"><a class="footnote-ref" href="#fn-name">3</a></sup>.
In other words, when thinking in terms of multiplication
and addition,
you don’t want to care about the bit representation.
For more examples of “abstractions shouldn’t leak”
I recommend the book
<a href="https://algebradriven.design/">algebra driven design</a>.</p>
<p>The final point is “We should look for elegance”.
Which should serve as a compass upon iteration.
Here again I’ve an example from just after Zurich hack:
The overflow bit in our addition record bugged me.
It kind off exposes the internals of addition.
So I decided to delete it in favor of doing a full co-product instead.
Doing this would break both multiplication and addition records,
the proofs have to be redone,
I’m not even sure if it’s possible.
So there is definitely a cost. <sup id="fnref-real-engineering"><a class="footnote-ref" href="#fn-real-engineering">4</a></sup>
However I like that design, it’s more elegant because
this would make multiplication have a product type as input,
and addition a sum type.
Which would have some nice symmetry.
This is something we didn’t drive home enough in the
Zurich hack presentation.
The proof we presented looked impressive,
but this isn’t something you necessarily want.
An elegant proof and design is what you want.</p>
<h2 id="proofs-and-programs">Proofs and programs</h2>
<p>I pondered on proofs in software
and how dependent types interplay.
Is the design I dreamed up correct?
How do you know this?
We used a property called a <a href="https://en.wikipedia.org/wiki/Homomorphism">homomorphism</a>
to prove correctness.
In mortal words,
our chip design was interpreted into <a href="https://en.wikipedia.org/wiki/Natural_number">natural numbers</a>
and we showed that
addition and multiplication would be the same for our chip,
as it is in natural numbers.</p>
<p>To start talking about proofs,
we need a design and implementation to proof correctness for.
In our case this was a chip design in Agda.
This Adder<sup id="fnref-addition-chip"><a class="footnote-ref" href="#fn-addition-chip">6</a></sup> is something we settled upon after several iterations
of design, but I’m cutting that part out for brevity:<sup id="fnref-in-zurich-hack"><a class="footnote-ref" href="#fn-in-zurich-hack">5</a></sup></p>
<div class="highlight"><pre><span></span><span class="kr">record</span> Adder <span class="o">{</span>τ <span class="ow">:</span> <span class="kt">Set</span><span class="o">}</span> <span class="o">{</span>size <span class="ow">:</span> ℕ<span class="o">}</span> <span class="o">(</span>μ <span class="ow">:</span> τ <span class="ow">→</span> Fin size<span class="o">)</span> <span class="ow">:</span> <span class="kt">Set</span> <span class="kr">where</span>
<span class="kr">constructor</span> adds
<span class="kr">field</span>
<span class="nf">add</span> <span class="ow">:</span> Fin <span class="mi">2</span> × τ × τ <span class="ow">→</span> τ × Fin <span class="mi">2</span> <span class="c1">-- 1</span>
<span class="nf">zeroA</span> <span class="ow">:</span> τ <span class="c1">-- 2</span>
proof-add <span class="c1">-- 3</span>
<span class="ow">:</span> <span class="o">(</span>mnp <span class="ow">:</span> Fin <span class="mi">2</span> × τ × τ<span class="o">)</span>
<span class="ow">→</span> toℕ <span class="o">(</span>digitize <span class="o">(</span>P.map μ id <span class="o">(</span>add mnp<span class="o">)))</span>
≡ toℕ <span class="o">(</span>addF'3 <span class="o">(</span>P.map id <span class="o">(</span>P.map μ μ<span class="o">)</span> mnp<span class="o">))</span>
<span class="kr">open</span> Adder
</pre></div>
<p>Here we’re saying, to define an adder you need 3 things.
You need an add operation <code>1</code>,
you need a zero <code>2</code> <sup id="fnref-nathan-zero"><a class="footnote-ref" href="#fn-nathan-zero">8</a></sup>,
and you need a proof of addition <code>3</code>.
The proof of addition is the
homomorphism from our chip to the natural numbers.
Furthermore we don’t specify the input type, which is represented by <code>τ</code>.
This is done because we want a baseless chip design.
We’re fine with arbitrary inputs,
As long as we can interpreted this, represented by <code>μ</code>.
In <code>μ</code> the <code>Fin size</code> indicates a finite size,
which we need because we want to map our design to the real world,
which is finite.</p>
<p>Not shown from the record is that
we’re able to make a bigger adder out of a smaller one
trough composition of two adders.
composition chips allows us to “grow” this size
without needing to re-prove correctness.
We only need to prove composition is correct.
All of this also holds for our eventual multiplication
design. which is build from an adder and composition
of other multiplication chips.
We shall see that proving correctness of composition isn’t
easy. Once done, all compositions will provably correct however. <sup id="fnref-didn't-finish"><a class="footnote-ref" href="#fn-didn't-finish">7</a></sup></p>
<p>A concrete example of <code>μ</code> would be an interpertation
into binary values:</p>
<div class="highlight"><pre><span></span><span class="nf">interpretBF</span> <span class="ow">:</span> Bool <span class="ow">→</span> Fin <span class="mi">2</span>
interpretBF false <span class="ow">=</span> zero
interpretBF true <span class="ow">=</span> suc zero
</pre></div>
<p>If we put this into the Adder then <code>τ = Bool</code>.
<code>interpretBF</code> in this case interprets our code as
a boolean value in natural numbers.
In other words the homomorphism.
This says what value a true and a false are in natural numbers.
Now can define how to add bits:</p>
<div class="highlight"><pre><span></span><span class="nf">add2</span> <span class="ow">:</span> Adder interpretBF <span class="c1">-- 1</span>
add add2 <span class="o">(</span>zero , false , false<span class="o">)</span> <span class="ow">=</span> false , zero <span class="c1">-- 2</span>
<span class="ow">...</span>
add add2 <span class="o">(</span>suc zero , true , true<span class="o">)</span> <span class="ow">=</span> true , suc zero
zeroA add2 <span class="ow">=</span> false <span class="c1">-- 3</span>
</pre></div>
<p>Here we first define the type of <code>add2</code> at <code>1</code>.
This uses the previously defined <code>interpretBF</code> to set <code>τ = Bool</code>.
Then we start giving an implementation for <code>add</code> at <code>2</code>,
which is a simple pattern match into values.
finally we give an implementation of <code>zeroA</code> at <code>3</code>.</p>
<p>You can do a quick correctness
check before doing a full prove.
For example, in multiplication
we didn’t do the full on homorphism prove at first.
It looked daunting,
so we settled on making a <a href="https://github.com/isovector/denotational-arithmetic-zurihac/commit/4eb494ad84a1ede2202b036379d8525a391eecbb#diff-201315dac0498e664f0dccffd803e509020bf7d50ce3509d27566a3c26e5cb38R273">unit test</a>
instead.
If we map out all possible inputs to all possible outputs we
got a crummy proof:</p>
<div class="highlight"><pre><span></span><span class="nf">_</span> <span class="ow">:</span> <span class="o">(</span>V.map <span class="c1">-- 1</span>
<span class="o">(</span>toℕ ∘ pairμ <span class="o">(</span>pairμ interpretBF<span class="o">)</span> ∘ uncurry <span class="o">(</span>mult mul2x2<span class="o">))</span> $
composeTheValues allBools2x2 allBools2x2
<span class="o">)</span>
≡ <span class="o">(</span><span class="mi">0</span> ∷ <span class="mi">0</span> ∷ <span class="mi">0</span> ∷ <span class="mi">0</span> ∷ <span class="c1">-- 2</span>
<span class="mi">0</span> ∷ <span class="mi">1</span> ∷ <span class="mi">2</span> ∷ <span class="mi">3</span> ∷
<span class="mi">0</span> ∷ <span class="mi">2</span> ∷ <span class="mi">4</span> ∷ <span class="mi">6</span> ∷
<span class="mi">0</span> ∷ <span class="mi">3</span> ∷ <span class="mi">6</span> ∷ <span class="mi">9</span> ∷ []<span class="o">)</span>
_ <span class="ow">=</span> refl
</pre></div>
<p>At <code>1</code> we put our chip design into the interpretation,
and at <code>2</code> we expect a multiplication table as result.
Is this test complete?
No, this only works for binary values up to 9,
we’ve not tested for trits or pentits or higher values.
We did prove however the chip behaves like we expect
for these values.
If you’re building a chip company where you only need
multiplication up to 9 in base 2 this is good enough.
As far unit tests go this is incredibly thorough because we’re testing against
all possible values in the chip design.
The more common approach is to sample a couple values and call it a day.</p>
<p>Which lead to an alternative approach called property testing.
Here you would generate two random inputs on one side,
interpret it trough the homomorphism
and then see if the multiplication in natural
numbers is the same as the test.
We didn’t do this because
it’s sort of difficult to do in Agda.<sup id="fnref-agda-noob"><a class="footnote-ref" href="#fn-agda-noob">9</a></sup>
Now you need to figure out how to get your source
of randomness.
Also time was a serious constraint,
and we had a more powerful and interesting technique,
proving!
A proof doesn’t have to be hard, for example consider the correctness
prove for our add2 chip:</p>
<div class="highlight"><pre><span></span> proof-add add2 <span class="o">(</span>zero , false , false<span class="o">)</span> <span class="ow">=</span> refl
<span class="ow">...</span>
proof-add add2 <span class="o">(</span>suc zero , true , true<span class="o">)</span> <span class="ow">=</span> refl
</pre></div>
<p><code>refl</code> means, <a href="https://en.wikipedia.org/wiki/Reflexive_relation">reflexivity</a>.
In other words, the statement is simple enough that Agda can
just look at the definition to figure out what it means.
In this case all we do is list out all possible input values,
and tell agda to look at the definition.
<code>proof-add</code><span class="quo">‘</span>s type signature ensures the implementation is correct.
This is only possible because Agda is dependently typed.
What we proved is that the homorphism is the same under composition for the addition.</p>
<p>This will work for the add2 chip.
However the add2 chip is kindoff useless by itself since it can
only add 2 bits.
Our idea was to compose these adders into bigger adders so that
any size can be represented.
To do this we first define the type signature:</p>
<div class="highlight"><pre><span></span><span class="nf">bigger-adder</span> <span class="ow">:</span> <span class="o">{</span>σ τ <span class="ow">:</span> <span class="kt">Set</span><span class="o">}</span> <span class="o">{</span>σ-size τ-size <span class="ow">:</span> ℕ<span class="o">}</span> <span class="o">{</span>μ <span class="ow">:</span> σ <span class="ow">→</span> Fin σ-size<span class="o">}</span> <span class="o">{</span>ν <span class="ow">:</span> τ <span class="ow">→</span> Fin τ-size<span class="o">}</span>
<span class="ow">→</span> Adder μ <span class="c1">-- 1</span>
<span class="ow">→</span> Adder ν <span class="c1">-- 2</span>
<span class="ow">→</span> Adder <span class="o">(</span>uncurry combine ∘ P.map μ ν<span class="o">)</span> <span class="c1">-- 3</span>
</pre></div>
<p>Here we’re putting in a low adder <code>1</code>, a high adder <code>2</code>
which results into a combined adder <code>3</code>.
<code>μ</code> and <code>ν</code> are placeholders for different adders.
This allows us to for example add trits to bits.
The resulting adder <code>3</code> maps over both sides of the resulting
tuple<sup id="fnref-product"><a class="footnote-ref" href="#fn-product">15</a></sup> with the interpertation.</p>
<p>If you squint a little, the implementation looks like a circuit: </p>
<div class="highlight"><pre><span></span>add <span class="o">(</span>bigger-adder x y<span class="o">)</span> <span class="c1">-- 1</span>
<span class="o">(</span>cin , <span class="o">(</span>mhi , mlo<span class="o">)</span> , <span class="o">(</span>nhi , nlo<span class="o">))</span> <span class="c1">-- 2</span>
<span class="ow">=</span> <span class="kr">let</span> <span class="c1">-- 3</span>
<span class="o">(</span>lo , cmid<span class="o">)</span> <span class="ow">=</span> y <span class="ow">.</span>add $ cin , mlo , nlo
<span class="o">(</span>hi , cout<span class="o">)</span> <span class="ow">=</span> x <span class="ow">.</span>add $ cmid , mhi , nhi
<span class="kr">in</span> <span class="o">((</span>hi , lo<span class="o">)</span> , cout<span class="o">)</span> <span class="c1">-- 4</span>
</pre></div>
<p>At <code>1</code> we’re copattern matching on bigger-adder so we can
get the underlying <code>μ</code> and <code>ν</code> adders as <code>x</code> and <code>y</code> respectively.
At <code>2</code> we’re getting the actual arguments into the bigger adder.
The type of this is $ Fin _2 \times (\sigma \times \tau) \times (\sigma \times \tau)$
this is where the carry comes into play as argument since
an adder needs to be able to tell when it overflows <sup id="fnref-unnesscary"><a class="footnote-ref" href="#fn-unnesscary">14</a></sup>.
The actual addition in <code>3</code> we pawn off to the underlying adders <code>x</code> and <code>y</code>,
all we do is hook in the carry<sup id="fnref-jappie-ventures"><a class="footnote-ref" href="#fn-jappie-ventures">16</a></sup>.
In <code>4</code> we emit the results.</p>
<p>Once we were reasonably confident of our implementation,
we want to prove this correctness.
This is a bit more involved than proving the boolean interpretation:</p>
<div class="highlight"><pre><span></span>proof-add <span class="o">(</span>bigger-adder <span class="o">{</span>σ-size <span class="ow">=</span> σ-size<span class="o">}</span> <span class="o">{</span>τ-size <span class="ow">=</span> τ-size<span class="o">}</span> <span class="o">{</span>μ <span class="ow">=</span> μ<span class="o">}</span> <span class="o">{</span>ν <span class="ow">=</span> ν<span class="o">}</span> x y<span class="o">)</span>
<span class="o">(</span>cin , <span class="o">(</span>mhi , mlo<span class="o">)</span> , <span class="o">(</span>nhi , nlo<span class="o">))</span>
<span class="kr">with</span> y <span class="ow">.</span>add <span class="o">(</span>cin , mlo , nlo<span class="o">)</span> <span class="kr">in</span> y-eq
<span class="ow">...</span> <span class="ow">|</span> <span class="o">(</span>lo , cmid<span class="o">)</span> <span class="kr">with</span> x <span class="ow">.</span>add <span class="o">(</span>cmid , mhi , nhi<span class="o">)</span> <span class="kr">in</span> x-eq
<span class="ow">...</span> <span class="ow">|</span> <span class="o">(</span>hi , cout<span class="o">)</span> <span class="ow">=</span>
<span class="kr">let</span> x-proof <span class="ow">=</span> proof-add x <span class="o">(</span>cmid , mhi , nhi<span class="o">)</span> <span class="c1">-- 3</span>
y-proof <span class="ow">=</span> proof-add y <span class="o">(</span>cin , mlo , nlo<span class="o">)</span>
size <span class="ow">=</span> σ-size
<span class="kr">in</span> begin
begin
toℕ <span class="o">(</span>cast _ <span class="o">(</span>combine cout <span class="o">(</span>combine <span class="o">(</span>μ hi<span class="o">)</span> <span class="o">(</span>μ lo<span class="o">))))</span> <span class="c1">-- 1</span>
≡⟨ toℕ-cast _ <span class="o">(</span>combine cout <span class="o">(</span>combine <span class="o">(</span>μ hi<span class="o">)</span> <span class="o">(</span>μ lo<span class="o">)))</span> ⟩
toℕ <span class="o">(</span>combine cout <span class="o">(</span>combine <span class="o">(</span>μ hi<span class="o">)</span> <span class="o">(</span>μ lo<span class="o">)))</span>
≡⟨ toℕ-combine cout _ ⟩
size * size * toℕ cout + toℕ <span class="o">(</span>combine <span class="o">(</span>μ hi<span class="o">)</span> <span class="o">(</span>μ lo<span class="o">))</span>
≡⟨ cong <span class="o">(</span>\ φ <span class="ow">→</span> size * size * toℕ cout + φ<span class="o">)</span> <span class="o">(</span>toℕ-combine <span class="o">(</span>μ hi<span class="o">)</span> <span class="o">(</span>μ lo<span class="o">))</span> ⟩
size * size * toℕ cout + <span class="o">(</span>size * toℕ <span class="o">(</span>μ hi<span class="o">)</span> + toℕ <span class="o">(</span>μ lo<span class="o">))</span>
≡⟨ <span class="c c-Directive">{! taneb !}</span> ⟩
toℕ <span class="o">(</span>addF' cin <span class="o">(</span>combine <span class="o">(</span>μ mhi<span class="o">)</span> <span class="o">(</span>μ mlo<span class="o">)))</span> + toℕ <span class="o">(</span>combine <span class="o">(</span>μ nhi<span class="o">)</span> <span class="o">(</span>μ nlo<span class="o">))</span>
≡⟨ sym $ toℕ-addF' <span class="o">(</span>addF' cin <span class="o">(</span>combine <span class="o">(</span>μ mhi<span class="o">)</span> <span class="o">(</span>μ mlo<span class="o">)))</span> <span class="o">(</span>combine <span class="o">(</span>μ nhi<span class="o">)</span> <span class="o">(</span>μ nlo<span class="o">))</span> ⟩
toℕ <span class="o">(</span>addF' <span class="o">(</span>addF' cin <span class="o">(</span>combine <span class="o">(</span>μ mhi<span class="o">)</span> <span class="o">(</span>μ mlo<span class="o">)))</span> <span class="o">(</span>combine <span class="o">(</span>μ nhi<span class="o">)</span> <span class="o">(</span>μ nlo<span class="o">)))</span> <span class="c1">-- 2</span>
∎
</pre></div>
<p>Note I drastically shortened this proof to make it fit <sup id="fnref-full-proof"><a class="footnote-ref" href="#fn-full-proof">12</a></sup>.
What do is making the first line (indicated by <code>1</code>)
be the same as the last line (indicated by <code>2</code>)
through steps with equational reasoning.
Every step is small,
and the process is almost fully mechanical pattern matching.
A step is anything within <code>≡⟨ ⟩</code>,
which does some small syntax transformation.
The <code>≡⟨ {! taneb !} ⟩</code> is a missing step, called a hole.
In this case we request taneb<sup id="fnref-ring-solver"><a class="footnote-ref" href="#fn-ring-solver">10</a></sup>, to figure out what goes here.
In <code>3</code>, we’re summoning the proofs from the <code>x</code> and <code>y</code> adders
to use in the proof later.
We’re making a bigger proof out of smaller ones.</p>
<p>If this proof is incorrect, you’ll get a compile error.<sup id="fnref-incorrectness"><a class="footnote-ref" href="#fn-incorrectness">11</a></sup>
This is similar to property tests,
although it doesn’t use randomness and shrinking,
but rather the structure of the implementation
through dependent types.
This is a big step up in terms off correctness compared to property tests.
No longer can you have stochastic issues like insufficient sampling,
or biased distributions.
Furthermore smaller proofs compose into larger ones (with the right design).
We can see that for example with <code>x-proof</code> in the above block.
Not just that, but every step between <code>≡⟨ ⟩</code> is a prove being re-used.
Property tests however aren’t as composable as proofs.
A value generator may be re-used, however care must
be taken the sampling and bias doesn’t become unacceptable.
Finally we’re able to prove on polymorphic type variables,
which property tests can’t do.
If you have software that /needs/ to be correct,
I think this dependently typed prove approach is a very good option to consider.
I also think Agda is an good choice for a language that supports that.</p>
<h2 id="parting-words">Parting words</h2>
<p>Denotational design is an excellent topic of study if you’re struggling with questions like
“how do I make my code be more pretty?”,
or “how do I design nice and easy to understand libraries?”.
Furthermore, even
for commercial code bases we can have correctness proofs.
This is a much more powerful technique than mere property tests,
and puts all that hype around dependent types to work.
We don’t need to rely on hand wavy laws asserted merely by
stochastic approximations of proofs,
we can do the real deal!
Please reach out if you’re in a domain where correctness
like this is important.
I’d love to chat :).</p>
<p>Finally I especially want to thank to both
Nathan and Sandy for giving useful feedback on my humble writings.
I also wish thank all other volunteers who participated,
I had a great time.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-i'm-on-left">
<p>I’m on the left. <a class="footnote-backref" href="#fnref-i'm-on-left" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-i-am-not-an-expert">
<p>I’m not really an expert on this at all, I just put it in my own mistaken words. Feel free to correct me. <a class="footnote-backref" href="#fnref-i-am-not-an-expert" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-name">
<p>As the name implies. This place is 30 minutes or so driving from Zurich. <a class="footnote-backref" href="#fnref-name" title="Jump back to footnote 3 in the text">↩</a><a class="footnote-backref" href="#fnref2-name" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn-real-engineering">
<p>In a commercial setting we’d decide if it’s worth investing additional
in this design.
The one presented at Zurich hack works.
But if this is intended to be used in a larger system,
iterating upon the design may help, if the business can afford it. <a class="footnote-backref" href="#fnref-real-engineering" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn-in-zurich-hack">
<p>In zurich hack we sortoff started out in a classroom with just random ideas.
One was quite funny where we somehow ended up with a design that was equivalent
to tallying the ones and zeros.
But we went in all kinds of directions before settling on using a record.
I guess that’s the point you’ve to just try a bunch of stuff
and not put to much ego into it. <a class="footnote-backref" href="#fnref-in-zurich-hack" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn-addition-chip">
<p>Addition chip <a class="footnote-backref" href="#fnref-addition-chip" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn-didn't-finish">
<p>Unfortunately during the hackaton we didn’t finish this for multiplication,
but we did for addition. <a class="footnote-backref" href="#fnref-didn't-finish" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn-nathan-zero">
<p>When I asked Nathan to review this post, he mentioned he still hasn’t figured out why we needed a zero.
I’m not really sure either, so it’s quite likely this isn’t needed at all!
This maybe an artifact of the time crunch at play. <a class="footnote-backref" href="#fnref-nathan-zero" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn-agda-noob">
<p>For me that is, because rember, I’m quite new to this all. <a class="footnote-backref" href="#fnref-agda-noob" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
<li id="fn-ring-solver">
<p>Nathan, a magical ring solver, or flesh and blood person, whichever interpretation suits you better. <a class="footnote-backref" href="#fnref-ring-solver" title="Jump back to footnote 10 in the text">↩</a></p>
</li>
<li id="fn-incorrectness">
<p>So what if you’re stuck on a proof?
This either means you can’t think of a function,
or it means the thing you’re trying to do is impossible.
Here you’d need to think really hard if the thing you’re designing
is correct.
You could also try to discuss the issue with a friend or ask
on <a href="https://wiki.portal.chalmers.se/agda/Main/Community">the internet</a>.
Plenty of people eager to help. <a class="footnote-backref" href="#fnref-incorrectness" title="Jump back to footnote 11 in the text">↩</a></p>
</li>
<li id="fn-full-proof">
<p>The full proof can be seen in the <a href="https://github.com/isovector/denotational-arithmetic-zurihac">github repository</a>,
although we made some additional changes to the project after the presentation as well. <a class="footnote-backref" href="#fnref-full-proof" title="Jump back to footnote 12 in the text">↩</a></p>
</li>
<li id="fn-useless">
<p>I guess we had no hope of succeeding,
which made it all the more worth while trying in my mind.
After all I spend all year being productive,
now was a time to do something cool. <a class="footnote-backref" href="#fnref-useless" title="Jump back to footnote 13 in the text">↩</a></p>
</li>
<li id="fn-unnesscary">
<p>As I mentioned in the design section,
I don’t believe this is necessary,
but this is an open research question. <a class="footnote-backref" href="#fnref-unnesscary" title="Jump back to footnote 14 in the text">↩</a></p>
</li>
<li id="fn-product">
<p>P stands for product in this case.
So it’s a bimap over a tuple (due to uncurry). <a class="footnote-backref" href="#fnref-product" title="Jump back to footnote 15 in the text">↩</a></p>
</li>
<li id="fn-jappie-ventures">
<p>Someone may or may not have opened <a href="https://nandgame.com/">nandgame</a>
during zurich hack to show a schema off an adder when we were struggling
with correctness. <a class="footnote-backref" href="#fnref-jappie-ventures" title="Jump back to footnote 16 in the text">↩</a></p>
</li>
</ol>
</div>Restoring mysql innodb on windows.2022-06-16T22:30:00+02:002023-01-17T17:00:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2022-06-16:/restoring-mysql-innodb-on-windows.html<p>Over the weekend a company had a power outage,
causing corruption to the on premise hosted mysql innodb database.
This means the company can’t do any work,
so I had to fix this fast before they opened again on Monday.
It’s not trivial because the system runs on …</p><p>Over the weekend a company had a power outage,
causing corruption to the on premise hosted mysql innodb database.
This means the company can’t do any work,
so I had to fix this fast before they opened again on Monday.
It’s not trivial because the system runs on
windows, which means the online guides don’t quite work.
Therefore I wrote these notes for future me.</p>
<p>We can boot the database with <a href="https://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html">force recovery</a>
In my case I had to set it to 3, try as low as possible.
It puts everything in read only but that’s good enough.
The recovery is a process of exporting these read only databases
to sql,
creating an entire new instalation with a new datafolder,
and reading these back into that.
The linux instruction can be found <a href="https://dba.stackexchange.com/questions/65728/forcing-innodb-recovery-of-a-corrupted-database">here</a>.
This post is specifically about windows.
We need to run from cmd not powershell,
powershell doesn’t understand <code><</code>, and I’ve doubts about it understanding <code>></code>.</p>
<p>To get a working installation I set <code>innod_force_recovery = 3</code> in <code>mysql\bin\my.ini</code>,
which can be found in xampp.
First we have to restart mysql so we can get the data out.
Now we can dump the tables:</p>
<div class="highlight"><pre><span></span>./mysqldump.exe -u root -pwhatever --skip-lock-tables -A > all.sql
</pre></div>
<p>And shut down the server.</p>
<div class="highlight"><pre><span></span>./mysqladmin -u root -pwhatever shut
</pre></div>
<p>I just emptied the datafolder with explorer,
by navigating to <code>T:/xampp/mysql/data</code>, and deleting everything in there.
Now I ran:</p>
<div class="highlight"><pre><span></span>./mysql_istall_db.exe --datadir<span class="o">=</span><span class="s2">"T:\xampp\mysql\data"</span>
</pre></div>
<p>I set <code>innod_force_recovery</code> to a comment again
and restart mysql from xampp,
I also checked if it ran in process monitor.</p>
<p>set the root password correctly again:</p>
<div class="highlight"><pre><span></span>mysql -u root
SET PASSWORD FOR <span class="s1">'root'</span>@<span class="s1">'localhost'</span> <span class="o">=</span> PASSWORD<span class="o">(</span><span class="s1">'whatever'</span><span class="o">)</span><span class="p">;</span>
FLUSH PRIVILEGES<span class="p">;</span>
</pre></div>
<p>restore:</p>
<div class="highlight"><pre><span></span>mysql -u root -pwhatever < all.sql
</pre></div>Failing in Haskell2022-02-27T00:00:00+01:002022-03-05T03:40:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2022-02-27:/failing-in-haskell.html<p><img alt="I'm a Haskell failure, don't tell anyone!" src="images/2022/failure.png"></p>
<p>Recently I encountered some dubious error handling code.
Not only was it failing, it was failing <span class="caps">WRONG</span> <sup id="fnref-anti-patterns"><a class="footnote-ref" href="#fn-anti-patterns">1</a></sup>.
This frustrates me because doing failing correctly
in Haskell is quite easy,
so why was it implemented wrongly?
I believe no-one has addressed failing with an opinion.
Plenty of people describe the …</p><p><img alt="I'm a Haskell failure, don't tell anyone!" src="images/2022/failure.png"></p>
<p>Recently I encountered some dubious error handling code.
Not only was it failing, it was failing <span class="caps">WRONG</span> <sup id="fnref-anti-patterns"><a class="footnote-ref" href="#fn-anti-patterns">1</a></sup>.
This frustrates me because doing failing correctly
in Haskell is quite easy,
so why was it implemented wrongly?
I believe no-one has addressed failing with an opinion.
Plenty of people describe the
<a href="http://www.randomhacks.net/2007/03/10/haskell-8-ways-to-report-errors/">various</a>
<a href="https://www.stackbuilders.com/blog/errors-and-exceptions-in-haskell/">ways</a>
<a href="https://www.fpcomplete.com/haskell/tutorial/exceptions/">you</a>
<a href="https://www.schoolofhaskell.com/school/starting-with-haskell/basics-of-haskell/10_Error_Handling">can</a>
<a href="https://wiki.haskell.org/Handling_errors_in_Haskell">fail</a>.
But none give opinions on why certain ways are better than others.
Therefore I shall share my failing expertise,
and describe the correct way to fail in Haskell.
In essence, this is an answer to Eric Kidd’s
<a href="http://www.randomhacks.net/2007/03/10/haskell-8-ways-to-report-errors/">plea for consistency</a>
<sup id="fnref-15-years-ago"><a class="footnote-ref" href="#fn-15-years-ago">2</a></sup>.
These are the properties we want from failure:</p>
<ol>
<li>Preciseness, vague errors are bad.</li>
<li>Locality, we need to know where errors come from.</li>
<li>Recoverability, the program should be able to recover after an error.</li>
<li>Change ability, introduction of new error cases should be easy.</li>
</ol>
<p>We want all these properties to make debugging easier.
This allows you to solve complicated bugs within minutes.
We structure the program so that it tells what goes wrong,
where and why.
This isn’t magic. It takes effort, but not magic at all.</p>
<p>I’ll describe how to achieve the above properties with some
example code for both pure and side effectful code.
However these principles apply to other languages as well <sup id="fnref-other-langs"><a class="footnote-ref" href="#fn-other-langs">3</a></sup>.
We’ll also go over some anti patterns and discuss their mediation.</p>
<h1 id="pure">Pure</h1>
<p>Haskell programmers prefer so called ‘pure’ code.
By which we mean immutable memory computations<sup id="fnref-memory"><a class="footnote-ref" href="#fn-memory">4</a></sup>.
Therefore we start with the pure case.
Ideally pure code fail management looks like this:</p>
<div class="highlight"><pre><span></span><span class="kr">newtype</span> <span class="kt">LookupError</span> <span class="ow">=</span> <span class="kt">NotFound</span> <span class="kt">Text</span>
<span class="nf">lookup</span> <span class="ow">::</span> <span class="kt">Map</span> <span class="kt">Text</span> <span class="kt">Double</span> <span class="ow">-></span> <span class="kt">Text</span> <span class="ow">-></span> <span class="kt">Either</span> <span class="kt">LookupError</span> <span class="kt">Double</span>
<span class="nf">lookup</span> <span class="n">env</span> <span class="n">argA</span> <span class="ow">=</span> <span class="n">maybe</span> <span class="p">(</span><span class="kt">Left</span> <span class="p">(</span><span class="kt">NotFound</span> <span class="n">argA</span><span class="p">))</span> <span class="kt">Right</span> <span class="o">$</span> <span class="kt">Map</span><span class="o">.</span><span class="n">lookup</span> <span class="n">argA</span> <span class="n">env</span>
<span class="kr">data</span> <span class="kt">DivideErrors</span> <span class="ow">=</span> <span class="kt">DivideLookup</span> <span class="kt">LookupError</span>
<span class="o">|</span> <span class="kt">DivisionByZero</span> <span class="kt">Double</span> <span class="kt">Double</span>
<span class="o">|</span> <span class="kt">DivNegativeDivision</span> <span class="kt">Double</span> <span class="kt">Double</span>
<span class="o">|</span> <span class="kt">DivNumberThreeIsBad</span> <span class="kt">Double</span> <span class="kt">Double</span>
<span class="nf">divide</span> <span class="ow">::</span> <span class="kt">Map</span> <span class="kt">Text</span> <span class="kt">Double</span> <span class="ow">-></span> <span class="kt">Text</span> <span class="ow">-></span> <span class="kt">Text</span> <span class="ow">-></span> <span class="kt">Either</span> <span class="kt">DivideErrors</span> <span class="kt">Double</span>
<span class="nf">divide</span> <span class="n">env</span> <span class="n">argA</span> <span class="n">argB</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">valA</span> <span class="ow"><-</span> <span class="n">first</span> <span class="kt">DivideLookup</span> <span class="o">$</span> <span class="n">lookup</span> <span class="n">env</span> <span class="n">argA</span>
<span class="n">valB</span> <span class="ow"><-</span> <span class="n">first</span> <span class="kt">DivideLookup</span> <span class="o">$</span> <span class="n">lookup</span> <span class="n">env</span> <span class="n">argB</span>
<span class="n">when</span> <span class="p">(</span><span class="n">valB</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="o">$</span> <span class="kt">Left</span> <span class="o">$</span> <span class="kt">DivisionByZero</span> <span class="n">valA</span> <span class="n">valB</span>
<span class="n">when</span> <span class="p">(</span><span class="n">valB</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="o">$</span> <span class="kt">Left</span> <span class="o">$</span> <span class="kt">DivNegativeDivision</span> <span class="n">valA</span> <span class="n">valB</span>
<span class="n">when</span> <span class="p">(</span><span class="n">valA</span> <span class="o">==</span> <span class="mf">3.0</span> <span class="o">||</span> <span class="n">valB</span> <span class="o">==</span> <span class="mf">3.0</span><span class="p">)</span> <span class="o">$</span> <span class="kt">Left</span> <span class="o">$</span> <span class="kt">DivNumberThreeIsBad</span> <span class="n">valA</span> <span class="n">valB</span>
<span class="n">pure</span> <span class="o">$</span> <span class="n">valA</span> <span class="o">/</span> <span class="n">valB</span>
</pre></div>
<p>Here we introduce the divide function.
Which takes an environment map,
looks up the values of said map,
and then performs division after doing some checks.
We introduce the environment as a convenient common
“this can fail” lookup.
But business may give us other in-variants
such as <code>DivNumberThreeIsBad</code>,
which also neatly fits in this “pattern”.
Errors are emitted by using the <code>Left</code> constructor,
which is the error branch according to <code>Either</code><span class="quo">‘</span>s
<code>Monad</code> instance.</p>
<p>This code will tell you exactly what went wrong
if something goes wrong.
Locality in this case can be improved a little by adding
a different constructor for each <code>lookup</code> call,
but in this case I’d argue locality is close enough.
The developer has the opportunity to recover from these errors,
a simple pattern match would suffice.
Furthermore if business demands yet another weird constraint,
such as <code>NumberFourIsBad</code>, pattern matches at call sites will emit
a <a href="https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/using-warnings.html#ghc-flag--Wincomplete-patterns"><code>-Wincomplete-patterns</code></a> warning.</p>
<p>If we need to compose these errors in a larger program we can
simply wrap previous errors in a bigger sumtype,
consider the following function <code>plusDiv</code> which emulates <code>(a + b) / c</code></p>
<div class="highlight"><pre><span></span><span class="kr">data</span> <span class="kt">PlusErrors</span> <span class="ow">=</span> <span class="kt">PlusLookup</span> <span class="kt">LookupError</span>
<span class="o">|</span> <span class="kt">PlusNoZeroResults</span>
<span class="o">|</span> <span class="kt">PlusNumberThreeIsBad</span> <span class="kt">Double</span> <span class="kt">Double</span>
<span class="nf">plus</span> <span class="ow">::</span> <span class="kt">Map</span> <span class="kt">Text</span> <span class="kt">Double</span> <span class="ow">-></span> <span class="kt">Text</span> <span class="ow">-></span> <span class="kt">Text</span> <span class="ow">-></span> <span class="kt">Either</span> <span class="kt">PlusErrors</span> <span class="kt">Double</span>
<span class="nf">plus</span> <span class="n">env</span> <span class="n">argA</span> <span class="n">argB</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">valA</span> <span class="ow"><-</span> <span class="n">first</span> <span class="kt">PlusLookup</span> <span class="o">$</span> <span class="n">lookup</span> <span class="n">env</span> <span class="n">argA</span>
<span class="n">valB</span> <span class="ow"><-</span> <span class="n">first</span> <span class="kt">PlusLookup</span> <span class="o">$</span> <span class="n">lookup</span> <span class="n">env</span> <span class="n">argB</span>
<span class="n">when</span> <span class="p">(</span><span class="n">valA</span> <span class="o">==</span> <span class="mf">3.0</span> <span class="o">||</span> <span class="n">valB</span> <span class="o">==</span> <span class="mf">3.0</span><span class="p">)</span> <span class="o">$</span>
<span class="kt">Left</span> <span class="o">$</span> <span class="kt">PlusNumberThreeIsBad</span> <span class="n">valA</span> <span class="n">valB</span>
<span class="kr">let</span> <span class="n">res</span> <span class="ow">=</span> <span class="n">valA</span> <span class="o">+</span> <span class="n">valB</span>
<span class="n">when</span> <span class="p">(</span><span class="n">res</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="o">$</span> <span class="kt">Left</span> <span class="kt">PlusNoZeroResults</span>
<span class="n">pure</span> <span class="o">$</span> <span class="n">valA</span> <span class="o">/</span> <span class="n">valB</span>
<span class="kr">data</span> <span class="kt">PlusDivErrors</span> <span class="ow">=</span> <span class="kt">PDPlusError</span> <span class="kt">PlusErrors</span>
<span class="o">|</span> <span class="kt">PDDivErrors</span> <span class="kt">DivideErrors</span>
<span class="o">|</span> <span class="kt">PDLookup</span> <span class="kt">LookupError</span>
<span class="c1">-- | (a + b) / c</span>
<span class="nf">plusDiv</span> <span class="ow">::</span> <span class="kt">Map</span> <span class="kt">Text</span> <span class="kt">Double</span>
<span class="ow">-></span> <span class="kt">Text</span> <span class="ow">-></span> <span class="kt">Text</span> <span class="ow">-></span> <span class="kt">Text</span>
<span class="ow">-></span> <span class="kt">Either</span> <span class="kt">PlusDivErrors</span> <span class="kt">Double</span>
<span class="nf">plusDiv</span> <span class="n">env</span> <span class="n">argA</span> <span class="n">argB</span> <span class="n">argC</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">res</span> <span class="ow"><-</span> <span class="n">first</span> <span class="kt">PDPlusError</span> <span class="o">$</span> <span class="n">plus</span> <span class="n">env</span> <span class="n">argA</span> <span class="n">argB</span>
<span class="n">cres</span> <span class="ow"><-</span> <span class="n">first</span> <span class="kt">PDLookup</span> <span class="o">$</span> <span class="n">lookup</span> <span class="n">env</span> <span class="n">argC</span>
<span class="n">first</span> <span class="kt">PDDivErrors</span> <span class="o">$</span>
<span class="n">divide</span> <span class="p">(</span><span class="kt">Map</span><span class="o">.</span><span class="n">fromList</span> <span class="p">[(</span><span class="s">"one"</span><span class="p">,</span> <span class="n">res</span><span class="p">),</span> <span class="p">(</span><span class="s">"two"</span><span class="p">,</span> <span class="n">cres</span><span class="p">)])</span> <span class="s">"one"</span> <span class="s">"two"</span>
</pre></div>
<p>The higher ‘level’ function <code>plusDiv</code> absorbs all errors from other functions
with extra constructors.
This isn’t the most ergonomic approach because variables
have to go through the environment map.
In a real program you could factor out this function.
For example with an <code>innerPlus :: Double -> Double -> Either PlusErrors Double</code>.
This adds additional type safety as well,
because the <code>Double</code> type is “smaller” then a
<code>Text</code> type in both memory and cardinality.
However I think for an example or early prototype it’s good.
Once more this tells us exactly what part of the computation failed,
if any.
In this case, the <a href="https://hackage.haskell.org/package/bifunctors-5/docs/Data-Bifunctor.html#v:first">first</a>
function is being used like
<a href="https://hackage.haskell.org/package/transformers-0.6.0.2/docs/Control-Monad-Trans-Class.html#v:lift">lift</a>:
Transforming the function it wraps to run in the “higher level” <code>Either</code> environment.
Some may even call this a natural transformation. </p>
<p>In certain cases you can use <a href="https://hackage.haskell.org/package/validation-1.1.2/docs/Data-Validation.html">Data.Validation</a>
instead of <code>Either</code>.
Which can collect more then one error.
But usage of that is out of the scope of this blogpost.</p>
<h1 id="io-and-exceptions"><code>IO</code> and exceptions</h1>
<p>It’s more difficult to recover from an exception than it is from a pure error value.
As a rule of thumb you can use exceptions when you expect the program to stop.
For example when you can’t find a critical resources from the database.
Another consideration is the value of the error.
If it’s important that an error be handled correctly,
then exceptions should be avoided.
However, this should only be done within a monad stack that has <code>IO</code> as base,
because exceptions are part of the <code>IO</code> <span class="quo">‘</span>contract’<sup id="fnref-throw"><a class="footnote-ref" href="#fn-throw">5</a></sup>.
This contract extends to any transformer stack that has <code>IO</code> as base.
For convenience however, I’ll write out the example in plain <code>IO</code>:</p>
<div class="highlight"><pre><span></span><span class="kr">data</span> <span class="kt">DivideException</span> <span class="kr">where</span>
<span class="kt">MkDivideException</span> <span class="ow">::</span> <span class="kt">HasCallStack</span> <span class="ow">=></span>
<span class="kt">DivideErrors</span> <span class="ow">-></span> <span class="kt">DivideException</span>
<span class="kr">deriving</span> <span class="kr">instance</span> <span class="kt">Exception</span> <span class="kt">DivideException</span>
<span class="kr">instance</span> <span class="kt">Show</span> <span class="kt">DivideException</span> <span class="kr">where</span>
<span class="n">show</span> <span class="p">(</span><span class="kt">MkDivideException</span> <span class="n">errors</span><span class="p">)</span> <span class="ow">=</span>
<span class="n">renderExceptionWithCallstack</span> <span class="n">errors</span> <span class="s">"MkDivideException"</span>
<span class="nf">renderExceptionWithCallstack</span> <span class="ow">::</span>
<span class="p">(</span><span class="kt">HasCallStack</span><span class="p">,</span> <span class="kt">Show</span> <span class="n">a</span><span class="p">)</span> <span class="ow">=></span> <span class="n">a</span> <span class="ow">-></span> <span class="kt">String</span> <span class="ow">-></span> <span class="kt">String</span>
<span class="nf">renderExceptionWithCallstack</span> <span class="n">errors</span> <span class="n">valueConstructor</span> <span class="ow">=</span> <span class="s">"("</span> <span class="o"><></span> <span class="n">valueConstructor</span> <span class="o"><></span> <span class="s">" $ "</span>
<span class="o"><></span> <span class="n">show</span> <span class="n">errors</span>
<span class="o"><></span> <span class="s">"/*"</span>
<span class="o"><></span> <span class="n">prettyCallStack</span> <span class="n">callStack</span>
<span class="o"><></span> <span class="s">" */)"</span>
<span class="nf">throwDivide</span> <span class="ow">::</span> <span class="kt">HasCallStack</span> <span class="ow">=></span> <span class="kt">Either</span> <span class="kt">DivideErrors</span> <span class="n">a</span> <span class="ow">-></span> <span class="kt">IO</span> <span class="n">a</span>
<span class="nf">throwDivide</span> <span class="ow">=</span> <span class="n">either</span> <span class="p">(</span><span class="n">throwIO</span> <span class="o">.</span> <span class="kt">MkDivideException</span><span class="p">)</span> <span class="n">pure</span>
<span class="nf">myEnv</span> <span class="ow">::</span> <span class="kt">Map</span> <span class="kt">Text</span> <span class="kt">Double</span>
<span class="nf">myEnv</span> <span class="ow">=</span> <span class="kt">Map</span><span class="o">.</span><span class="n">fromList</span> <span class="p">[(</span><span class="s">"zero"</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">),</span> <span class="p">(</span><span class="s">"one"</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.0</span><span class="p">),</span>
<span class="p">(</span><span class="s">"two"</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">),</span> <span class="p">(</span><span class="s">"three"</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">)]</span>
<span class="nf">main</span> <span class="ow">::</span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="nf">main</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">result1</span> <span class="ow"><-</span> <span class="n">throwDivide</span> <span class="o">$</span> <span class="n">divide</span> <span class="n">myEnv</span> <span class="s">"one"</span> <span class="s">"two"</span>
<span class="n">result2</span> <span class="ow"><-</span> <span class="n">throwDivide</span> <span class="o">$</span> <span class="n">divide</span> <span class="n">myEnv</span> <span class="s">"one"</span> <span class="s">"three"</span> <span class="c1">-- throws</span>
<span class="n">print</span> <span class="p">(</span><span class="n">result1</span> <span class="p">,</span> <span class="n">result2</span><span class="p">)</span>
</pre></div>
<p>All this boilerplate attaches the callstack to our exception.
We also put the entire pure error type
directly into the exception.
It composes.
This is why pure error handling is preferable,
but if you don’t have time to do this, exceptions like those described above are good too.
This idea of attaching call stacks to your exceptions is
explained further in <a href="https://maksbotan.github.io/posts/2021-01-20-callstacks.html">this blogpost</a></p>
<p>If that’s to much work, the <a href="https://hackage.haskell.org/package/base/docs/Prelude.html#v:error"><code>error</code></a>
call also has a stack trace,
although its type allows vagueness unfortunately.
Also make sure to attach it to <span class="caps">IO</span>,
or the <a href="#throw">nullpointer</a> anti pattern may occur.
I think the <a href="https://hackage.haskell.org/package/extra-1.7.10/docs/Control-Exception-Extra.html#v:errorIO"><code>errorIO</code></a>
function is a good idea instead of <code>error</code>
because it avoids that nullpointer scenario entirely.
Don’t try to catch errors,
use exceptions if you need to catch.</p>
<h2 id="mtl"><span class="caps">MTL</span></h2>
<p>I recently blogged about <a href="https://jappieklooster.nl/a-brief-intro-to-mtl.html">mtl</a>,
so I’ll briefly cover how to modify this code into mtl style as well:</p>
<div class="highlight"><pre><span></span><span class="nf">throwDivide</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">HasCallStack</span><span class="p">,</span> <span class="kt">MonadIO</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kt">ExceptT</span> <span class="kt">DivideFailures</span> <span class="n">a</span> <span class="ow">-></span> <span class="n">m</span> <span class="n">a</span>
<span class="nf">throwDivide</span> <span class="n">meow</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">res</span> <span class="ow"><-</span> <span class="n">runExceptT</span> <span class="n">meow</span>
<span class="kr">case</span> <span class="n">res</span> <span class="kr">of</span>
<span class="kt">Left</span> <span class="n">err</span> <span class="ow">-></span> <span class="n">liftIO</span> <span class="o">$</span> <span class="n">throwIO</span> <span class="o">$</span> <span class="kt">MkDivideException</span> <span class="n">err</span>
<span class="kt">Right</span> <span class="n">res</span> <span class="ow">-></span> <span class="n">pure</span> <span class="n">res</span>
<span class="nf">myDBFunc</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">MonadError</span> <span class="kt">DivideFailures</span> <span class="n">m</span><span class="p">,</span> <span class="kt">MonadDB</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="n">m</span> <span class="kt">Double</span>
<span class="nf">myDBFunc</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">myEnv</span> <span class="ow"><-</span> <span class="n">getMyEnv</span>
<span class="n">divide</span> <span class="n">myEnv</span> <span class="s">"one"</span> <span class="s">"two"</span>
<span class="nf">main</span> <span class="ow">::</span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="nf">main</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">result1</span> <span class="ow"><-</span> <span class="n">runDB</span> <span class="o">$</span> <span class="n">throwDivide</span> <span class="o">$</span> <span class="n">myDBFunc</span>
<span class="n">print</span> <span class="n">result1</span>
</pre></div>
<p><code>throwDivide</code> works quite similarly as in the previous example but now
it works with any transformer stack based on <code>IO</code>.
This works because we pretend the <code>ExceptT</code> exists at the call site,
which makes it come true.
This is explained thoroughly in the previous <a href="https://jappieklooster.nl/a-brief-intro-to-mtl.html">blog post</a>.</p>
<h1 id="anti-patterns">Anti patterns</h1>
<p>Now I’ll cover several anti patterns I’ve seen and
discuss what to do differently.
Some of these patterns occur in the wild,
for example <a href="https://hackage.haskell.org/package/aeson-2.0.3.0/docs/Data-Aeson.html#v:eitherDecode">aeson</a>
famously exposes a <code>String</code> for errors,
which is problematic for a library.
The next section explains why.</p>
<h2 id="text-in-left-branch-of-either">Text in left branch of Either</h2>
<p>These examples contain text in the left branch:</p>
<div class="highlight"><pre><span></span><span class="nf">y</span> <span class="ow">::</span> <span class="kt">Either</span> <span class="kt">Text</span> <span class="n">a</span>
<span class="nf">x</span> <span class="ow">::</span> <span class="kt">Either</span> <span class="kt">String</span> <span class="n">a</span>
</pre></div>
<p>The problem is that we break the recoverability
property.
There is no way to make a closed pattern match on a string.
Instead we should create a sumtype for possible errors:</p>
<div class="highlight"><pre><span></span><span class="kr">data</span> <span class="kt">YErrors</span> <span class="ow">=</span> <span class="kt">YErrorOne</span>
<span class="o">|</span> <span class="kt">YErrorTwo</span> <span class="kt">Double</span>
<span class="nf">y</span> <span class="ow">::</span> <span class="kt">Either</span> <span class="kt">YErrors</span> <span class="n">a</span>
</pre></div>
<p>This way client code can pattern match on all possible
branches, and if the additional errors get introduced
the compiler notifies the developer through
<a href="https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/using-warnings.html#ghc-flag--Wincomplete-patterns"><code>-Wincomplete-patterns</code> warning</a>.</p>
<h2 id="throw"><code>throw</code></h2>
<p><code>throw</code> allows throwing exceptions in pure code.
This is very wrong because the code ends up behaving like a null
pointer in Java, due to Haskell’s non strict evaluation.
In other words, this breaks the locality of errors.
Consider for example:</p>
<div class="highlight"><pre><span></span><span class="nf">myInts</span> <span class="ow">::</span> <span class="p">[</span><span class="kt">Int</span><span class="p">]</span>
<span class="nf">myInts</span> <span class="ow">=</span> <span class="kt">[]</span>
<span class="kr">data</span> <span class="kt">EmptyException</span> <span class="ow">=</span> <span class="kt">MkEmptyException</span>
<span class="kr">deriving</span> <span class="p">(</span><span class="kt">Exception</span><span class="p">,</span> <span class="kt">Show</span><span class="p">)</span>
<span class="nf">myHead</span> <span class="ow">::</span> <span class="p">[</span><span class="kt">Int</span><span class="p">]</span> <span class="ow">-></span> <span class="kt">Int</span>
<span class="nf">myHead</span> <span class="kt">[]</span> <span class="ow">=</span> <span class="n">throw</span> <span class="o">$</span> <span class="kt">MkEmptyException</span>
<span class="nf">myHead</span> <span class="p">(</span><span class="n">x</span> <span class="kt">:</span> <span class="n">y</span><span class="p">)</span> <span class="ow">=</span> <span class="n">x</span>
<span class="nf">main</span> <span class="ow">::</span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="nf">main</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="kr">let</span> <span class="n">x</span> <span class="ow">=</span> <span class="n">myHead</span> <span class="n">myInts</span>
<span class="n">print</span> <span class="o">$</span> <span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span> <span class="o"><></span> <span class="n">myInts</span>
<span class="n">print</span> <span class="n">x</span>
</pre></div>
<p>It fails on that last line.
Even though the error was at the <code>x</code> binding.
Much better is to use <a href="https://hackage.haskell.org/package/base/docs/Control-Exception.html#v:throwIO"><code>throwIO</code></a>,
modifying the above example:</p>
<div class="highlight"><pre><span></span><span class="nf">myInts</span> <span class="ow">::</span> <span class="p">[</span><span class="kt">Int</span><span class="p">]</span>
<span class="nf">myInts</span> <span class="ow">=</span> <span class="kt">[]</span>
<span class="kr">data</span> <span class="kt">EmptyException</span> <span class="ow">=</span> <span class="kt">MkEmptyException</span>
<span class="kr">deriving</span> <span class="kt">Exception</span>
<span class="nf">myHead</span> <span class="ow">::</span> <span class="p">[</span><span class="kt">Int</span><span class="p">]</span> <span class="ow">-></span> <span class="kt">IO</span> <span class="kt">Int</span>
<span class="nf">myHead</span> <span class="kt">[]</span> <span class="ow">=</span> <span class="n">throwIO</span> <span class="o">$</span> <span class="kt">MkEmptyException</span>
<span class="nf">myHead</span> <span class="p">(</span><span class="n">x</span> <span class="kt">:</span> <span class="n">y</span><span class="p">)</span> <span class="ow">=</span> <span class="n">pure</span> <span class="n">x</span>
<span class="nf">main</span> <span class="ow">::</span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="nf">main</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">x</span> <span class="ow"><-</span> <span class="n">myHead</span> <span class="n">myInts</span>
<span class="n">print</span> <span class="o">$</span> <span class="p">[</span><span class="mi">0</span><span class="o">..</span><span class="mi">5</span><span class="p">]</span> <span class="o"><></span> <span class="n">myInts</span>
<span class="n">print</span> <span class="n">x</span>
</pre></div>
<p>This will indeed fail on the first line.
We lost purity,
in this case <code>Either</code> could have been used as
described above, which is even better.
Alternatively we can use the <code>HasCallStack</code>
trick, if we stick with exceptions.</p>
<h2 id="generic-app-exceptions">Generic app exceptions</h2>
<p>I’m talking about something like this:</p>
<div class="highlight"><pre><span></span><span class="kr">data</span> <span class="kt">AppException</span> <span class="ow">=</span> <span class="kt">MkAppException</span> <span class="kt">Text</span>
<span class="kr">deriving</span> <span class="kt">Exception</span>
</pre></div>
<p>This is bad because it ends up being thrown
at many places with no good way of recovering.
Exceptions are already difficult to recover from,
but if they’re re-used a lot,
it becomes even more difficult.
Now you’d have to pattern match on the <code>Text</code>
to handle the correct error and hope no one
re-uses that particular error text, ever!
Furthermore the error may be vague,
depending on what’s being put in the <code>Text</code> field.</p>
<p>It’s better to define a custom exception per situation.
For example:</p>
<div class="highlight"><pre><span></span><span class="kr">data</span> <span class="kt">DBUserNotFound</span> <span class="ow">=</span> <span class="kt">MkDBUserNotFound</span> <span class="p">{</span> <span class="n">userId</span> <span class="ow">::</span> <span class="kt">UUID</span> <span class="p">}</span>
<span class="kr">deriving</span> <span class="kt">Exception</span>
<span class="kr">data</span> <span class="kt">AwsImgResourceNotFound</span> <span class="ow">=</span> <span class="kt">MkImgResourceNotFound</span> <span class="p">{</span><span class="n">name</span> <span class="ow">::</span> <span class="kt">Text</span><span class="p">,</span> <span class="n">awsId</span> <span class="ow">::</span> <span class="kt">UUID</span> <span class="p">}</span>
<span class="kr">deriving</span> <span class="kt">Exception</span>
</pre></div>
<p>These are precise and type safe, we no longer can be vague because
you have to provide a <code>UUID</code>.
This forces the throw site to capture this <code>UUID</code> from
the environment, making the error precise trough type safety.</p>
<p>However, you’ll likely already have generic app
exception being called from different 400 places.
Fortunately we can recover some of the locality property by rewriting
the exception <a href="https://maksbotan.github.io/posts/2021-01-20-callstacks.html#capturing-stacks">in a <span class="caps">GADT</span></a>
and using the <code>HasCallStack</code> trick.</p>
<h2 id="squashing-errors">Squashing errors</h2>
<p>This can occur when using bind <code>>>=</code> on <code>Maybe</code>, for example:</p>
<div class="highlight"><pre><span></span><span class="nf">x</span> <span class="ow">::</span> <span class="kt">Maybe</span> <span class="kt">Int</span>
<span class="nf">y</span> <span class="ow">::</span> <span class="kt">Maybe</span> <span class="kt">Int</span>
<span class="nf">z</span> <span class="ow">::</span> <span class="kt">Maybe</span> <span class="kt">Int</span>
<span class="nf">z</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">x'</span> <span class="ow"><-</span> <span class="n">x</span>
<span class="n">y'</span> <span class="ow"><-</span> <span class="n">y</span>
<span class="n">pure</span> <span class="o">$</span> <span class="n">x'</span> <span class="o">+</span> <span class="n">y'</span>
</pre></div>
<p><code>Nothing</code> won’t tell us why <code>z</code> failed.
This is wrong because it breaks preciseness.
instead we should restructure like so:</p>
<div class="highlight"><pre><span></span><span class="nf">x</span> <span class="ow">::</span> <span class="kt">Maybe</span> <span class="kt">Int</span>
<span class="nf">y</span> <span class="ow">::</span> <span class="kt">Maybe</span> <span class="kt">Int</span>
<span class="kr">data</span> <span class="kt">ZFailures</span> <span class="ow">=</span> <span class="kt">NoX</span>
<span class="o">|</span> <span class="kt">NoY</span>
<span class="nf">z</span> <span class="ow">::</span> <span class="kt">Either</span> <span class="kt">ZFailures</span> <span class="kt">Int</span>
<span class="nf">z</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">x'</span> <span class="ow"><-</span> <span class="n">maybe</span> <span class="p">(</span><span class="kt">Left</span> <span class="kt">NoX</span><span class="p">)</span> <span class="kt">Right</span> <span class="o">$</span> <span class="n">x</span>
<span class="n">y'</span> <span class="ow"><-</span> <span class="n">maybe</span> <span class="p">(</span><span class="kt">Left</span> <span class="kt">NoY</span><span class="p">)</span> <span class="kt">Right</span> <span class="o">$</span> <span class="n">y</span>
<span class="n">pure</span> <span class="o">$</span> <span class="n">x'</span> <span class="o">+</span> <span class="n">y'</span>
</pre></div>
<p>This will tell you exactly what went wrong,
while retaining most of the power of bind.
It’s fine to use <code>Maybe</code> if there is a single
error case,
but often this isn’t the case.</p>
<h1 id="conclusion">Conclusion</h1>
<p>The preference relation of failing is like this:</p>
<div class="highlight"><pre><span></span>pure error handling > exceptions *with* stack traces > errorIO calls
</pre></div>
<p>Anything else is most likely <em>wrong</em> and should be avoided.
Structure your programs so that it tells what goes wrong,
where and why.
I think most software would benefit from doing this,
and these concepts are easy.
Let <a href="mailto:hi@jappie.me">me know</a> if you disagree, or need help with this.</p>
<h1 id="references">References</h1>
<ul>
<li>https://github.com/waddlaw/haskell-stack-trace-plugin</li>
<li>https://www.stackbuilders.com/blog/errors-and-exceptions-in-haskell/</li>
<li>https://wiki.haskell.org/Handling_errors_in_Haskell</li>
<li>http://www.randomhacks.net/2007/03/10/haskell-8-ways-to-report-errors/</li>
<li>https://maksbotan.github.io/posts/2021-01-20-callstacks.html</li>
<li>https://hackage.haskell.org/package/validation-1.1.2/docs/Data-Validation.html</li>
<li>https://www.parsonsmatt.org/2018/11/03/trouble_with_typed_errors.html</li>
</ul>
<div class="footnote">
<hr>
<ol>
<li id="fn-anti-patterns">
<p>Wrong as in they were following some of the ani patterns listed below. <a class="footnote-backref" href="#fnref-anti-patterns" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-15-years-ago">
<p>This is 15 years ago, ah well, better late then never I suppose. <a class="footnote-backref" href="#fnref-15-years-ago" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-other-langs">
<p>When I was writing this I realized I do the exact same program structuring
in Java or <span class="caps">PHP</span>.
It is more work in Java or <span class="caps">PHP</span>, but possible and saves so much debugging time. <a class="footnote-backref" href="#fnref-other-langs" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn-memory">
<p>Accessing memory is for no particular reason considered pure,
although one wonders if this could be changed
to make managing memory bounds easier. <a class="footnote-backref" href="#fnref-memory" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn-throw">
<p>See the throw <a href="#throw">anti pattern</a> <a class="footnote-backref" href="#fnref-throw" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
</ol>
</div>Installing a NixOS desktop tracked with git2022-01-30T16:20:00+01:002022-02-01T01:00:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2022-01-30:/installing-a-nixos-desktop-tracked-with-git.html<p>A few years ago I wrote a post on installing
NixOS <a href="https://jappieklooster.nl/nixos-on-encrypted-btrfs.html">on encrypted btrfs</a>.
I recently went trough that guide to install
NixOS once more.
It is good, but it has some issues:</p>
<ol>
<li>btrfs: Which I no longer use due to performance concerns.</li>
<li><a href="https://git-scm.com/">git</a>: This requires some special attention,
but …</li></ol><p>A few years ago I wrote a post on installing
NixOS <a href="https://jappieklooster.nl/nixos-on-encrypted-btrfs.html">on encrypted btrfs</a>.
I recently went trough that guide to install
NixOS once more.
It is good, but it has some issues:</p>
<ol>
<li>btrfs: Which I no longer use due to performance concerns.</li>
<li><a href="https://git-scm.com/">git</a>: This requires some special attention,
but composes really well with nix and nixos.</li>
<li>It doesn’t explain how to deal with secrets.</li>
</ol>
<p>I’ll address these concerns here,
since I <em>just</em> bricked an install due to git usage <sup id="fnref-hardware"><a class="footnote-ref" href="#fn-hardware">1</a></sup>.
And I also wasted some time re-figuring out secrets.</p>
<p>Why use git if it introduces complexity?
For one it serves as an excellent backup tool,
furthermore it allows managing multiple deployments
side by side trough branches.
Finally having a log of changes can be useful when things
break.
Things <em>will</em> break. Such is the life of a tinkerer. </p>
<p>I imagine some people still want to use btrfs,
so I’ll leave the old guide in place.
However, I’ll copy over parts which were good in here for convenience.
This updated guide still uses encrypted disks,
I’ve had no problems with this at all,
and I recommend disk encryption to all.</p>
<h1 id="getting-started">Getting started</h1>
<p>Get yourself a NixOS <a href="https://nixos.org/download.html#download-nixos">live usb</a>.
I use the minimal <span class="caps">ISO</span>, because the graphical <span class="caps">ISO</span> slows booting and gives no advantage
aside from being pretty. Use</p>
<div class="highlight"><pre><span></span>cat minimal-nixos.iso > /dev/sdX
</pre></div>
<p>where <code>X</code> is the usb drive found by <code>lsblk</code>.
<code>X</code> should be a letter, numbers indicate partitions,
which we don’t want to cat upon because the <span class="caps">ISO</span> already
contains a partitioning scheme.</p>
<p>Boot into it on the target machine.
Become root with <code>sudo -i</code>.</p>
<h1 id="internet">Internet</h1>
<p>Next step is to setup <span class="caps">WIFI</span>, you can skip this if you’re on Ethernet:</p>
<div class="highlight"><pre><span></span>wpa_passphrase SSID PASS > /etc/wpa_supplicant.conf
systemctl restart wpa_supplicant
</pre></div>
<p>The first command creates a config for wpa_supplicant.
The reader must fill in <span class="caps">SSID</span> and <span class="caps">PASS</span> of his target wifi network.
The second command tells systemd to go restart wpa_supplicant and use the new config.</p>
<p>Ask google if you’re online:</p>
<div class="highlight"><pre><span></span>curl google.com
</pre></div>
<p>Should return a 301 Moved.
If it hangs or refuses the connection you likely have no internet.
There is no point proceeding until you have internet access.</p>
<h1 id="partitioning">Partitioning</h1>
<p>Now to setup the partitioning on the <span class="caps">RIGHT</span> device.
Choose carefully.
Use <code>lsblk</code> to figure out which device is <span class="caps">RIGHT</span>.
You’ll know it’s the <span class="caps">WRONG</span> device if you lose data after partitioning.
The <span class="caps">RIGHT</span> device will be called <code>$dev</code> hence forward.</p>
<p>There are no other partitioning tools than gdisk.
Only heretics believe there are.
Therefore we use gdisk:</p>
<div class="highlight"><pre><span></span>gdisk <span class="nv">$dev</span>
</pre></div>
<h2 id="gdisk-cheat-sheet">Gdisk cheat sheet</h2>
<table>
<thead>
<tr>
<th>Command</th>
<th>Effect</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>p</code></td>
<td>For printing, to see what’s going on.</td>
</tr>
<tr>
<td><code>d</code></td>
<td>For deletion, you should start out with deleting everything on <code>$dev</code>.</td>
</tr>
<tr>
<td><code>n</code></td>
<td>Is used for creating new partitions.</td>
</tr>
<tr>
<td><code>w</code></td>
<td>is used for writing once finished.</td>
</tr>
</tbody>
</table>
<p>This table just describes the commands needed for the intended partitioning.</p>
<h2 id="intended-partitioning">Intended partitioning</h2>
<table>
<thead>
<tr>
<th>Number</th>
<th>type</th>
<th>size</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>ef00</td>
<td>+500M</td>
</tr>
<tr>
<td>2</td>
<td>8200</td>
<td>+$(SIZE_RAM+ a little)G</td>
</tr>
<tr>
<td>3</td>
<td>8300</td>
<td>(rest of disk)</td>
</tr>
</tbody>
</table>
<p>The first partition will be boot,
the second swap<sup id="fnref-optional"><a class="footnote-ref" href="#fn-optional">2</a></sup>,
the third will be everything else.
We will encrypt everything else.
With type <code>ef00</code> we will use <span class="caps">UEFI</span> for booting.
Don’t worry. nix will handle that, mostly.
Done. Onwards!</p>
<h1 id="encryption">Encryption</h1>
<p>We use <code>cryptsetup</code> for encryption.
Make sure to select the right partition.
We do not want to encrypt the boot partition because then we can’t boot.
So if you followed above instructions it will be either <code>3</code> or <code>p3</code>
(depending on device type).
We’ll call it <code>3</code>.</p>
<div class="highlight"><pre><span></span>cryptsetup luksFormat <span class="s2">"</span><span class="nv">$dev</span><span class="s2">"</span><span class="m">3</span>
cryptsetup open <span class="s2">"</span><span class="nv">$dev</span><span class="s2">"</span><span class="m">3</span> nixenc
</pre></div>
<p>The first command does the actual formatting,
the second one opens up the formatted disk.
You’ll need to provide the right password in both cases.
Choose one you can remember but is strong.
Once decrypted the disk will be mapped to <code>/dev/mapper/nixenc</code>,
note that we supplied that final part in the last command.</p>
<h1 id="formatting-filesystems">Formatting filesystems</h1>
<p>Partitioning is a distinct step from setting up filesystems.</p>
<div class="highlight"><pre><span></span>mkfs.vfat -n boot <span class="s2">"</span><span class="nv">$dev</span><span class="s2">"</span><span class="m">1</span>
mkswap <span class="s2">"</span><span class="nv">$dev</span><span class="s2">"</span><span class="m">2</span>
swapon <span class="s2">"</span><span class="nv">$dev</span><span class="s2">"</span><span class="m">2</span>
mkfs.ext4 -L root /dev/mapper/nixenc
</pre></div>
<p>The boot partition will be <code>vfat</code> because <a href="https://wiki.archlinux.org/index.php/EFI_system_partition"><span class="caps">UEFI</span> tells us to</a>.
The everything else partition will be <code>ext4</code>.
Note that we point it at the mapped file,
if the <code>"$dev"3</code>device were to be used directly we’d remove the encryption.</p>
<h1 id="mounting">Mounting</h1>
<p>Here we mount all partitions.</p>
<div class="highlight"><pre><span></span>mount /dev/mapper/nixenc /mnt/
mkdir /mnt/boot
mount <span class="s2">"</span><span class="nv">$dev</span><span class="s2">"</span><span class="m">1</span> /mnt/boot
</pre></div>
<p>This makes them detectable by the nix config generation script.
Furthermore it allows the script to write the config on
the proper disk.</p>
<h2 id="did-i-do-everything-right">Did I do everything right?</h2>
<p>The second time I ran trough this post everything went
quite quickly,
so I became skeptical.
To verify everything was sane I used the following commands:</p>
<div class="highlight"><pre><span></span>mount <span class="p">|</span> grep /mnt
ls /mnt
</pre></div>
<p>The first command is to check if the encrypted volume and boot is mounted at
the right paths.
The second one to verify the folders are created.</p>
<h1 id="configure-nix-part-1-in-the-live-environment">Configure nix part 1, in the live environment</h1>
<p>We use a script to generate an intial nix configuration,
which detects the hardware for us:</p>
<div class="highlight"><pre><span></span>nixos-generate-config --root /mnt
</pre></div>
<p>This will give us a default config
which we can <a href="https://nixos.org/nixos/manual/index.html#sec-changing-config">customize</a> now.
You can also pick up ideas from <a href="https://github.com/jappeace/linux-config/blob/work-machine/configuration.nix">my config</a>, although its a mess.
On a wireless laptop,
it’s highly recommended to enable <a href="https://nixos.wiki/wiki/Wpa_supplicant">wpa_supplicant</a>:</p>
<div class="highlight"><pre><span></span>networking<span class="o">.</span>wireless<span class="o">.</span><span class="ss">enable =</span> <span class="no">true</span>
</pre></div>
<p>Furthermore we’re going to need the packages vim and git
to modify the config after booting into the system:</p>
<div class="highlight"><pre><span></span> environment = {
systemPackages = [
pkgs.git
pkgs.vim
];
};
</pre></div>
<p>Make sure to set the channel to the same as in your
tracked git configuration,
or alternatively be ready to deal with upgrades.
For example on a live <span class="caps">ISO</span> of <code>21.11</code> I downgraded to <code>21.05</code> with:</p>
<div class="highlight"><pre><span></span>nix-channel --add https://nixos.org/channels/nixos-21.05 nixos
nix-channel --update
</pre></div>
<p>and then I changed the <code>configuration.nix</code> to:</p>
<div class="highlight"><pre><span></span> system = {
stateVersion = "21.05";
};
</pre></div>
<p>Dealing with channels is quite fragile,
so I have these commands copied as comments in my <code>configuration.nix</code>.</p>
<p>Once configuration is done we can install nix:</p>
<div class="highlight"><pre><span></span>nixos-install
</pre></div>
<p>Don’t worry, we can use <code>nixos-rebuild switch</code> to reconfigure nix whenever,
once we’re booted into it.
Hopefully we boot successfully:</p>
<div class="highlight"><pre><span></span>reboot
</pre></div>
<p>Booting is hard, don’t worry if this goes wrong the first <s>10</s> 30 times.</p>
<p>You may need to enable <span class="caps">UEFI</span> in your <span class="caps">BIOS</span>.
It’s up to the reader to figure that part out <sup id="fnref-f-keys"><a class="footnote-ref" href="#fn-f-keys">3</a></sup><sup id="fnref-boot-issue"><a class="footnote-ref" href="#fn-boot-issue">4</a></sup>.
Alternatively one could setup grub. Good luck with that.
You can’t read the rest of this post until you’ve booted.
Go back if you haven’t booted, you messed up.</p>
<h1 id="configure-nix-part-2-with-git">Configure nix part 2, with git</h1>
<p>Once we’re booted into the installation,
the absolute paths for the symlinks are different.
For example root is no longer under <code>/mnt</code>, but under, well, root <code>/</code>.
So login as root<sup id="fnref-display-manager"><a class="footnote-ref" href="#fn-display-manager">5</a></sup> and clone your config project:</p>
<div class="highlight"><pre><span></span><span class="nb">cd</span> /
git clone https://github.com/jappeace/linux-config
chown jappie:users -R /linux-config
</pre></div>
<p>This puts the linux-config project on the <code>/linux-config</code> path.
Some old time linux/unix users may puke in their mouths
upon seeing the “standard” directories being ignored, but fuck them.
We mustn’t forget to copy over the hardware generated
config into our git project:</p>
<div class="highlight"><pre><span></span>cp /etc/nixos/hardware-configuration.nix /linux-config/hardware/branch-name.nix
</pre></div>
<p>Where branch name the name is for the git branch you’ll use for this deployment.
More on that in the branches section.
Also we want to include this as a module in the <code>configuration.nix</code></p>
<div class="highlight"><pre><span></span> <span class="ss">imports =</span> <span class="p">[</span>
<span class="o">.</span><span class="l">/hardware/branch-name.nix</span>
<span class="p">];</span>
</pre></div>
<p>The tracked configuration should be used by the system.
What I usually do is login as my own user,
and run the script <a href="https://github.com/jappeace/linux-config/blob/work-machine/scripts/nixos-setup.sh">setup-nixos.sh</a>:</p>
<div class="highlight"><pre><span></span><span class="nb">exit</span>
<span class="nb">cd</span> /linux-config/scripts/
./setup-nixos.sh
</pre></div>
<p>Which sets up the symlink from the git tracked configuration to
the standard location, in other words it does this:</p>
<div class="highlight"><pre><span></span>ln -sf /linux-config/configuration.nix /etc/nixos/configuration.nix
</pre></div>
<p>This script also symlinks all relevant
dotfiles from linux config into the home folder.
You may want prefer <a href="https://github.com/nix-community/home-manager">home manager</a>
to symlinking dotfiles.
But this works for me.</p>
<h2 id="multiple-machines-desktops">Multiple machines desktops</h2>
<p>Branches are ideal for managing multiple machines,
because it allows you to diverge oddities such as hardware specific configurations.
For example I also have this crummy display switch script which is only relevant
for the <span class="caps">PC</span>.
On the laptop it has to be slightly different (if used at all).</p>
<p>This branch setup also allows merging back configuration changes from other machines.
Which involves solving an ordinary git conflict.
I recommend the reader to use merges rather then rebases,
because that way git remembers how conflicts are resolved.</p>
<p>However I recently learned that a friend of mine handles this trough
a <a href="https://github.com/erikbackman/nixos-config/blob/master/flake.nix#L54">module</a>
system.
He then runs that with <code>nixos-rebuild --flake .#machine-name</code>.
So rather then using branches he has an entrypoint per machine
which he calls out directly with the flake.
I find his setup interesting, and may move over to something like that in the future,
although since flakes are still experimental, I’ll hold off.</p>
<p>Another alternative is multiple configuration.nix files
in a repository and let the symlink decide which should be used
for what machine.
This would avoid any merge conflicts,
although the versions between machines need to be similar.
I think I prefer merge conflicts.</p>
<h1 id="secrets">Secrets</h1>
<p>I have three major secret sources.
1. ssh keys
2. gpg keys
3. The keepassxc database</p>
<p>The database in <a href="https://syncthing.net/">synchting</a>,
this gives me access to all services so that I can
simply generate new gpg and ssh keys per deployment.
Any other file manage service would do, but I like
syncthing because it’s decentralized.</p>
<p>I go to https://localhost:8384 on some device that has the database,
and the target device on the same address.
I type over the device id into that screen to start syncing.
after syncing completes I have access to the keepasscx database.
Now I can generate new ssh keys and gpg keys,
and login to services to update those.
With that finished the installation is complete.</p>
<p>Aside from getting the keypass database up and running,
it’s important to add your newly generated public key to
the services you manage.
For example this website is hosted on the <a href="https://jappieklooster.nl/the-nix-mutli-monolith-machine-nmmm.html">nixos multi monolith</a>.
It be prudent to add the ssh key via a machine that already
has access to it.
Syncthing can also be used for this.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-hardware">
<p>I had uuid for my disks, but I swapped the disks so the boot bricked. <a class="footnote-backref" href="#fnref-hardware" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-optional">
<p>This one is optional but allows hibernation.
Which is very convenient for laptops.
It can also make your system <a href="https://askubuntu.com/questions/291378/do-we-still-need-swap-partitions-on-servers">more stable</a>. <a class="footnote-backref" href="#fnref-optional" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-f-keys">
<p>press some f keys on boot, f11 or f2 maybe? <a class="footnote-backref" href="#fnref-f-keys" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn-boot-issue">
<p>An issue I encountered was that rather then selecting <span class="caps">EUFI</span> boot,
the bios did a traditional boot on the disk.
So <span class="caps">EUFI</span> was correctly installed,
I just had to select the same disk but with the <span class="caps">EUFI</span> label from the bios.
Yup, all kinds of stuff can go wrong with booting. <a class="footnote-backref" href="#fnref-boot-issue" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn-display-manager">
<p>Once rebooted you may be stuck at the display manager.
Use <code>Alt+f1</code> to switch to another <span class="caps">TTY</span> and login as root,
then use <code>passwd your-user-name</code> to set an initial password for that user.
Use <code>Alt+f7</code> to go back to the display manager. <a class="footnote-backref" href="#fnref-display-manager" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
</ol>
</div>A brief intro to MTL2021-08-10T00:05:00+02:002021-08-10T09:50:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2021-08-10:/a-brief-intro-to-mtl.html<p><img alt="mtl-header" src="images/2021/mtl.png"></p>
<p>Recently a blog post came out which I quite like,
it describes how to use the concrete
base <a href="https://blog.cofree.coffee/2021-08-05-a-brief-intro-to-monad-transformers/">transformers</a>.
It’s very thorough and gives a concrete example for using
transformers.
Although it looks quite low level and
I think you’ll get more out of transformers by
using full …</p><p><img alt="mtl-header" src="images/2021/mtl.png"></p>
<p>Recently a blog post came out which I quite like,
it describes how to use the concrete
base <a href="https://blog.cofree.coffee/2021-08-05-a-brief-intro-to-monad-transformers/">transformers</a>.
It’s very thorough and gives a concrete example for using
transformers.
Although it looks quite low level and
I think you’ll get more out of transformers by
using full <span class="caps">MTL</span>.</p>
<p>That blogpost inspired me to write this,
because I can’t find a succinct description on how to
use <span class="caps">MTL</span><sup id="fnref-best-mtl"><a class="footnote-ref" href="#fn-best-mtl">1</a></sup>.
I learned <span class="caps">MTL</span> by staring at <a href="https://hackage.haskell.org/package/reflex">reflex</a>
for days, if not weeks.
Which is an uncomfortable learning process.
To make <span class="caps">MTL</span> more accessible I’ll give a brief overview of this style<sup id="fnref-mtl-vs-transformers"><a class="footnote-ref" href="#fn-mtl-vs-transformers">3</a></sup>.
I’ll write down how <span class="caps">MTL</span> works from the ground up,
so people can <em>read</em> how to use it rather
then struggling with code for days like I did.</p>
<p>If you like video presentations, I also presented how to use <span class="caps">MTL</span> in a
<a href="https://www.youtube.com/watch?v=MPlrAe-XYMU&t=300s">video format</a>.</p>
<h2 id="the-type-variable-m"><a id="intro"></a> The type variable <code>m</code></h2>
<p>We start by introducing <code>m</code>.
Which could’ve been named <code>monad</code> or <code>x</code>,
but the community settled on using <code>m</code> for type variables
with the <code>Monad</code> constraint, so I will use this too.
Normally we use type variable’s in concrete types,
for example <code>Maybe a</code> or <code>[a]</code>.
However, Instead of having our type variable
inside a concrete type, we can also flip it around:</p>
<div class="highlight"><pre><span></span><span class="nf">moreMonad</span> <span class="ow">::</span> <span class="kt">Monad</span> <span class="n">m</span> <span class="ow">=></span> <span class="n">m</span> <span class="kt">Int</span>
<span class="nf">moreMonad</span> <span class="ow">=</span> <span class="n">return</span> <span class="mi">5</span>
</pre></div>
<p>This compiles because the <code>Monad</code> constraint on <code>m</code>
gives us the <a href="https://hackage.haskell.org/package/base-4.15.0.0/docs/Control-Monad.html#v:return"><code>return</code></a>
function <sup id="fnref-nonsense"><a class="footnote-ref" href="#fn-nonsense">2</a></sup>.
After you’re convinced this is a valid definition,
let’s use it.
What can we do with this <code>moreMonad</code> binding?
Well, we can pattern match on it:</p>
<div class="highlight"><pre><span></span><span class="nf">fiveTroughMaybe</span> <span class="ow">::</span> <span class="kt">Int</span>
<span class="nf">fiveTroughMaybe</span> <span class="ow">=</span> <span class="kr">case</span> <span class="n">moreMonad</span> <span class="kr">of</span>
<span class="kt">Just</span> <span class="n">x</span> <span class="ow">-></span> <span class="n">x</span>
<span class="kt">Nothing</span> <span class="ow">-></span> <span class="mi">9</span>
</pre></div>
<p><span class="caps">GHC</span> will give
<code>moreMonad</code> the type <code>Maybe</code>
at this call site.
<span class="caps">GHC</span> reasons backwards from the pattern match up to the <code>case moreMonad of</code>
definition to figure out the type.
In my head I describe this backwards reasoning process as:
“pretending you have a <code>Maybe</code> which makes it becomes true”.
So <code>fiveTroughMaybe</code>results in 5 because <code>return</code> is implemented as <code>Just</code>
on the <code>Maybe</code> type’s <code>Monad</code> instance.</p>
<p>This is valid.
You should convince yourself it’s valid.
To convince yourself I’m not lying
paste this code into <span class="caps">GHCI</span> before continuing,
and gain some confidence,
because yonder be dragons.</p>
<p>Continuing now with the same module we can also pattern match on <code>Either</code>:</p>
<div class="highlight"><pre><span></span><span class="nf">fiveTroughEither</span> <span class="ow">::</span> <span class="kt">Int</span>
<span class="nf">fiveTroughEither</span> <span class="ow">=</span> <span class="kr">case</span> <span class="n">moreMonad</span> <span class="kr">of</span>
<span class="kt">Right</span> <span class="n">x</span> <span class="ow">-></span> <span class="n">x</span>
<span class="kt">Left</span> <span class="n">_y</span> <span class="ow">-></span> <span class="mi">9</span>
</pre></div>
<p>Both <code>fiveTroughMaybe</code> and <code>fiveTroughEither</code> will result in <code>5</code>.
This is allowed in the same module because <code>moreMonad</code>
will only get assigned the type at the <a href="https://en.wikipedia.org/wiki/Call_site">call site</a>.
The compiler figures out the type of <code>moreMonad</code> by looking at usage per
call site.
This backwards ‘figuring out’ is normal for type variables.</p>
<h3 id="optional-mastery-exercise">(optional) mastery exercise</h3>
<ul>
<li>Can we always pattern match on every possible monad type like we just did with <code>Just</code> or <code>Either</code>?
Can we always get the value out without being in the same monad?
The answer is in the footnote. <sup id="fnref-no-pattern-match"><a class="footnote-ref" href="#fn-no-pattern-match">4</a></sup></li>
</ul>
<!-- + GHC figures out here the types on callsite, what mechanism can be used
to flip the reasoning direction of GHC?
RankNTypes? Existentials?
-->
<h2 id="transformers-as-constraints-on-m">Transformers as constraints on <code>m</code></h2>
<p>With that brief introduction,
we can start applying this idea to the ‘<a href="https://blog.cofree.coffee/2021-08-05-a-brief-intro-to-monad-transformers/">“A Brief Intro to Monad Transformers” blogpost</a>‘
which inspired me to write this.
In that blogpost,
a newtype is constructed to hold the entire monad transformer stack like this:</p>
<div class="highlight"><pre><span></span><span class="kr">newtype</span> <span class="kt">AppM</span> <span class="n">a</span> <span class="ow">=</span> <span class="kt">AppM</span> <span class="p">{</span>
<span class="n">runAppM</span> <span class="ow">::</span> <span class="kt">ExceptT</span> <span class="kt">String</span> <span class="p">(</span><span class="kt">State</span> <span class="p">(</span><span class="kt">M</span><span class="o">.</span><span class="kt">Map</span> <span class="kt">VariableName</span> <span class="kt">Int</span><span class="p">))</span> <span class="n">a</span>
<span class="p">}</span>
<span class="kr">deriving</span> <span class="kr">newtype</span> <span class="p">(</span><span class="kt">Functor</span><span class="p">,</span> <span class="kt">Applicative</span><span class="p">,</span> <span class="kt">Monad</span><span class="p">,</span> <span class="kt">MonadError</span> <span class="kt">String</span><span class="p">,</span>
<span class="kt">MonadState</span> <span class="p">(</span><span class="kt">M</span><span class="o">.</span><span class="kt">Map</span> <span class="kt">VariableName</span> <span class="kt">Int</span><span class="p">))</span>
</pre></div>
<p>I call this <code>AppM</code> a concrete type because there is only one way to pattern match on it.
We’re not allowed to pretend it’s a <code>Maybe</code> for example.
This definition is used in the <code>assignIndexToVariables</code> function:</p>
<div class="highlight"><pre><span></span><span class="nf">assignIndexToVariables</span> <span class="ow">::</span> <span class="kt">AST</span> <span class="kt">VariableName</span> <span class="ow">-></span> <span class="kt">Variables</span> <span class="ow">-></span> <span class="kt">AppM</span> <span class="p">(</span><span class="kt">AST</span> <span class="kt">Int</span><span class="p">)</span>
</pre></div>
<p>Instead of using the concrete type <code>AppM</code>,
we could use <span class="caps">MTL</span> type classes to describe what is needed.
These type classes will become constraints on <code>m</code>,
similarly to how <code>Monad</code> was a constraint on <code>m</code> in the <a href="#intro">introduction</a>.
Which means we want to have <a href="https://hackage.haskell.org/package/mtl-2.2.2/docs/Control-Monad-Except.html#t:MonadError"><code>MonadError String</code></a>
as replacement for <code>ExceptT String</code> <sup id="fnref-why-different-name"><a class="footnote-ref" href="#fn-why-different-name">5</a></sup>,
and <a href="https://hackage.haskell.org/package/mtl-2.2.2/docs/Control-Monad-State-Lazy.html#t:MonadState"><code>MonadState (M.Map VariableName Int)</code></a>
as replacement for <code>State (M.Map VariableName Int)</code>.
Doing
this will change the type signature of <code>assignIndexToVariables</code> as follows:</p>
<div class="highlight"><pre><span></span><span class="nf">assignIndexToVariables</span> <span class="ow">::</span>
<span class="kt">MonadError</span> <span class="kt">String</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">MonadState</span> <span class="p">(</span><span class="kt">M</span><span class="o">.</span><span class="kt">Map</span> <span class="kt">VariableName</span> <span class="kt">Int</span><span class="p">)</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">AST</span> <span class="kt">VariableName</span>
<span class="ow">-></span> <span class="kt">Variables</span>
<span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">AST</span> <span class="kt">Int</span><span class="p">)</span>
</pre></div>
<p>So what’s the difference?
The type signature is more verbose,
although we no longer need the newtype.
In trade for this verbosity,
we can use the monad stack in any order
at the call site.
Both invocations of running this code are now allowed:</p>
<div class="highlight"><pre><span></span><span class="nf">main</span> <span class="ow">::</span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="nf">main</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="o">...</span>
<span class="n">print</span> <span class="o">$</span> <span class="n">flip</span> <span class="n">evalState</span> <span class="n">mempty</span> <span class="o">$</span> <span class="n">runExceptT</span> <span class="o">$</span>
<span class="n">assignIndexToVariables</span> <span class="n">ast</span> <span class="n">vars</span>
<span class="n">print</span> <span class="o">$</span> <span class="n">runExcept</span> <span class="o">$</span> <span class="n">flip</span> <span class="n">evalStateT</span> <span class="n">mempty</span> <span class="o">$</span>
<span class="n">assignIndexToVariables</span> <span class="n">ast</span> <span class="n">vars</span>
</pre></div>
<p>This wasn’t possible in the original code.
We’ve told the compiler that the <em>order</em>
of a monad stack doesn’t matter.
Which makes sense because consider two monad stacks:</p>
<div class="highlight"><pre><span></span><span class="nf">at</span> <span class="ow">::</span> <span class="kt">Char</span> <span class="ow">-></span> <span class="kt">ExceptT</span> <span class="kt">String</span> <span class="p">(</span><span class="kt">State</span> <span class="p">(</span><span class="kt">M</span><span class="o">.</span><span class="kt">Map</span> <span class="kt">VariableName</span> <span class="kt">Int</span><span class="p">))</span> <span class="kt">Int</span>
<span class="nf">bt</span> <span class="ow">::</span> <span class="kt">Int</span> <span class="ow">-></span> <span class="kt">StateT</span> <span class="p">(</span><span class="kt">M</span><span class="o">.</span><span class="kt">Map</span> <span class="kt">VariableName</span> <span class="kt">Int</span><span class="p">)</span> <span class="p">(</span><span class="kt">Except</span> <span class="kt">String</span><span class="p">)</span> <span class="kt">String</span>
</pre></div>
<p>These describe the same capabilities,
however the compiler says <code>a</code> and <code>b</code> should <em>not</em> compose.
It’s impossible to write:</p>
<div class="highlight"><pre><span></span><span class="nf">ct</span> <span class="ow">::</span> <span class="kt">Char</span> <span class="ow">-></span> <span class="kr">_</span> <span class="kt">String</span>
<span class="nf">ct</span> <span class="ow">=</span> <span class="n">at</span> <span class="o">>=></span> <span class="n">bt</span>
</pre></div>
<p>We don’t know what goes at <code>_</code> for <code>ct</code> because <code>at</code> and <code>bt</code>
have concrete types.
We can write this composition however if
these signatures are defined in <span class="caps">MTL</span> style:</p>
<div class="highlight"><pre><span></span><span class="nf">am</span> <span class="ow">::</span> <span class="kt">MonadError</span> <span class="kt">String</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">MonadState</span> <span class="p">(</span><span class="kt">M</span><span class="o">.</span><span class="kt">Map</span> <span class="kt">VariableName</span> <span class="kt">Int</span><span class="p">)</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">Char</span> <span class="ow">-></span> <span class="n">m</span> <span class="kt">Int</span>
<span class="nf">bm</span> <span class="ow">::</span> <span class="kt">MonadState</span> <span class="p">(</span><span class="kt">M</span><span class="o">.</span><span class="kt">Map</span> <span class="kt">VariableName</span> <span class="kt">Int</span><span class="p">)</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">MonadError</span> <span class="kt">String</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">Int</span> <span class="ow">-></span> <span class="n">m</span> <span class="kt">String</span>
<span class="nf">cm</span> <span class="ow">::</span> <span class="kt">MonadState</span> <span class="p">(</span><span class="kt">M</span><span class="o">.</span><span class="kt">Map</span> <span class="kt">VariableName</span> <span class="kt">Int</span><span class="p">)</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">MonadError</span> <span class="kt">String</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">Char</span> <span class="ow">-></span> <span class="n">m</span> <span class="kt">String</span>
<span class="nf">cm</span> <span class="ow">=</span> <span class="n">am</span> <span class="o">>=></span> <span class="n">bm</span>
</pre></div>
<p>Because constraints don’t specify an order on transformers,
the <span class="caps">MTL</span> style function definition can compose. <sup id="fnref-not-solved-anything"><a class="footnote-ref" href="#fn-not-solved-anything">6</a></sup></p>
<p>This section described the core idea of <span class="caps">MTL</span>.
In the following sections we’re going to extend <span class="caps">MTL</span>
using type error driven development.
This all sound like madness if you don’t try it out with a compiler.
And I feel understanding errors is a large part of understanding <span class="caps">MTL</span>.
The type errors are difficult to decipher.
I’ve made an <a href="https://github.com/jappeace/mtl-src/blob/master/src/Lib.hs">reference project</a>
so the reader can verify the truth of my claims.</p>
<p>Even though you may doubt me dear reader,
let’s go deeper into the <a href="images/2021/abyss.jpeg">abyss</a>.</p>
<h3 id="optional-mastery-exercises">(optional) mastery exercises</h3>
<ul>
<li>What does the function <a href="https://hackage.haskell.org/package/transformers-0.6.0.2/docs/Control-Monad-Trans-Class.html#v:lift"><code>lift</code></a> do?
The answer is in the footnotes. <sup id="fnref-lift-function"><a class="footnote-ref" href="#fn-lift-function">8</a></sup></li>
<li>Is the order of a transformer stack always irrelevant? The answer is in the footnotes. <sup id="fnref-irrelevant"><a class="footnote-ref" href="#fn-irrelevant">7</a></sup></li>
<li>
<p>Say <code>ct</code> has this type signature:</p>
<div class="highlight"><pre><span></span>ct :: Char
-> ExceptT String (State (M.Map VariableName Int)) String
</pre></div>
<p>Call <code>at</code> and then <code>bt</code> from within <code>ct</code> such that it composes like the fish operator <a href="https://hackage.haskell.org/package/base-4.15.0.0/docs/Control-Monad.html#v:-62--61--62-"><code>>=></code></a>
would
with help of <a href="https://hackage.haskell.org/package/monad-control-1.0.3.1/docs/Control-Monad-Trans-Control.html#v:liftWith"><code>liftWith</code></a>.
For additional background see <a href="https://lexi-lambda.github.io/blog/2019/09/07/demystifying-monadbasecontrol/">this blogpost</a>.
The answer can be found in the <a href="https://github.com/jappeace/mtl-src/blob/master/src/Lib.hs">reference project</a> under the binding <code>answer</code>.</p>
</li>
</ul>
<h2 id="lose-and-tight-constraints"><a id="lose-tight-constraints"></a> Lose and tight constraints</h2>
<p>Say we want to use <code>moreMonad</code> from the <a href="#intro">introduction</a> in <code>assignIndexToVariables</code>.
I call <code>moreMonad</code> a tightly constrained binding.
It’s only allowed to use what <code>Monad</code> typeclass provides.
<code>assignIndexToVariables</code> on the other hand is less tightly constrained,
since it has <code>Monad</code> by implication, and also everything in <code>MonadError</code> and <code>MonadState</code>.
So let’s use it:</p>
<div class="highlight"><pre><span></span><span class="nf">assignIndexToVariables</span> <span class="ow">::</span>
<span class="kt">MonadError</span> <span class="kt">String</span> <span class="n">m</span> <span class="ow">=></span>
<span class="kt">MonadState</span> <span class="p">(</span><span class="kt">M</span><span class="o">.</span><span class="kt">Map</span> <span class="kt">VariableName</span> <span class="kt">Int</span><span class="p">)</span> <span class="n">m</span> <span class="ow">=></span>
<span class="kt">AST</span> <span class="kt">VariableName</span> <span class="ow">-></span> <span class="kt">Variables</span> <span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">AST</span> <span class="kt">Int</span><span class="p">)</span>
<span class="nf">assignIndexToVariables</span> <span class="kr">_</span> <span class="kr">_</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">_z</span> <span class="ow"><-</span> <span class="n">moreMonad</span>
<span class="o">...</span>
</pre></div>
<p>This would just work, because both <a href="https://hackage.haskell.org/package/mtl-2.2.2/docs/Control-Monad-Except.html"><code>MonadError</code></a> and
<a href="https://hackage.haskell.org/package/mtl-2.2.2/docs/Control-Monad-State-Class.html#t:MonadState"><code>MonadState</code></a> imply <code>Monad</code> in their definitions.
In mtl style, the functions with <em>tighter</em> constraints can be used in <em>more</em> situations without any refactoring.</p>
<p>Now we’re going to do the complete opposite.
The most lax constraint possible is <code>MonadIO</code>,
which gives access to arbitrary <code>IO</code> trough the <code>liftIO</code> function.
I’m not casting judgement,
I just want to show what happens.
So let’s add a <a href="https://en.wikipedia.org/wiki/MacGyver">MacGyver</a> <sup id="fnref-macgyver"><a class="footnote-ref" href="#fn-macgyver">10</a></sup>
logging function as follows:</p>
<div class="highlight"><pre><span></span><span class="nf">macGyverLog</span> <span class="ow">::</span> <span class="kt">MonadIO</span> <span class="n">m</span> <span class="ow">=></span> <span class="kt">String</span> <span class="ow">-></span> <span class="n">m</span> <span class="nb">()</span>
<span class="nf">macGyverLog</span> <span class="n">msg</span> <span class="ow">=</span> <span class="n">liftIO</span> <span class="o">$</span> <span class="n">putStrLn</span> <span class="n">msg</span>
</pre></div>
<p>The code is perfect.
Y’know, it misses some things you’d expect from your regular
logging library such as code positions, time stamping, etc.
But that’s why it’s called <code>macGyverLog</code> and not <code>kitchenSinkLog</code>.
And if we did some introspection we may find our code
tends to look a lot more like <code>macGyverLog</code> then <code>kitchenSinkLog</code>,
so let’s throw it into production:</p>
<div class="highlight"><pre><span></span><span class="nf">assignIndexToVariables</span> <span class="ow">::</span>
<span class="kt">MonadError</span> <span class="kt">String</span> <span class="n">m</span> <span class="ow">=></span>
<span class="kt">MonadState</span> <span class="p">(</span><span class="kt">M</span><span class="o">.</span><span class="kt">Map</span> <span class="kt">VariableName</span> <span class="kt">Int</span><span class="p">)</span> <span class="n">m</span> <span class="ow">=></span>
<span class="kt">AST</span> <span class="kt">VariableName</span> <span class="ow">-></span> <span class="kt">Variables</span> <span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">AST</span> <span class="kt">Int</span><span class="p">)</span>
<span class="nf">assignIndexToVariables</span> <span class="kr">_</span> <span class="kr">_</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">macGyverLog</span> <span class="s">"Starting reading more monad"</span>
<span class="n">_z</span> <span class="ow"><-</span> <span class="n">moreMonad</span>
<span class="n">macGyverLog</span> <span class="s">"End reading more monad"</span>
<span class="o">...</span>
</pre></div>
<p>This will cause the following type error:</p>
<div class="highlight"><pre><span></span>src/Lib.hs:31:5: error:
• Could not deduce (MonadIO m) arising from a use of ‘macGyverLog’
from the context: (MonadError String m,
MonadState (M.Map VariableName Int) m)
bound by the type signature for:
assignIndexToVariables :: forall (m :: * -> *).
(MonadError String m,
MonadState (M.Map VariableName Int) m) =>
AST VariableName -> Variables -> m (AST Int)
at src/Lib.hs:(26,1)-(29,46)
Possible fix:
add (MonadIO m) to the context of
the type signature for:
assignIndexToVariables :: forall (m :: * -> *).
(MonadError String m,
MonadState (M.Map VariableName Int) m) =>
AST VariableName -> Variables -> m (AST Int)
</pre></div>
<p>The possible fix read in the type error is correct,
but we have to know what a context is,
and a type signature to decipher that error message<sup id="fnref-why-dont-they"><a class="footnote-ref" href="#fn-why-dont-they">11</a></sup>.
The compiler is saying in incomprohensible error speak that you need to
add a constraint <code>MonadIO m</code> like so:</p>
<div class="highlight"><pre><span></span><span class="nf">assignIndexToVariables</span> <span class="ow">::</span>
<span class="kt">MonadIO</span> <span class="n">m</span> <span class="ow">=></span>
<span class="kt">MonadError</span> <span class="kt">String</span> <span class="n">m</span> <span class="ow">=></span>
<span class="kt">MonadState</span> <span class="p">(</span><span class="kt">M</span><span class="o">.</span><span class="kt">Map</span> <span class="kt">VariableName</span> <span class="kt">Int</span><span class="p">)</span> <span class="n">m</span> <span class="ow">=></span>
<span class="kt">AST</span> <span class="kt">VariableName</span> <span class="ow">-></span> <span class="kt">Variables</span> <span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">AST</span> <span class="kt">Int</span><span class="p">)</span>
</pre></div>
<p>Cryptic as it may be, the reason I’m writing about this isn’t that particular error message,
it’s the next one:</p>
<div class="highlight"><pre><span></span>src/Lib.hs:51:50: error:
• No instance for (MonadIO Data.Functor.Identity.Identity)
arising from a use of ‘assignIndexToVariables’
• In the second argument of ‘($)’, namely
‘assignIndexToVariables ast vars’
In the second argument of ‘($)’, namely
‘runExceptT $ assignIndexToVariables ast vars’
In the second argument of ‘($)’, namely
‘flip evalState mempty
$ runExceptT $ assignIndexToVariables ast vars’
|
51 | print $ flip evalState mempty $ runExceptT $ assignIndexToVariables ast vars
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
</pre></div>
<p>Here you may start thinking, <span class="caps">WTF</span>. Rightfully so.
The reason it starts talking about identity is because evalState
<em>runs in identity</em>.
Look at the haddocks for <a href="https://hackage.haskell.org/package/transformers-0.6.0.2/docs/Control-Monad-Trans-State-Lazy.html#v:evalState"><code>evalState</code></a>, then click on <a href="https://hackage.haskell.org/package/transformers-0.6.0.2/docs/Control-Monad-Trans-State-Lazy.html#t:State"><code>State</code></a>.
The base monad is identity.
How do we fix this?
First we replace <code>Identity</code> with a gap by invoking <code>evalStateT</code> instead of <code>evalState</code> (note the <code>T</code>).
This will result in the following beauty <a id="shes-a-beauty"></a>:</p>
<div class="highlight"><pre><span></span>src/Lib.hs:51:5: error:
• Ambiguous type variable ‘m0’ arising from a use of ‘print’
prevents the constraint ‘(Show
(m0 (Either String (AST Int))))’ from being solved.
Probable fix: use a type annotation to specify what ‘m0’ should be.
These potential instances exist:
instance (Show a, Show b) => Show (Either a b)
-- Defined in ‘Data.Either’
instance (Show k, Show a) => Show (M.Map k a)
-- Defined in ‘Data.Map.Internal’
instance Show a => Show (AST a) -- Defined at src/Lib.hs:21:13
...plus 18 others
...plus 9 instances involving out-of-scope types
(use -fprint-potential-instances to see them all)
• In a stmt of a 'do' block:
print
$ flip evalStateT mempty
$ runExceptT $ assignIndexToVariables ast vars
In the expression:
do print
$ flip evalStateT mempty
$ runExceptT $ assignIndexToVariables ast vars
print (eitherFive, maybeFive)
In the expression:
let
vars = S.fromList [...]
ast
= Node (Leaf "a") (Node (Leaf "b") (Node (Leaf "a") (Leaf "c")))
in
do print
$ flip evalStateT mempty
$ runExceptT $ assignIndexToVariables ast vars
print (eitherFive, maybeFive)
|
51 | print $ flip evalStateT mempty $ runExceptT $ assignIndexToVariables ast vars
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/Lib.hs:51:18: error:
• Ambiguous type variable ‘m0’ arising from a use of ‘evalStateT’
prevents the constraint ‘(Monad m0)’ from being solved.
Probable fix: use a type annotation to specify what ‘m0’ should be.
These potential instances exist:
instance Monad (Either e) -- Defined in ‘Data.Either’
instance Monad IO -- Defined in ‘GHC.Base’
instance [safe] Monad m => Monad (ExceptT e m)
-- Defined in ‘Control.Monad.Trans.Except’
...plus six others
...plus 16 instances involving out-of-scope types
(use -fprint-potential-instances to see them all)
• In the first argument of ‘flip’, namely ‘evalStateT’
In the expression: flip evalStateT mempty
In the second argument of ‘($)’, namely
‘flip evalStateT mempty
$ runExceptT $ assignIndexToVariables ast vars’
|
51 | print $ flip evalStateT mempty $ runExceptT $ assignIndexToVariables ast vars
| ^^^^^^^^^^
src/Lib.hs:51:51: error:
• Ambiguous type variable ‘m0’ arising from a use of ‘assignIndexToVariables’
prevents the constraint ‘(MonadIO m0)’ from being solved.
Probable fix: use a type annotation to specify what ‘m0’ should be.
These potential instances exist:
instance [safe] MonadIO IO -- Defined in ‘Control.Monad.IO.Class’
instance [safe] MonadIO m => MonadIO (ExceptT e m)
-- Defined in ‘Control.Monad.Trans.Except’
instance [safe] MonadIO m => MonadIO (StateT s m)
-- Defined in ‘Control.Monad.Trans.State.Lazy’
...plus 11 instances involving out-of-scope types
(use -fprint-potential-instances to see them all)
• In the second argument of ‘($)’, namely
‘assignIndexToVariables ast vars’
In the second argument of ‘($)’, namely
‘runExceptT $ assignIndexToVariables ast vars’
In the second argument of ‘($)’, namely
‘flip evalStateT mempty
$ runExceptT $ assignIndexToVariables ast vars’
|
51 | print $ flip evalStateT mempty $ runExceptT $ assignIndexToVariables ast vars
| ^^^^^^^^^^^^^^^^^^^
</pre></div>
<!-- Oh ghci, I love it when you talk to me this way. -->
<p>All these <code>m0</code> occur because we introduced the gap,
<span class="caps">GHC</span> has no idea what the base monad is at this point.
Which is what we want.
Let’s solve it all in one change:</p>
<div class="highlight"><pre><span></span> <span class="n">print</span> <span class="o">=<<</span> <span class="n">flip</span> <span class="n">evalStateT</span> <span class="n">mempty</span>
<span class="p">(</span><span class="n">runExceptT</span> <span class="o">$</span> <span class="n">assignIndexToVariables</span> <span class="n">ast</span> <span class="n">vars</span><span class="p">)</span>
</pre></div>
<p>Note that we replaced the application of <code>$</code> to a bind <code>=<<</code>.
The base monad is now <code>IO</code> instead of <code>Identity</code>,
which solves everything.
It’s solved because <code>IO</code>, surprise, surprise, has an instance of <code>MonadIO</code>!
Who would’ve thought <code>liftIO</code> could be <code>id</code>.
The readers’ keen eye spots a pattern:
We merely select instances with types that have code attached to them.
It’s code generation based on type selection.</p>
<h2 id="reinterpreting-io-or-the-mtl-style">Reinterpreting <span class="caps">IO</span> or the <span class="caps">MTL</span> style</h2>
<p>In most situations <code>MonadIO</code> is fine.
However it doesn’t allow us to reinterpret effects that are
using <code>IO</code>.
An example of reinterpretation is
a test where you would want to measure how often the
<code>mcGyverLog</code> function is being called.
This section will show you how to do that.</p>
<p>First we start by <span class="caps">MTL</span>-izing our <code>IO</code> based code.
How can we rewrite <code>mcGyverLog</code> so that it doesn’t use <code>IO</code> explicetly?
The function that needs <code>IO</code> is <code>putStrLn</code>.
It’s type signature is <code>String -> IO ()</code>.
We want that <code>IO</code> to be an <code>m</code>,
so we introduce a new typeclass for our <a href="https://hackage.haskell.org/package/monad-logger-0.3.36/docs/Control-Monad-Logger.html#t:MonadLogger">not invented here</a> log:</p>
<div class="highlight"><pre><span></span><span class="kr">class</span> <span class="p">(</span><span class="kt">Monad</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kt">NotInventedHereLog</span> <span class="n">m</span> <span class="kr">where</span>
<span class="n">nihLog</span> <span class="ow">::</span> <span class="kt">String</span> <span class="ow">-></span> <span class="n">m</span> <span class="nb">()</span>
</pre></div>
<p>We already know what implementation <code>m</code> has if it’s an
<code>IO</code> instance:</p>
<div class="highlight"><pre><span></span><span class="kr">instance</span> <span class="kt">NotInventedHereLog</span> <span class="kt">IO</span> <span class="kr">where</span>
<span class="n">nihLog</span> <span class="ow">::</span> <span class="kt">String</span> <span class="ow">-></span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="n">nihLog</span> <span class="ow">=</span> <span class="n">putStrLn</span>
</pre></div>
<p>To make our previous example work we need to replace <code>MonadIO</code> in our
function definition with <code>NotInventedHereLog</code>.
Because we renamed the <code>macGyverLog</code> function to <code>nihLog</code>,
we need to replace those calls with <code>nihLog</code> as well:</p>
<div class="highlight"><pre><span></span><span class="nf">assignIndexToVariables2</span> <span class="ow">::</span>
<span class="kt">NotInventedHereLog</span> <span class="n">m</span> <span class="ow">=></span>
<span class="kt">MonadError</span> <span class="kt">String</span> <span class="n">m</span> <span class="ow">=></span>
<span class="kt">MonadState</span> <span class="p">(</span><span class="kt">M</span><span class="o">.</span><span class="kt">Map</span> <span class="kt">VariableName</span> <span class="kt">Int</span><span class="p">)</span> <span class="n">m</span> <span class="ow">=></span>
<span class="kt">AST</span> <span class="kt">VariableName</span> <span class="ow">-></span> <span class="kt">Variables</span> <span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">AST</span> <span class="kt">Int</span><span class="p">)</span>
<span class="nf">assignIndexToVariables2</span> <span class="n">ast</span> <span class="n">variables</span> <span class="ow">=</span> <span class="n">forM</span> <span class="n">ast</span> <span class="o">$</span> <span class="nf">\</span><span class="n">var</span> <span class="ow">-></span> <span class="kr">do</span>
<span class="n">nihLog</span> <span class="s">"start more monad"</span>
<span class="n">_z</span> <span class="ow"><-</span> <span class="n">moreMonad</span>
<span class="o">...</span>
</pre></div>
<p>Running this will give us the following type error:</p>
<div class="highlight"><pre><span></span>src/Lib.hs:54:52: error:
• No instance for (NotInventedHereLog
(StateT (M.Map VariableName Int) (ExceptT [Char] IO)))
arising from a use of ‘assignIndexToVariables2’
• In the second argument of ‘($)’, namely
‘assignIndexToVariables2 ast vars’
In the first argument of ‘runExceptT’, namely
‘(flip evalStateT mempty $ assignIndexToVariables2 ast vars)’
In the second argument of ‘(=<<)’, namely
‘runExceptT
(flip evalStateT mempty $ assignIndexToVariables2 ast vars)’
|
54 | print =<< runExceptT (flip evalStateT mempty $ assignIndexToVariables2 ast vars)
</pre></div>
<p>This looks scary, maybe this is the one, the one type error I can’t
solve?
After all, I’ve been waiting for that one perfect type error
for over four years now.
But no,
this is known as the <a href="http://felixmulder.com/writing/2020/08/08/Revisiting-application-structure#the-n2-issue">$n^2$-instances problem</a><sup id="fnref-not-aprove"><a class="footnote-ref" href="#fn-not-aprove">12</a></sup>,
which sounds very impressive.
However for now we completely ignore that problem
by providing an instance which solves this type error:</p>
<div class="highlight"><pre><span></span><span class="kr">instance</span> <span class="p">(</span><span class="kt">NotInventedHereLog</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kt">NotInventedHereLog</span> <span class="p">(</span><span class="kt">StateT</span> <span class="n">s</span> <span class="n">m</span><span class="p">)</span> <span class="kr">where</span>
<span class="n">nihLog</span> <span class="ow">=</span> <span class="n">lift</span> <span class="o">.</span> <span class="n">nihLog</span>
</pre></div>
<p>This code says:
If you’re a <code>StateT</code> and your base monad already has a <code>NotInventedHereLog</code>
constraint,
you also have <code>NotInvnetedHereLog</code> instance with help of <a href="https://hackage.haskell.org/package/transformers-0.6.0.2/docs/Control-Monad-Trans-Class.html#v:lift"><code>lift</code></a>.
By providing this instance we’re generating lift calls over <code>StateT</code>
for all occurrences of <code>niLog</code>.</p>
<p>Moving on we get the same error for <code>ExceptT</code>:</p>
<div class="highlight"><pre><span></span>src/Lib.hs:54:52: error:
• No instance for (NotInventedHereLog (ExceptT [Char] IO))
arising from a use of ‘assignIndexToVariables2’
• In the second argument of ‘($)’, namely
‘assignIndexToVariables2 ast vars’
In the first argument of ‘runExceptT’, namely
‘(flip evalStateT mempty $ assignIndexToVariables2 ast vars)’
In the second argument of ‘(=<<)’, namely
‘runExceptT
(flip evalStateT mempty $ assignIndexToVariables2 ast vars)’
|
54 | print =<< runExceptT (flip evalStateT mempty $ assignIndexToVariables2 ast vars)
</pre></div>
<p>We will keep getting these errors for every unique transformer we use
because of the
<a href="http://felixmulder.com/writing/2020/08/08/Revisiting-application-structure#the-n2-issue">$n^2$-instances problem</a><sup id="fnref2-not-aprove"><a class="footnote-ref" href="#fn-not-aprove">12</a></sup>.
This blogpost isn’t about solving the $n^2$
problem, so I will ignore it.
The solution to the type error however is pretty much the same:</p>
<div class="highlight"><pre><span></span><span class="kr">instance</span> <span class="p">(</span><span class="kt">NotInventedHereLog</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kt">NotInventedHereLog</span> <span class="p">(</span><span class="kt">ExceptT</span> <span class="n">e</span> <span class="n">m</span><span class="p">)</span> <span class="kr">where</span>
<span class="n">nihLog</span> <span class="ow">=</span> <span class="n">lift</span> <span class="o">.</span> <span class="n">nihLog</span>
</pre></div>
<p>The example code will compile with these two additional instances.
However we still don’t know how to reinterpret this purely,
and we have introduced code duplication.
This duplication can be removed with the default mechanism described in
Alexis King her <a href="https://lexi-lambda.github.io/blog/2017/04/28/lifts-for-free-making-mtl-typeclasses-derivable/">blogpost</a>.</p>
<p>For the pure interpretation
we need to introduce a newtype, which is used to attach an instance for <code>NotInventedHereLog</code>.
This type’s instance will collect the logged messages.
Turns out that the <code>WriterT</code> monad transformer does exactly what we want.
So the pure code will look like this:</p>
<div class="highlight"><pre><span></span><span class="kr">newtype</span> <span class="kt">NihLogT</span> <span class="n">m</span> <span class="n">a</span> <span class="ow">=</span> <span class="kt">MkNihLogT</span> <span class="p">{</span>
<span class="n">runNihLog</span> <span class="ow">::</span> <span class="kt">WriterT</span> <span class="p">[</span><span class="kt">String</span><span class="p">]</span> <span class="n">m</span> <span class="n">a</span>
<span class="p">}</span> <span class="kr">deriving</span> <span class="p">(</span><span class="kt">Functor</span><span class="p">,</span> <span class="kt">Applicative</span><span class="p">,</span> <span class="kt">Monad</span><span class="p">,</span> <span class="kt">MonadTrans</span><span class="p">,</span> <span class="kt">MonadWriter</span> <span class="p">[</span><span class="kt">String</span><span class="p">])</span>
<span class="kr">instance</span> <span class="kt">Monad</span> <span class="n">m</span> <span class="ow">=></span> <span class="kt">NotInventedHereLog</span> <span class="p">(</span><span class="kt">NihLogT</span> <span class="n">m</span><span class="p">)</span> <span class="kr">where</span>
<span class="n">nihLog</span> <span class="n">msg</span> <span class="ow">=</span> <span class="n">tell</span> <span class="p">[</span><span class="n">msg</span><span class="p">]</span>
</pre></div>
<p>The instance simply <a href="https://hackage.haskell.org/package/mtl-2.2.2/docs/src/Control.Monad.Writer.Class.html#tell"><code>tell</code></a>‘s the message, which <code>mappends</code> it to the list.
To run this code we use <code>runNihLog</code>:</p>
<div class="highlight"><pre><span></span> <span class="kr">let</span> <span class="n">pureCode</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">Either</span> <span class="kt">String</span> <span class="p">(</span><span class="kt">AST</span> <span class="kt">Int</span><span class="p">),</span> <span class="p">[</span><span class="kt">String</span><span class="p">])</span>
<span class="n">pureCode</span> <span class="ow">=</span> <span class="n">runWriter</span> <span class="o">$</span> <span class="n">runNihLog</span> <span class="o">$</span> <span class="n">runExceptT</span> <span class="p">(</span><span class="n">flip</span> <span class="n">evalStateT</span> <span class="n">mempty</span> <span class="o">$</span> <span class="n">assignIndexToVariables2</span> <span class="n">ast</span> <span class="n">vars</span><span class="p">)</span>
<span class="p">(</span><span class="n">eitherAst</span><span class="p">,</span> <span class="n">allLoggedMessages</span><span class="p">)</span> <span class="ow">=</span> <span class="n">pureCode</span>
<span class="n">print</span> <span class="n">pureCode</span>
</pre></div>
<p>Now <code>allLoggedMessages</code> will contain all messages emitted by <code>nihLog</code>.
With this you can write property tests on <code>assignIndexToVariables2</code>.
For example you could assert that an <span class="caps">AST</span> of size 20 should emit at least 40 log messages.
Obviously this isn’t limited to tests or purity,
you could also add a newtype that has a connection pool to send
the messages to some database for example.</p>
<p>I’ll tap out here.
This was supposed to be a short note on someone else’s blogpost,
which spiraled into dumping all my knowledge here on <span class="caps">MTL</span> and extending it a bit as well.
For example I didn’t even know about the defaults mechanism <a href="https://lexi-lambda.github.io/blog/2017/04/28/lifts-for-free-making-mtl-typeclasses-derivable/">Alexis wrote about</a>.
I’ll tap out here, thanks for reading.</p>
<p>Let me know if you have any strong opinions on this style.
Love it or hate it, I’d like to know!
Or if you need any help using it.
I’m interested in effect systems in general.
Also let me know about your favorite effect system that
I didn’t acknowledge.</p>
<h2 id="links">Links</h2>
<ul>
<li>Inspirational <a href="https://blog.cofree.coffee/2021-08-05-a-brief-intro-to-monad-transformers/">blogpost</a></li>
<li>A functional example is available <a href="https://github.com/jappeace/mtl-src/blob/master/src/Lib.hs">here</a></li>
<li><a href="https://www.youtube.com/watch?v=MPlrAe-XYMU&t=300s">A video presentation on the exact same topic.</a></li>
<li>Full mtl style reinterpretation test <a href="https://github.com/lexi-lambda/mtl-style-example">example</a></li>
<li>And a <a href="https://lexi-lambda.github.io/blog/2017/06/29/unit-testing-effectful-haskell-with-monad-mock/">library</a> for mocking around mtl style, also gives more background.</li>
<li>If I didn’t manage to exhaust you, here is more <a href="https://ocharles.org.uk/posts/2016-01-26-transformers-free-monads-mtl-laws.html">background</a> and alternatives.</li>
</ul>
<div class="footnote">
<hr>
<ol>
<li id="fn-best-mtl">
<p>The best I could find is this <a href="http://felixmulder.com/writing/2020/08/08/Revisiting-application-structure#the-n2-issue">blogpost</a>,
which is more an experience report rather
then an explanation of what’s going on.
<a href="https://ocharles.org.uk/posts/2016-01-26-transformers-free-monads-mtl-laws.html">Ocharles</a>
could also be said to be talking about the subject,
but it’s not an introduction, it’s an expert reference. <a class="footnote-backref" href="#fnref-best-mtl" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-nonsense">
<p>Note that although this is valid haskell,
in practice it’s nonsense to write anything
with just a <code>Monad</code> constraint.
I’m saying this because the constraint is <a href="#lose-tight-constraints">too tight</a>!
You can’t do anything in just a monad.
I’m only showing it here to raise a point.
In this situation it would’ve made more
sense to drop the entire monad sharade and assign 5 directly with type of <code>Int</code>. <a class="footnote-backref" href="#fnref-nonsense" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-mtl-vs-transformers">
<p>Aren’t transformers the same as mtl? No!
Back in the stone ages, people weren’t sure what the right
approach would be to doing this, and frankly people still aren’t sure.
The base package is the <a href="https://hackage.haskell.org/package/transformers">transformers package</a>.
Although the <a href="https://hackage.haskell.org/package/mtl">mtl package</a>
is most popular, there is also <a href="https://hackage.haskell.org/package/mtl-tf">mtl-tf</a>.
<a href="https://hackage.haskell.org/package/polysemy">Polysemy</a> also builds on top of transformers.
I think the subject of best approach is a subject for debate.
I’d like to point out as well that in technology, popularity means
nothing. Cobol used to be more popular then C up till the 1990’s,
and is practically dead now.
JavaScript is also more popular then Haskell, is it therefore better? <a class="footnote-backref" href="#fnref-mtl-vs-transformers" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn-no-pattern-match">
<p>No it’s not possible if a constructor isn’t exposed. Reflex uses this with <a href="https://hackage.haskell.org/package/reflex-0.8.1.0/docs/Reflex-Dynamic.html#t:Dynamic"><code>Dynamic t</code></a>
to create a smart destructor.
In reflex you can only use a value in a dynamic by putting an entire monadic action inside the dynamic, and then using <a href="https://hackage.haskell.org/package/reflex-dom-core-0.6.2.0/docs/Reflex-Dom-Widget-Basic.html#v:dyn"><code>dyn</code></a> to get it out (for reflex-dom at least, sdl uses something different).
I’m highlighting this because I feel this is a powerful idea. <a class="footnote-backref" href="#fnref-no-pattern-match" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn-why-different-name">
<p>Why MonadError and ExceptT don’t share their names?
It’s explained <a href="https://www.reddit.com/r/haskell/comments/3ded39/why_cant_we_have_an_eithert_in_transformers/ct4mnk1/">here</a>. <a class="footnote-backref" href="#fnref-why-different-name" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn-not-solved-anything">
<p>We’ve not solved the <a href="https://blog.cofree.coffee/2021-08-05-a-brief-intro-to-monad-transformers/#cb5">problem</a>
here of being able to write a
<code>Compose</code> <code>Monad</code> instance at all.
We just kind of worked around it by not talking about order.
It’s truly an awful hack, which are the best. <a class="footnote-backref" href="#fnref-not-solved-anything" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn-irrelevant">
<p>No! It just happens to be irrelevant for most transformers in the transformers package.
For example there exist the transformer <a href="https://hackage.haskell.org/package/resourcet">ResourceT</a>.
I encountered a production bug due to wrong order of running the transformer stack.
Only large queries crashed with an opaque message <code>StatementAlreadyFinalized</code>.
The <a href="https://hackage.haskell.org/package/persistent-2.13.1.1/docs/Database-Persist-Class.html#v:selectSourceRes"><code>selectSource</code></a>
exposes the resource constraint.
We also had a typeclass to embed that <code>ReaderT backend</code> into the constraints,
since it’s just a regular reader after all (it’s not quite).
We also had an instance that said order of this <code>ResourceT</code> and <code>ReaderT backend</code> didn’t matter,
but it did.
This instance caused the running of the query to leak outside of the <code>runResourceT</code> block.
Removing the instance and being explicit about order solved the bug. <a class="footnote-backref" href="#fnref-irrelevant" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn-lift-function">
<p>It allows concrete monad transformer selection,
For example :
<code>haskell
x :: LogginT (ReaderT AppContext Maybe) String
x = do
callInLoggingT
lift $ callInReaderT
lift $ lift $ callInMaybe</code>
AppM from the original blogpost didn’t have to
do this because the deriving mechanism generated the instances that in turn generate the lift calls. <a class="footnote-backref" href="#fnref-lift-function" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn-value-function">
<p>Aren’t values just functions with 0 arguments?
Especially in haskell were it may not have been evaluated yet. <a class="footnote-backref" href="#fnref-value-function" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
<li id="fn-macgyver">
<p>MacGyver is a <span class="caps">TV</span> character who was known to get out of
precarious situations with absurdly little in terms of
resources and a lot of improvisation, our logging function feels similar at this point.
Another term that could’ve been used was Not Invented Here,
but I liked MacGyver better. <a class="footnote-backref" href="#fnref-macgyver" title="Jump back to footnote 10 in the text">↩</a></p>
</li>
<li id="fn-why-dont-they">
<p>The thing that baffles me is that <span class="caps">GHC</span> already manages to print your own
code, they figure out the right constraint,
why don’t they just give the example like I’m doing
here with the message “try doing this:”.
Instead they talk about constraints and contexts.
Do they hate people using their language or something?
I feel George Orwell’s
<a href="https://lhsblogs.typepad.com/files/george-orwell-on-writing.pdf">guide to writing</a>
well also applies to error messages. <a class="footnote-backref" href="#fnref-why-dont-they" title="Jump back to footnote 11 in the text">↩</a></p>
</li>
<li id="fn-not-aprove">
<p>I’m linking to this blog as an explenation of the n^2 instances problem.
I definitely don’t endorse using Overlappable as a solution for it.
I’d prefer using <a href="https://lexi-lambda.github.io/blog/2017/04/28/lifts-for-free-making-mtl-typeclasses-derivable/">Alexis</a>
method which is very copy paste-able. <a class="footnote-backref" href="#fnref-not-aprove" title="Jump back to footnote 12 in the text">↩</a><a class="footnote-backref" href="#fnref2-not-aprove" title="Jump back to footnote 12 in the text">↩</a></p>
</li>
</ol>
</div>The Nix mutli-monolith machine (NMMM).2021-07-31T15:05:00+02:002021-08-01T15:09:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2021-07-31:/the-nix-mutli-monolith-machine-nmmm.html<p><img alt="a NMMM computer" src="images/2021/computer-nixos-monolith.png"></p>
<p>I redid how my services are structured.
Instead of running each project on a separate <span class="caps">VM</span>,
they’re now all running on a dedicated Hetzner machine.
This is what I call the nix multi monolith machine
(hence forth called <span class="caps">NMMM</span>,
pronounced like tasting something delicious prefixed with an N).
There …</p><p><img alt="a NMMM computer" src="images/2021/computer-nixos-monolith.png"></p>
<p>I redid how my services are structured.
Instead of running each project on a separate <span class="caps">VM</span>,
they’re now all running on a dedicated Hetzner machine.
This is what I call the nix multi monolith machine
(hence forth called <span class="caps">NMMM</span>,
pronounced like tasting something delicious prefixed with an N).
There are some really big advantages to the <span class="caps">NMMM</span> approach.</p>
<h2 id="why-use-a-nmmm">Why use a <span class="caps">NMMM</span>?</h2>
<p>Setting up <span class="caps">NMMM</span> has a couple of advantages:</p>
<ul>
<li>the <span class="caps">NMMM</span> is cheap <sup id="fnref-cheap"><a class="footnote-ref" href="#fn-cheap">1</a></sup></li>
<li>the <span class="caps">NMMM</span> is simple <sup id="fnref-simple"><a class="footnote-ref" href="#fn-simple">2</a></sup></li>
<li>the <span class="caps">NMMM</span> doesn’t need nixops while benefiting from nix <sup id="fnref-no-nixops"><a class="footnote-ref" href="#fn-no-nixops">3</a></sup></li>
<li>the <span class="caps">NMMM</span> spins out new services fast <sup id="fnref-fast"><a class="footnote-ref" href="#fn-fast">4</a></sup></li>
</ul>
<p>Obviously there are also some disadvantages
to this approach,
which get answered in the <a href="#discussion">discussion section</a>.
But for a typical startup situation,
where money is tight and there is no
product market fit yet,
and time to market is important,
I think the <span class="caps">NMMM</span> is the best.</p>
<h2 id="what-is-nmmm">What is <span class="caps">NMMM</span></h2>
<p>The core of the Nix mutli-monolith machine (<span class="caps">NMMM</span>) is
a dedicated machine.
A dedicated machine means no virtualization,
in other word it’s bear metal.</p>
<p>The second major part of this configuration is nix.
This means that every piece of software is described in nix files.
Just like the configuration of this software.
More on that in the <a href="#nix-config">nix config</a> section.</p>
<p>The third major part is multi-monoliths.
Meaning that you can have multiple,
unrelated services running on the same machine.
This is something different from micro services.
<a href="https://microservices.io/">Microservices</a> communicate with each other at some point
to provide a consistent frontend.
The <a href="https://microservices.io/">link </a> calls this ‘loosly coupled’,
or as I liked to call it: They chat with each other.
Monoliths on the other hands should <em>not</em> communicate
with each other and are isolated.
They have independent frontends and backends.
In other words, no chatting between monoliths.
They just stand there <a href="https://www.youtube.com/watch?v=cHWs3c3YNs4">silent and ominous</a>.
I feel this approach falls inline with the <a href="https://martinfowler.com/bliki/MonolithFirst.html">monlith first</a>
approach,
but rather then a single monolith I can deploy many
completely unrelated projects.</p>
<h2 id="nix-config"><a id="nix-config"></a> Nix config</h2>
<p>I build this by relying on the <a href="https://nixos.wiki/wiki/Module">module system</a>.
The main entrypoint
is an ordinary nixos <a href="https://nixos.org/manual/nixos/stable/index.html#sec-configuration-file">configuration file</a>,
other modules in general are also shaped like this even
though the entry point depends on them.
So we got a very versatile one trick pony!</p>
<p>Assuming <code>/</code> is the root of the project,
the root configuration looks like this:</p>
<div class="highlight"><pre><span></span><span class="c1"># /nix/hetzner/configuration.nix</span>
<span class="p">{</span> config<span class="p">,</span> pkgs<span class="p">,</span> <span class="o">...</span> <span class="p">}:</span>
<span class="p">{</span>
<span class="ss">imports =</span>
<span class="p">[</span>
<span class="o">.</span><span class="l">/hardware-configuration.nix</span>
<span class="o">..</span><span class="l">/../videocut.org/nix/videocut.nix</span>
<span class="o">..</span><span class="l">/../massapp.org/nix/massapp.nix</span>
<span class="o">..</span><span class="l">/../raster.click/nixops/raster.nix</span>
<span class="p">];</span>
<span class="o">...</span>
</pre></div>
<p>That has the same structure as the <code>configuration.nix</code> I use on
<a href="https://github.com/jappeace/linux-config/blob/lenovo-amd/configuration.nix#L35">my laptop</a>.
The difference this example has with my laptop is that
I’m pulling in a bunch of additional modules aside from the hardware config
trough the <code>imports</code> mechanism.
<code>imports</code> tells nixos to also include those other configuration files.
However before looking into those specific files, I need to explain how to run this entrypoint.
I call this configuration from my <code>/makefile</code>:</p>
<div class="highlight"><pre><span></span><span class="nf">deploy</span><span class="o">:</span>
nix-shell nix/nixpkgs-shell.nix --run <span class="s2">"make deploy_"</span>
<span class="nv">ROOT_DIR</span><span class="o">:=</span><span class="k">$(</span>shell dirname <span class="k">$(</span>realpath <span class="k">$(</span>firstword <span class="k">$(</span>MAKEFILE_LIST<span class="k">))))</span>
<span class="nf">deploy_</span><span class="o">:</span>
<span class="nv">NIXOS_CONFIG</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span>ROOT_DIR<span class="k">)</span><span class="s2">/nix/hetzner/configuration.nix"</span> nixos-rebuild switch --target-host root@videocut.org --show-trace
</pre></div>
<p><code>make deploy</code> will deploy if called form the root of the project <code>/</code>.
It runs <code>make deploy_</code> from a shell that sets the <code>NIX_PATH</code>
to a <a href="https://jappieklooster.nl/pinning-nixops-builds.html">pinned nixpkgs</a>.
This uses <code>nixos-rebuild switch</code>, just like on my laptop.
However I specify the target host to be one of the
domains hosted on the Hetzner machine.
All domains domains lead to the Hetzner machine.</p>
<p>But how does the machine decide which <span class="caps">HTTP</span> request goes to what service?
After all they all arrive on the same machine now,
so something has to make a decision on this.
<a href="https://www.nginx.com/">Nginx</a> can do that!
this is a bit later in the entry point file:</p>
<div class="highlight"><pre><span></span> <span class="c1"># /nix/hetzner/configuration.nix</span>
<span class="o">...</span>
services<span class="o">.</span><span class="ss">nginx =</span> <span class="p">{</span>
<span class="ss">virtualHosts =</span> pkgs<span class="o">.</span>lib<span class="o">.</span>foldr <span class="p">(</span>x<span class="p">:</span> prev<span class="p">:</span> pkgs<span class="o">.</span>lib<span class="o">.</span>recursiveUpdate <span class="p">(</span><span class="nb">import</span> x<span class="p">)</span> prev<span class="p">)</span> <span class="p">{}</span>
<span class="p">[</span><span class="o">..</span><span class="l">/../videocut.org/nix/vhost.nix</span>
<span class="o">..</span><span class="l">/../massapp.org/nix/vhost.nix</span>
<span class="o">..</span><span class="l">/../raster.click/nixops/vhost.nix</span>
<span class="o">..</span><span class="l">/../blog/vhost.nix</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="o">...</span>
</pre></div>
<p>We let the individual services decide how to configure the virtual
hosts.
Virtual hosts allow us to specify configurations per <a href="http://nginx.org/en/docs/http/request_processing.html">domain name</a>.
For example the <a href="https://massapp.org/">massapp.org</a>
host looks like this:</p>
<div class="highlight"><pre><span></span> <span class="c1"># /massapp.org/nix/vhost.nix</span>
<span class="n">lib</span> <span class="o">=</span> <span class="kn">import</span> <span class="o">./</span><span class="n">lib</span><span class="o">.</span><span class="n">nix</span><span class="p">;</span>
<span class="n">sslBools</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">forceSSL</span> <span class="o">=</span> <span class="n">true</span><span class="p">;</span>
<span class="n">enableACME</span> <span class="o">=</span> <span class="n">true</span><span class="p">;</span> <span class="c1"># E</span>
<span class="p">};</span>
<span class="n">base</span> <span class="o">=</span> <span class="n">locations</span><span class="p">:</span>
<span class="p">{</span>
<span class="n">inherit</span> <span class="n">locations</span><span class="p">;</span>
<span class="p">}</span> <span class="o">//</span> <span class="n">sslBools</span><span class="p">;</span> <span class="c1"># C</span>
<span class="n">proxy</span> <span class="o">=</span> <span class="n">base</span> <span class="p">{</span>
<span class="s2">"/"</span><span class="o">.</span><span class="n">proxyPass</span> <span class="o">=</span> <span class="s2">"http://127.0.0.1:${</span>
<span class="n">toString</span> <span class="n">lib</span><span class="o">.</span><span class="n">massapp_port</span> <span class="c1"># D</span>
<span class="p">}</span><span class="o">/</span><span class="s2">"; </span>
<span class="p">};</span>
<span class="n">redirect</span> <span class="o">=</span> <span class="p">{</span> <span class="n">globalRedirect</span> <span class="o">=</span> <span class="s2">"${lib.massappDomain}"</span><span class="p">;</span> <span class="p">}</span> <span class="o">//</span> <span class="n">sslBools</span><span class="p">;</span>
<span class="ow">in</span> <span class="p">{</span>
<span class="s2">"www.${lib.massappDomain}"</span> <span class="o">=</span> <span class="n">redirect</span><span class="p">;</span> <span class="c1"># B</span>
<span class="s2">"${lib.massappDomain}"</span> <span class="o">=</span> <span class="n">proxy</span><span class="p">;</span> <span class="c1"># A</span>
</pre></div>
<p>The main thing we’re saying at <code>A</code> is that the <a href="https://massapp.org/">massapp.org</a>
domain should point to the port defined at <code>D</code>.
Furthermore in <code>B</code> we’re redirecting all <code>www</code> traffic to <code>A</code>.
Which strips off <code>www</code> from <code>www.masssapp.org</code> resulting into <code>massapp.org</code>.
For some reason people still yearn to type <code>www</code>.
With this redirect we trash the peoples’ pointless dreams and desires.
Finally in <code>C</code> we strip of the <span class="caps">HTTPS</span>,
which the proxy will remake into an <span class="caps">HTTP</span> connection.
Traffic at this point is internal, so <a href="http://www.steves-internet-guide.com/ssl-certificates-explained/"><span class="caps">SSL</span></a>
has served it’s purpose
and we can safely strip it.
In practice this means our application doesn’t have to deal with certificates or <span class="caps">SSL</span>.
This leverages nixos <a href="https://nixos.wiki/wiki/Nginx">built-in</a> <a href="https://letsencrypt.org/">let’s encrypt</a>
support without even thinking about it in <code>E</code> <sup id="fnref-breaks-often"><a class="footnote-ref" href="#fn-breaks-often">5</a></sup>.</p>
<p>Since we just bound all incoming domains to a unique port,
we have to bind a program to that port as well.
This program is our main application code, or the monolith.
For example let’s look at the massapp module.
Herein I register
a systemd service which runs the main massapp executable.
This is another file like <code>configuration.nix</code>
(just like my laptop! The one trick pony):</p>
<div class="highlight"><pre><span></span><span class="c1"># /massapp.org/nix/massapp.nix</span>
<span class="p">{</span> config<span class="p">,</span> pkgs<span class="p">,</span> <span class="o">...</span> <span class="p">}:</span>
<span class="k">let</span>
<span class="ss">massapp =</span> pkgs<span class="o">.</span>callPackage <span class="o">..</span><span class="l">/webservice/default.nix</span> <span class="p">{</span> <span class="p">};</span> <span class="c1"># A</span>
<span class="k">in</span> <span class="p">{</span>
<span class="o">...</span>
systemd<span class="o">.</span>services<span class="o">.</span><span class="ss">massapp =</span> <span class="c1"># D</span>
<span class="p">{</span>
<span class="ss">description =</span> <span class="s2">"Massapp webservice"</span><span class="p">;</span>
<span class="ss">serviceConfig =</span> <span class="p">{</span>
<span class="ss">Type =</span> <span class="s2">"simple"</span><span class="p">;</span>
<span class="ss">ExecStart =</span> <span class="s2">"</span><span class="si">${</span> massapp <span class="si">}</span><span class="s2">/bin/massapp"</span><span class="p">;</span> <span class="c1"># B</span>
<span class="p">};</span>
<span class="ss">wantedBy =</span> <span class="p">[</span> <span class="s2">"multi-user.target"</span> <span class="p">];</span>
<span class="ss">requires =</span> <span class="p">[</span> <span class="s2">"postgresql.service"</span> <span class="p">];</span>
<span class="ss">environment =</span> <span class="p">{</span>
<span class="ss">PORT =</span> <span class="s2">"</span><span class="si">${</span><span class="nb">toString</span> lib<span class="o">.</span>massapp_port<span class="si">}</span><span class="s2">"</span><span class="p">;</span> <span class="c1"># C</span>
<span class="o">...</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="p">}</span>
</pre></div>
<p>Within <code>A</code> I load the main binary of the service.
We tell systemd about that program at <code>B</code>.
Finally we make the program aware of the correct port in <code>C</code>
by setting the environment variable <code>PORT</code>.
<a href="https://www.yesodweb.com/">Yesod</a> has a configuration built in for that by default,
and massapp is a <a href="https://www.yesodweb.com/">Yesod</a> application.
This would work for any other application, you
can even pass <span class="caps">CLI</span> arguments like this just by modifying the <code>ExecStart</code>.
At <code>D</code> we give this systemd unit the name massapp,
by doing this std out is logged in journalctl and tagged with the unit name.
Nixos-rebuild will now also know if a service failed trough exit codes.
For example if the service can’t find the database.
Which we discuss how to setup in the <a href="#database">database</a> section.</p>
<h2 id="database-integration"><a id="database"></a> Database integration</h2>
<p>This is the database configuration, again in the entrypoint
<code>configuration.nix</code> file:</p>
<div class="highlight"><pre><span></span> <span class="c1"># /nix/hetzner/configuration.nix</span>
<span class="o">...</span>
services<span class="o">.</span><span class="ss">postgresql =</span> <span class="p">{</span>
<span class="ss">enable =</span> <span class="no">true</span><span class="p">;</span>
<span class="c1"># A</span>
<span class="ss">authentication =</span> pkgs<span class="o">.</span>lib<span class="o">.</span>mkOverride <span class="mi">10</span> <span class="s1">''</span>
<span class="s1"> local all all trust</span>
<span class="s1"> host all all ::1/128 trust</span>
<span class="s1"> host all all 0.0.0.0/0 md5</span>
<span class="s1"> host all all ::/0 md5</span>
<span class="s1"> ''</span><span class="p">;</span>
<span class="ss">settings =</span> <span class="p">{</span> <span class="c1"># B</span>
<span class="ss">log_connections =</span> <span class="s2">"yes"</span><span class="p">;</span>
<span class="ss">log_statement =</span> <span class="s2">"all"</span><span class="p">;</span>
<span class="ss">log_disconnections =</span> <span class="s2">"yes"</span><span class="p">;</span>
<span class="p">};</span>
<span class="c1"># C</span>
<span class="ss">initialScript =</span> pkgs<span class="o">.</span>writeText <span class="s2">"backend-initScript"</span> <span class="s1">''</span>
<span class="s1"> CREATE USER massapp_prod WITH PASSWORD </span><span class="err">'</span><span class="s1">someinitialpassword</span><span class="err">'</span><span class="s1">;</span>
<span class="s1"> CREATE DATABASE massapp_prod;</span>
<span class="s1"> ALTER USER massapp_prod WITH SUPERUSER;</span>
<span class="s1"> ''</span><span class="p">;</span>
<span class="o">...</span>
<span class="p">}</span>
</pre></div>
<p>The initial script<code>C</code> is only run when the database
is started for the first time,
after that you need to manage users by hand.
However I still add these users to script to keep track
of them if I ever need to abandon this system.
Like this at least the users will exist on the new machine.
I give every monoliths’ user superuser access in <code>C</code>.
This isn’t the best for security, but it’s really convenient.
Besides if a hacker manages to get a shell for the system
or a shell for the database it’s already to late for me anyway.
I’d just scrap this system and start over elsewhere.</p>
<p>In <code>B</code> I set some additional logging options,
which are just convenient when things break<sup id="fnref-it-breaks-always"><a class="footnote-ref" href="#fn-it-breaks-always">6</a></sup>.
With statement logging I can still see what happened
in the database if the application didn’t emit enough information.
logging all statements <a href="https://dba.stackexchange.com/questions/30144/performance-impact-of-setting-postgresql-to-log-all-statements">slows down</a> the database.
However, my services are hardly taxed at the moment,
they are taxed but no where near justifying not logging.</p>
<p>Localhost connections can authenticate without password in <code>A</code>.
Outside connections are ignored, they’re not even asked for a password.</p>
<h2 id="discussion"><a id="discussion"></a> Discussion</h2>
<p>This setup goes against <a href="https://www.reddit.com/r/sysadmin/comments/92qhuu/should_i_put_multiple_services_on_a_single_vm_or/">advice</a>
from sysadmins, where they recommend
you split up everything across <span class="caps">VM</span>’s as much as possible.
I reject <a href="https://www.youtube.com/watch?v=nyErR1FS1_0">their hypothesis</a>.
I mean they argue for splitting services,
but the tool used for splitting shouldn’t be dogmatic
and always result in choosing a <span class="caps">VM</span>.
Using multiprocessing,
and tenanting for specific software packages is good enough.
<a href="https://www.reddit.com/r/sysadmin/comments/92qhuu/should_i_put_multiple_services_on_a_single_vm_or/e37r7pm/">myron semacks</a>
at least gives some arguments on why he chooses to use a <span class="caps">VM</span>.
However his reasons for choosing a <span class="caps">VM</span>’s
revolve around windows related oddities.
For example:</p>
<blockquote>
<p>Think about what happens when you need to upgrade the <span class="caps">OS</span>, which typically means you make a new <span class="caps">VM</span> to replace the old one.</p>
</blockquote>
<p>We can seamlessly upgrade nixos pretty much always.
I’ve had configuration files change which caused errors when upgrading nixos stable versions,
but these errors need to be solved before deploying.
The systems I use with nixos themselves have always been stable. Or: </p>
<blockquote>
<p>Think about what happens when there is a problem and the <span class="caps">VM</span> is down. (Bad Windows update, you fat fingered a network setting, etc)</p>
</blockquote>
<p>How would I even fat finger a network setting?
I have to modify a file, commit the file and deploy to do this.
At this point you can’t even call it fat fingering.
And even if you somehow screw up a network setting deliberately,
you can reboot from an older generation and the problem is gone.
The difference here is that this deployment is nix based,
which allows me to manage the complexity of the monolith
much more efficiently and reliably.</p>
<p><span class="caps">NMMM</span> can also be used to impress potential clients
by quickly creating new websites.
What are the steps?</p>
<ol>
<li>Get the new domain, for example <code>newdomain.com</code>,</li>
<li>copy over the <code>massapp.org</code> folder into <code>newdomain.com</code>,</li>
<li>hook it into the main config<code>/nix/hetzner/configuration.nix</code>,</li>
<li>setup the database,</li>
<li>redo branding </li>
<li>deploy.</li>
</ol>
<p>That’s it.
That’s two hours of work, maybe three if something goes wrong.
Nothing is more impressive then have a functioning website,
<em>the next day</em>.
I guess aside from startups,
consultancies should also look into this approach.</p>
<p>On several occasions I’ve mentioned that <code>configuration.nix</code> is
just like on my laptop.
Originally I even copied pasted the Postgres
configuration from my laptop to this machine.
It just works.
This is one of the big benefits you get out of nix,
it’s called the <a href="https://www.youtube.com/watch?v=OyfBQmvr2Hc&t=2305s">copy-paste monad</a>.</p>
<p>If you’re having doubts about nixops and want to replace it,
you may run into secret management issues.
Fortunately there is an alternative project for secret management called
<a href="https://github.com/ryantm/agenix">agenix</a>.
Although I’m a bit skeptical over <a href="https://github.com/FiloSottile/age">age</a>,
it seems to new to trust.
You should do your diligence before using that in production.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I described this <span class="caps">NMMM</span> setup that’s working really well for me.
Furthermore I think I gave compelling reasons for
others to try this out.
Perhaps I liberated some people from their
<a href="https://www.youtube.com/watch?v=4JkIs37a2JE">virtual insanity</a>.
Let me know if I’ve inspired you to change your course of action,
or you’ve some compelling reasons not to use this approach.
Because how to structure services is a large decision,
I find this all fascinating.</p>
<h2 id="further-reading">Further reading</h2>
<ul>
<li><a href="https://nixos.org/manual/nixos/stable/index.html#sec-configuration-file">The NixOS configuration file explained</a></li>
<li><a href="https://nixos.wiki/wiki/Module">NixOS module system</a></li>
<li><a href="https://gist.github.com/nh2/ebc27311731f53ee623ae781ca25103f">Setting up a Hetzner machine with NixOS script</a></li>
<li>General nix advice:<ul>
<li><a href="https://nix.dev/">nix.dev</a></li>
<li><a href="https://nixos.org/guides/nix-pills/">nix ‘pills’</a></li>
</ul>
</li>
</ul>
<div class="footnote">
<hr>
<ol>
<li id="fn-cheap">
<p>The price of a <a href="https://www.hetzner.com/dedicated-rootserver">Hetzner</a>
machine is <code>€ 45</code> euro’s per month.
This gives you 1 terabyte of raid-1 disks,
12 threads and 64g of ram.
You <a href="https://aws.amazon.com/blogs/aws/new-t3-instances-burstable-cost-effective-performance/">currently</a>
pay around <code>$ 240</code> dollar per month for a t3.2xlarge instance.
That’s halve the amount of ram and only 8 vCPU’s (not dedicated threads)
and that doesn’t include network cost or storage.
It’s safe to say that Hetzner is <em>cheap</em>.
<p>Of course on <span class="caps">AWS</span> you can get a much smaller machine for a cheaper price per month,
which may sound good initially,
but as soon as you need larger or more machines,
a dedicated machine become much more cost effective.
And you’ll need a larger machine eventually.
If your idea fails you probably want to keep it online for your resume,
which would make it one of the monoliths,
or it succeeds and you need a bigger machine to deal with memory volatility.
I happened to need a large machine to do video editing for <a href="https://videocut.org/">videocut.org</a>.
</p> <a class="footnote-backref" href="#fnref-cheap" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-simple">
<p>The setup is simple because you
don’t have to do networking.
Everything is on the same machine,
which means you only have to do
inter process communication.
Furthermore, all services communicate to the same database
process,
which uses the databases’ internal tenanting system
to ensure the data remains isolated.
<p>The same goes for anything else services need,
if a service needs the filesystem,
just prefix some folder with their domain name.
I don’t have to deal with buckets or networking issues.
and if something doesn’t work,
9 out of 10 times systemd will tell me exactly what’s broken,
no need to dive into the <span class="caps">AWS</span> <span class="caps">CLI</span>.</p> <a class="footnote-backref" href="#fnref-simple" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-no-nixops">
<p>For me nixops causes a lot of needles friction
which I like to avoid.
For example,
the nixops developers decided to
change how the <span class="caps">CLI</span> works from the bash based counterparts.
<code>nixops scp</code> only works with <code>--from</code> and <code>--to</code> flags.
It refuses to accept the ordinary scp syntax, for no reason.
<p>There are countless other examples of this kind of friction.
I just get frustrated by writing about it, so I won’t.
However the more important thing this does
is to cut out one of the variables during deployment.
I can cut out both Nixops and the <span class="caps">AWS</span> variables,
after all I don’t need to manage <span class="caps">AWS</span>, so why would I use nixops? </p> <a class="footnote-backref" href="#fnref-no-nixops" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn-fast">
<p>Finally the last advantage
is that setting up a new service is <em>fast</em>.
No need to wait ages for an instance to boot,
starting a process is nearly instant after all.
This is sort off the same as <code>3</code>, but in this case we’re saying <span class="caps">AWS</span>
is slow, rather then nixops having poor <span class="caps">UX</span>. <a class="footnote-backref" href="#fnref-fast" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn-breaks-often">
<p>It’s built in but this used to break quite a lot,
this is because it’s enterly reliant on the graces of let’s encrypt service.
Which may refuse you to give a certificate for all <a href="https://letsencrypt.org/docs/rate-limits/">kinds of reasons</a>. <a class="footnote-backref" href="#fnref-breaks-often" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn-it-breaks-always">
<p>When do things break you ask? Well of course all software is broken so we always need to enable this. <a class="footnote-backref" href="#fnref-it-breaks-always" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
</ol>
</div>Using git for templates2020-10-24T15:00:00+02:002020-10-24T15:00:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2020-10-24:/using-git-for-templates.html<p>Over the past few years I’ve started using git as a template
management tool<sup id="fnref-9"><a class="footnote-ref" href="#fn-9">1</a></sup>.
For example, I clone my <a href="https://github.com/jappeace/haskell-template-project">haskell template project</a>,
edit the names, edit the <code>readme</code> end re-setup the remotes:</p>
<div class="highlight"><pre><span></span>git remote add template git@github.com:jappeace/haskell-template-project.git
git remote set-url origin git@github …</pre></div><p>Over the past few years I’ve started using git as a template
management tool<sup id="fnref-9"><a class="footnote-ref" href="#fn-9">1</a></sup>.
For example, I clone my <a href="https://github.com/jappeace/haskell-template-project">haskell template project</a>,
edit the names, edit the <code>readme</code> end re-setup the remotes:</p>
<div class="highlight"><pre><span></span>git remote add template git@github.com:jappeace/haskell-template-project.git
git remote set-url origin git@github.com:jappeace/new-project.git
</pre></div>
<p>This allows me to keep my downstream project up to date with the template.
When I discover another tool, this can be merged into the downstream project for example.
It allows evolving the template over time as I learn more about
whatever programming language I’m using.</p>
<p>It also gives me default branding.
Which makes it look more ‘professional’.
Although I don’t like branding, I know this is necessary.
Having it there by default makes it so that I don’t forget to do this.</p>
<p>When starting a project you also gain some free authoritativeness,
because you have a longer commit history the project looks
like it is more mature.
Although this practice is a little deceptive,
there is a core of truth, because the template
has gotten a lot of attention.
If you feel offended by this practice, ask yourself first,
why are you trusting git histories? These can be <a href="https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History">rewritten anyway</a>.
This authoritativeness also gives me a small moral boost.
It just <em>feels</em> better to have a project with well thought
out tools being started and some history
rather then the ‘normal’ stuff everyone starts with.</p>
<h2 id="alternatives">Alternatives</h2>
<p>The alternatives are using tools that spit out templates.
In the haskell world these are <a href="https://github.com/commercialhaskell/stack-templates">stack new</a>,
<a href="https://cabal.readthedocs.io/en/latest/developing-packages.html#using-cabal-init">cabal init</a>
or <a href="https://github.com/kowainik/summoner">summoner</a>.
These tools are simply a glorified <code>sed</code> on projects folders:
They find and replace variables.
There is no versioning, there is no personalization except what limited
variables give.</p>
<p>Because these tools don’t take into account history, the first commit will be a
large commit with a message like ‘Initial’.
I find this problematic because it squashes all the
work that went into creating the template.
Information is lost, for no reason.</p>
<!-- TODO verify this-->
<p>These tools don’t use git to track their templates.
They could, but they don’t.
Maybe they wanted to be <span class="caps">VCS</span><sup id="fnref-1"><a class="footnote-ref" href="#fn-1">2</a></sup> agnostic?
I’m speculating,
but even if you wanted to be that,
you could still use any version system to track the template
and recreate the history in the <span class="caps">VCS</span> desired by the user.
This at least gives the user their history of the template back.
While also giving them the choice of <span class="caps">VCS</span><sup id="fnref-8"><a class="footnote-ref" href="#fn-8">3</a></sup></p>
<h2 id="a-template-hierarchy">A template hierarchy</h2>
<p>What I described in the intro can be taken a step further.
A template of a template can be used to create another new template.
For example, say you’re a fan of <a href="https://nixos.org/">nix</a> or <a href="https://www.gnu.org/software/make/manual/make.html">make</a>,
so why not put a makefile and a <a href="https://nix.dev/tutorials/towards-reproducibility-pinning-nixpkgs.html#pinning-nixpkgs">nix pin</a>
in some template of templates project.
These tools don’t care about which language you use,
so their configuration can be shared among Python, Java and Haskell projects for example.
Maybe for github it could even have a partial travis or github actions <span class="caps">CI</span> setup.</p>
<p>Now, say you’re starting a new Python project, but have
no template yet for that programming language.
You simply clone this template of templates and customize it to
your own <a href="https://stackoverflow.com/questions/25011078/what-does-pythonic-mean#:~:text=Pythonic%20means%20code%20that%20doesn,is%20intended%20to%20be%20used.">Pythonic</a> needs.
You can still pull in the upstream changes from your template of templates,
for example to upgrade the nix pin.
The big advantage of course is that a change in the template of templates
can easily merged into all downstream projects with git.</p>
<h1 id="conclusion">Conclusion</h1>
<p>I convinced the reader that using git is the superior way for template management.
It’s simpler, and more flexible.
I also convinced the authors of stack, cabal and summoner to drop their current way of managing
templates and use git instead.</p>
<p>Or, more likely, no-one read this far.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-9">
<p>This started out as having no better alternative, due to my nixos choice. But now I realise, this is the best alternative. <a class="footnote-backref" href="#fnref-9" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-1">
<p>Version Control System <a class="footnote-backref" href="#fnref-1" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-8">
<p>I wouldn’t bother with this choice unless someone is asking for it.
I think git has won the <span class="caps">VCS</span> battle by a wide <a href="https://softwareengineering.stackexchange.com/questions/136079/are-there-any-statistics-that-show-the-popularity-of-git-versus-svn">margin</a>.
Of course that’s easy for me the say as neither maintainer nor user.
25% <span class="caps">SVN</span> market share is a lot more then I expected (the absolute <span class="caps">SVN</span> repo count is still going up!). <a class="footnote-backref" href="#fnref-8" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>Hacking atom2020-08-16T21:12:00+02:002020-08-16T23:09:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2020-08-16:/hacking-atom.html<p>On my <a href="https://www.twitch.tv/jappiejappie">twitch stream</a> I had project where I made
a <a href="https://en.wikipedia.org/wiki/Scan_line"><span class="caps">CRT</span> scan line</a> monitor in atom:</p>
<iframe class="video" src="https://www.youtube.com/embed/FdYISkTBVuw" frameborder="0" allow="accelerometer; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>The inspiration was the fallout <a href="https://fallout.fandom.com/wiki/Pip-Boy">pip-boy</a>.
I though it would be cool to have this be in your editor,
maybe some of the gray beards are nostalgic for this?</p>
<p>What amazes me is …</p><p>On my <a href="https://www.twitch.tv/jappiejappie">twitch stream</a> I had project where I made
a <a href="https://en.wikipedia.org/wiki/Scan_line"><span class="caps">CRT</span> scan line</a> monitor in atom:</p>
<iframe class="video" src="https://www.youtube.com/embed/FdYISkTBVuw" frameborder="0" allow="accelerometer; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<p>The inspiration was the fallout <a href="https://fallout.fandom.com/wiki/Pip-Boy">pip-boy</a>.
I though it would be cool to have this be in your editor,
maybe some of the gray beards are nostalgic for this?</p>
<p>What amazes me is that this is possible with just <span class="caps">CSS</span>.
Atom has become my favorite editor after doing this project
(although I hardly use it),
and my love for <span class="caps">CSS</span> has grown even more!</p>Ghcid for multi package projects2019-10-17T21:30:00+02:002019-12-13T21:58:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2019-10-17:/ghcid-for-multi-package-projects.html<p><img alt="GHCID magic" src="/images/2019/fire.svg"></p>
<p>When I tried <a href="https://github.com/ndmitchell/ghcid">Ghcid</a>for a
<a href="http://hackage.haskell.org/package/reflex">reflex</a> project,
it wouldn’t rebuild on file change.
This is because reflex has a multi package project setup by default.
Recently I found that it is possible to use
<a href="https://github.com/ndmitchell/Ghcid">Ghcid</a> for multi package project builds.
The trick is to create an executable that …</p><p><img alt="GHCID magic" src="/images/2019/fire.svg"></p>
<p>When I tried <a href="https://github.com/ndmitchell/ghcid">Ghcid</a>for a
<a href="http://hackage.haskell.org/package/reflex">reflex</a> project,
it wouldn’t rebuild on file change.
This is because reflex has a multi package project setup by default.
Recently I found that it is possible to use
<a href="https://github.com/ndmitchell/Ghcid">Ghcid</a> for multi package project builds.
The trick is to create an executable that includes all the sources.
For example in <a href="https://github.com/sol/hpack">hpack</a> style:</p>
<div class="highlight"><pre><span></span><span class="l l-Scalar l-Scalar-Plain">tests</span><span class="p p-Indicator">:</span>
<span class="l l-Scalar l-Scalar-Plain">unit</span><span class="p p-Indicator">:</span>
<span class="l l-Scalar l-Scalar-Plain">main</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">Spec.hs</span>
<span class="l l-Scalar l-Scalar-Plain">source-dirs</span><span class="p p-Indicator">:</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">test</span>
<span class="c1"># including extra source dirs allows Ghcid to watch</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">src</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">../common/src</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">../frontend/src</span>
</pre></div>
<p>Now Ghcid will watch all the sources.
By targeting the test suite you can also immediately use this to run
unit tests.
If your other project use additional dependencies you may need to add
them to that test as well.</p>
<p>However because I’m doing reflex,
I had to compile JavaScript after Ghcid
typed checked everything.
I also had some unit tests to run.
Luckily Ghcid supports both a run and a test flag.
I simply used the <code>run</code> flag to run the tests and the <code>test</code>
flag to start <span class="caps">GHCJS</span> compilation.
Which brings us to this spell:</p>
<div class="highlight"><pre><span></span> ghcid -s <span class="s2">"import Main"</span> <span class="se">\</span>
-c <span class="s2">"make update-cabal && cabal new-repl"</span> <span class="se">\</span>
-T <span class="s2">":! cd .. && make after-native-build"</span> <span class="se">\</span>
--run test:unit
</pre></div>
<p>The run flag can also be used to run your code if you don’t have a weird
<span class="caps">GHCJS</span> requirement like me.
Running like that gives fast feedback because it’s interpreted.</p>
<h1 id="conclusion">Conclusion</h1>
<p>This setup is an order of magnitude faster for debugging in my case.
If I have a compile error it now notifies me instantaneously,
whereas before it would take a second or two.
If I need to do a full build it takes about 10 seconds,
whereas before it was somewhere around a minute or two.</p>Pinning nixops builds2019-10-17T17:35:00+02:002019-11-25T23:10:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2019-10-17:/pinning-nixops-builds.html<p><img alt="Pinned nixops" src="/images/2019/pinned-nixos.png"></p>
<p>On one machine my <a href="https://nixos.org/nixops/">Nixops</a> builds and deploys, but on the other one it fails. Why?
Isn’t nix supposed to deliver <a href="https://www.software.ac.uk/blog/2017-10-05-reproducible-environments-nix">reproducible builds</a>?
Turns out nixops uses by default your system
configurations’ <a href="https://github.com/NixOS/nixops/issues/736#issuecomment-333399151">channels</a>
rather then a <a href="https://vaibhavsagar.com/blog/2018/05/27/quick-easy-nixpkgs-pinning/">pinned nix packages</a>.
Which is not <a href="https://medium.com/@ejpcmac/about-using-nix-in-my-development-workflow-12422a1f2f4c">why you’re using nix</a>.
You want …</p><p><img alt="Pinned nixops" src="/images/2019/pinned-nixos.png"></p>
<p>On one machine my <a href="https://nixos.org/nixops/">Nixops</a> builds and deploys, but on the other one it fails. Why?
Isn’t nix supposed to deliver <a href="https://www.software.ac.uk/blog/2017-10-05-reproducible-environments-nix">reproducible builds</a>?
Turns out nixops uses by default your system
configurations’ <a href="https://github.com/NixOS/nixops/issues/736#issuecomment-333399151">channels</a>
rather then a <a href="https://vaibhavsagar.com/blog/2018/05/27/quick-easy-nixpkgs-pinning/">pinned nix packages</a>.
Which is not <a href="https://medium.com/@ejpcmac/about-using-nix-in-my-development-workflow-12422a1f2f4c">why you’re using nix</a>.
You want a reproducible build, in other words,
either both machines fail for the same reason or both succeed.
We will discuss how to do this for nixops in this post trough a pin.</p>
<p>To pin a nixops deployment we create a shell<sup id="fnref-alternative"><a class="footnote-ref" href="#fn-alternative">1</a></sup> from which we run nixops:</p>
<div class="highlight"><pre><span></span><span class="k">let</span>
<span class="ss">pkgs =</span> <span class="nb">import</span> <span class="o">.</span><span class="l">/pin.nix</span><span class="p">;</span>
<span class="k">in</span>
pkgs<span class="o">.</span>stdenv<span class="o">.</span>mkDerivation<span class="p">{</span>
<span class="ss">name =</span> <span class="s2">"nixops-env"</span><span class="p">;</span>
<span class="ss">NIX_PATH=</span><span class="s2">"nixpkgs=</span><span class="si">${</span>pkgs<span class="o">.</span>path<span class="si">}</span><span class="s2">"</span><span class="p">;</span>
<span class="ss">buildInputs =</span> <span class="p">[</span> pkgs<span class="o">.</span>nixops <span class="p">];</span>
<span class="p">}</span>
</pre></div>
<p>Where <code>pin.nix</code> <a href="https://nixos.wiki/wiki/FAQ/Pinning_Nixpkgs">looks like</a> this:</p>
<div class="highlight"><pre><span></span><span class="k">let</span>
<span class="ss">hostPkgs =</span> <span class="nb">import</span> <span class="l"><nixpkgs></span> <span class="p">{};</span>
<span class="ss">pinnedPkgs =</span> hostPkgs<span class="o">.</span>fetchFromGitHub <span class="p">{</span>
<span class="ss">owner =</span> <span class="s2">"NixOS"</span><span class="p">;</span>
<span class="ss">repo =</span> <span class="s2">"nixpkgs-channels"</span><span class="p">;</span>
<span class="ss">rev =</span> <span class="s2">"1601f559e89ba71091faa26888711d4dd24c2d4d"</span><span class="p">;</span>
<span class="ss">sha256 =</span> <span class="s2">"0iayyz9617mz6424spwbi9qvmcl8hiql42czxg8mi4ycq4p1k0dx"</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">in</span>
<span class="nb">import</span> pinnedPkgs <span class="p">{</span>
config<span class="o">.</span><span class="ss">allowUnfree =</span> <span class="no">true</span><span class="p">;</span> <span class="c1"># took me too long to figure out</span>
<span class="p">}</span>
</pre></div>
<p>This is also possible for <a href="https://github.com/reflex-frp/reflex-platform">reflex-platform</a>
projects where we simply depend on the provided pin:</p>
<div class="highlight"><pre><span></span> <span class="ss">pkgs =</span> <span class="p">((</span><span class="nb">import</span> <span class="o">.</span><span class="l">/reflex</span><span class="p">)</span> <span class="p">{</span> <span class="p">})</span><span class="o">.</span>nixpkgs<span class="p">;</span>
</pre></div>
<h1 id="conclusion">Conclusion</h1>
<p>In this post we discussed how to pin nixops builds.
There are other ways to do pinning, which is
described <a href="https://discourse.nixos.org/t/nixops-pinning-nixpkgs/734">here</a>.
I found it hard to get a working solution from that thread,
therefore I recorded my own solution here.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-alternative">
<p>For my reflex project I simply made an alternative shell,
because reflex-platform doesn’t make the shell easily modifiable.
you can run a different shell by providing the filename to the nix-shell command:
<code>nix-shell nixops-shell.nix</code> <a class="footnote-backref" href="#fnref-alternative" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Reflex server side html rendering2019-08-05T19:05:00+02:002019-08-05T19:05:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2019-08-05:/reflex-server-side-html-rendering.html<p><a href="https://jappieklooster.nl/fullstack-haskell-reflex-and-servant.html">Reflex</a>
is a single page app framework written in Haskell compiled to JavaScript.
A major concern with reflex is the slow loading times,
this can be mediated however by doing server side rendering<sup id="fnref-brag"><a class="footnote-ref" href="#fn-brag">1</a></sup>.
This blog post will discuss how to do that<sup id="fnref-oblisk"><a class="footnote-ref" href="#fn-oblisk">2</a></sup>.</p>
<p><img alt="Bob doing SSR" src="/images/2019/bob-busy.jpeg"></p>
<p>The main idea is that we …</p><p><a href="https://jappieklooster.nl/fullstack-haskell-reflex-and-servant.html">Reflex</a>
is a single page app framework written in Haskell compiled to JavaScript.
A major concern with reflex is the slow loading times,
this can be mediated however by doing server side rendering<sup id="fnref-brag"><a class="footnote-ref" href="#fn-brag">1</a></sup>.
This blog post will discuss how to do that<sup id="fnref-oblisk"><a class="footnote-ref" href="#fn-oblisk">2</a></sup>.</p>
<p><img alt="Bob doing SSR" src="/images/2019/bob-busy.jpeg"></p>
<p>The main idea is that we can share most of our single page app code
on both the frontend as well as the backend.
The backend will create a static <span class="caps">HTML</span> page,
whereas the frontend will be a JavaScript client.
The browser can display this <span class="caps">HTML</span> content very fast without having to
interpret any JavaScript.
This is where all the speed gain comes from. <sup id="fnref-server"><a class="footnote-ref" href="#fn-server">3</a></sup>
JavaScript developers call this technique <a href="https://medium.com/capital-one-tech/why-everyone-is-talking-about-isomorphic-universal-JavaScript-and-why-it-matters-38c07c87905">‘isomporhic’</a><sup id="fnref-category"><a class="footnote-ref" href="#fn-category">4</a></sup>
JavaScript.
This blogpost will tell you how to do it with reflex.
I found it surprisingly easy, and the results very impressive.</p>
<p>The <a href="http://hackage.haskell.org/package/reflex-dom-core-0.5/docs/Reflex-Dom-Builder-Static.html#v:renderStatic">renderStatic</a>
function does the <span class="caps">HTML</span> rendering.
However to use it we need to
jump trough some oddly-shaped,
<a href="http://hackage.haskell.org/package/mtl"><span class="caps">MTL</span></a> flavoured,
hoops<sup id="fnref-hoops"><a class="footnote-ref" href="#fn-hoops">5</a></sup>.
Doing this is not much harder than writing a regular reflex app.
Jumping trough these solves handling
browser native primitives,
such as <span class="caps">XHR</span> calls.
This is done
with help of <a href="https://hackage.haskell.org/package/reflex-dom-core-0.5/docs/Reflex-Dom-Prerender.html#v:prerender">prerender</a>.
Which is also the key to displaying authenticated content fast,
but we have to rewire the the initial app state to do that.
Fortunately this rewiring caused the initial state management to become more elegant
then it was in the
<a href="https://jappieklooster.nl/authentication-in-reflex-servant.html">authentication</a> post.</p>
<p>The source code is available in the
<a href="https://github.com/jappeace/awesome-project-name/tree/prerender">reference project</a>.</p>
<h1 id="strange-mtl-hoops">Strange <span class="caps">MTL</span> Hoops</h1>
<p>We need to get rid of
<a href="https://hackage.haskell.org/package/reflex-dom-core-0.5/docs/src/Reflex.Dom.Old.html#MonadWidgetConstraints">WidgetMonad</a>
and replace it with the underlying builders.
We do this because the WidgetMonad has a nasty equality constraint
on GhcjsDomSpace that prevents the use of <code>renderStatic</code>:</p>
<div class="highlight"><pre><span></span><span class="kr">type</span> <span class="kt">MonadWidgetConstraints</span> <span class="n">t</span> <span class="n">m</span> <span class="ow">=</span>
<span class="p">(</span> <span class="kt">DomBuilder</span> <span class="n">t</span> <span class="n">m</span>
<span class="p">,</span> <span class="kt">DomBuilderSpace</span> <span class="n">m</span> <span class="o">~</span> <span class="kt">GhcjsDomSpace</span> <span class="c1">-- <-- nasty constraint</span>
<span class="o">...</span>
<span class="p">)</span>
<span class="kr">class</span> <span class="kt">MonadWidgetConstraints</span> <span class="n">t</span> <span class="n">m</span> <span class="ow">=></span> <span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span>
<span class="kr">instance</span> <span class="kt">MonadWidgetConstraints</span> <span class="n">t</span> <span class="n">m</span> <span class="ow">=></span> <span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span>
</pre></div>
<p>A good strategy is replacing it with
<a href="http://hackage.haskell.org/package/reflex-dom-core-0.5/docs/Reflex-Dom-Builder-Class.html#t:DomBuilder">DomBuilder</a>
and add more constraints based on compiler requests.
<span class="caps">MTL</span> monads push outwards. That is to say,
the low level widgets determine the
constraints of the higher level widgets.</p>
<p>Unfortunately this is not the only place for that equality constraint.
The input and textarea widgets indirectly cause this constraint
too because of their return types.
For example in input:</p>
<div class="highlight"><pre><span></span><span class="kr">data</span> <span class="kt">TextInput</span> <span class="n">t</span>
<span class="ow">=</span> <span class="kt">TextInput</span> <span class="p">{</span>
<span class="o">...</span>
<span class="p">,</span> <span class="n">_textInput_builderElement</span>
<span class="ow">::</span> <span class="kt">InputElement</span> <span class="kt">EventResult</span> <span class="kt">GhcjsDomSpace</span> <span class="n">t</span> <span class="c1">-- <-- nasty fundep</span>
<span class="p">}</span>
</pre></div>
<p>There is no direct equality constraint,
but it gets asserted anyway because of
<a href="https://wiki.haskell.org/Functional_dependencies">functional dependencies</a>.
However I made an alternative to both these widgets in the Bulmex library.
In which I just
copied the <a href="https://hackage.haskell.org/package/reflex-dom-core-0.5/docs/Reflex-Dom-Widget-Input.html">original</a>
and removed the constraining records from the result types.
They can be found
<a href="https://hackage.haskell.org/package/bulmex-2.0.0/docs/Reflex-Bulmex-Input-Polymorphic.html">here</a>.</p>
<h1 id="prerender-parts">Prerender Parts</h1>
<p>As mentioned before,
we intent to share most code between the client and the server.
In case of the client we use a JavaScript executable entry point
and get a <span class="caps">GHCJS</span> <span class="caps">DOM</span> environment.
In case of the server we use a servant endpoint as entry point and get a static
<span class="caps">DOM</span> builder environment.
When the <span class="caps">GHCJS</span> environment is accessed reflex needs to know what to do in the other environment and vice versa.
In the browser for example we’ll happily do <span class="caps">XHR</span> calls,
but when doing server side rendering we can simply ignore the request<sup id="fnref-prerender"><a class="footnote-ref" href="#fn-prerender">9</a></sup>.
This use case between environments is handled by <a href="https://hackage.haskell.org/package/reflex-dom-core-0.5/docs/Reflex-Dom-Prerender.html">prerender</a>:</p>
<div class="highlight"><pre><span></span><span class="nf">prerender</span> <span class="ow">::</span> <span class="n">m</span> <span class="n">a</span>
<span class="ow">-></span> <span class="p">(</span><span class="kt">PrerenderClientConstraint</span> <span class="n">js</span> <span class="n">t</span> <span class="p">(</span><span class="kt">Client</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kt">Client</span> <span class="n">m</span> <span class="n">a</span><span class="p">)</span>
<span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Dynamic</span> <span class="n">t</span> <span class="n">a</span><span class="p">)</span>
</pre></div>
<p>Depending on the entry point we use we get a different monad environment.
On the server we get a StaticDomBuilder environment (<code>m a</code>),
and in the JavaScript executable frontend we get a <span class="caps">GHCJS</span> <span class="caps">DOM</span> environment
(<code>Client m a</code>).
The first monad is the one being displayed on the server,
the second one has access to the <span class="caps">GHCJS</span> <span class="caps">DOM</span> environment.
The resulting value will be captured in a dynamic and can be used by the
shared code.</p>
<p>Now we can enhance all servant-reflex calls with prerender calls,
to make it do nothing in the static environment and do the <span class="caps">XHR</span> calls on the <span class="caps">GHCJS</span> dom environment.
For example the <code>postLogin</code> function from the <a href="https://jappieklooster.nl/authentication-in-reflex-servant.html">authentication blogpost</a>
now looks like:</p>
<div class="highlight"><pre><span></span><span class="nf">postLogin</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">DomBuilder</span> <span class="n">t</span> <span class="n">m</span><span class="p">,</span> <span class="kt">Prerender</span> <span class="n">js</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span>
<span class="ow">=></span> <span class="kt">Dynamic</span> <span class="n">t</span> <span class="p">(</span><span class="kt">Either</span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span> <span class="kt">User</span><span class="p">)</span>
<span class="ow">-></span> <span class="kt">Event</span> <span class="n">t</span> <span class="nb">()</span>
<span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Event</span> <span class="n">t</span> <span class="p">(</span><span class="kt">ReqResult</span> <span class="nb">()</span> <span class="p">(</span><span class="kt">AuthCookies</span> <span class="kt">NoContent</span><span class="p">)))</span>
<span class="nf">postLogin</span> <span class="n">dd</span> <span class="n">yy</span> <span class="ow">=</span> <span class="n">fmap</span> <span class="n">switchDyn</span> <span class="o">$</span> <span class="n">prerender</span> <span class="p">(</span><span class="n">pure</span> <span class="n">never</span><span class="p">)</span> <span class="o">$</span> <span class="n">postLogin'</span> <span class="n">dd</span> <span class="n">yy</span>
</pre></div>
<p>Here <code>postLogin'</code> is the generated endpoint from <a href="https://hackage.haskell.org/package/servant-reflex">servant-reflex</a>.
In case of a server StaticDomBuilder environment we do nothing never
<code>(pure never)</code><sup id="fnref-side"><a class="footnote-ref" href="#fn-side">6</a></sup>,
in case of a <span class="caps">GHCJS</span> <span class="caps">DOM</span> environment we do the <code>postLogin'</code> function.
The fmap switchDyn part is to get rid of the dynamic,
which is unnecessary because we already return an event.
Similar code can be used for all generated endpoints<sup id="fnref-points"><a class="footnote-ref" href="#fn-points">7</a></sup>.</p>
<h1 id="backend-servant-endpoint">Backend servant endpoint</h1>
<p>We need to add an endpoint to servant which will serve the
statically rendered <span class="caps">HTML</span>.
To add it to servant we make an authenticated entry point, which returns
<a href="http://hackage.haskell.org/package/servant-fiat-content-1.0.0/docs/Servant-HTML-Fiat.html"><span class="caps">HTML</span></a>
as a <code>ByteString</code>.
Because it’s authenticated we can choose what to serve.
The authenticated app, if the user has the permission, or
the public site, if the user doesn’t have that.
It looks like this:</p>
<div class="highlight"><pre><span></span><span class="kr">type</span> <span class="kt">Webservice</span> <span class="ow">=</span> <span class="o">...</span>
<span class="kt">:<|></span> <span class="kt">Auth</span> <span class="kt">'[Cookie, JWT]</span> <span class="kt">User</span> <span class="kt">:></span> <span class="p">(</span><span class="kt">Get</span> <span class="kt">'[HTML]</span> <span class="kt">BS</span><span class="o">.</span><span class="kt">ByteString</span><span class="p">)</span>
<span class="o">...</span>
</pre></div>
<p>In the previous blogpost we’d simply return a ‘401 unauthorized’ status code.
Servant auth allows us to decide how to respond.
Which we can see in the implementation:</p>
<div class="highlight"><pre><span></span><span class="nf">renderHtmlEndpoint</span> <span class="ow">::</span> <span class="kt">HeadSettings</span> <span class="ow">-></span> <span class="kt">AuthResult</span> <span class="kt">User</span> <span class="ow">-></span> <span class="kt">Handler</span> <span class="kt">BS</span><span class="o">.</span><span class="kt">ByteString</span>
<span class="nf">renderHtmlEndpoint</span> <span class="n">settings</span> <span class="n">authRes</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">fmap</span> <span class="n">snd</span> <span class="o">$</span> <span class="n">liftIO</span> <span class="o">$</span> <span class="n">renderStatic</span> <span class="o">$</span>
<span class="n">htmlWidget</span> <span class="n">settings</span> <span class="o">$</span> <span class="n">main</span> <span class="o">$</span> <span class="kt">IniState</span> <span class="o">$</span> <span class="n">toMaybe</span> <span class="n">authRes</span>
</pre></div>
<p>The <code>main</code> function is the entry point for the reflex code.
We wrap it around in <a href="https://hackage.haskell.org/package/bulmex-2.0.0/docs/Reflex-Bulmex-Html.html#v:htmlWidget"><code>htmlWidget</code></a>,
which has <a href="https://hackage.haskell.org/package/bulmex-2.0.0/docs/Reflex-Bulmex-Html.html#t:HeadSettings"><code>HeadSettings</code></a>
to load the JavaScript client as a simple <a href="https://hackage.haskell.org/package/network-uri-2.6.1.0/docs/Network-URI.html#t:URI"><code>URI</code></a>.
The decision of what to show is completely pushed into the reflex code,
the only thing we do at server side is give it the <code>IniState</code>.
Next we’ll discuss how the reflex app decides what to show in this environment.</p>
<h1 id="reflex-app">Reflex App</h1>
<p>So now we have an <code>IniState</code> for the app,
we can read or write it to the <span class="caps">DOM</span> depending on environment.
I wrote a function for that.
On the app side this is done by <a href="https://hackage.haskell.org/package/bulmex-2.0.0/docs/Reflex-Bulmex-Html.html#v:writeReadDom"><code>writeReadDom</code></a>:</p>
<div class="highlight"><pre><span></span><span class="nf">writeReadDom</span> <span class="ow">::</span>
<span class="p">(</span> <span class="kt">FromJSON</span> <span class="n">a</span>
<span class="p">,</span> <span class="kt">ToJSON</span> <span class="n">a</span>
<span class="p">,</span> <span class="kt">Dom</span><span class="o">.</span><span class="kt">DomBuilder</span> <span class="n">t</span> <span class="n">m</span>
<span class="p">,</span> <span class="kt">Dom</span><span class="o">.</span><span class="kt">Prerender</span> <span class="n">js</span> <span class="n">t</span> <span class="n">m</span>
<span class="p">)</span> <span class="ow">=></span>
<span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span> <span class="ow">-></span> <span class="n">a</span> <span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Dynamic</span> <span class="n">t</span> <span class="n">a</span><span class="p">)</span>
</pre></div>
<p>The first value is the <span class="caps">HTML</span> id,
the second one is the app state as received by it’s respective endpoint.
The result behaves roughly like this:</p>
<div class="highlight"><pre><span></span><span class="c"><!-- Static dom builder environment (server) --></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">id</span><span class="o">=</span><span class="s">"Text.Text"</span> <span class="na">type</span><span class="o">=</span><span class="s">"application/json"</span><span class="p">></span>
<span class="o"><?</span><span class="nx">php</span> <span class="nx">encode</span> <span class="nx">a</span> <span class="o">?></span>
<span class="p"></</span><span class="nt">script</span><span class="p">></span>
<span class="c"><!-- GHCJS environment (browser) --></span>
JSON.parse(document.getElementById("Text.Text"));
</pre></div>
<p>Of course the JavaScript will look completely different
and the result is put in a dynamic.
But the resulting behavior is the same, the server encodes the <span class="caps">JSON</span>
and the client will decodes it giving us the app state from the server
without additional requests.
Under the hood <code>prerender</code> figures out if we need to read or write.
All the client needs to do is provide said value,
in case of the <a href="https://github.com/jappeace/awesome-project-name/tree/prerender">example app</a>
this is just a <code>Maybe User</code>.
The decision of what to display is all integrated within
the same reflex code like this:</p>
<div class="highlight"><pre><span></span><span class="nf">main</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">AWidget</span> <span class="n">js</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kt">IniState</span> <span class="ow">-></span> <span class="n">m</span> <span class="nb">()</span>
<span class="nf">main</span> <span class="n">iniState</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">iniDyn</span> <span class="ow"><-</span> <span class="n">writeReadDom</span> <span class="s">"iniState"</span> <span class="n">iniState</span>
<span class="n">rec</span> <span class="n">loginEvt</span> <span class="ow"><-</span> <span class="n">elDynAttr</span> <span class="s">"div"</span> <span class="n">loginAttr</span> <span class="o">$</span> <span class="n">loginWidget</span> <span class="n">iniDyn</span>
<span class="n">loginAttr</span> <span class="ow"><-</span> <span class="n">holdDyn</span> <span class="p">(</span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="p">)</span> <span class="o">$</span> <span class="n">hidden</span> <span class="o"><$</span> <span class="n">loginEvt</span>
<span class="n">void</span> <span class="o">$</span> <span class="n">holdEvent</span> <span class="nb">()</span> <span class="n">loginEvt</span> <span class="n">authenticatedWidget</span>
<span class="nf">authenticatedWidget</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">AWidget</span> <span class="n">js</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kt">User</span> <span class="ow">-></span> <span class="n">m</span> <span class="nb">()</span>
<span class="nf">authenticatedWidget</span> <span class="ow">=</span> <span class="o">...</span>
</pre></div>
<p>We always get an <code>IniState</code>,
but we make sure to first put it in <code>writeReadDom</code>.
This ensures that the <code>IniState</code> is the actual <code>IniState</code> from
the server and not the one we hardcoded
in the executable to make the compiler happy<sup id="fnref-nohappy"><a class="footnote-ref" href="#fn-nohappy">8</a></sup>.
This <code>iniDyn</code> then gets used for the <code>loginWidget</code> which returns a
<code>loginEvt</code>.
If the event fires it has a user,
with which we can display the <code>authenticatedWidget</code>.
This is done with help of <code>holdEvent</code>.
Note that if no event is received, no authentication is done.
<code>loginAttr</code> will however become hidden as soon as there is a <code>loginEvt</code>,
hiding the login form.
The login widget itself looks like this now:</p>
<div class="highlight"><pre><span></span><span class="nf">loginWidget</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">AWidget</span> <span class="n">js</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kt">Dynamic</span> <span class="n">t</span> <span class="kt">IniState</span> <span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Event</span> <span class="n">t</span> <span class="kt">User</span><span class="p">)</span>
<span class="nf">loginWidget</span> <span class="n">iniDyn</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">pb</span> <span class="ow"><-</span> <span class="n">getPostBuild</span>
<span class="n">formEvt</span> <span class="ow"><-</span> <span class="n">loginForm</span>
<span class="n">pure</span> <span class="o">$</span> <span class="n">leftmost</span> <span class="p">[</span><span class="n">formEvt</span><span class="p">,</span>
<span class="n">noNothing</span> <span class="o">$</span> <span class="n">updated</span> <span class="o">$</span> <span class="n">userDyn</span><span class="p">,</span>
<span class="n">noNothing</span> <span class="o">$</span> <span class="n">current</span> <span class="n">userDyn</span> <span class="o"><@</span> <span class="n">pb</span><span class="p">]</span>
<span class="kr">where</span>
<span class="n">userDyn</span> <span class="ow">=</span> <span class="n">unpackUser</span> <span class="o"><$></span> <span class="n">iniDyn</span>
</pre></div>
<p>Compared to the <a href="https://jappieklooster.nl/authentication-in-reflex-servant.html">previous blogpost</a>
we got rid of <code>autoLogin</code> and use the <code>IniState</code> instead.
The post build event is used to sample the current value of <code>userDyn</code>.
This ensures a user event fires if there is one available.
That mechanism also works for static rendering.</p>
<h1 id="conclusion">Conclusion</h1>
<p>Loading times was one of reflex major weaknesses and this blog post discussed how to
mostly remove it.
It still takes time before the app becomes interactive,
but at least now useful information can be shown to the user instead
of a loading screen.
I’m really happy with the end result, and I think anyone with a reflex app should
seriously consider doing this.</p>
<h1 id="references">References</h1>
<ul>
<li><a href="https://github.com/jappeace/awesome-project-name/tree/prerender">Reference project github</a></li>
<li><a href="http://docs.reflex-frp.org/en/latest/reflex_dom_docs.html#prerendering-server-side-rendering">Official reflex prerender docs</a></li>
<li><a href="https://hackage.haskell.org/package/reflex-dom-core-0.5/docs/Reflex-Dom-Prerender.html">Prerender on hackage</a></li>
<li><a href="https://hackage.haskell.org/package/reflex-dom-core-0.5/docs/Reflex-Dom-Builder-Static.html#v:renderStatic">Render static on hackage</a></li>
<li><a href="https://github.com/obsidiansystems/obelisk">Obelisk does already do <span class="caps">SSR</span> out of the box</a></li>
</ul>
<div class="footnote">
<hr>
<ol>
<li id="fn-brag">
<p>Going from 8 seconds in my own app to about 0.5 seconds. <a class="footnote-backref" href="#fnref-brag" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-oblisk">
<p>A prebaked solution is available in <a href="https://github.com/obsidiansystems/obelisk">oblisk</a>, one should consider using that. <a class="footnote-backref" href="#fnref-oblisk" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-server">
<p>The server doesn’t need to interpret JavaScript,
Haskell can also target native as compile target.
But even if it were running JavaScript you can still get
speed gains by doing page caching. <a class="footnote-backref" href="#fnref-server" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn-category">
<p>So it’s not isomorpism categorical terms, but JavaScript terms. <a class="footnote-backref" href="#fnref-category" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn-hoops">
<p>I figured out all this stuff on <a href="https://www.youtube.com/channel/UCQxmXSQEYyCeBC6urMWRPVw?view_as=subscriber">stream</a> <a class="footnote-backref" href="#fnref-hoops" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn-side">
<p>I could imagine an alternative design:
For example with a micro service setup where you just let the app decide what to call.
You’d still need to use a different client on the server side though because it’s native code.
But this can be done for with <a href="http://hackage.haskell.org/package/servant-client">servant-client</a>.
You define your micro services around the ‘servant’ based <span class="caps">REST</span> contracts and let the app pull it in. <a class="footnote-backref" href="#fnref-side" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn-points">
<p>It may also be possible to make a more generic function to do this,
but I didn’t want to spend the time
(I’ve been working on this for days). <a class="footnote-backref" href="#fnref-points" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn-nohappy">
<p>I don’t know of a type safe way around this inital state problem.
Note that the JavaScript loading by itself is also not type safe,
a <code>URI</code> is used.
With that in mind we can kind off consider this the ‘edge’
of the system. <a class="footnote-backref" href="#fnref-nohappy" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn-prerender">
<p>Special thanks to lumie for pointing this out. <a class="footnote-backref" href="#fnref-prerender" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
</ol>
</div>Citrix XenCenter 7.6 notes2019-04-18T00:00:00+02:002019-04-18T00:00:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2019-04-18:/citrix-xencenter-76-notes.html<p>A client had a Citrix environment running.
To debug the deployment of a machine, I copied the Citrix environment locally
in VirtualBox.
Note that the server doesn’t run any machines inside citrix,
but it does get the XenCenter operational.
It appears that in nixos you can just setup a …</p><p>A client had a Citrix environment running.
To debug the deployment of a machine, I copied the Citrix environment locally
in VirtualBox.
Note that the server doesn’t run any machines inside citrix,
but it does get the XenCenter operational.
It appears that in nixos you can just setup a local <a href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/xen-dom0.nix">xen server</a>.
The configuration described here will setup a Citrix XenServer that installs and runs,
it just can’t run other <span class="caps">VM</span>’s. It says it misses <span class="caps">HVM</span>.</p>
<p>This post contains my notes on getting XenCenter operational in VirtualBox.
It’s also a guide for people familiar with Linux,
but not with Citrix.</p>
<h1 id="citrix-xenserver">Citrix XenServer</h1>
<p>It’s actually CentOS with XenServer installed on it.
The <span class="caps">VM</span> shell will drop you into a ncurses menu which has an option to go to a bash shell.</p>
<p>The branding says it’s a ‘smaller’ operating system, looks pretty complete to me though.
I think they mean that they have kernel modules that may allow the guest <span class="caps">VM</span>’s
to use hardware directly.
To do this you have to modify the host as well as the guest <span class="caps">OS</span>.
They both have to understand the virtualization situation.</p>
<p>If you want to do this properly you should seriously consider <a href="https://www.youtube.com/watch?v=xXWaECk9XqM">containers</a>.
Even though virtual machines are <a href="https://www.infoworld.com/article/3071679/linux-containers-vs-vms-a-security-comparison.html">more secure</a>
and virtual machines are more portable<sup id="fnref-mix"><a class="footnote-ref" href="#fn-mix">1</a></sup>,
the trade off however is that virtual machines are really inneficient.
Giving them direct access to hardware won’t
change the nature of <span class="caps">OS</span> kernels.
To hog all resources, which they have to do to manage them.
A container setup will simply have a single kernel, yet many operating systems (user spaces).</p>
<h2 id="get-xenserver">Get XenServer</h2>
<p>Go to <a href="https://www.citrix.com">citrix.com</a>, you must get it from there.
Don’t bother googling this.
Google doesn’t find the download.
You <em>have</em> to make an account to download it.</p>
<p>Then go for the <code>citrix hypervisor</code> in the downloads area.
By the way: <code>XenServer = citrix hypervisor</code>
I think the only thing they did to the XenServer is add branding.</p>
<h2 id="get-client">Get client</h2>
<p>The XenServer also contains the client.
No need to download any of the other stuff.
XenServer is one of the many products hosted by citrix.</p>
<h1 id="you-can-run-xenserver-in-virtual-box">You can run XenServer in virtual box</h1>
<p>You need enough <span class="caps">RAM</span> though, just create a <a href="https://linuxize.com/post/create-a-linux-swap-file/">swap file</a>
if you run into trouble.</p>
<p>Make sure to <a href="https://superuser.com/questions/227505/what-is-the-difference-between-nat-bridged-host-only-networking">bridge</a>
it on a device like <span class="caps">WIFI</span>.
This allows one <span class="caps">VM</span> to access another <span class="caps">VM</span>.
Also put promiscuous mode on allow all.
I think this is why I couldn’t transfer the <span class="caps">VHD</span>.</p>
<p>Install it, go into the root shell and type <code>ip addr</code> to get the ip.
We need this for the client.</p>
<h2 id="creating-an-extra-storage-repository">Creating an extra storage repository:</h2>
<p>For some reason the XenServer created a very small 8G repository,
I just added another disk with VirtualBox and then
ran this on the XenServer vm shell:</p>
<div class="highlight"><pre><span></span>xe sr-create name-label<span class="o">=</span><Storage ID> <span class="nv">shared</span><span class="o">=</span><span class="nb">false</span> device-config:device<span class="o">=</span><Path of the Storage device> <span class="nv">type</span><span class="o">=</span>lvm content-type<span class="o">=</span>user
</pre></div>
<p>Actually I attempted to partition it, but I think the command already does that.
If not see my other post where <a href="https://jappieklooster.nl/nixos-on-encrypted-btrfs.html">gdisk</a> is explained.</p>
<h1 id="use-the-windows-client">Use the windows client</h1>
<p>I couldn’t get the citrix Linux client to work on nixos.
I attempted <a href="https://libvirt.org/">libvirt</a> because it supports remote but it
seemed fairly complicated.
I ran the XenCenter client in the <a href="https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/">edge iso</a>
provided by <span class="caps">MS</span>.
This <span class="caps">VM</span> was bridged too.</p>
<p>Once inside the <span class="caps">VM</span>,
click on a <span class="caps">VMDK</span> to import it into XenCenter.
I could find no way of doing this in the XenCenter program itself.
Also make sure to have a server selected before importing it,
otherwise you’ll get a null reference error on the storage screen.
It apparently uses the selected one instead of the location from previous screen.</p>
<h2 id="log-for-notices">Log for notices</h2>
<p>If you ever get a “generic error”, check the log:</p>
<div class="highlight"><pre><span></span> C:\Users\IEUser\AppData\Roaming\Citrix\XenCenter\logs
</pre></div>
<p>There is a ton of information in that.
Pasting the relevant part into google will get you more
progress than “generic error”.</p>
<h1 id="virtualbox-vmdks">VirtualBox VMDKs</h1>
<p>Citrix XenServer does not appear to accept VirtualBox VMDKs.
I got <span class="caps">VHD</span> to work locally, but for some reason
it didn’t work at the client.
If you want a NixOS <span class="caps">VHD</span> checkout <a href="https://jappieklooster.nl/nixos-notes.html">this post</a></p>
<div class="footnote">
<hr>
<ol>
<li id="fn-mix">
<p>If you mix <span class="caps">OS</span>’es for some reason you’re right in using virtual machines <a class="footnote-backref" href="#fnref-mix" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Nixos notes2019-04-18T00:00:00+02:002019-04-18T00:00:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2019-04-18:/nixos-notes.html<p>This is a post of things I wanted to do in nixos
but isn’t described anywhere.
I had to read source code to figure these things out.
By explaining here what is going on I make things easier for other people.</p>
<p><img alt="Nixos notes" src="/images/2019/nixos_notes.svg"></p>
<h1 id="nix-custom-image">Nix custom image</h1>
<p>It’s possible to bypass …</p><p>This is a post of things I wanted to do in nixos
but isn’t described anywhere.
I had to read source code to figure these things out.
By explaining here what is going on I make things easier for other people.</p>
<p><img alt="Nixos notes" src="/images/2019/nixos_notes.svg"></p>
<h1 id="nix-custom-image">Nix custom image</h1>
<p>It’s possible to bypass virtual box and make a bunch of different image
formats directly, I used two files <code>image.nix</code>:</p>
<div class="highlight"><pre><span></span><span class="p">{</span> config<span class="p">,</span> <span class="o">...</span> <span class="p">}:</span>
<span class="k">let</span>
<span class="ss">pkgs =</span> <span class="nb">import</span> <span class="o">.</span><span class="l">/pin.nix</span> <span class="p">{</span> <span class="p">};</span>
<span class="k">in</span>
<span class="p">{</span>
system<span class="o">.</span>build<span class="o">.</span><span class="ss">image =</span> <span class="nb">import</span> <span class="l"><nixpkgs/nixos/lib/make-disk-image.nix></span> <span class="p">{</span>
<span class="ss">name =</span> <span class="s2">"nixos-vmdk-</span><span class="si">${</span>config<span class="o">.</span>system<span class="o">.</span>nixos<span class="o">.</span>label<span class="si">}</span><span class="s2">-</span><span class="si">${</span>pkgs<span class="o">.</span>stdenv<span class="o">.</span>hostPlatform<span class="o">.</span>system<span class="si">}</span><span class="s2">"</span><span class="p">;</span>
<span class="ss">format =</span> <span class="s2">"vpc"</span><span class="p">;</span>
<span class="k">inherit</span> pkgs config<span class="p">;</span>
<span class="ss">lib =</span> pkgs<span class="o">.</span>lib<span class="p">;</span>
<span class="ss">partitionTableType =</span> <span class="s2">"legacy"</span><span class="p">;</span>
<span class="ss">diskSize =</span> <span class="mi">11</span> <span class="err">*</span> <span class="mi">1024</span><span class="p">;</span>
<span class="p">};</span>
<span class="o">....</span> <span class="c1"># remaining config, same as configuration.nix</span>
<span class="p">}</span>
</pre></div>
<p>and the <code>disk.nix</code>:</p>
<div class="highlight"><pre><span></span><span class="p">{</span> nixos <span class="o">?</span> <span class="l"><nixpkgs/nixos></span>
<span class="p">,</span> system <span class="o">?</span> <span class="nb">builtins</span><span class="o">.</span>currentSystem
<span class="p">}:</span>
<span class="k">let</span>
<span class="ss">machine-configuration =</span> <span class="nb">import</span> <span class="o">.</span><span class="l">/image.nix</span><span class="p">;</span>
<span class="ss">machine =</span> <span class="nb">import</span> nixos <span class="p">{</span>
<span class="k">inherit</span> system<span class="p">;</span>
<span class="ss">configuration =</span> machine-configuration<span class="p">;</span>
<span class="p">};</span>
<span class="k">in</span>
machine<span class="o">.</span>config<span class="o">.</span>system<span class="o">.</span>build<span class="o">.</span>image
</pre></div>
<p>If you run <code>nix-build disk.nix</code> you’ll get a <span class="caps">VHD</span> with the configuration from <code>image.nix</code>
which is just a nixos standard <code>configuraiton.nix</code>.</p>
<h1 id="nixos-rebuild-remote">Nixos-rebuild remote</h1>
<p>We can use nixos-rebuild to do in place updates of a running system remotely.
If your deployment is a single <span class="caps">VM</span> this is significantly easier than using
nixops.
I ended up with this make file:</p>
<div class="highlight"><pre><span></span><span class="nv">IP</span><span class="o">=</span><span class="s2">"192.168.0.39"</span>
deploy:
<span class="nv">NIXOS_CONFIG</span><span class="o">=</span><span class="k">$(</span>shell <span class="nb">pwd</span><span class="k">)</span><span class="s2">"/image.nix"</span> nixos-rebuild switch --target-host root@<span class="k">$(</span>IP<span class="k">)</span>
</pre></div>
<p>You may also want to set <code>--build-host</code>, because by default it will build on <code>target-host</code>.</p>
<h1 id="install-nix-on-running-nix">Install nix on running nix</h1>
<p>For some reason my boot disk gets corrupted after switching a couple of times.
A solution is just to never reboot, however
you can also just fix this while running the system.
Switching doesn’t do this apparently.
I know this because it didn’t boot.
This doesn’t matter because you can install your currently running system!</p>
<p>Mount root as <code>/mnt</code>.
Format your boot partition and mount it on <code>/mnt/boot</code>.
Then run <code>nixos-install</code>.
For example:</p>
<div class="highlight"><pre><span></span>mount <span class="c1"># list everything mounted</span>
mount /dev/nvme0n1p2 /mnt
umount /boot
mkfs.vfat -n boot /dev/nvme0n1p1
mount /dev/nvme0n1p1 /boot
mount /dev/nvme0n1p1 /mnt/boot
</pre></div>Authentication in Reflex & Servant2019-02-28T15:00:00+01:002019-02-28T15:00:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2019-02-28:/authentication-in-reflex-servant.html<p>In the previous <a href="https://jappieklooster.nl/fullstack-haskell-reflex-and-servant.html">blog post</a>
we saw interaction with servant in reflex.
Although that covers the basics,
there are several more hurdles to overcome to get comfortable with Reflex.
I think most of these are encountered by building a simple login system.
So let’s build something like:</p>
<div class="highlight"><pre><span></span> +------------------+
+-----------+ | . - |
| +--------+| | . - .. . |
| +--------+| | . .-- - + m..-. |
| +--------+| \ | m …</pre></div><p>In the previous <a href="https://jappieklooster.nl/fullstack-haskell-reflex-and-servant.html">blog post</a>
we saw interaction with servant in reflex.
Although that covers the basics,
there are several more hurdles to overcome to get comfortable with Reflex.
I think most of these are encountered by building a simple login system.
So let’s build something like:</p>
<div class="highlight"><pre><span></span> +------------------+
+-----------+ | . - |
| +--------+| | . - .. . |
| +--------+| | . .-- - + m..-. |
| +--------+| \ | m# + .-..% - |
| +--------+| \ | + .+ #.+...+ |
| login ----+------X| .. -.- * |
+-----------+ / | .. ..-#+ + |
/ | . - -. . |
| . . |
| - |
| |
+------------------+
Awesome app
</pre></div>
<p>I’ve experienced that this is hard for the first time.
With this blog post I hope that setting up authentication becomes easier,
considering the following pain points:</p>
<ol>
<li><span class="quo">‘</span>Switching screens’ after login, requires <a href="https://wiki.haskell.org/MonadFix">recursive do</a>.</li>
<li><a href="http://hackage.haskell.org/package/servant-auth-server-0.4.3.0/docs/Servant-Auth-Server.html">Dealing with cookies</a>
yourself is pretty hard, many pesky (security) details.</li>
<li>Rendering widgets inside <span class="caps">FRP</span> constructs requires <a href="https://hackage.haskell.org/package/reflex-dom-core-0.4/docs/Reflex-Dom-Widget-Basic.html#v:widgetHold">widgetHold</a>
or ‘dyn’.</li>
</ol>
<p>I’m of course a world expert on login systems, so in production do everything
exactly as I do.
Trust me, I’m from the internet.
But seriously If you see something dubious,
do contact me.
I’ll happily rectify mistakes,
and put you on the <a href="/pages/page-of-honour.html">page of honour</a> (if you want).</p>
<p>I left some code out of this blog post for succinctness.
But the full source
is available on <a href="https://github.com/jappeace/awesome-project-name/tree/auth">github</a>.</p>
<h1 id="api-endpoints"><span class="caps">API</span> Endpoints</h1>
<div class="highlight"><pre><span></span><span class="kr">type</span> <span class="kt">ServiceAPI</span> <span class="ow">=</span> <span class="kt">PublicAPI</span> <span class="kt">:<|></span> <span class="kt">Auth</span> <span class="kt">'[Cookie, JWT]</span> <span class="kt">User</span> <span class="kt">:></span> <span class="kt">AuthAPI</span>
</pre></div>
<p>First we need to split our <span class="caps">API</span> into two types.
The <code>PublicAPI</code> and the <code>AuthAPI</code>.
Once the user is logged in he gets access to the <code>AuthAPI</code>.
We’ll secure this with a <span class="caps">JWT</span> cookie.
If the user does not have a proper cookie,
he’ll get a <a href="https://tools.ietf.org/html/rfc7235#section-3.1">401 unauthorized</a>
status code.</p>
<div class="highlight"><pre><span></span><span class="kr">type</span> <span class="kt">PublicAPI</span> <span class="ow">=</span> <span class="s">"api"</span> <span class="kt">:></span> <span class="s">"1.0"</span> <span class="kt">:></span> <span class="s">"login"</span> <span class="kt">:></span> <span class="kt">ReqBody</span> <span class="kt">'[JSON]</span> <span class="kt">User</span>
<span class="kt">:></span> <span class="kt">Post</span> <span class="kt">'[JSON]</span> <span class="p">(</span><span class="kt">AuthCookies</span> <span class="kt">NoContent</span><span class="p">)</span>
</pre></div>
<p>This is our entire public api, we only expose the login endpoint.
The result contains no content, but cookies.
Servant assumes all status codes are possible in every request.
Therefore we don’t have to mention the 401 status code.</p>
<div class="highlight"><pre><span></span><span class="kr">type</span> <span class="kt">AuthAPI</span> <span class="ow">=</span>
<span class="s">"api"</span> <span class="kt">:></span> <span class="s">"1.0"</span> <span class="kt">:></span> <span class="s">"me"</span> <span class="kt">:></span> <span class="kt">Get</span> <span class="kt">'[JSON]</span> <span class="kt">User</span>
<span class="kt">:<|></span> <span class="s">"api"</span> <span class="kt">:></span> <span class="s">"1.0"</span> <span class="kt">:></span> <span class="s">"users"</span> <span class="kt">:></span> <span class="kt">Get</span> <span class="kt">'[JSON]</span> <span class="p">[</span><span class="kt">User</span><span class="p">]</span>
<span class="kt">:<|></span> <span class="s">"api"</span> <span class="kt">:></span> <span class="s">"1.0"</span> <span class="kt">:></span> <span class="s">"message"</span> <span class="kt">:></span> <span class="kt">ReqBody</span> <span class="kt">'[JSON]</span> <span class="kt">Message</span>
<span class="kt">:></span> <span class="kt">Post</span> <span class="kt">'[JSON]</span> <span class="p">[</span><span class="kt">Message</span><span class="p">]</span>
</pre></div>
<p>The <code>AuthAPI</code> is similar to the <code>ServiceAPI</code>
from the <a href="https://jappieklooster.nl/fullstack-haskell-reflex-and-servant.html">previous blog post</a>,
which only contained the <code>users</code> and <code>message</code> endpoints.
Now we’ve extended it with a <code>getme</code> endpoint.
The <code>getme</code> endpoint is a hack to do auto login with cookies.
It allows us to do a request on initial page load to see
if we have the cookies or not.
Technically we shouldn’t have to do this trough a request,
but it works for version <code>0.1</code>.</p>
<p>Next we’ll implement these types into handlers:</p>
<div class="highlight"><pre><span></span><span class="nf">login</span> <span class="ow">::</span> <span class="kt">ApiSettings</span> <span class="ow">-></span> <span class="kt">User</span> <span class="ow">-></span> <span class="kt">Handler</span> <span class="p">(</span><span class="kt">AuthCookies</span> <span class="kt">NoContent</span><span class="p">)</span>
<span class="nf">login</span> <span class="n">settings</span> <span class="n">user</span> <span class="ow">=</span> <span class="kr">if</span> <span class="n">elem</span> <span class="n">user</span> <span class="n">users</span> <span class="kr">then</span> <span class="kr">do</span>
<span class="n">withCookies</span> <span class="ow"><-</span> <span class="n">liftIO</span> <span class="o">$</span>
<span class="n">acceptLogin</span> <span class="n">cookies</span> <span class="p">(</span><span class="n">jwtSettings</span> <span class="n">settings</span><span class="p">)</span> <span class="n">user</span>
<span class="n">pure</span> <span class="o">$</span> <span class="n">maybe</span> <span class="p">(</span><span class="n">clearSession</span> <span class="n">cookies</span> <span class="kt">NoContent</span><span class="p">)</span> <span class="p">(</span><span class="nf">\</span><span class="n">x</span> <span class="ow">-></span> <span class="n">x</span> <span class="kt">NoContent</span><span class="p">)</span>
<span class="n">withCookies</span>
<span class="kr">else</span> <span class="n">throwAll</span> <span class="n">err401</span> <span class="c1">-- unauthorized</span>
<span class="kr">where</span>
<span class="n">cookies</span> <span class="ow">=</span> <span class="n">cookieSettings</span> <span class="n">settings</span>
</pre></div>
<p>We added a new login handler, which checks if the user
exists within the users list.
If the user is in the list, we use <code>acceptLogin</code> to create a <span class="caps">JWT</span> from the user.
<code>[A]cceptLogin</code> returns maybe a function which applies the <span class="caps">JWT</span> cookie.
In the success branch of maybe we apply this function to <code>NoContent</code> to get an <code>AuthCookies NoContent</code>.
The <code>Nothing</code> branch also produces <code>AuthCookies NoConent</code>,
but it sets the cookie values with <code>clearSession</code> resulting in nothing instead of a <span class="caps">JWT</span>.</p>
<p>The <code>ApiSettings</code> is just a data type with various configurations:</p>
<div class="highlight"><pre><span></span><span class="kr">data</span> <span class="kt">ApiSettings</span> <span class="ow">=</span> <span class="kt">ApiSettings</span>
<span class="p">{</span> <span class="n">cookieSettings</span> <span class="ow">::</span> <span class="kt">CookieSettings</span>
<span class="p">,</span> <span class="n">jwtSettings</span> <span class="ow">::</span> <span class="kt">JWTSettings</span>
<span class="p">,</span> <span class="n">connection</span> <span class="ow">::</span> <span class="kt">Connection</span>
<span class="p">}</span>
</pre></div>
<p>You can’t use connection like this in production,
it needs to be a <a href="http://hackage.haskell.org/package/resource-pool-0.2.3.2/docs/Data-Pool.html">pool</a>,
because servant is <a href="https://www.reddit.com/r/haskell/comments/4tq2q0/does_servant_run_on_multicore_cpu/">fully concurrent</a>.
You’ll end up with data races if you use a plain connection.</p>
<p>For the <code>cookieSettings</code> I modified the defaults quite a bit:</p>
<div class="highlight"><pre><span></span><span class="nf">cookieConf</span> <span class="ow">=</span>
<span class="n">defaultCookieSettings</span>
<span class="p">{</span> <span class="n">cookieIsSecure</span> <span class="ow">=</span> <span class="kt">NotSecure</span>
<span class="p">,</span> <span class="n">cookieMaxAge</span> <span class="ow">=</span> <span class="kt">Just</span> <span class="o">$</span> <span class="n">secondsToDiffTime</span> <span class="o">$</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">24</span> <span class="o">*</span> <span class="mi">365</span>
<span class="p">,</span> <span class="n">cookieXsrfSetting</span> <span class="ow">=</span> <span class="kt">Just</span> <span class="o">$</span>
<span class="n">def</span> <span class="p">{</span> <span class="n">xsrfCookieName</span> <span class="ow">=</span> <span class="kt">Text</span><span class="o">.</span><span class="n">encodeUtf8</span> <span class="n">cookieName</span>
<span class="p">,</span> <span class="n">xsrfHeaderName</span> <span class="ow">=</span> <span class="kt">Text</span><span class="o">.</span><span class="n">encodeUtf8</span> <span class="n">headerName</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>Cookies are set to <code>NotSecure</code> to allow it to work on <span class="caps">HTTP</span>.
This is required for local testing,
and avoids confusion about why your
cookies, and your entire login system, don’t work locally.
You should simply disable <span class="caps">HTTP</span> in production anyway.
There is no good reason for using plain <span class="caps">HTTP</span> on a live website,
ever since <a href="https://letsencrypt.org/">let’s encrypt</a>
became a thing.</p>
<p>The max age is simply an auto sign out after a period,
a year in this case.
This is a bit more secure because we don’t trust
someone’s login forever.</p>
<p><span class="caps">XSRF</span> settings are set to use the names from the Common <span class="caps">XSRF</span> module.
This ensures the requests from the frontend use the same names as
servant-auth expects.</p>
<div class="highlight"><pre><span></span><span class="nf">authenticatedServer</span> <span class="ow">::</span> <span class="kt">ApiSettings</span> <span class="ow">-></span> <span class="kt">AuthResult</span> <span class="kt">User</span> <span class="ow">-></span> <span class="kt">Server</span> <span class="kt">AuthAPI</span>
<span class="nf">authenticatedServer</span> <span class="n">settings</span> <span class="p">(</span><span class="kt">Authenticated</span> <span class="n">user</span><span class="p">)</span> <span class="ow">=</span>
<span class="p">(</span><span class="n">pure</span> <span class="n">user</span> <span class="kt">:<|></span> <span class="n">pure</span> <span class="n">users</span> <span class="kt">:<|></span> <span class="n">messages</span> <span class="p">(</span><span class="n">connection</span> <span class="n">settings</span><span class="p">))</span>
<span class="nf">authenticatedServer</span> <span class="kr">_</span> <span class="kr">_</span> <span class="ow">=</span> <span class="n">throwAll</span> <span class="n">err401</span> <span class="c1">-- unauthorized</span>
</pre></div>
<p>The <code>authenticatedServer</code> handles the endpoints for the authenticated <span class="caps">API</span>.
The only new one is <code>getMe</code>, which just returns the authenticated user.
All authenticated endpoints now have access to the user who was
authenticated. This user is decoded from the <span class="caps">JWT</span> by servant-auth.</p>
<p>We get an <code>AuthResult</code> from servant-auth-server to work with.
If the user is authenticated, we give access to the <span class="caps">API</span>,
if not we return a 401 status code.
This is done manually, which means the 401 response is not mandatory.</p>
<div class="highlight"><pre><span></span><span class="nf">server</span> <span class="ow">::</span> <span class="kt">ApiSettings</span> <span class="ow">-></span> <span class="kt">FilePath</span> <span class="ow">-></span> <span class="kt">Server</span> <span class="kt">Webservice</span>
<span class="nf">server</span> <span class="n">settings</span> <span class="n">staticFolder</span> <span class="ow">=</span>
<span class="p">(</span><span class="n">login</span> <span class="n">settings</span> <span class="kt">:<|></span> <span class="n">authenticatedServer</span> <span class="n">settings</span><span class="p">)</span>
<span class="kt">:<|></span> <span class="n">serveDirectoryFileServer</span> <span class="n">staticFolder</span>
</pre></div>
<p>The server function now has to split our <span class="caps">API</span> on authenticated and public parts.
It’s similar to the previous blog post.
We still serve the static folder for testing.</p>
<div class="highlight"><pre><span></span><span class="nf">app</span> <span class="ow">::</span> <span class="kt">ApiSettings</span> <span class="ow">-></span> <span class="kt">FilePath</span> <span class="ow">-></span> <span class="kt">Application</span>
<span class="nf">app</span> <span class="n">settings</span> <span class="n">staticFolder</span> <span class="ow">=</span>
<span class="n">serveWithContext</span> <span class="n">webservice</span> <span class="n">context</span> <span class="o">$</span> <span class="n">server</span> <span class="n">settings</span> <span class="n">staticFolder</span>
<span class="kr">where</span>
<span class="n">context</span> <span class="ow">=</span> <span class="n">cookieSettings</span> <span class="n">settings</span> <span class="kt">:.</span> <span class="n">jwtSettings</span> <span class="n">settings</span> <span class="kt">:.</span> <span class="kt">EmptyContext</span>
</pre></div>
<p>We now serve with context,
this is the servant-auth entry point for
decoding of the <span class="caps">JWT</span> from the Cookie.</p>
<p>This is probably not the way you want to do login on the sever side
for the following reasons:</p>
<ul>
<li>We don’t handle passwords, period. (Authentication by trust is a thing?)</li>
<li>One shouldn’t use <span class="caps">JWT</span>’s for <a href="http://cryto.net/~joepie91/blog/2016/06/13/stop-using-jwt-for-sessions/">sessions</a></li>
</ul>
<p>Solving these issues is out of the scope of this article<sup id="fnref-scope"><a class="footnote-ref" href="#fn-scope">1</a></sup>.</p>
<p>The client <span class="caps">API</span> is much simpler:</p>
<div class="highlight"><pre><span></span><span class="nf">postLogin</span> <span class="ow">::</span> <span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">Dynamic</span> <span class="n">t</span> <span class="p">(</span><span class="kt">Either</span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span> <span class="kt">User</span><span class="p">)</span>
<span class="ow">-></span> <span class="kt">Event</span> <span class="n">t</span> <span class="nb">()</span>
<span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Event</span> <span class="n">t</span> <span class="p">(</span><span class="kt">ReqResult</span> <span class="nb">()</span> <span class="p">(</span><span class="kt">AuthCookies</span> <span class="kt">NoContent</span><span class="p">)))</span>
<span class="nf">getUsers</span> <span class="ow">::</span> <span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">Event</span> <span class="n">t</span> <span class="nb">()</span>
<span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Event</span> <span class="n">t</span> <span class="p">(</span><span class="kt">ReqResult</span> <span class="nb">()</span> <span class="p">[</span><span class="kt">User</span><span class="p">]))</span>
<span class="nf">getMe</span> <span class="ow">::</span> <span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">Event</span> <span class="n">t</span> <span class="nb">()</span>
<span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Event</span> <span class="n">t</span> <span class="p">(</span><span class="kt">ReqResult</span> <span class="nb">()</span> <span class="kt">User</span><span class="p">))</span>
<span class="nf">postMessage</span> <span class="ow">::</span> <span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">Dynamic</span> <span class="n">t</span> <span class="p">(</span><span class="kt">Either</span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span> <span class="kt">Message</span><span class="p">)</span>
<span class="ow">-></span> <span class="kt">Event</span> <span class="n">t</span> <span class="nb">()</span>
<span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Event</span> <span class="n">t</span> <span class="p">(</span><span class="kt">ReqResult</span> <span class="nb">()</span> <span class="p">[</span><span class="kt">Message</span><span class="p">]))</span>
<span class="p">(</span><span class="n">postLogin</span> <span class="kt">:<|></span> <span class="p">(</span><span class="n">getMe</span> <span class="kt">:<|></span> <span class="n">getUsers</span> <span class="kt">:<|></span> <span class="n">postMessage</span><span class="p">))</span> <span class="ow">=</span> <span class="n">apiClients</span>
</pre></div>
<p>The clients are still generated, the only thing I’ve done is spell
out the type signatures.</p>
<h1 id="reflex">Reflex</h1>
<p>Time for some reflex.
The reflex changes are rather small but dense:</p>
<div class="highlight"><pre><span></span><span class="nf">loginWidget</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Event</span> <span class="n">t</span> <span class="kt">User</span><span class="p">)</span>
<span class="nf">loginWidget</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">autoLoginEvt</span> <span class="ow"><-</span> <span class="n">autoLogin</span>
<span class="n">formEvt</span> <span class="ow"><-</span> <span class="n">loginForm</span>
<span class="n">pure</span> <span class="o">$</span> <span class="n">leftmost</span> <span class="p">[</span><span class="n">formEvt</span><span class="p">,</span> <span class="n">autoLoginEvt</span><span class="p">]</span>
</pre></div>
<p>This function performs either an auto login,
or shows a login form.
leftmost is a function that combines events,
by using the value of whichever fires.
If they fire both,
the element that occurs first in the list is used.
Hence the name leftmost.</p>
<p>Now let’s dive into it’s sub components,
first <code>autologin</code>:</p>
<div class="highlight"><pre><span></span><span class="nf">autoLogin</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Event</span> <span class="n">t</span> <span class="kt">User</span><span class="p">)</span>
<span class="nf">autoLogin</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">pb</span> <span class="ow"><-</span> <span class="n">getPostBuild</span>
<span class="n">withSuccess</span> <span class="o"><$></span> <span class="n">getMe</span> <span class="n">pb</span>
</pre></div>
<p>We get the <a href="https://hackage.haskell.org/package/reflex-0.5/docs/Reflex-PostBuild-Class.html">post build event</a>,
an event that fires after the widget is placed in the <span class="caps">DOM</span>.
We use that to greedily to make the <code>getMe</code> request.
If successful that is used as the resulting event.
However the loginWidget makes a login form regardless of success:</p>
<div class="highlight"><pre><span></span><span class="nf">userInput</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Dynamic</span> <span class="n">t</span> <span class="kt">User</span><span class="p">)</span>
<span class="nf">userInput</span> <span class="ow">=</span> <span class="o">...</span>
<span class="nf">loginForm</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Event</span> <span class="n">t</span> <span class="kt">User</span><span class="p">)</span>
<span class="nf">loginForm</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">user</span> <span class="ow"><-</span> <span class="n">userInput</span>
<span class="n">buttonEvt</span> <span class="ow"><-</span> <span class="n">button</span> <span class="s">"login"</span>
<span class="n">postResult</span> <span class="ow"><-</span> <span class="n">postLogin</span> <span class="p">(</span><span class="kt">Right</span> <span class="o"><$></span> <span class="n">user</span><span class="p">)</span> <span class="n">buttonEvt</span>
<span class="n">void</span> <span class="o">$</span> <span class="n">flash</span> <span class="n">postResult</span> <span class="o">$</span> <span class="n">text</span> <span class="o">.</span> <span class="kt">Text</span><span class="o">.</span><span class="n">pack</span> <span class="o">.</span> <span class="n">show</span> <span class="o">.</span> <span class="n">reqFailure</span>
<span class="n">pure</span> <span class="o">$</span> <span class="n">current</span> <span class="n">user</span> <span class="o"><@</span> <span class="n">withSuccess</span> <span class="n">postResult</span>
</pre></div>
<p>The <code>userInput</code> has remained the same as in the
<a href="https://jappieklooster.nl/fullstack-haskell-reflex-and-servant.html">previous blog post</a>.
After the <code>userInput</code> form we create a button,
which gives us a resulting <code>buttonEvt</code> event.
This event only fires if the button is pressed.
We use the <code>buttonEvt</code> to call <code>postLogin</code>.
As input we use the dynamic <code>user</code> from the <code>userInput</code> form.
This gives us a <code>postResultEvt</code>,
an event that only fires on request completion.
Remember <code>postLogin</code> is a function generated from our <span class="caps">API</span> type signature:</p>
<div class="highlight"><pre><span></span><span class="nf">postLogin</span> <span class="ow">::</span> <span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">Dynamic</span> <span class="n">t</span> <span class="p">(</span><span class="kt">Either</span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span> <span class="kt">User</span><span class="p">)</span>
<span class="ow">-></span> <span class="kt">Event</span> <span class="n">t</span> <span class="nb">()</span>
<span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Event</span> <span class="n">t</span> <span class="p">(</span><span class="kt">ReqResult</span> <span class="nb">()</span> <span class="p">(</span><span class="kt">AuthCookies</span> <span class="kt">NoContent</span><span class="p">)))</span>
</pre></div>
<p>The <code>ReqResult</code> is a container for dealing with <span class="caps">HTTP</span> status codes,
connection errors and decoding issues <sup id="fnref-clearcache"><a class="footnote-ref" href="#fn-clearcache">2</a></sup>.
In case of failure the <code>postResultEvt</code> is ‘flashed’,
or shown briefly for a couple of seconds.
In case of success we tag the <code>postResultEvt</code> with the <code>user</code>
and use that as resulting event.</p>
<p>Note that <code><@</code>, is the same as <code><$</code>, except it works on behaviors:
The event gets the value of whatever the behavior is at the time of the event.
A behavior is something that always has a value, but which can
change at any moment.
Whereas an event is something that happens at a point in time with some
value.
A mouse position is an example of behavior,
whereas a mouse click is an event<sup id="fnref-dynamics"><a class="footnote-ref" href="#fn-dynamics">3</a></sup>.</p>
<p>Next we move onto the function that ties everything together:</p>
<div class="highlight"><pre><span></span><span class="nf">reflex</span> <span class="ow">::</span> <span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span> <span class="ow">=></span> <span class="n">m</span> <span class="nb">()</span>
<span class="nf">reflex</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">rec</span> <span class="n">loginEvt</span> <span class="ow"><-</span> <span class="n">elDynAttr</span> <span class="s">"div"</span> <span class="n">loginAttr</span> <span class="n">loginWidget</span>
<span class="n">loginAttr</span> <span class="ow"><-</span> <span class="n">holdDyn</span> <span class="p">(</span><span class="kt">Map</span><span class="o">.</span><span class="n">empty</span><span class="p">)</span> <span class="o">$</span> <span class="n">hidden</span> <span class="o"><$</span> <span class="n">loginEvt</span>
<span class="n">void</span> <span class="o">$</span> <span class="n">holdEvent</span> <span class="nb">()</span> <span class="n">loginEvt</span> <span class="n">authenticatedWidget</span>
</pre></div>
<p>Once the login event happens
this function will hide the <code>loginWidget</code>
and put the <code>authenticatedWidget</code> on the <span class="caps">DOM</span>.
<code>loginAttr</code> is used before it’s assigned however.
This is impossible in a normal do block,
but it is possible within a <code>rec</code> block.
The <code>rec</code> keyword is from <a href="https://wiki.haskell.org/MonadFix">recursive do</a>.
It allows referencing of variables ‘higher’ in a do block.
In our case we need to have <code>loginAttr</code> within the <code>elDynAttr</code> function.</p>
<p>I don’t want to go into how <code>rec</code> works<sup id="fnref-howitworks"><a class="footnote-ref" href="#fn-howitworks">5</a></sup>,
but I do want to make clear why it’s needed.
There is a reference cycle<sup id="fnref-breakcycle"><a class="footnote-ref" href="#fn-breakcycle">4</a></sup>.
Look closely at <code>loginAttr</code>.
It depends on <code>loginEvt</code>.
Now look at how the <code>loginEvt</code> is made.
It comes from a div that requires <code>loginAttr</code>.
A cycle.
I don’t know of any other way to solve this than using
recursive do.</p>
<p>So the login widget resides in a <code>div</code> with dynamic attributes.
These attributes are set on the next line,
which starts out as an empty Map, no attributes.
Once the login event happens, it becomes the <code>hidden</code>,
which sets the style to <code>display:none</code>.</p>
<p><code>holdEvent</code> is used to extract the user as a value from the event
and render <code>authenticatedWidget</code> as a new part of the <span class="caps">DOM</span>.
The <code>holdEvent</code> functions is a convenience function for
<a href="https://hackage.haskell.org/package/reflex-dom-core-0.4/docs/Reflex-Dom-Widget-Basic.html#v:widgetHold">widgetHold</a>:</p>
<div class="highlight"><pre><span></span><span class="nf">widgetHold</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">DomBuilder</span> <span class="n">t</span> <span class="n">m</span><span class="p">,</span> <span class="kt">MonadHold</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span>
<span class="ow">=></span> <span class="n">m</span> <span class="n">a</span>
<span class="ow">-></span> <span class="kt">Event</span> <span class="n">t</span> <span class="p">(</span><span class="n">m</span> <span class="n">a</span><span class="p">)</span>
<span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Dynamic</span> <span class="n">t</span> <span class="n">a</span><span class="p">)</span>
<span class="nf">holdEvent</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">Dom</span><span class="o">.</span><span class="kt">DomBuilder</span> <span class="n">t</span> <span class="n">m</span><span class="p">,</span> <span class="kt">MonadHold</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span>
<span class="ow">=></span> <span class="n">b</span>
<span class="ow">-></span> <span class="kt">Event</span> <span class="n">t</span> <span class="n">a</span>
<span class="ow">-></span> <span class="p">(</span><span class="n">a</span> <span class="ow">-></span> <span class="n">m</span> <span class="n">b</span><span class="p">)</span>
<span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Dynamic</span> <span class="n">t</span> <span class="n">b</span><span class="p">)</span>
<span class="nf">holdEvent</span> <span class="n">val</span> <span class="n">evt</span> <span class="n">fun</span> <span class="ow">=</span>
<span class="kt">Dom</span><span class="o">.</span><span class="n">widgetHold</span> <span class="p">(</span><span class="n">pure</span> <span class="n">val</span><span class="p">)</span> <span class="o">$</span> <span class="n">fun</span> <span class="o"><$></span> <span class="n">evt</span>
</pre></div>
<p><code>widgetHold</code> will show the first given widget,
until the event happens which has a widget as value.
Then the widget within that event is put onto the <span class="caps">DOM</span>
instead of the original.
It’s a bit like <a href="http://hackage.haskell.org/package/base-4.12.0.0/docs/Data-Traversable.html#v:sequence">sequence</a><sup id="fnref-lookingfor"><a class="footnote-ref" href="#fn-lookingfor">6</a></sup>.
In any case it returns the widget value as a dynamic.</p>
<p><code>holdEvent</code> however assumes we initially don’t want to render
anything on the <span class="caps">DOM</span>.
Then it asks you to provide an event with any value and finally
a function that consumes the value to produce the widget.
It will execute the function and display the resulting widget
on the <span class="caps">DOM</span> instead of nothing.</p>
<p>The first argument of <code>holdEvent</code> is the default value.
The second argument is the event which we want to hold.
The final argument is the function we want to execute producing a widget.
The function keeps returning the default value
until the event fires for the first time,
then it will keep on displaying the fired event.</p>
<p>Note that <code>widgetHold</code> is slow <sup id="fnref-dynSlow"><a class="footnote-ref" href="#fn-dynSlow">8</a></sup> because it modifies the <span class="caps">DOM</span> <sup id="fnref-heretics"><a class="footnote-ref" href="#fn-heretics">9</a></sup>.
It’s much better to use <a href="https://hackage.haskell.org/package/reflex-dom-core-0.4/docs/Reflex-Dom-Widget-Basic.html#v:dynText">dynText</a>
and <a href="https://hackage.haskell.org/package/reflex-dom-core-0.4/docs/Reflex-Dom-Widget-Basic.html#v:elDynAttr">elDynAttr</a>
to modify the dom/layout.
However, <code>widgetHold</code> is really convenient to get
access to values within events.
I also think that the parts inside a widgetHold function
don’t get evaluated until the event occurs.
This is really convenient for login.
Now you don’t have to evaluate the bulk of your app on initial page load.
<code>widgetHold</code> can postpone evaluating large parts of your app.
Which makes that initial render much faster<sup id="fnref-codeSplitting"><a class="footnote-ref" href="#fn-codeSplitting">7</a></sup>.</p>
<p>Anyway as we can see from the type signature,
in this case the <code>b ~ ()</code>, and the <code>a ~ User</code>.
Which leads us to authenticatedWidget:</p>
<div class="highlight"><pre><span></span><span class="nf">authenticatedWidget</span> <span class="ow">::</span> <span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span> <span class="ow">=></span> <span class="kt">User</span> <span class="ow">-></span> <span class="n">m</span> <span class="nb">()</span>
<span class="nf">authenticatedWidget</span> <span class="n">user</span> <span class="ow">=</span>
<span class="n">el</span> <span class="s">"div"</span> <span class="o">$</span> <span class="kr">do</span>
<span class="n">getUsersWidget</span>
<span class="n">sendMsgWidget</span> <span class="n">user</span>
</pre></div>
<p>This is the same as the app discussed in the previous blog post.
Although now we use the logged in user to send messages.</p>
<h1 id="xsrf"><span class="caps">XSRF</span></h1>
<p>To make servant reflex work nicely with servant-auth we need
to modify the requests a bit, servant reflex supports this with
<code>ClientOptions</code>:</p>
<div class="highlight"><pre><span></span><span class="nf">clientOpts</span> <span class="ow">::</span> <span class="kt">ClientOptions</span>
<span class="nf">clientOpts</span> <span class="ow">=</span> <span class="kt">ClientOptions</span> <span class="o">$</span> <span class="n">tweakReq</span>
<span class="kr">where</span>
<span class="n">tweakReq</span> <span class="n">r</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">mayCookie</span> <span class="ow"><-</span> <span class="n">findCookie</span> <span class="n">cookieName</span>
<span class="n">return</span> <span class="o">$</span> <span class="n">r</span> <span class="o">&</span> <span class="n">headerMod</span> <span class="n">headerName</span> <span class="o">.~</span> <span class="n">mayCookie</span> <span class="c1">-- forgive lenses</span>
<span class="n">headerMod</span> <span class="n">d</span> <span class="ow">=</span> <span class="n">xhrRequest_config</span> <span class="o">.</span> <span class="n">xhrRequestConfig_headers</span> <span class="o">.</span> <span class="n">at</span> <span class="n">d</span>
<span class="nf">apiClients</span> <span class="ow">::</span> <span class="n">forall</span> <span class="n">t</span> <span class="n">m</span><span class="o">.</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kr">_</span>
<span class="nf">apiClients</span> <span class="ow">=</span> <span class="n">clientWithOpts</span>
<span class="n">serviceAPI</span> <span class="p">(</span><span class="kt">Proxy</span> <span class="o">@</span><span class="n">m</span><span class="p">)</span> <span class="p">(</span><span class="kt">Proxy</span> <span class="o">@</span><span class="nb">()</span><span class="p">)</span> <span class="p">(</span><span class="n">constDyn</span> <span class="n">url</span><span class="p">)</span> <span class="n">clientOpts</span>
</pre></div>
<p>The client options lives in the <code>JSM</code> monad and gives us an opportunity
to modify the <code>XHRRequest</code> how we want.
We make sure the names are the same by using the ones defined
in the common module.</p>
<h1 id="conclusion">Conclusion</h1>
<p>So there you have it. Authentication.
Not the most exciting thing in the world,
but once this is done you can start making something cool.
I hope I helped you get trough this ordeal fast,
and explain some of the finer reflex points.
Now I hope to see many cool reflex projects popping up.
<span class="caps">PM</span> me your cool projects.</p>
<h1 id="references">References</h1>
<p>With the release of reflex <code>0.5</code> we now have updated docs!</p>
<ul>
<li><a href="https://github.com/jappeace/awesome-project-name/tree/auth">Source code</a></li>
<li><a href="https://jappieklooster.nl/fullstack-haskell-reflex-and-servant.html">Previous blog</a></li>
<li><a href="https://hackage.haskell.org/package/reflex">Reflex</a></li>
<li><a href="https://hackage.haskell.org/package/reflex-dom-core-0.4">Reflex dom</a></li>
<li><a href="http://hackage.haskell.org/package/servant-reflex-0.3.4">Servant reflex</a></li>
<li><a href="https://wiki.haskell.org/MonadFix">MonadFix</a></li>
<li><a href="http://hackage.haskell.org/package/base-4.12.0.0/docs/Control-Monad-Fix.html">Control.Monad.Fix</a></li>
<li><a href="https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/glasgow_exts.html#the-recursive-do-notation">Recursive do language extension</a></li>
<li><a href="http://hackage.haskell.org/package/reflex-dom-core-0.4/docs/Reflex-Dom-Xhr.html">Reflex dom <span class="caps">XHR</span></a></li>
</ul>
<div class="footnote">
<hr>
<ol>
<li id="fn-scope">
<p>My articles tend to snowball anyway, for example invented how to do <span class="caps">XSRF</span> specifically for this article.
I Don’t want to cargo cult a bunch of <span class="caps">XSRF</span> vulnerable websites. <a class="footnote-backref" href="#fnref-scope" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-clearcache">
<p>What I’m doing in production is a clear cache refres on any
4xx status code and decode/encoding errors.
This automatically will fix any stale clients without any mental overhead.
I just attach a monad to most api calls which does that. <a class="footnote-backref" href="#fnref-clearcache" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-dynamics">
<p>For completeness:
The <code>userInput</code> form returns a dynamic user,
which is both a behavior as well as an event:
the event fires whenever the behavior changes value. <a class="footnote-backref" href="#fnref-dynamics" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn-breakcycle">
<p>I currently believe that
in reflex the
cycle can be broken because we start with the preposition that a
dynamic and behavior always have a value.
Even in this case we can just set it to empty, until the event fires,
which is by definition after rendering.
This also explains why you use a sampled value from a behavior in a <code>rec</code>
block. <a class="footnote-backref" href="#fnref-breakcycle" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn-howitworks">
<p><a href="http://hackage.haskell.org/package/base-4.12.0.0/docs/Control-Monad-Fix.html">This</a>
explains how it works, note the thesis on value recursion too. <a class="footnote-backref" href="#fnref-howitworks" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn-lookingfor">
<p>sequence doesn’t work because event isn’t foldable,
and it will never be foldable because that breaks <span class="caps">FRP</span> semantics. <a class="footnote-backref" href="#fnref-lookingfor" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn-codeSplitting">
<p>I still think reflex load times are too slow, especially on mobile.
I know about the app option, but an app is just a dirty hack to get your crappy website to speed up.
It’s much better to have everything as a website, it results in less maintenance and better <span class="caps">UX</span>
(if you can pull it off).
Nobody should have to install anything in 2019.
But the mobile web will remain slow for
<a href="https://www.youtube.com/watch?v=4bZvq3nodf4">good reasons</a>
and if I knew how I’d happily help speeding up <span class="caps">GHCJS</span> and reflex.
I think for example that widgetHold and dyn may be a good candidates for code
splitting entry points.
But I believe you’d need to make the compiler aware of that somehow.
I also believe you probably don’t need the entire Haskell runtime immediately,
lazily loading exceptions would be good for example. <a class="footnote-backref" href="#fnref-codeSplitting" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn-dynSlow">
<p>And <a href="https://hackage.haskell.org/package/reflex-dom-core-0.4/docs/Reflex-Dom-Widget-Basic.html#v:dyn">dyn</a> for that matter <a class="footnote-backref" href="#fnref-dynSlow" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn-heretics">
<p>Even though some <a href="https://www.reddit.com/r/javascript/comments/6115ay/why_do_developers_think_the_dom_is_slow/">reddit thread</a>
disagrees with me thinking the <span class="caps">DOM</span> is slow and saying it’s because of
<a href="https://gist.github.com/paulirish/5d52fb081b3570c81e3a">layout trashing</a>.
Heretics. <a class="footnote-backref" href="#fnref-heretics" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
</ol>
</div>Lens into wrapped newtypes2018-12-30T21:34:00+01:002018-12-30T21:34:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2018-12-30:/lens-into-wrapped-newtypes.html<p><img alt="Categorical representation of the NT iso" src="/images/2018/nt-iso.svg"></p>
<blockquote>
<p>All newtypes are isomorphisms</p>
<p><br /></p>
<p>— My mother</p>
</blockquote>
<p><a href="http://hackage.haskell.org/package/lens-4.17/docs/Control-Lens-Wrapped.html">Control.Lens.Wrapped</a>
uses the isomorphism property to introduce a type class <code>Wrapped</code>.
Let’s explore use cases, because after all, it doesn’t appear to do much at first glance.
What’s the point of formalizing wrapping and unwrapping of types?</p>
<p>Instance boilerplate …</p><p><img alt="Categorical representation of the NT iso" src="/images/2018/nt-iso.svg"></p>
<blockquote>
<p>All newtypes are isomorphisms</p>
<p><br /></p>
<p>— My mother</p>
</blockquote>
<p><a href="http://hackage.haskell.org/package/lens-4.17/docs/Control-Lens-Wrapped.html">Control.Lens.Wrapped</a>
uses the isomorphism property to introduce a type class <code>Wrapped</code>.
Let’s explore use cases, because after all, it doesn’t appear to do much at first glance.
What’s the point of formalizing wrapping and unwrapping of types?</p>
<p>Instance boilerplate will be reduced in this blog post.
In my use case this will include orphan instances.
Furthermore, I believe that this technique will make using newtypes more attractive.</p>
<h1 id="newtype">Newtype</h1>
<p>Consider the following common code in
a <a href="https://jappieklooster.nl/fullstack-haskell-reflex-and-servant.html">fullstack haskell webapp</a>:</p>
<div class="highlight"><pre><span></span><span class="kr">data</span> <span class="kt">Login</span> <span class="ow">=</span> <span class="kt">Login</span>
<span class="p">{</span> <span class="n">email</span> <span class="ow">::</span> <span class="kt">Text</span>
<span class="p">,</span> <span class="n">password</span> <span class="ow">::</span> <span class="kt">Text</span>
<span class="p">}</span>
</pre></div>
<p>Looks good? No of course not.
This common code is shared between both client and server,
therefore we should be pendantic about these record field.
We wrap common occurrences such as Text in newtypes
so we don’t accidentally mix up the fields.
This would be a better representation:</p>
<div class="highlight"><pre><span></span><span class="kr">newtype</span> <span class="kt">Email</span> <span class="ow">=</span> <span class="kt">Email</span> <span class="p">{</span> <span class="n">unEmail</span> <span class="ow">::</span> <span class="kt">Text</span> <span class="p">}</span>
<span class="kr">newtype</span> <span class="kt">Password</span> <span class="ow">=</span> <span class="kt">Password</span> <span class="p">{</span> <span class="n">unPassword</span> <span class="ow">::</span> <span class="kt">Text</span> <span class="p">}</span>
<span class="kr">data</span> <span class="kt">Login</span> <span class="ow">=</span> <span class="kt">Login</span>
<span class="p">{</span> <span class="n">email</span> <span class="ow">::</span> <span class="kt">Email</span>
<span class="p">,</span> <span class="n">password</span> <span class="ow">::</span> <span class="kt">Password</span>
<span class="p">}</span>
</pre></div>
<p>Of course this extra safety comes at the cost of more boilerplate,
but these few lines add a lot of safety:
Mixing of email and password becomes less likely,
from frontend input to the backend.
This is analogous to having integration tests on user input fields,
<span class="caps">AJAX</span> calls, <span class="caps">HTTP</span> endpoints, and database insertion.
That’s a lot of safety for two extra lines,
therefore we accept this trade and move on.</p>
<h1 id="database">Database</h1>
<p>Now we can put the fields of login directly into our
user table schema for the database:</p>
<div class="highlight"><pre><span></span><span class="c1">-- database</span>
<span class="kr">data</span> <span class="kt">UserT</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">User</span>
<span class="p">{</span> <span class="n">email</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Email</span>
<span class="p">,</span> <span class="n">password</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Password</span>
<span class="p">}</span>
<span class="c1">-- etc beam boilerplate...</span>
</pre></div>
<p>Although this is what we want, it doesn’t compile,
we need to tell beam how to get the right underlying type
so it can produce the right queries and schema:</p>
<div class="highlight"><pre><span></span><span class="c1">-- backend orphanage</span>
<span class="kr">instance</span> <span class="kt">HasSqlEqualityCheck</span> <span class="kt">PgExpressionSyntax</span> <span class="kt">Email</span>
</pre></div>
<p>This instance allows us to use the <a href="http://hackage.haskell.org/package/beam-core-0.7.2.2/docs/Database-Beam-Query.html#v:-61--61-."><code>==.</code></a>
operator on beam expressions directly.
We can now compare the column email with a client email.
I don’t know why an instance is needed for this, but the
compiler wanted it whenever I used that operator.</p>
<div class="highlight"><pre><span></span><span class="c1">-- backend orphanage</span>
<span class="kr">instance</span> <span class="kt">HasSqlValueSyntax</span> <span class="kt">PgValueSyntax</span> <span class="kt">Email</span> <span class="kr">where</span>
<span class="n">sqlValueSyntax</span> <span class="ow">=</span> <span class="n">sqlValueSyntax</span> <span class="o">.</span> <span class="n">unEmail</span>
</pre></div>
<p>Here we use <code>unEmail</code> to the underlying type their sqlValuesyntax.</p>
<div class="highlight"><pre><span></span><span class="c1">-- backend orphanage</span>
<span class="kr">instance</span> <span class="kt">FromBackendRow</span> <span class="kt">Postgres</span> <span class="kt">Email</span>
</pre></div>
<p>I believe this tells beam we want to be able to use a postgres database
on email.
We don’t have to instantiate sqlite instances
(beam can do multiple backends).</p>
<div class="highlight"><pre><span></span><span class="c1">-- backend orphanage</span>
<span class="kr">instance</span> <span class="kt">FromField</span> <span class="kt">Email</span> <span class="kr">where</span>
<span class="n">fromField</span> <span class="n">a</span> <span class="n">b</span> <span class="ow">=</span> <span class="kt">Email</span> <span class="o"><$></span> <span class="n">fromField</span> <span class="n">a</span> <span class="n">b</span>
</pre></div>
<p><a href="https://hackage.haskell.org/package/postgresql-simple-0.5.4.0/docs/Database-PostgreSQL-Simple-FromField.html#t:FromField">FromField</a>
is a type class from <a href="https://hackage.haskell.org/package/postgresql-simple-0.5.4.0/docs/Database-PostgreSQL-Simple.html">PgSimple</a>.
Here, we’re telling the compiler to just use the from field from the underlying type,
and once it’s done we can wrap it back into an email.</p>
<div class="highlight"><pre><span></span><span class="c1">-- backend orphanage</span>
<span class="kr">instance</span> <span class="kt">HasDefaultSqlDataTypeConstraints</span> <span class="kt">PgColumnSchemaSyntax</span> <span class="kt">Email</span>
</pre></div>
<p>This is a constraint coming from beam migrate,
if you want to have automatic schema generation,
or be able to step trough various schemas you need this.</p>
<div class="highlight"><pre><span></span><span class="c1">-- backend orphanage</span>
<span class="kr">instance</span> <span class="kt">HasDefaultSqlDataType</span> <span class="kt">PgDataTypeSyntax</span> <span class="kt">Email</span> <span class="kr">where</span>
<span class="n">defaultSqlDataType</span> <span class="n">proxy</span> <span class="ow">=</span> <span class="n">defaultSqlDataType</span> <span class="o">$</span> <span class="n">unEmail</span> <span class="o"><$></span> <span class="n">proxy</span>
</pre></div>
<p>This is also needed for migrations.
Here we’re removing the newtype from the proxy to tell it to use the
underlying type.
Note that a proxy isn’t holding any data,
those functions will never be executed,
in case of proxies we’re just interested in type.</p>
<p>All of this is repeated for Password to,
and any other newtypes you want:</p>
<div class="highlight"><pre><span></span><span class="c1">-- backend orphanage</span>
<span class="kr">instance</span> <span class="kt">HasSqlEqualityCheck</span> <span class="kt">PgExpressionSyntax</span> <span class="kt">Password</span>
<span class="kr">instance</span> <span class="kt">HasSqlValueSyntax</span> <span class="kt">PgValueSyntax</span> <span class="kt">Password</span> <span class="kr">where</span>
<span class="n">sqlValueSyntax</span> <span class="ow">=</span> <span class="n">sqlValueSyntax</span> <span class="o">.</span> <span class="n">unPassword</span>
<span class="kr">instance</span> <span class="kt">FromBackendRow</span> <span class="kt">Postgres</span> <span class="kt">Password</span>
<span class="kr">instance</span> <span class="kt">FromField</span> <span class="kt">Password</span> <span class="kr">where</span>
<span class="n">fromField</span> <span class="n">a</span> <span class="n">b</span> <span class="ow">=</span> <span class="kt">Password</span> <span class="o"><$></span> <span class="n">fromField</span> <span class="n">a</span> <span class="n">b</span>
<span class="kr">instance</span> <span class="kt">HasDefaultSqlDataTypeConstraints</span> <span class="kt">PgColumnSchemaSyntax</span> <span class="kt">Password</span>
<span class="kr">instance</span> <span class="kt">HasDefaultSqlDataType</span> <span class="kt">PgDataTypeSyntax</span> <span class="kt">Password</span> <span class="kr">where</span>
<span class="n">defaultSqlDataType</span> <span class="n">proxy</span> <span class="ow">=</span> <span class="n">defaultSqlDataType</span> <span class="o">$</span> <span class="n">unPassword</span> <span class="o"><$></span> <span class="n">proxy</span>
</pre></div>
<p>I put these instances into an orphanage
(dedicated file for orphan instances)
because I want to put the newtypes directly into the database.
However I don’t want our common code to be dependent on beam,
that would mean the frontend JavaScript suddenly would pull
in beam as a dependency for no reason.
We’ll eliminate the need for these orphans later.</p>
<p>What did we gain?
The ability to put these newtypes in the database,
what did we lose?
Well we now have a lot of extra boilerplate to content with.</p>
<h1 id="wrapped">Wrapped</h1>
<p>Let’s kill the boilerplate!</p>
<div class="highlight"><pre><span></span><span class="c1">-- common</span>
<span class="kr">newtype</span> <span class="kt">Email</span> <span class="ow">=</span> <span class="kt">Email</span> <span class="p">{</span> <span class="n">unEmail</span> <span class="ow">::</span> <span class="kt">Text</span> <span class="p">}</span> <span class="kr">deriving</span> <span class="kt">Generic</span>
<span class="kr">newtype</span> <span class="kt">Password</span> <span class="ow">=</span> <span class="kt">Password</span> <span class="p">{</span> <span class="n">unPassword</span> <span class="ow">::</span> <span class="kt">Text</span> <span class="p">}</span> <span class="kr">deriving</span> <span class="kt">Generic</span>
<span class="kr">instance</span> <span class="kt">Wrapped</span> <span class="kt">Email</span>
<span class="kr">instance</span> <span class="kt">Wrapped</span> <span class="kt">Password</span>
</pre></div>
<p>If you can provide an <a href="http://hackage.haskell.org/package/lens-4.17/docs/Control-Lens-Iso.html#t:Iso-39-">Iso’</a>,
then you instantiate the Wrapped type class.
If a newtype has derived generic we get an instance for free by just declaring it, and using the default.
This is possible because generic knows about constructors.</p>
<div class="highlight"><pre><span></span><span class="c1">-- backend orphanage</span>
<span class="nf">wrappedSqlValueSyntax</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">Wrapped</span> <span class="n">a</span><span class="p">,</span> <span class="kt">HasSqlValueSyntax</span> <span class="n">b</span> <span class="p">(</span><span class="kt">Unwrapped</span> <span class="n">a</span><span class="p">))</span> <span class="ow">=></span> <span class="n">a</span> <span class="ow">-></span> <span class="n">b</span>
<span class="nf">wrappedSqlValueSyntax</span> <span class="ow">=</span> <span class="n">sqlValueSyntax</span> <span class="o">.</span> <span class="n">view</span> <span class="n">_Wrapped'</span>
<span class="nf">fromWrappedField</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">Wrapped</span> <span class="n">a</span><span class="p">,</span> <span class="kt">FromField</span> <span class="p">(</span><span class="kt">Unwrapped</span> <span class="n">a</span><span class="p">))</span> <span class="ow">=></span> <span class="kt">FieldParser</span> <span class="n">a</span>
<span class="nf">fromWrappedField</span> <span class="n">a</span> <span class="n">b</span> <span class="ow">=</span> <span class="n">review</span> <span class="n">_Wrapped'</span> <span class="o"><$></span> <span class="n">fromField</span> <span class="n">a</span> <span class="n">b</span>
<span class="nf">wrappedDefaultSqlDataType</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">Wrapped</span> <span class="n">a</span><span class="p">,</span> <span class="kt">HasDefaultSqlDataType</span> <span class="n">b</span> <span class="p">(</span><span class="kt">Unwrapped</span> <span class="n">a</span><span class="p">))</span> <span class="ow">=></span> <span class="kt">Proxy</span> <span class="n">a</span> <span class="ow">-></span> <span class="kt">Bool</span> <span class="ow">-></span> <span class="n">b</span>
<span class="nf">wrappedDefaultSqlDataType</span> <span class="n">proxy</span> <span class="ow">=</span> <span class="n">defaultSqlDataType</span> <span class="o">$</span> <span class="n">view</span> <span class="n">_Wrapped'</span> <span class="o"><$></span> <span class="n">proxy</span>
</pre></div>
<p>These functions pull out the essence of wrapping.
If <code>a</code> is wrapped, we can speak about it’s unwrapped form (which is why we need the type class).
If <code>a</code> his unwrapped form for example implements <code>FromField</code>, we can make a <code>FieldParser</code> for it.</p>
<div class="highlight"><pre><span></span><span class="c1">-- backend orphanage</span>
<span class="kr">instance</span> <span class="kt">HasSqlEqualityCheck</span> <span class="kt">PgExpressionSyntax</span> <span class="kt">Email</span>
<span class="kr">instance</span> <span class="kt">HasSqlValueSyntax</span> <span class="kt">PgValueSyntax</span> <span class="kt">Email</span> <span class="kr">where</span>
<span class="n">sqlValueSyntax</span> <span class="ow">=</span> <span class="n">wrappedSqlValueSyntax</span>
<span class="kr">instance</span> <span class="kt">FromBackendRow</span> <span class="kt">Postgres</span> <span class="kt">Email</span>
<span class="kr">instance</span> <span class="kt">FromField</span> <span class="kt">Email</span> <span class="kr">where</span>
<span class="n">fromField</span> <span class="ow">=</span> <span class="n">fromWrappedField</span>
<span class="kr">instance</span> <span class="kt">HasDefaultSqlDataTypeConstraints</span> <span class="kt">PgColumnSchemaSyntax</span> <span class="kt">Email</span>
<span class="kr">instance</span> <span class="kt">HasDefaultSqlDataType</span> <span class="kt">PgDataTypeSyntax</span> <span class="kt">Email</span> <span class="kr">where</span>
<span class="n">defaultSqlDataType</span> <span class="ow">=</span> <span class="n">wrappedDefaultSqlDataType</span>
<span class="kr">instance</span> <span class="kt">HasSqlEqualityCheck</span> <span class="kt">PgExpressionSyntax</span> <span class="kt">Password</span>
<span class="kr">instance</span> <span class="kt">HasSqlValueSyntax</span> <span class="kt">PgValueSyntax</span> <span class="kt">Password</span> <span class="kr">where</span>
<span class="n">sqlValueSyntax</span> <span class="ow">=</span> <span class="n">wrappedSqlValueSyntax</span>
<span class="kr">instance</span> <span class="kt">FromBackendRow</span> <span class="kt">Postgres</span> <span class="kt">Password</span>
<span class="kr">instance</span> <span class="kt">FromField</span> <span class="kt">Password</span> <span class="kr">where</span>
<span class="n">fromField</span> <span class="ow">=</span> <span class="n">fromWrappedField</span>
<span class="kr">instance</span> <span class="kt">HasDefaultSqlDataTypeConstraints</span> <span class="kt">PgColumnSchemaSyntax</span> <span class="kt">Password</span>
<span class="kr">instance</span> <span class="kt">HasDefaultSqlDataType</span> <span class="kt">PgDataTypeSyntax</span> <span class="kt">Password</span> <span class="kr">where</span>
<span class="n">defaultSqlDataType</span> <span class="ow">=</span> <span class="n">wrappedDefaultSqlDataType</span>
</pre></div>
<p>The instances themselves do the same thing over and over,
they wrap or unwrap types to get the underlying interesting value.
Here this is obvious by having the instances of both
Email and Password point to the same functions.</p>
<p>At first glance, this implementation does not look better.
However we now can clearly see that the wrapping is indeed
the same operation because the instances all point toward the same
functions.
The FromField instance for Password is implemented with fromWrappedField,
so does the Email instance.
This is possible because both Email and Password have instantiated the
Wrapped instance.</p>
<p>This change is a lot better if you consider that there is no more
logic being repeated here.
Which means there are no more logic bugs in the repetition.
The boilerplate is now in it’s purest form: Dumb repetition.
By itself I wouldn’t consider the current state to be that bad anymore.
However, these are still orphans,
which can cause <a href="https://stackoverflow.com/questions/3079537/orphaned-instances-in-haskell">bad problems</a>.
We should kill all orphans!</p>
<h1 id="a-general-instance">A general instance</h1>
<p>Can’t we make a generalized instance that does all of this wrapping
for all newtypes?
My first attempt was rather crazy looking back.
I wanted to create the ultimate orphan.
a polymorphic instance that was kept in check by constraints
such as <code>Wrapped</code> and the fact underlying types would have
these beam instances implemented.
I attempted this but did got very far, I wasn’t very sure what was
going on anymore with the type errors I got out of that.
But after a bit of searching I realized that what I attempted
to do was ridiculous and dangerous.
Of course that wouldn’t work, now all previous and future <code>Wrapped</code>
newtypes would have to be able to be fit into postgres or fail.
This piece of code would break all existing libraries that would’ve
had a <code>Wrapped</code> newtype.
No this was an absurd idea.</p>
<p>Rather than solving the problem for all newtypes,
I stepped back, and added yet another newtype:</p>
<div class="highlight"><pre><span></span><span class="c1">-- wrapped.hs</span>
<span class="kr">newtype</span> <span class="kt">DBFieldWrap</span> <span class="n">a</span> <span class="ow">=</span> <span class="kt">DBFieldWrap</span>
<span class="p">{</span> <span class="n">_unField</span> <span class="ow">::</span> <span class="n">a</span>
<span class="p">}</span> <span class="kr">deriving</span> <span class="p">(</span><span class="kt">Generic</span><span class="p">,</span> <span class="kt">Show</span><span class="p">)</span>
<span class="kr">instance</span> <span class="kt">Wrapped</span> <span class="n">a</span> <span class="ow">=></span> <span class="kt">Wrapped</span> <span class="p">(</span><span class="kt">DBFieldWrap</span> <span class="n">a</span><span class="p">)</span>
<span class="kr">instance</span> <span class="p">(</span><span class="kt">Wrapped</span> <span class="n">a</span><span class="p">,</span> <span class="kt">BeamBackend</span> <span class="n">be</span><span class="p">,</span>
<span class="kt">BackendFromField</span> <span class="n">be</span> <span class="p">(</span><span class="kt">DBFieldWrap</span> <span class="n">a</span><span class="p">),</span>
<span class="kt">FromBackendRow</span> <span class="n">be</span> <span class="p">(</span><span class="kt">Unwrapped</span> <span class="n">a</span><span class="p">)</span>
<span class="p">)</span> <span class="ow">=></span>
<span class="kt">FromBackendRow</span> <span class="n">be</span> <span class="p">(</span><span class="kt">DBFieldWrap</span> <span class="n">a</span><span class="p">)</span>
</pre></div>
<p>That final instance shows the core idea.
We add the wrapped restriction on <code>a</code>,
which allows us to speak about the unwrapped form of <code>a</code>.
The unwrapped type of <code>Email</code> would be Text.
Beam has already made a FromField instance for Text,
so we’re done.</p>
<div class="highlight"><pre><span></span><span class="kr">instance</span> <span class="p">(</span> <span class="kt">IsSql92ExpressionSyntax</span> <span class="n">be</span>
<span class="p">,</span> <span class="kt">Wrapped</span> <span class="n">a</span>
<span class="p">,</span> <span class="kt">HasSqlEqualityCheck</span> <span class="n">be</span> <span class="p">(</span><span class="kt">Unwrapped</span> <span class="n">a</span><span class="p">)</span>
<span class="p">)</span> <span class="ow">=></span>
<span class="kt">HasSqlEqualityCheck</span> <span class="n">be</span> <span class="p">(</span><span class="kt">DBFieldWrap</span> <span class="n">a</span><span class="p">)</span>
<span class="kr">instance</span> <span class="p">(</span><span class="kt">Wrapped</span> <span class="n">a</span><span class="p">,</span> <span class="kt">FromField</span> <span class="p">(</span><span class="kt">Unwrapped</span> <span class="n">a</span><span class="p">))</span> <span class="ow">=></span> <span class="kt">FromField</span> <span class="p">(</span><span class="kt">DBFieldWrap</span> <span class="n">a</span><span class="p">)</span> <span class="kr">where</span>
<span class="n">fromField</span> <span class="n">a</span> <span class="n">b</span> <span class="ow">=</span> <span class="n">review</span> <span class="p">(</span><span class="n">_Wrapped'</span> <span class="o">.</span> <span class="n">_Wrapped'</span><span class="p">)</span> <span class="o"><$></span> <span class="n">fromField</span> <span class="n">a</span> <span class="n">b</span>
</pre></div>
<p>The <a href="http://hackage.haskell.org/package/lens-4.17/docs/Control-Lens-Review.html#v:review">review function</a>
just calls the constructor.
We have to call two <code>_Wrapped'</code>s with it because we need to put
it in the <code>Email</code> or <code>Password</code>, and then we need to put it into
the <code>DBField</code>.</p>
<div class="highlight"><pre><span></span><span class="kr">instance</span> <span class="p">(</span><span class="kt">Wrapped</span> <span class="n">a</span><span class="p">,</span> <span class="kt">HasSqlValueSyntax</span> <span class="n">be</span> <span class="p">(</span><span class="kt">Unwrapped</span> <span class="n">a</span><span class="p">))</span> <span class="ow">=></span>
<span class="kt">HasSqlValueSyntax</span> <span class="n">be</span> <span class="p">(</span><span class="kt">DBFieldWrap</span> <span class="n">a</span><span class="p">)</span> <span class="kr">where</span>
<span class="n">sqlValueSyntax</span> <span class="ow">=</span> <span class="n">sqlValueSyntax</span> <span class="o">.</span> <span class="n">view</span> <span class="p">(</span><span class="n">_Wrapped'</span> <span class="o">.</span> <span class="n">_Wrapped'</span><span class="p">)</span>
</pre></div>
<p>The view function is just an alias for <code>^.</code>, a getter.
We get the result of wrapping twice.</p>
<div class="highlight"><pre><span></span><span class="kr">instance</span> <span class="p">(</span> <span class="kt">IsSql92ColumnSchemaSyntax</span> <span class="n">be</span>
<span class="p">,</span> <span class="kt">Wrapped</span> <span class="n">a</span>
<span class="p">,</span> <span class="kt">HasDefaultSqlDataTypeConstraints</span> <span class="n">be</span> <span class="p">(</span><span class="kt">Unwrapped</span> <span class="n">a</span><span class="p">)</span>
<span class="p">)</span> <span class="ow">=></span>
<span class="kt">HasDefaultSqlDataTypeConstraints</span> <span class="n">be</span> <span class="p">(</span><span class="kt">DBFieldWrap</span> <span class="n">a</span><span class="p">)</span>
<span class="kr">instance</span> <span class="p">(</span> <span class="kt">IsSql92DataTypeSyntax</span> <span class="n">be</span>
<span class="p">,</span> <span class="kt">Wrapped</span> <span class="n">a</span>
<span class="p">,</span> <span class="kt">HasDefaultSqlDataType</span> <span class="n">be</span> <span class="p">(</span><span class="kt">Unwrapped</span> <span class="n">a</span><span class="p">)</span>
<span class="p">)</span> <span class="ow">=></span>
<span class="kt">HasDefaultSqlDataType</span> <span class="n">be</span> <span class="p">(</span><span class="kt">DBFieldWrap</span> <span class="n">a</span><span class="p">)</span> <span class="kr">where</span>
<span class="n">defaultSqlDataType</span> <span class="n">proxy</span> <span class="ow">=</span>
<span class="n">defaultSqlDataType</span> <span class="o">$</span> <span class="n">view</span> <span class="p">(</span><span class="n">_Wrapped'</span> <span class="o">.</span> <span class="n">_Wrapped'</span><span class="p">)</span> <span class="o"><$></span> <span class="n">proxy</span>
</pre></div>
<p>Apparently you can
<code>fmap</code> into a proxy to get the right type out.
Even though a proxy has no data, it will change type.</p>
<p>This does exactly the same thing as the independent functions
did in case of the orphanage,
the only difference is that they wrap twice.
We essentially tell the type checker to look for the beam instance
two levels deeper.
We tell it by using restrictions on the instances (the stuff before <code>=></code>).</p>
<p>Now we can insert our newtypes directly into database
without having to implement all those Beam instances:</p>
<div class="highlight"><pre><span></span><span class="c1">-- database</span>
<span class="kr">data</span> <span class="kt">UserT</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">User</span>
<span class="p">{</span> <span class="n">email</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="p">(</span><span class="kt">DBFieldWrap</span> <span class="kt">Email</span><span class="p">)</span>
<span class="p">,</span> <span class="n">password</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="p">(</span><span class="kt">DBFieldWrap</span> <span class="kt">Password</span><span class="p">)</span>
<span class="p">}</span>
<span class="c1">-- etc beam boilerplate...</span>
</pre></div>
<p>Adding an additional newtype for the database is now easy:</p>
<div class="highlight"><pre><span></span><span class="c1">-- common</span>
<span class="kr">newtype</span> <span class="kt">DateOfBirth</span> <span class="ow">=</span> <span class="kt">DateOfBirth</span> <span class="p">{</span> <span class="n">unEmail</span> <span class="ow">::</span> <span class="kt">Day</span> <span class="p">}</span> <span class="kr">deriving</span> <span class="kt">Generic</span>
<span class="kr">instance</span> <span class="kt">Wrapped</span> <span class="kt">DateOfBirth</span>
<span class="kr">data</span> <span class="kt">UserT</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">User</span>
<span class="p">{</span> <span class="n">email</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="p">(</span><span class="kt">DBFieldWrap</span> <span class="kt">Email</span><span class="p">)</span>
<span class="p">,</span> <span class="n">password</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="p">(</span><span class="kt">DBFieldWrap</span> <span class="kt">Password</span><span class="p">)</span>
<span class="p">,</span> <span class="n">dob</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="p">(</span><span class="kt">DBFieldWrap</span> <span class="kt">DateOfBirth</span><span class="p">)</span>
<span class="p">}</span>
</pre></div>
<p>Note that this technique doesn’t just work for beam instances,
one could do the same for Aeson,
or any other library that requires many instances on newtypes.
The <code>Wrapped</code> instance can be re-used.</p>
<h1 id="conclusion">Conclusion</h1>
<p>So what did we gain?</p>
<ul>
<li>We no longer have orphans!</li>
<li>Boilerplate has been reduced to just the wrapped instance.</li>
</ul>
<p>What did we lose?</p>
<ul>
<li>Unfortunately we need to unwrap and wrap at the call sites (beam queries).</li>
<li>To use this we need to depend on lens.</li>
<li>The beam schema is a little bit more verbose.</li>
</ul>
<p>Because all those negative points are really small,
I’m calling this a win.
The sources can be found on <a href="https://github.com/jappeace/dbfield">github</a> and <a href="https://hackage.haskell.org/package/beam-newtype-field">hackage</a>.</p>Fullstack Haskell: Reflex and Servant2018-10-09T12:08:00+02:002018-10-09T12:08:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2018-10-09:/fullstack-haskell-reflex-and-servant.html<p>In the <a href="/tag/pragmatic-haskell.html">pragmatic haskell</a> series, we saw how to setup a simple webserver with database.
But at some point you still need a frontend.
If it were 2005 you may have been able to get away with just <a href="http://hackage.haskell.org/package/blaze-html">blaze</a>.
But we are in 2018+, and <a href="https://wiki.haskell.org/The_JavaScript_Problem">JavaScript is a problem</a>.
In …</p><p>In the <a href="/tag/pragmatic-haskell.html">pragmatic haskell</a> series, we saw how to setup a simple webserver with database.
But at some point you still need a frontend.
If it were 2005 you may have been able to get away with just <a href="http://hackage.haskell.org/package/blaze-html">blaze</a>.
But we are in 2018+, and <a href="https://wiki.haskell.org/The_JavaScript_Problem">JavaScript is a problem</a>.
In this blog post we will explore how to deal with JavaScript trough reflex and <span class="caps">GHCJS</span>.
An alternative to consider is <a href="https://github.com/dmjio/miso">miso</a>,
which uses the <a href="https://guide.elm-lang.org/architecture/">elm architecture</a>
(or <a href="https://redux.js.org/">redux</a> if you’re from <span class="caps">JS</span>),
here is a <a href="https://www.reddit.com/r/haskell/comments/7nxni9/reflexdom_vs_miso/">comparison</a>.
Obviously I chose reflex.</p>
<p><img alt="fancy db image" src="/images/2018/reflex-and-servant.svg"></p>
<h1 id="preparation">Preparation</h1>
<p>First we need to setup the dev environment.
This time we’ll double down on <a href="https://nixos.org/nix/">nix</a> because reflex does
that too and fighting build tools is no fun.
This has the advantage that the <a href="https://github.com/jappeace/awesome-project-name/tree/reflex">resulting code on github</a>
is reproducible.
All need to be done is setup the file watch for which I wrote a make command:</p>
<div class="highlight"><pre><span></span>make file-watch
</pre></div>
<p>This rebuilds both the Haskell back end and JavaScript front end incrementally.</p>
<p>There are two separate environments now, one is for the native Haskell target (x86),
and the other is the JavaScript target.
We can enter the shell environment for the native target with <code>make enter</code>
and the JavaScript target with <code>make enter-js</code>.
This is convenient for doing one of commands.</p>
<p>The biggest issue I had when setting this up was figuring out how to add extra dependencies
not in the nix repo.
I found out by reading the nix code that this can be done with the overrides flag.
Another issue was tools for shells,
such as hpack which generates cabal files from the <code>package.yaml</code> file.
I really wanted to use that as I didn’t want to learn cabal,
besides, hpack’s is much more succinct, it doesn’t require explicit module exports.
there is a shellOverrides attribute for that.</p>
<div class="highlight"><pre><span></span> <span class="ss">overrides =</span> self<span class="p">:</span> super<span class="p">:</span> <span class="k">rec</span> <span class="p">{</span>
<span class="ss">beam-core =</span> self<span class="o">.</span>callPackage <span class="o">.</span><span class="l">/packages/beam-core.nix</span> <span class="p">{</span> <span class="p">};</span>
<span class="o">...</span>
<span class="p">};</span>
<span class="o">...</span>
<span class="ss">shellToolOverrides =</span> ghc<span class="p">:</span> super<span class="p">:</span> <span class="p">{</span>
<span class="k">inherit</span> <span class="p">(</span>ghc<span class="p">)</span> hpack<span class="p">;</span>
<span class="ss">fswatcher =</span> pkgs<span class="o">.</span>inotify-tools<span class="p">;</span>
<span class="o">...</span>
<span class="p">};</span>
</pre></div>
<h1 id="back-end">Back end</h1>
<p>The backend is mostly the same as the result of the <a href="/pragmatic-haskell.html">pragmatic haskell</a>
series.
We moved the <span class="caps">API</span> endpoints that need to be accessed by client to the e common code,
and added an additional endpoint for hosting the html.
Normally we wouldn’t use Haskell to deliver static assets and use a
specialized program such as <a href="https://www.nginx.com/">nginx</a>.
Since this is for experimentation however we made an exception:</p>
<div class="highlight"><pre><span></span><span class="kr">type</span> <span class="kt">Webservice</span> <span class="ow">=</span> <span class="kt">ServiceAPI</span>
<span class="kt">:<|></span> <span class="kt">Raw</span> <span class="c1">-- JS entry point</span>
<span class="nf">webservice</span> <span class="ow">::</span> <span class="kt">Proxy</span> <span class="kt">Webservice</span>
<span class="nf">webservice</span> <span class="ow">=</span> <span class="kt">Proxy</span>
<span class="o">...</span>
<span class="nf">server</span> <span class="ow">::</span> <span class="kt">Connection</span> <span class="ow">-></span> <span class="kt">Server</span> <span class="kt">Webservice</span>
<span class="nf">server</span> <span class="n">conn</span><span class="ow">=</span>
<span class="p">(</span><span class="n">pure</span> <span class="n">users</span> <span class="kt">:<|></span> <span class="n">messages</span> <span class="n">conn</span><span class="p">)</span> <span class="kt">:<|></span> <span class="n">serveDirectoryFileServer</span> <span class="s">"dist-ghcjs/build/x86_64-linux/ghcjs-0.2.1/frontend-0.1.0.0/c/webservice/build/webservice/webservice.jsexe/"</span>
</pre></div>
<p>The webservice type definition has the aditional <code>Raw</code> endpoint,
which <a href="http://haskell-servant.github.io/servant/Servant-API-Raw.html">allows hosting</a>
of custom wai apps.
The <code>serveDirectoryFileServer</code> is that custom wai app and just hosts the JavaScript
output of the client.</p>
<h1 id="common-code">Common code</h1>
<p>This is where the shared code between client and server lives.
We put the <span class="caps">API</span> definition in here.
Since servant can create both servers and clients it’s a great library
for this use case.</p>
<p>Any change in <span class="caps">API</span> will now cause the type checker to tell us where this is affected
in both client and server.
Type safety becomes amplified,
making bugs more obvious and increasing developer productivity.</p>
<p>The actual <a href="#commonsrccommonhs">content</a> of this module isn’t that interesting,
it’s just the <span class="caps">API</span> definition.
Common code gets compiled within the JavaScript client.
This means it’s public, one should not put any passwords or trade secrets in here.</p>
<h1 id="front-end">Front end</h1>
<p>I started with trying to get reflex to work with servant because it seemed the most uncertain.
After this I intended to use <code>servant-client</code> for generating the client functions,
here I ran into another hurdle as the latest servant wasn’t available.
Apparently reflex is pinned to an old hackage repository,
I attempted to upgrade but abandoned that endeavour as it required more nix modifications
and I’d prefer to keep the same pin as upstream so I could get help when I needed it.
Using the older servant, I hit a run time exception:</p>
<div class="highlight"><pre><span></span>uncaught exception in Haskell main thread: ReferenceError: h$hsnet_getaddrinfo is not defined
</pre></div>
<p>This is because servant client uses a system call for networking
which is unavailable in the browser sandbox.
A bit of googling led me to <a href="https://github.com/imalsogreg/servant-reflex">servant reflex</a>.
Using this was hard because are no official <a href="http://hackage.haskell.org/package/servant-reflex-0.3.3/candidate">haddocs</a>
since it hasn’t been released yet.
Finding an <a href="https://github.com/meditans/haskell-webapps/tree/master/UI/ReflexFRP/mockLoginPage">example</a>
was of great help which led me to this client definition:</p>
<div class="highlight"><pre><span></span><span class="nf">apiClients</span> <span class="ow">::</span> <span class="n">forall</span> <span class="n">t</span> <span class="n">m</span><span class="o">.</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kr">_</span>
<span class="nf">apiClients</span> <span class="ow">=</span> <span class="n">client</span> <span class="n">serviceAPI</span> <span class="p">(</span><span class="kt">Proxy</span> <span class="o">@</span><span class="n">m</span><span class="p">)</span> <span class="p">(</span><span class="kt">Proxy</span> <span class="o">@</span><span class="nb">()</span><span class="p">)</span> <span class="p">(</span><span class="n">constDyn</span> <span class="n">url</span><span class="p">)</span>
<span class="kr">where</span> <span class="n">url</span> <span class="ow">::</span> <span class="kt">BaseUrl</span>
<span class="n">url</span> <span class="ow">=</span> <span class="kt">BasePath</span> <span class="s">"/"</span>
</pre></div>
<p>This creates both functions for querying the <code>serviceAPI</code> from the common module.
All this seems to do is getting that <code>m</code> in scope and applying it to the client with a proxy.
The partial type signature was left from the example this way intentionally,
because it’s formalized below anyway and it’s rather big.</p>
<div class="highlight"><pre><span></span><span class="nf">getUsers</span> <span class="ow">::</span> <span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">Event</span> <span class="n">t</span> <span class="nb">()</span>
<span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Event</span> <span class="n">t</span> <span class="p">(</span><span class="kt">ReqResult</span> <span class="nb">()</span> <span class="p">[</span><span class="kt">User</span><span class="p">]))</span>
<span class="nf">postMessage</span> <span class="ow">::</span> <span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">Dynamic</span> <span class="n">t</span> <span class="p">(</span><span class="kt">Either</span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span> <span class="kt">Message</span><span class="p">)</span>
<span class="ow">-></span> <span class="kt">Event</span> <span class="n">t</span> <span class="nb">()</span>
<span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Event</span> <span class="n">t</span> <span class="p">(</span><span class="kt">ReqResult</span> <span class="nb">()</span> <span class="p">[</span><span class="kt">Message</span><span class="p">]))</span>
<span class="p">(</span><span class="n">getUsers</span> <span class="kt">:<|></span> <span class="n">postMessage</span><span class="p">)</span> <span class="ow">=</span> <span class="n">apiClients</span>
</pre></div>
<p>This pulls out the functions from <code>apiClients</code>,
we also get there final signature here.
The entire file can be seen in the <a href="#frontendsrcservantclienths">sources</a>.</p>
<h2 id="reflex">Reflex</h2>
<p>After getting the <span class="caps">API</span> to function I started working on making an actual <span class="caps">UI</span>.
Which is what this code does for the <code>getUsers</code> function:</p>
<div class="highlight"><pre><span></span><span class="nf">reflex</span> <span class="ow">::</span> <span class="kt">IO</span><span class="nb">()</span>
<span class="nf">reflex</span> <span class="ow">=</span>
<span class="n">mainWidget</span> <span class="o">$</span>
<span class="n">el</span> <span class="s">"div"</span> <span class="o">$</span> <span class="kr">do</span>
<span class="c1">-- babys steps, get users from memory</span>
<span class="n">intButton</span> <span class="ow"><-</span> <span class="n">button</span> <span class="s">"Get Users"</span>
<span class="n">serverInts</span> <span class="ow"><-</span> <span class="n">fmapMaybe</span> <span class="n">reqSuccess</span> <span class="o"><$></span> <span class="n">getUsers</span> <span class="n">intButton</span>
<span class="n">display</span> <span class="o">=<<</span> <span class="n">holdDyn</span> <span class="p">([</span><span class="kt">User</span> <span class="s">"none"</span> <span class="s">"none"</span><span class="p">])</span> <span class="n">serverInts</span>
</pre></div>
<p>mainWidget is the root of reflex, we use <code>el</code> to specify <span class="caps">HTML</span> elements that surround other elements.
the button functions creates a button (no surprise).
This is within the monad widget,
interaction between components is handled trough that monad.</p>
<p>On the next line we use the intButton immediately.
If we look at the getUsers type signature we see that it requires an <code>Event t ()</code> argument,
this is satisfied by the button. In other words <code>getUsers</code> will triggered on the button event.
The result is once more put in the monadwidget.
Finally we map the result to assume success or Nothing,
it will be just a list of users now.
The holdDyn function is then used to give a default value to the resulting event in case of nothing,
we always display either the default or the request result.</p>
<h2 id="markup-with-reflex">Markup with reflex</h2>
<p>For the <code>postMessage</code> function I made a form with text inputs:</p>
<div class="highlight"><pre><span></span> <span class="c1">-- Post a usermessage and display results</span>
<span class="n">input</span> <span class="ow"><-</span> <span class="n">messageInput</span>
<span class="n">sendMsg</span> <span class="ow"><-</span> <span class="n">button</span> <span class="s">"Send Message"</span>
<span class="n">messages</span> <span class="ow"><-</span> <span class="n">fmapMaybe</span> <span class="n">reqSuccess</span> <span class="o"><$></span> <span class="n">postMessage</span> <span class="p">(</span><span class="kt">Right</span> <span class="o"><$></span> <span class="n">input</span><span class="p">)</span> <span class="n">sendMsg</span>
<span class="n">resulting</span> <span class="ow"><-</span> <span class="n">holdDyn</span>
<span class="p">([</span><span class="kt">Message</span> <span class="p">(</span><span class="kt">User</span> <span class="s">"none"</span> <span class="s">"none"</span><span class="p">)</span> <span class="s">"ddd"</span><span class="p">])</span> <span class="c1">-- what to show if nothing</span>
<span class="n">messages</span> <span class="c1">-- source of messages (if any)</span>
<span class="kr">_</span> <span class="ow"><-</span> <span class="n">el</span> <span class="s">"div"</span> <span class="o">$</span>
<span class="n">simpleList</span> <span class="n">resulting</span> <span class="n">fancyMsg</span>
</pre></div>
<p><code>messageInput</code> is a function that returns a “dynamic message” (see below),
the button is for sending of messages.
To display we use a similar pattern however this time we’ll mark it up in <span class="caps">HTML</span>
with fancy messages.
We traverse over the dynamic list with the <code>simpleList</code> function,
here I expected traverse to work.</p>
<div class="highlight"><pre><span></span> <span class="kr">where</span>
<span class="n">fancyMsg</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kt">Dynamic</span> <span class="n">t</span> <span class="kt">Message</span> <span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Element</span> <span class="kt">EventResult</span> <span class="kt">GhcjsDomSpace</span> <span class="n">t</span><span class="p">)</span>
<span class="n">fancyMsg</span> <span class="n">msg</span> <span class="ow">=</span> <span class="n">elClass</span> <span class="s">"div"</span> <span class="s">"message"</span> <span class="o">$</span> <span class="kr">do</span>
<span class="kr">_</span> <span class="ow"><-</span> <span class="n">elDynHtml'</span> <span class="s">"h1"</span> <span class="o">$</span> <span class="kt">Text</span><span class="o">.</span><span class="n">pack</span> <span class="o">.</span> <span class="n">name</span> <span class="o">.</span> <span class="n">from</span> <span class="o"><$></span> <span class="n">msg</span>
<span class="n">elDynHtml'</span> <span class="s">"span"</span> <span class="o">$</span> <span class="kt">Text</span><span class="o">.</span><span class="n">pack</span> <span class="o">.</span> <span class="n">content</span> <span class="o"><$></span> <span class="n">msg</span>
</pre></div>
<p>Every message is put in a <code>div</code> element,
for displaying dynamic content however we need to use <code>elDynhtml'</code> function,
there is no way of getting a value out of dynamic,
we can only show it to the user.
This is a strong safety guarantee.</p>
<h2 id="reflex-react-component">Reflex “react component”</h2>
<p>Input fields can be combined together into larger components,
which is showcased in the Message form:</p>
<div class="highlight"><pre><span></span><span class="nf">messageInput</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Dynamic</span> <span class="n">t</span> <span class="kt">Message</span><span class="p">)</span>
<span class="nf">messageInput</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">user</span> <span class="ow"><-</span> <span class="n">userInput</span>
<span class="n">message</span> <span class="ow"><-</span> <span class="n">labeledInput</span> <span class="s">"message"</span>
<span class="n">pure</span> <span class="o">$</span> <span class="p">(</span><span class="kt">Message</span> <span class="o"><$></span> <span class="n">user</span><span class="p">)</span> <span class="o"><*></span> <span class="p">(</span><span class="kt">Text</span><span class="o">.</span><span class="n">unpack</span> <span class="o"><$></span> <span class="n">_textInput_value</span> <span class="n">message</span><span class="p">)</span>
<span class="nf">userInput</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Dynamic</span> <span class="n">t</span> <span class="kt">User</span><span class="p">)</span>
<span class="nf">userInput</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">username</span> <span class="ow"><-</span> <span class="n">labeledInput</span> <span class="s">"username"</span>
<span class="n">emailInput</span> <span class="ow"><-</span> <span class="n">labeledInput</span> <span class="s">"email"</span>
<span class="n">pure</span> <span class="o">$</span> <span class="kt">User</span> <span class="o">.</span> <span class="kt">Text</span><span class="o">.</span><span class="n">unpack</span> <span class="o"><$></span> <span class="n">_textInput_value</span> <span class="n">username</span> <span class="o"><*></span> <span class="p">(</span><span class="kt">Text</span><span class="o">.</span><span class="n">unpack</span> <span class="o"><$></span> <span class="n">_textInput_value</span> <span class="n">emailInput</span><span class="p">)</span>
<span class="nf">labeledInput</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span> <span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">TextInput</span> <span class="n">t</span><span class="p">)</span>
<span class="nf">labeledInput</span> <span class="n">label</span> <span class="ow">=</span> <span class="n">elClass</span> <span class="s">"div"</span> <span class="s">"field"</span> <span class="o">$</span> <span class="kr">do</span>
<span class="n">elClass</span> <span class="s">"label"</span> <span class="s">"label"</span> <span class="o">$</span> <span class="n">text</span> <span class="n">label</span>
<span class="n">elClass</span> <span class="s">"div"</span> <span class="s">"control"</span> <span class="o">$</span> <span class="n">textInput</span> <span class="p">(</span><span class="n">def</span> <span class="o">&</span> <span class="n">textInputConfig_attributes</span> <span class="o">.~</span> <span class="n">constDyn</span> <span class="p">(</span><span class="kt">Text</span><span class="o">.</span><span class="n">pack</span> <span class="s">"class"</span> <span class="o">=:</span> <span class="kt">Text</span><span class="o">.</span><span class="n">pack</span> <span class="s">"input"</span><span class="p">))</span>
</pre></div>
<p>This is done with applicative fmap <code><$></code> and spaceship <code><*></code>.
That last line sets some extra confiugrations can be set withLenses for textInput,
which is another <a href="http://hackage.haskell.org/package/lens">rabithole</a>.
They can simply be thought of as getters and setters for haskell, although more powerfull.</p>
<p>Note that these functions are analogue to a react component, and see the difference!
They compose perfectly and will function independently.</p>
<p>Feel the power.</p>
<p>My only complaint is that the resulting JavaScript binary is huge,
<span class="caps">8MB</span>, <span class="caps">2MB</span> after using the <a href="https://github.com/ghcjs/ghcjs/wiki/Deployment">closure compiler</a>.</p>
<h1 id="conclusion">Conclusion</h1>
<p>I’m very pleased with reflex,
now I don’t have to deal with JavaScript,
I can prototype my <span class="caps">API</span> rapidly and I’m not restricted to an architecture.
It is better than I expected, the core seems really well designed.
The only downside is the large binary.
None the less I’m willing to use this for a larger project.</p>
<h1 id="links">Links</h1>
<p>For convenience here is a list of used resources:</p>
<ul>
<li><a href="https://github.com/jappeace/awesome-project-name/tree/reflex">Complete sources for this post</a></li>
<li><a href="https://github.com/reflex-frp/reflex">Reflex</a></li>
<li><a href="https://github.com/reflex-frp/reflex-platform/blob/develop/docs/project-development.md">Reflex project development</a></li>
<li><a href="https://github.com/meditans/haskell-webapps/tree/master/UI/ReflexFRP/mockLoginPage">Example frp app</a></li>
<li><a href="http://hackage.haskell.org/package/servant-reflex-0.3.3/candidate">Servant reflex release candidate docs</a></li>
<li><a href="http://docs.reflex-frp.org/en/latest/">Explenation of things</a></li>
</ul>
<h1 id="sources">Sources</h1>
<p>The project has become too big to share all files,
as always there is the <a href="https://github.com/jappeace/awesome-project-name/tree/reflex">github link</a>.
I will however put all discussed code in complete form in here.</p>
<h2 id="backendsrclibhs">backend/src/Lib.hs</h2>
<div class="highlight"><pre><span></span><span class="cm">{-# LANGUAGE DataKinds #-}</span>
<span class="cm">{-# LANGUAGE TypeOperators #-}</span>
<span class="cm">{-# OPTIONS_GHC -Wno-missing-monadfail-instances #-}</span>
<span class="kr">module</span> <span class="nn">Lib</span>
<span class="p">(</span> <span class="nf">webAppEntry</span>
<span class="p">)</span> <span class="kr">where</span>
<span class="kr">import</span> <span class="nn">Servant</span>
<span class="kr">import</span> <span class="nn">Common</span>
<span class="kr">import</span> <span class="nn">Control.Monad.IO.Class</span><span class="p">(</span><span class="n">liftIO</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Network.Wai</span><span class="p">(</span><span class="kt">Application</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Network.Wai.Handler.Warp</span><span class="p">(</span><span class="n">run</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Database.PostgreSQL.Simple</span> <span class="p">(</span><span class="kt">Connection</span><span class="p">)</span>
<span class="kr">import</span> <span class="k">qualified</span> <span class="nn">DB</span> <span class="k">as</span> <span class="n">DB</span>
<span class="kr">import</span> <span class="nn">Database.Beam.Backend.SQL.BeamExtensions</span> <span class="p">(</span><span class="nf">runInsertReturningList</span><span class="p">)</span>
<span class="kr">import</span> <span class="k">qualified</span> <span class="nn">Database.Beam</span> <span class="k">as</span> <span class="n">Beam</span>
<span class="kr">import</span> <span class="k">qualified</span> <span class="nn">Database.Beam.Postgres</span> <span class="k">as</span> <span class="n">PgBeam</span>
<span class="kr">import</span> <span class="nn">Data.Text</span><span class="p">(</span><span class="n">pack</span><span class="p">,</span> <span class="n">unpack</span><span class="p">)</span>
<span class="kr">type</span> <span class="kt">Webservice</span> <span class="ow">=</span> <span class="kt">ServiceAPI</span>
<span class="kt">:<|></span> <span class="kt">Raw</span> <span class="c1">-- JS entry point</span>
<span class="nf">webservice</span> <span class="ow">::</span> <span class="kt">Proxy</span> <span class="kt">Webservice</span>
<span class="nf">webservice</span> <span class="ow">=</span> <span class="kt">Proxy</span>
<span class="nf">users</span> <span class="ow">::</span> <span class="p">[</span><span class="kt">User</span><span class="p">]</span>
<span class="nf">users</span> <span class="ow">=</span>
<span class="p">[</span> <span class="kt">User</span> <span class="s">"Isaac Newton"</span> <span class="s">"isaac@newton.co.uk"</span>
<span class="p">,</span> <span class="kt">User</span> <span class="s">"Albert Einstein"</span> <span class="s">"ae@mc2.org"</span>
<span class="p">]</span>
<span class="nf">messages</span> <span class="ow">::</span> <span class="kt">Connection</span> <span class="ow">-></span> <span class="kt">Message</span> <span class="ow">-></span> <span class="kt">Handler</span> <span class="p">[</span><span class="kt">Message</span><span class="p">]</span>
<span class="nf">messages</span> <span class="n">conn</span> <span class="n">message</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">fromDb</span> <span class="ow"><-</span> <span class="n">liftIO</span> <span class="o">$</span>
<span class="kt">PgBeam</span><span class="o">.</span><span class="n">runBeamPostgres</span> <span class="n">conn</span> <span class="o">$</span> <span class="kr">do</span>
<span class="kr">let</span> <span class="n">user</span> <span class="ow">=</span> <span class="n">from</span> <span class="n">message</span>
<span class="p">[</span><span class="n">foundUser</span><span class="p">]</span> <span class="ow"><-</span> <span class="n">runInsertReturningList</span> <span class="p">(</span><span class="kt">DB</span><span class="o">.</span><span class="n">_ausers</span> <span class="kt">DB</span><span class="o">.</span><span class="n">awesomeDB</span><span class="p">)</span> <span class="o">$</span>
<span class="kt">Beam</span><span class="o">.</span><span class="n">insertExpressions</span> <span class="p">[</span><span class="kt">DB</span><span class="o">.</span><span class="kt">User</span>
<span class="kt">Beam</span><span class="o">.</span><span class="n">default_</span>
<span class="p">(</span><span class="kt">Beam</span><span class="o">.</span><span class="n">val_</span> <span class="p">(</span><span class="n">pack</span> <span class="o">$</span> <span class="n">name</span> <span class="o">$</span> <span class="n">user</span> <span class="p">))</span>
<span class="p">(</span><span class="kt">Beam</span><span class="o">.</span><span class="n">val_</span> <span class="p">(</span><span class="n">pack</span> <span class="o">$</span> <span class="n">email</span> <span class="o">$</span> <span class="n">user</span> <span class="p">))</span>
<span class="p">]</span>
<span class="kr">_</span> <span class="ow"><-</span> <span class="n">runInsertReturningList</span> <span class="p">(</span><span class="kt">DB</span><span class="o">.</span><span class="n">_messages</span> <span class="kt">DB</span><span class="o">.</span><span class="n">awesomeDB</span><span class="p">)</span> <span class="o">$</span>
<span class="kt">Beam</span><span class="o">.</span><span class="n">insertExpressions</span>
<span class="p">[</span><span class="kt">DB</span><span class="o">.</span><span class="kt">Message</span>
<span class="kt">Beam</span><span class="o">.</span><span class="n">default_</span>
<span class="p">(</span><span class="kt">Beam</span><span class="o">.</span><span class="n">val_</span> <span class="p">(</span><span class="kt">Beam</span><span class="o">.</span><span class="n">pk</span> <span class="n">foundUser</span><span class="p">))</span>
<span class="p">(</span><span class="kt">Beam</span><span class="o">.</span><span class="n">val_</span> <span class="p">(</span><span class="n">pack</span> <span class="o">$</span> <span class="n">content</span> <span class="n">message</span><span class="p">))</span>
<span class="p">]</span>
<span class="kt">Beam</span><span class="o">.</span><span class="n">runSelectReturningList</span> <span class="o">$</span> <span class="kt">Beam</span><span class="o">.</span><span class="n">select</span> <span class="o">$</span> <span class="kr">do</span>
<span class="n">usr</span> <span class="ow"><-</span> <span class="p">(</span><span class="kt">Beam</span><span class="o">.</span><span class="n">all_</span> <span class="p">(</span><span class="kt">DB</span><span class="o">.</span><span class="n">_ausers</span> <span class="kt">DB</span><span class="o">.</span><span class="n">awesomeDB</span><span class="p">))</span>
<span class="n">msg</span> <span class="ow"><-</span> <span class="kt">Beam</span><span class="o">.</span><span class="n">oneToMany_</span> <span class="p">(</span><span class="kt">DB</span><span class="o">.</span><span class="n">_messages</span> <span class="kt">DB</span><span class="o">.</span><span class="n">awesomeDB</span><span class="p">)</span> <span class="kt">DB</span><span class="o">.</span><span class="n">_from</span> <span class="n">usr</span>
<span class="n">pure</span> <span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">usr</span><span class="p">)</span>
<span class="n">pure</span> <span class="o">$</span>
<span class="n">fmap</span> <span class="p">(</span>
<span class="nf">\</span><span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">usr</span><span class="p">)</span> <span class="ow">-></span> <span class="kt">Message</span>
<span class="p">(</span><span class="kt">User</span>
<span class="p">(</span><span class="n">unpack</span> <span class="o">$</span> <span class="kt">DB</span><span class="o">.</span><span class="n">_name</span> <span class="n">usr</span><span class="p">)</span>
<span class="p">(</span><span class="n">unpack</span> <span class="o">$</span> <span class="kt">DB</span><span class="o">.</span><span class="n">_email</span> <span class="n">usr</span><span class="p">))</span>
<span class="p">(</span><span class="n">unpack</span> <span class="o">$</span> <span class="kt">DB</span><span class="o">.</span><span class="n">_content</span> <span class="n">msg</span><span class="p">)</span>
<span class="p">)</span> <span class="n">fromDb</span>
<span class="nf">server</span> <span class="ow">::</span> <span class="kt">Connection</span> <span class="ow">-></span> <span class="kt">Server</span> <span class="kt">Webservice</span>
<span class="nf">server</span> <span class="n">conn</span><span class="ow">=</span>
<span class="p">(</span><span class="n">pure</span> <span class="n">users</span> <span class="kt">:<|></span> <span class="n">messages</span> <span class="n">conn</span><span class="p">)</span> <span class="kt">:<|></span> <span class="n">serveDirectoryFileServer</span> <span class="s">"dist-ghcjs/build/x86_64-linux/ghcjs-0.2.1/frontend-0.1.0.0/c/webservice/build/webservice/webservice.jsexe/"</span>
<span class="nf">app</span> <span class="ow">::</span> <span class="kt">Connection</span> <span class="ow">-></span> <span class="kt">Application</span>
<span class="nf">app</span> <span class="n">conn</span> <span class="ow">=</span> <span class="n">serve</span> <span class="n">webservice</span> <span class="p">(</span><span class="n">server</span> <span class="n">conn</span><span class="p">)</span>
<span class="nf">webAppEntry</span> <span class="ow">::</span> <span class="kt">Connection</span> <span class="ow">-></span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="nf">webAppEntry</span> <span class="n">conn</span> <span class="ow">=</span> <span class="n">run</span> <span class="mi">6868</span> <span class="p">(</span><span class="n">app</span> <span class="n">conn</span><span class="p">)</span>
</pre></div>
<h2 id="commonsrccommonhs">common/src/Common.hs</h2>
<div class="highlight"><pre><span></span><span class="cm">{-# LANGUAGE DataKinds #-}</span>
<span class="cm">{-# LANGUAGE TypeOperators #-}</span>
<span class="cm">{-# LANGUAGE DeriveGeneric #-}</span>
<span class="kr">module</span> <span class="nn">Common</span> <span class="kr">where</span>
<span class="kr">import</span> <span class="nn">GHC.Generics</span><span class="p">(</span><span class="kt">Generic</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Servant.API</span>
<span class="kr">import</span> <span class="nn">Data.Proxy</span>
<span class="kr">import</span> <span class="nn">Data.Aeson</span><span class="p">(</span><span class="kt">ToJSON</span><span class="p">,</span> <span class="kt">FromJSON</span><span class="p">)</span>
<span class="kr">type</span> <span class="kt">ServiceAPI</span> <span class="ow">=</span> <span class="s">"api"</span> <span class="kt">:></span> <span class="s">"1.0"</span> <span class="kt">:></span> <span class="s">"users"</span> <span class="kt">:></span> <span class="kt">Get</span> <span class="kt">'[JSON]</span> <span class="p">[</span><span class="kt">User</span><span class="p">]</span>
<span class="kt">:<|></span> <span class="s">"api"</span> <span class="kt">:></span> <span class="s">"1.0"</span> <span class="kt">:></span> <span class="s">"message"</span> <span class="kt">:></span> <span class="kt">ReqBody</span> <span class="kt">'[JSON]</span> <span class="kt">Message</span> <span class="kt">:></span> <span class="kt">Post</span> <span class="kt">'[JSON]</span> <span class="p">[</span><span class="kt">Message</span><span class="p">]</span>
<span class="kr">data</span> <span class="kt">Message</span> <span class="ow">=</span> <span class="kt">Message</span> <span class="p">{</span>
<span class="n">from</span> <span class="ow">::</span> <span class="kt">User</span><span class="p">,</span>
<span class="n">content</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">}</span> <span class="kr">deriving</span> <span class="p">(</span><span class="kt">Eq</span><span class="p">,</span> <span class="kt">Show</span><span class="p">,</span> <span class="kt">Generic</span><span class="p">)</span>
<span class="kr">instance</span> <span class="kt">ToJSON</span> <span class="kt">Message</span>
<span class="kr">instance</span> <span class="kt">FromJSON</span> <span class="kt">Message</span>
<span class="kr">data</span> <span class="kt">User</span> <span class="ow">=</span> <span class="kt">User</span>
<span class="p">{</span> <span class="n">name</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">,</span> <span class="n">email</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">}</span> <span class="kr">deriving</span> <span class="p">(</span><span class="kt">Eq</span><span class="p">,</span> <span class="kt">Show</span><span class="p">,</span> <span class="kt">Generic</span><span class="p">)</span>
<span class="kr">instance</span> <span class="kt">ToJSON</span> <span class="kt">User</span>
<span class="kr">instance</span> <span class="kt">FromJSON</span> <span class="kt">User</span>
<span class="nf">serviceAPI</span> <span class="ow">::</span> <span class="kt">Proxy</span> <span class="kt">ServiceAPI</span>
<span class="nf">serviceAPI</span> <span class="ow">=</span> <span class="kt">Proxy</span>
</pre></div>
<h2 id="frontendsrclibhs">frontend/src/Lib.hs</h2>
<div class="highlight"><pre><span></span><span class="cm">{-# LANGUAGE OverloadedStrings #-}</span>
<span class="cm">{-# OPTIONS_GHC -fprint-explicit-kinds -Wpartial-type-signatures #-}</span>
<span class="kr">module</span> <span class="nn">Lib</span>
<span class="p">(</span> <span class="nf">reflex</span>
<span class="p">)</span> <span class="kr">where</span>
<span class="kr">import</span> <span class="nn">Reflex</span>
<span class="kr">import</span> <span class="nn">Reflex.Dom</span>
<span class="kr">import</span> <span class="k">qualified</span> <span class="nn">Data.Text</span> <span class="k">as</span> <span class="n">Text</span>
<span class="kr">import</span> <span class="nn">Control.Applicative</span> <span class="p">((</span><span class="o"><*></span><span class="p">),</span> <span class="p">(</span><span class="o"><$></span><span class="p">))</span>
<span class="kr">import</span> <span class="nn">Common</span>
<span class="kr">import</span> <span class="nn">Servant.Reflex</span>
<span class="kr">import</span> <span class="nn">ServantClient</span>
<span class="nf">reflex</span> <span class="ow">::</span> <span class="kt">IO</span><span class="nb">()</span>
<span class="nf">reflex</span> <span class="ow">=</span>
<span class="n">mainWidget</span> <span class="o">$</span>
<span class="n">el</span> <span class="s">"div"</span> <span class="o">$</span> <span class="kr">do</span>
<span class="c1">-- babys steps, get users from memory</span>
<span class="n">intButton</span> <span class="ow"><-</span> <span class="n">button</span> <span class="s">"Get Users"</span>
<span class="n">serverInts</span> <span class="ow"><-</span> <span class="n">fmapMaybe</span> <span class="n">reqSuccess</span> <span class="o"><$></span> <span class="n">getUsers</span> <span class="n">intButton</span>
<span class="n">display</span> <span class="o">=<<</span> <span class="n">holdDyn</span> <span class="p">([</span><span class="kt">User</span> <span class="s">"none"</span> <span class="s">"none"</span><span class="p">])</span> <span class="n">serverInts</span>
<span class="c1">-- Post a usermessage and display results</span>
<span class="n">input</span> <span class="ow"><-</span> <span class="n">messageInput</span>
<span class="n">sendMsg</span> <span class="ow"><-</span> <span class="n">button</span> <span class="s">"Send Message"</span>
<span class="n">messages</span> <span class="ow"><-</span> <span class="n">fmapMaybe</span> <span class="n">reqSuccess</span> <span class="o"><$></span> <span class="n">postMessage</span> <span class="p">(</span><span class="kt">Right</span> <span class="o"><$></span> <span class="n">input</span><span class="p">)</span> <span class="n">sendMsg</span>
<span class="n">resulting</span> <span class="ow"><-</span> <span class="n">holdDyn</span>
<span class="p">([</span><span class="kt">Message</span> <span class="p">(</span><span class="kt">User</span> <span class="s">"none"</span> <span class="s">"none"</span><span class="p">)</span> <span class="s">"ddd"</span><span class="p">])</span> <span class="c1">-- what to show if nothing</span>
<span class="n">messages</span> <span class="c1">-- source of messages (if any)</span>
<span class="kr">_</span> <span class="ow"><-</span> <span class="n">el</span> <span class="s">"div"</span> <span class="o">$</span>
<span class="n">simpleList</span> <span class="n">resulting</span> <span class="n">fancyMsg</span>
<span class="n">pure</span> <span class="nb">()</span>
<span class="kr">where</span>
<span class="n">fancyMsg</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kt">Dynamic</span> <span class="n">t</span> <span class="kt">Message</span> <span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Element</span> <span class="kt">EventResult</span> <span class="kt">GhcjsDomSpace</span> <span class="n">t</span><span class="p">)</span>
<span class="n">fancyMsg</span> <span class="n">msg</span> <span class="ow">=</span> <span class="n">elClass</span> <span class="s">"div"</span> <span class="s">"message"</span> <span class="o">$</span> <span class="kr">do</span>
<span class="kr">_</span> <span class="ow"><-</span> <span class="n">elDynHtml'</span> <span class="s">"h1"</span> <span class="o">$</span> <span class="kt">Text</span><span class="o">.</span><span class="n">pack</span> <span class="o">.</span> <span class="n">name</span> <span class="o">.</span> <span class="n">from</span> <span class="o"><$></span> <span class="n">msg</span>
<span class="n">elDynHtml'</span> <span class="s">"span"</span> <span class="o">$</span> <span class="kt">Text</span><span class="o">.</span><span class="n">pack</span> <span class="o">.</span> <span class="n">content</span> <span class="o"><$></span> <span class="n">msg</span>
<span class="nf">messageInput</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Dynamic</span> <span class="n">t</span> <span class="kt">Message</span><span class="p">)</span>
<span class="nf">messageInput</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">user</span> <span class="ow"><-</span> <span class="n">userInput</span>
<span class="n">message</span> <span class="ow"><-</span> <span class="n">labeledInput</span> <span class="s">"message"</span>
<span class="n">pure</span> <span class="o">$</span> <span class="p">(</span><span class="kt">Message</span> <span class="o"><$></span> <span class="n">user</span><span class="p">)</span> <span class="o"><*></span> <span class="p">(</span><span class="kt">Text</span><span class="o">.</span><span class="n">unpack</span> <span class="o"><$></span> <span class="n">_textInput_value</span> <span class="n">message</span><span class="p">)</span>
<span class="nf">userInput</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Dynamic</span> <span class="n">t</span> <span class="kt">User</span><span class="p">)</span>
<span class="nf">userInput</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">username</span> <span class="ow"><-</span> <span class="n">labeledInput</span> <span class="s">"username"</span>
<span class="n">emailInput</span> <span class="ow"><-</span> <span class="n">labeledInput</span> <span class="s">"email"</span>
<span class="n">pure</span> <span class="o">$</span> <span class="kt">User</span> <span class="o">.</span> <span class="kt">Text</span><span class="o">.</span><span class="n">unpack</span> <span class="o"><$></span> <span class="n">_textInput_value</span> <span class="n">username</span> <span class="o"><*></span> <span class="p">(</span><span class="kt">Text</span><span class="o">.</span><span class="n">unpack</span> <span class="o"><$></span> <span class="n">_textInput_value</span> <span class="n">emailInput</span><span class="p">)</span>
<span class="nf">labeledInput</span> <span class="ow">::</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span> <span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">TextInput</span> <span class="n">t</span><span class="p">)</span>
<span class="nf">labeledInput</span> <span class="n">label</span> <span class="ow">=</span> <span class="n">elClass</span> <span class="s">"div"</span> <span class="s">"field"</span> <span class="o">$</span> <span class="kr">do</span>
<span class="n">elClass</span> <span class="s">"label"</span> <span class="s">"label"</span> <span class="o">$</span> <span class="n">text</span> <span class="n">label</span>
<span class="n">elClass</span> <span class="s">"div"</span> <span class="s">"control"</span> <span class="o">$</span> <span class="n">textInput</span> <span class="p">(</span><span class="n">def</span> <span class="o">&</span> <span class="n">textInputConfig_attributes</span> <span class="o">.~</span> <span class="n">constDyn</span> <span class="p">(</span><span class="kt">Text</span><span class="o">.</span><span class="n">pack</span> <span class="s">"class"</span> <span class="o">=:</span> <span class="kt">Text</span><span class="o">.</span><span class="n">pack</span> <span class="s">"input"</span><span class="p">))</span>
</pre></div>
<h2 id="frontendsrcservantclienths">frontend/src/ServantClient.hs</h2>
<div class="highlight"><pre><span></span><span class="cm">{-# LANGUAGE OverloadedStrings #-}</span>
<span class="cm">{-# LANGUAGE RankNTypes #-}</span>
<span class="cm">{-# LANGUAGE TypeApplications #-}</span>
<span class="cm">{-# OPTIONS_GHC -Wno-partial-type-signatures #-}</span>
<span class="cm">{-# LANGUAGE NoMonomorphismRestriction #-}</span>
<span class="cm">{-# LANGUAGE PartialTypeSignatures #-}</span>
<span class="cm">{-# LANGUAGE ScopedTypeVariables #-}</span>
<span class="cm">{-# LANGUAGE TypeApplications #-}</span>
<span class="cm">{-# LANGUAGE TypeOperators #-}</span>
<span class="c1">-- | This modules purpose is just to generate the xhr clients.</span>
<span class="c1">-- there is some type magick going on generating these,</span>
<span class="c1">-- therefore the functions are isolated.</span>
<span class="kr">module</span> <span class="nn">ServantClient</span>
<span class="p">(</span> <span class="nf">postMessage</span><span class="p">,</span> <span class="nf">getUsers</span>
<span class="p">)</span> <span class="kr">where</span>
<span class="kr">import</span> <span class="nn">Reflex</span>
<span class="kr">import</span> <span class="nn">Reflex.Dom</span>
<span class="kr">import</span> <span class="k">qualified</span> <span class="nn">Data.Text</span> <span class="k">as</span> <span class="n">Text</span>
<span class="kr">import</span> <span class="nn">Servant.API</span>
<span class="kr">import</span> <span class="nn">Common</span>
<span class="kr">import</span> <span class="nn">Servant.Reflex</span>
<span class="kr">import</span> <span class="nn">Data.Proxy</span>
<span class="c1">-- | This intermediate definition is necisarry because the @m is similar for both clients,</span>
<span class="c1">-- they have the same wrapping monad however the containing type is different</span>
<span class="c1">-- (which is why we have the nomonomorphism restirction disabled)</span>
<span class="nf">apiClients</span> <span class="ow">::</span> <span class="n">forall</span> <span class="n">t</span> <span class="n">m</span><span class="o">.</span> <span class="p">(</span><span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span><span class="p">)</span> <span class="ow">=></span> <span class="kr">_</span>
<span class="nf">apiClients</span> <span class="ow">=</span> <span class="n">client</span> <span class="n">serviceAPI</span> <span class="p">(</span><span class="kt">Proxy</span> <span class="o">@</span><span class="n">m</span><span class="p">)</span> <span class="p">(</span><span class="kt">Proxy</span> <span class="o">@</span><span class="nb">()</span><span class="p">)</span> <span class="p">(</span><span class="n">constDyn</span> <span class="n">url</span><span class="p">)</span>
<span class="kr">where</span> <span class="n">url</span> <span class="ow">::</span> <span class="kt">BaseUrl</span>
<span class="n">url</span> <span class="ow">=</span> <span class="kt">BasePath</span> <span class="s">"/"</span>
<span class="nf">getUsers</span> <span class="ow">::</span> <span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">Event</span> <span class="n">t</span> <span class="nb">()</span> <span class="c1">-- ^ Trigger the XHR Request</span>
<span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Event</span> <span class="n">t</span> <span class="p">(</span><span class="kt">ReqResult</span> <span class="nb">()</span> <span class="p">[</span><span class="kt">User</span><span class="p">]))</span> <span class="c1">-- ^ Consume the answer</span>
<span class="nf">postMessage</span> <span class="ow">::</span> <span class="kt">MonadWidget</span> <span class="n">t</span> <span class="n">m</span>
<span class="ow">=></span> <span class="kt">Dynamic</span> <span class="n">t</span> <span class="p">(</span><span class="kt">Either</span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span> <span class="kt">Message</span><span class="p">)</span>
<span class="ow">-></span> <span class="kt">Event</span> <span class="n">t</span> <span class="nb">()</span>
<span class="ow">-></span> <span class="n">m</span> <span class="p">(</span><span class="kt">Event</span> <span class="n">t</span> <span class="p">(</span><span class="kt">ReqResult</span> <span class="nb">()</span> <span class="p">[</span><span class="kt">Message</span><span class="p">]))</span>
<span class="p">(</span><span class="n">getUsers</span> <span class="kt">:<|></span> <span class="n">postMessage</span><span class="p">)</span> <span class="ow">=</span> <span class="n">apiClients</span>
</pre></div>Back to the Netherlands!2018-09-23T18:21:00+02:002018-09-23T18:21:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2018-09-23:/back-to-the-netherlands.html<p>After a year in foreign countries, I’ve returned home.
Last week my working holiday visa expired,
and unlike a friend of mine,
I did not want to be deported in handcuffs.</p>
<p><img alt="Dutch flag" src="/images/2018/dutch-flag.svg"></p>
<p>I quite enjoyed my stay in Australia, the people are relax, life is good.
I’ve drank lots …</p><p>After a year in foreign countries, I’ve returned home.
Last week my working holiday visa expired,
and unlike a friend of mine,
I did not want to be deported in handcuffs.</p>
<p><img alt="Dutch flag" src="/images/2018/dutch-flag.svg"></p>
<p>I quite enjoyed my stay in Australia, the people are relax, life is good.
I’ve drank lots of coffee and ate many a Kangaroo, but it’s good to be back
and speak my own language.
The people just “<a href="https://stuffdutchpeoplelike.com/2010/11/26/no-56-normalcy-doe-normaal/">act normal</a>“,</p>
<p>First I was a month in Indonesia,
essentially looking for jobs in Australia (as you do),
and hanging out with my then girlfriend.
Then in Sydney I quickly found a job and place to live.
Three interviews were pre-arranged, they all got converted into offers.
The first one was <a href="https://stackchat.com/#contact-area">stackchat</a>,
they gave me a pretty bad offer, I needed money most yet they came with equity.
Then there was <a href="https://www.ayudasystems.com/">ayuda</a>,
I probably could’ve asked them lots more money as they were hardly strapped for cash.
<a href="https://www.openlearning.com/">Openlearning</a> offered most though, more then I asked for,
which was nice.
I took the highest.</p>
<p>Around new year the relationship with Jesiska was in crisis,
and I went back to Indonesia, where it failed.
Happy new year.</p>
<p>then, around February the company wanted to initiate sponsorship.
However I had become pretty unhappy with the working arrangement.
It looked like I was just being kept busy,
while having no impact on anything.
That working relationship also ended there.</p>
<p>At that point I seriously considered going back,
This was my lowest point in Australia.
I had nothing there except a little hobby project started with some friends.
ex-Colleagues at Open Learning, I was crestfallen.
Although they were being nice about my essentially destroyed life,
in Netherlands we would rip on each other at this point,
You could at least joke about it you know.
I’m too facetious I hear.</p>
<p>Here I decided that rather than giving up so easily I’d fight for staying in
Australia as a software engineer.
I just had arranged better housing, now I bet I could also arrange better work!
Easier said then done, I emailed so many companies,
I even made a little movie to sell myself.
Few replies but some did.</p>
<p>First there was Silicon Valley Chicken, they wanted to sell chicken trough an app with a game.
Completely aimed at Gamers, and only order trough the app.
I wasn’t too excited about that, they weren’t excited about me being on working holiday visa.
After that I met <a href="https://www.daisee.com/">Daisee</a> at the functional programming meetup.
I managed to get an interview.
Turned out it was a Haskell job.
I was super excited about that.
Then <a href="https://www.jointrunk.com/">Trunk</a> wanted to deliver version control to designers.
But I kind off neglected them in favor of the Haskell opportunity. </p>
<p>This was role reversal, now the friends had a pretty crappy job compared to doing Haskell.
Especially considering the company had taken up <a href="https://azure.microsoft.com/en-us/services/cosmos-db/">Cosmos db</a>
which required running windows.
They were both open source advocates,
requiring to run this made them.. Unhappy.</p>
<p>But I was happy. For a couple of months.
Daisee kept hiring people and we grew.
The more time passed the more responsibility I had to give away,
until I was focused upon the one project.
Which then got canned.
Not that it bothered me as I was planning for the next stage in my life.</p>
<p>Going to the Philippines! Except, everyone I spoke to was pretty negative about it,
and the offer from Daisee wasn’t as nice as I expected.
Back home I am,
not that it’s bad, just sudden.
I’ll be staying here for some time,
doing remote work for Daisee as a <a href="http://penguin.engineer/">company</a>!
Meanwhile Daisee is working on a work visa for Australia.
This seems much better than the Philippines,
which felt more like a golden prison.</p>
<p>This is all seems pretty random, oh well, such is life.
I had fun at least.</p>NixOS on encrypted btrfs2018-08-19T13:02:00+02:002022-01-30T13:18:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2018-08-19:/nixos-on-encrypted-btrfs.html<p>Nixos is heroin for tinkerers.
Paradise can be tinkered together and be freely shared among peers
because it’s fully reproducible!
Jappie wanted more, he wanted a secure disk <em>and</em> a <span class="caps">BTRFS</span>.
There used to be no guides for this, now there is.</p>
<p><img alt="Locked btrfs on nixos" src="/images/2018/locked_btrfs.svg"></p>
<p>The bullet was bitten, <span class="caps">BTRFS</span> was made …</p><p>Nixos is heroin for tinkerers.
Paradise can be tinkered together and be freely shared among peers
because it’s fully reproducible!
Jappie wanted more, he wanted a secure disk <em>and</em> a <span class="caps">BTRFS</span>.
There used to be no guides for this, now there is.</p>
<p><img alt="Locked btrfs on nixos" src="/images/2018/locked_btrfs.svg"></p>
<p>The bullet was bitten, <span class="caps">BTRFS</span> was made to work on a <span class="caps">LUKS</span> encrypted disk.
This isn’t hard, with care and precision.
To help a reader we document the journey towards <span class="caps">BTRFS</span>.
Commands compiled and included.</p>
<h1 id="getting-started">Getting started</h1>
<p>Get yourself a NixOS <a href="https://nixos.org/download.html#download-nixos">live usb</a>.
I use the minimal <span class="caps">ISO</span>, because the graphical <span class="caps">ISO</span> slows booting and gives no advantage
aside from being pretty. Use</p>
<div class="highlight"><pre><span></span>cat minimal-nixos.iso > /dev/sdX
</pre></div>
<p>where <code>X</code> is the usb drive found by <code>lsblk</code>.
<code>X</code> should be a letter, numbers indicate partitions, which we don’t want to cat upon.
Boot into it on the target machine.</p>
<h1 id="networking">Networking</h1>
<p>Next step is to setup <span class="caps">WIFI</span>, you can skip this if you’re on Ethernet:</p>
<div class="highlight"><pre><span></span>wpa_passphrase SSID PASS > /etc/wpa_supplicant.conf
systemctl restart wpa_supplicant
</pre></div>
<p>The first command creates a config for wpa_supplicant.
The reader must fill in <span class="caps">SSID</span> and <span class="caps">PASS</span> of his target wifi network.
The second command tells systemd to go restart wpa_supplicant and use the new config.</p>
<p>Ask google if you’re online: <code>curl google.com</code> should return a 301 redirect:</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">HTML</span><span class="p">><</span><span class="nt">HEAD</span><span class="p">><</span><span class="nt">meta</span> <span class="na">http-equiv</span><span class="o">=</span><span class="s">"content-type"</span> <span class="na">content</span><span class="o">=</span><span class="s">"text/html;charset=utf-8"</span><span class="p">></span>
<span class="p"><</span><span class="nt">TITLE</span><span class="p">></span>301 Moved<span class="p"></</span><span class="nt">TITLE</span><span class="p">></</span><span class="nt">HEAD</span><span class="p">><</span><span class="nt">BODY</span><span class="p">></span>
<span class="p"><</span><span class="nt">H1</span><span class="p">></span>301 Moved<span class="p"></</span><span class="nt">H1</span><span class="p">></span>
The document has moved
<span class="p"><</span><span class="nt">A</span> <span class="na">HREF</span><span class="o">=</span><span class="s">"http://www.google.com/"</span><span class="p">></span>here<span class="p"></</span><span class="nt">A</span><span class="p">></span>.
<span class="p"></</span><span class="nt">BODY</span><span class="p">></</span><span class="nt">HTML</span><span class="p">></span>
</pre></div>
<p>There is no point proceeding until you have networking.</p>
<h1 id="partitioning">Partitioning</h1>
<p>Now to setup the partitioning on the <span class="caps">RIGHT</span> device.
Choose carefully.
Use <code>lsblk</code> to figure out which device is <span class="caps">RIGHT</span>.
You’ll know it’s the <span class="caps">WRONG</span> device if you lose data after partitioning.
The <span class="caps">RIGHT</span> device will be called <code>$dev</code> hence forward.</p>
<p>There are no other partitioning tools than gdisk.
Only heretics believe there are.
Therefore we use gdisk:</p>
<div class="highlight"><pre><span></span>gdisk <span class="nv">$dev</span>
</pre></div>
<h2 id="gdisk-cheat-sheet">Gdisk cheat sheet</h2>
<table>
<thead>
<tr>
<th>Command</th>
<th>Effect</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>p</code></td>
<td>For printing, to see what’s going on.</td>
</tr>
<tr>
<td><code>d</code></td>
<td>For deletion, you should start out with deleting everything on <code>$dev</code>.</td>
</tr>
<tr>
<td><code>n</code></td>
<td>Is used for creating new partitions.</td>
</tr>
<tr>
<td><code>w</code></td>
<td>is used for writing once finished.</td>
</tr>
</tbody>
</table>
<p>This table just describes the commands needed for the intended partitioning.</p>
<h2 id="intended-partitioning">Intended partitioning</h2>
<table>
<thead>
<tr>
<th>Number</th>
<th>type</th>
<th>size</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>ef00</td>
<td>+500M</td>
</tr>
<tr>
<td>2</td>
<td>8200</td>
<td>+$(SIZE_RAM+ a little)G</td>
</tr>
<tr>
<td>3</td>
<td>8300</td>
<td>(rest of disk)</td>
</tr>
</tbody>
</table>
<p>The first partition will be boot,
the second swap<sup id="fnref-optional"><a class="footnote-ref" href="#fn-optional">1</a></sup>,
the third will be everything else.
We will encrypt everything else.
With type <code>ef00</code> we will use <span class="caps">UEFI</span> for booting.
Don’t worry. nix will handle that, mostly.
Done. Onwards!</p>
<h1 id="encryption">Encryption</h1>
<p>We use <code>cryptsetup</code> for encryption.
Make sure to select the right partition.
We do not want to encrypt the boot partition because then we can’t boot.
So if you followed above instructions it will be either <code>3</code> or <code>p3</code>
(depending on device type).
We’ll call it <code>3</code>.</p>
<div class="highlight"><pre><span></span>cryptsetup luksFormat <span class="s2">"</span><span class="nv">$dev</span><span class="s2">"</span><span class="m">3</span>
cryptsetup open <span class="s2">"</span><span class="nv">$dev</span><span class="s2">"</span><span class="m">3</span> nixenc
</pre></div>
<p>The first command does the actual formatting,
the second one opens up the formatted disk.
You’ll need to provide the right password in both cases.
Choose one you can remember but is strong.
Once decrypted the disk will be mapped to <code>/dev/mapper/nixenc</code>,
note that we supplied that final part in the last command.</p>
<h1 id="formatting-filesystems">Formatting filesystems</h1>
<p>Partitioning is a distinct step from setting up filesystems.</p>
<div class="highlight"><pre><span></span>mkfs.vfat -n boot <span class="s2">"</span><span class="nv">$dev</span><span class="s2">"</span><span class="m">1</span>
mkswap <span class="s2">"</span><span class="nv">$dev</span><span class="s2">"</span><span class="m">2</span>
swapon <span class="s2">"</span><span class="nv">$dev</span><span class="s2">"</span><span class="m">2</span>
mkfs.btrfs -L root /dev/mapper/nixenc
</pre></div>
<p>The boot partition will be <code>vfat</code> because <a href="https://wiki.archlinux.org/index.php/EFI_system_partition"><span class="caps">UEFI</span> tells us to</a>.
The everything else partition will be <code>btrfs</code>,
because why are you following this guide if not?
Note that we point it at the mapped file,
if the <code>"$dev"3</code>device were to be used directly we’d remove the encryption.</p>
<h1 id="moutning-and-subvolumes">Moutning and subvolumes</h1>
<p>Wouldn’t it be nice to have subvolumes on your <span class="caps">BTRFS</span>?
This is not <a href="https://en.wikipedia.org/wiki/Cargo_cult_programming">cargo culted</a>
at all.</p>
<div class="highlight"><pre><span></span>mount -t btrfs /dev/mapper/nixenc /mnt/
btrfs subvol create /mnt/nixos
umount /mnt
mount -t btrfs -o <span class="nv">subvol</span><span class="o">=</span>nixos /dev/mapper/nixenc /mnt
</pre></div>
<p>First we create a nixos subvolume below the root subvolume,
eg the nixos operating system will not be installed in the root,
but one node below the root,
allowing potentially more operating systems to be installed on the same
partition.
Reverse explaining cargo culting behavior,
this maybe a good idea.</p>
<div class="highlight"><pre><span></span>btrfs subvol create /mnt/home
</pre></div>
<p>Setting up a subvolume for <code>home</code> allows btrfs based backups.
I used too also make subvolumes for <code>tmp</code><sup id="fnref-cmod"><a class="footnote-ref" href="#fn-cmod">2</a></sup> and <code>var</code>,
but I don’t see the merit in that.</p>
<div class="highlight"><pre><span></span>mkdir /mnt/boot
mount <span class="s2">"</span><span class="nv">$dev</span><span class="s2">"</span><span class="m">1</span> /mnt/boot
</pre></div>
<p>Here we mount the boot partition.
Just to make it detectable by the nix config generation script.</p>
<h2 id="did-i-do-everything-right">Did I do everything right?</h2>
<p>The second time I ran trough this post everything went
quite quickly,
so I became skeptical.
To verify everything was sane I used the following commands:</p>
<div class="highlight"><pre><span></span>mount <span class="p">|</span> grep /mnt
ls /mnt
</pre></div>
<p>The first command is to check if the encrypted volume and boot is mounted at
the right paths.
The second one to verify the folders are created, which are subvolumes.
The subvolume command creates a folder so if it exists we presume it worked.
But if you’re really unsure you can use <code>btrfs subvol list /mnt/</code>.</p>
<h1 id="configure-nix">Configure nix</h1>
<p>We can use hardware detection to figure out how to setup nix on this setup:</p>
<div class="highlight"><pre><span></span>nixos-generate-config --root /mnt
</pre></div>
<p>Done.</p>
<p>Now the user needs to write his own <a href="https://nixos.org/nixos/manual/index.html#sec-changing-config">nix config</a>,
or <a href="https://github.com/jappeace/linux-config/blob/master/configuration.nix">copy mine</a>
or cherry pick whatever they need (recommend).
On a wireless laptop,
it’s highly recommended to enable <a href="https://nixos.wiki/wiki/Wpa_supplicant">wpa_supplicant</a>:</p>
<div class="highlight"><pre><span></span>networking<span class="o">.</span>wireless<span class="o">.</span><span class="ss">enable =</span> <span class="no">true</span>
</pre></div>
<p>Once configuration is done we can install nix:</p>
<div class="highlight"><pre><span></span>nixos-install
</pre></div>
<p>Don’t worry, we can use <code>nixos-rebuild switch</code> to reconfigure nix whenever once
we’re booted into it.
Hopefully we boot successfully:</p>
<div class="highlight"><pre><span></span>reboot
</pre></div>
<p>Booting is hard, don’t worry if this goes wrong the first <s>10</s> 30 times.</p>
<p>You may need to enable <span class="caps">UEFI</span> in your <span class="caps">BIOS</span>.
It’s up to the reader to figure that part out.
(press some f keys on boot, f11 maybe?).
Alternatively one could setup grub. Good luck with that.</p>
<p>You can’t read the rest of this post until you’ve booted,
go back if you haven’t, you messed up.</p>
<h1 id="final-steps">Final steps</h1>
<p>Once rebooted you may be stuck at the display manager.
Use <code>Alt+f1</code> to switch to another <span class="caps">TTY</span> and login as root,
then use <code>passwd your-user-name</code> to set an initial password.
Use <code>Alt+f7</code> to go back to the display manager.</p>
<p>I personally haven’t moved all my configuration into nix yet
(it’s a big project),
but I wrote a <a href="https://github.com/jappeace/linux-config/blob/master/scripts/nixos-setup.sh">script</a>
that symlinks all dotfiles, and hardlinks
the <code>configuration.nix</code> to my linux-config project.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-optional">
<p>This one is optional but allows hibernation.
Which is very convenient for laptops.
It can also make your system <a href="https://askubuntu.com/questions/291378/do-we-still-need-swap-partitions-on-servers">more stable</a>.
Note that
Swap files are <a href="https://wiki.archlinux.org/index.php/Btrfs#Swap_file">bad</a>
on <span class="caps">BTRFS</span>. <a class="footnote-backref" href="#fnref-optional" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-cmod">
<p>If you want a subvolume for /tmp, make sure to chmod it to 777
Otherwise various applications get upset.
Pulse audio for example doesn’t work well if it can’t write into <code>/tmp</code>. <a class="footnote-backref" href="#fnref-cmod" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>Pragmatic Haskell III: Beam Postgres DB2018-08-05T17:30:00+02:002023-08-19T12:32:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2018-08-05:/pragmatic-haskell-iii-beam-postgres-db.html<blockquote>
<p>Note, I no longer recommend using beam for business as <span class="caps">ORM</span>.
Please use <a href="https://www.yesodweb.com/book/persistent">persistent</a> instead.
Beam is far to complicated for it’s use case.
Migrations can be run with <a href="https://hackage.haskell.org/package/postgresql-migration">postgresql-migration</a> for example, using the suggested migrations from persistent.
I’d only recommend using beam for hobby projects or as …</p></blockquote><blockquote>
<p>Note, I no longer recommend using beam for business as <span class="caps">ORM</span>.
Please use <a href="https://www.yesodweb.com/book/persistent">persistent</a> instead.
Beam is far to complicated for it’s use case.
Migrations can be run with <a href="https://hackage.haskell.org/package/postgresql-migration">postgresql-migration</a> for example, using the suggested migrations from persistent.
I’d only recommend using beam for hobby projects or as a case study for <a href="https://okmij.org/ftp/tagless-final/JFP.pdf">finally tagless</a>.</p>
</blockquote>
<p>No need to read a book to use Haskell!
This post will get you going with a serious web application while
only sticking to the concepts that are encountered.
This is a Haskell safari with as end goal a working webapp with database.</p>
<ol>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-simple-servant-web-server.html">Pragmatic Haskell: Simple servant web server</a></li>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-ii-io-webservant.html">Pragmatic Haskell <span class="caps">II</span>: <span class="caps">IO</span> Webservant</a></li>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-iii-beam-postgres-db.html">Pragmatic Haskell <span class="caps">III</span>: Beam Postgres <span class="caps">DB</span></a></li>
</ol>
<p><img alt="fancy db image" src="/images/2018/haskell-beam-postgres.svg"></p>
<p>Web applications need to store data.
In the <a href="https://jappieklooster.nl/pragmatic-haskell-ii-io-webservant.html">previous blog post</a>
we did this in a file for simplicity.
Now we will use something more appropriate: A relational database.
The beam library is used for this because it is closest to the “<span class="caps">ORM</span>” way of
thinking: Model a schema, generate <span class="caps">SQL</span> to query that schema,
and have migrations to move between different versions of that schema.
Migrations are left for another post for simplicity.</p>
<p><a href="#complete-sources">For the inpatient: Resulting source</a></p>
<h1 id="preparation">Preparation</h1>
<p>Unfortunately this post requires us to do quite a bit of devops to get started.
We need to:</p>
<ol>
<li>Install Postgres</li>
<li>Create a user</li>
<li>Create db</li>
<li>Populate structure</li>
</ol>
<p>Installing Postgres is out of the scope of this post.
We sidestep using migrations for with the <code>data_model.sql</code> file
<a href="#data_modelsql">(see sources)</a>.
Use the following commands to prepare the database:</p>
<div class="highlight"><pre><span></span>sudo -u postgres createuser -s <span class="nv">$USER</span>
dropdb awesome_db
createdb awesome_db
psql -f ./data_model.sql -d awesome_db
</pre></div>
<p>Congratulations, devops was survived.
Note that using this sql file is not idiomatic to beam.
The schema should be managed by beam,
but getting migrations to function is currently hard (it’s a work in progress).</p>
<h1 id="creating-structure">Creating structure</h1>
<p>The beam library models our desired structure at type level.
This is done in a separate file called <code>DB.hs</code>.
It can be seen in <a href="#dbhs">the sources</a>.
With help of this code beam can inspect the definitions,
and it also provides type safety for the beam sql domain specific language.
In other words if the migrations work there would be a path towards bringing
the database up to date with the code base,
or a compile error.
This is very pleasant because we get a thight feedback loop.
Now let us carefully inspect that file to understand it.</p>
<h2 id="language-extensions">Language extensions</h2>
<div class="highlight"><pre><span></span><span class="cm">{-# LANGUAGE DeriveGeneric #-}</span>
<span class="cm">{-# LANGUAGE FlexibleInstances #-}</span>
<span class="cm">{-# LANGUAGE MultiParamTypeClasses #-}</span>
<span class="cm">{-# LANGUAGE OverloadedStrings #-}</span>
<span class="cm">{-# LANGUAGE StandaloneDeriving #-}</span>
<span class="cm">{-# LANGUAGE TypeFamilies #-}</span>
<span class="cm">{-# LANGUAGE DuplicateRecordFields #-}</span>
</pre></div>
<p>This looks daunting however these extensions are individually quite simple.
Generally they either make nicer <span class="caps">API</span>’s or an easier language to use.
We will go over each of them.</p>
<h3 id="standalonederiving">StandaloneDeriving</h3>
<p><code>StandaloneDeriving</code> allows us to use the derive mechanism outside of a data
declaration, for example:</p>
<div class="highlight"><pre><span></span><span class="kr">deriving</span> <span class="kr">instance</span> <span class="kt">Show</span> <span class="kt">Message</span>
</pre></div>
<p>This mechanism allows deriving (automatic code generation)
to be used more flexibly.
In this case we want to do this because the <code>MessageT f</code> type constructor
to derive (as <code>f</code> is unknown), but <code>MessageT Identity</code> is known
so we can derive that.
The <span class="caps">GHC</span> <a href="https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/glasgow_exts.html#stand-alone-deriving-declarations">manual lists</a>
more possible reasons to derive like this instead of the standard method.</p>
<h3 id="typefamilies">TypeFamilies</h3>
<p>Type families allow us to declare <code>data</code> inside an instance.
The <code>Table</code> class requires <code>TypeFamilies</code> to instantiate because
it needs a <code>data</code> called primary key in it’s instance:</p>
<div class="highlight"><pre><span></span><span class="kr">instance</span> <span class="kt">Table</span> <span class="kt">UserT</span> <span class="kr">where</span>
<span class="kr">data</span> <span class="kt">PrimaryKey</span> <span class="kt">UserT</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">UserId</span> <span class="p">(</span><span class="kt">Columnar</span> <span class="n">f</span> <span class="kt">Int</span><span class="p">)</span> <span class="kr">deriving</span> <span class="kt">Generic</span>
<span class="n">primaryKey</span> <span class="ow">=</span> <span class="kt">UserId</span> <span class="o">.</span> <span class="n">_id</span>
</pre></div>
<p>This example is a prime reason to use type families:
It allows beam to assume the primary key exist for all tables.</p>
<p>The Haskell <a href="https://wiki.haskell.org/GHC/Type_families">wiki</a>
goes more in depth on type families</p>
<h3 id="flexibleinstances">FlexibleInstances</h3>
<p>If we don’t enable <code>FlexibleInstances</code> we get the following error:</p>
<div class="highlight"><pre><span></span>/home/jappie/projects/haskell/awesome-project-name/src/DB.hs:36:10: error:
• Illegal instance declaration for ‘Beamable (PrimaryKey MessageT)’
(All instance types must be of the form (T a1 ... an)
where a1 ... an are *distinct type variables*,
and each type variable appears at most once in the instance head.
Use FlexibleInstances if you want to disable this.)
• In the instance declaration for ‘Beamable (PrimaryKey MessageT)’
|
36 | instance Beamable (PrimaryKey MessageT)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
</pre></div>
<p>Without Flexible instances parenthesis aren’t allowed.
We know the parenthesis are the problem because the following line
does not get an error:</p>
<div class="highlight"><pre><span></span><span class="kr">instance</span> <span class="kt">Beamable</span> <span class="kt">MessageT</span>
</pre></div>
<h3 id="multiparamtypeclasses">MultiParamTypeClasses</h3>
<p>If <code>MultiParamTypeClasses</code> is disabled an error appears:</p>
<div class="highlight"><pre><span></span>/home/jappie/projects/haskell/awesome-project-name/src/DB.hs:65:10: error:
• Illegal instance declaration for ‘Database be AwesomeDb’
(Only one type can be given in an instance head.
Use MultiParamTypeClasses if you want to allow more, or zero.)
• In the instance declaration for ‘Database be AwesomeDb’
|
65 | instance Database be AwesomeDb
| ^^^^^^^^^^^^^^^^^^^^^
</pre></div>
<p>Because we use two parameters for this instance (<code>be</code> and <code>AwesomeDb</code>).
By default Haskell only allows one.</p>
<h3 id="derivegeneric">DeriveGeneric</h3>
<p><code>DeriveGeneric</code> was discussed in a
<a href="https://jappieklooster.nl/pragmatic-haskell-simple-servant-web-server.html">previous blog post</a>.
In short: <code>Generic</code> allows for introspection of data structures using the
fact any data structure can be modeled in a regular (generic) pattern.</p>
<h3 id="overloadedstrings">OverloadedStrings</h3>
<p><code>OverloadedStrings</code> is probably the most common language extension.
It converts string automatically, for example <code>String -> ByteString</code>.
In our case it’s only used for connection string:</p>
<div class="highlight"><pre><span></span><span class="nf">connectionString</span> <span class="ow">::</span> <span class="kt">BS</span><span class="o">.</span><span class="kt">ByteString</span>
<span class="nf">connectionString</span> <span class="ow">=</span> <span class="s">"dbname=awesome_db"</span>
</pre></div>
<p>In this case it inserts automatically a function <code>String -> ByteString</code>.
Using this extension avoids tedious conversions.</p>
<h3 id="duplicaterecordfields">DuplicateRecordFields</h3>
<p><code>DuplicateRecordFields</code> allows creation of records with the same name.
For example both user and messages have an <code>_id</code> record.
Because they have the same name,
type annotations are used to determine which function is called,
for example in:</p>
<div class="highlight"><pre><span></span><span class="kr">instance</span> <span class="kt">Table</span> <span class="kt">UserT</span> <span class="kr">where</span>
<span class="kr">data</span> <span class="kt">PrimaryKey</span> <span class="kt">UserT</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">UserId</span> <span class="p">(</span><span class="kt">Columnar</span> <span class="n">f</span> <span class="kt">Int</span><span class="p">)</span> <span class="kr">deriving</span> <span class="kt">Generic</span>
<span class="n">primaryKey</span> <span class="ow">=</span> <span class="kt">UserId</span> <span class="o">.</span> <span class="p">(</span><span class="n">_id</span> <span class="ow">::</span> <span class="kt">UserT</span> <span class="n">f</span> <span class="ow">-></span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Int</span><span class="p">)</span>
</pre></div>
<h2 id="imports">Imports</h2>
<div class="highlight"><pre><span></span><span class="kr">import</span> <span class="k">qualified</span> <span class="nn">Data.ByteString</span> <span class="k">as</span> <span class="n">BS</span>
</pre></div>
<p>ByteString is required for the type signature of connection string.</p>
<div class="highlight"><pre><span></span><span class="kr">import</span> <span class="k">qualified</span> <span class="nn">Data.Text</span> <span class="k">as</span> <span class="n">Text</span>
</pre></div>
<p>Text is required for the column datatype.</p>
<div class="highlight"><pre><span></span><span class="kr">import</span> <span class="nn">Database.Beam</span>
</pre></div>
<p>We import all of beam, for convenience.
This entire module is a client of beam, there is no need
for explicit imports.</p>
<h2 id="user-table">User table</h2>
<p>We start with defining the structure of our user table.</p>
<div class="highlight"><pre><span></span><span class="kr">data</span> <span class="kt">UserT</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">User</span>
<span class="p">{</span> <span class="n">_id</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Int</span>
<span class="p">,</span> <span class="n">_name</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span>
<span class="p">,</span> <span class="n">_email</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span>
<span class="p">}</span>
<span class="kr">deriving</span> <span class="kt">Generic</span>
</pre></div>
<p>What are these <code>C</code> and <code>f</code><span class="quo">‘</span>s doing here?
The <code>C</code> is an abbreviation for
<a href="http://hackage.haskell.org/package/beam-core-0.7.2.2/docs/Database-Beam-Schema.html#t:Columnar">Columnar</a>,
which is a type that requires two other types to complete.
In this case <code>C</code> is given an <code>f</code>,
and a second argument with the actual type of the column.
<code>f</code> is not defined, instead it’s also an argument of <code>UserT</code>, therefore <code>UserT</code>
is of kind <code>* -> *</code>. What we know however is that this <code>f</code> is the same for all
columns in <code>UserT</code>.</p>
<p>Now this <code>f</code> can be thought of as a ‘gap’ that can be filled up with anything.
This gap allows the beam library to inspect the structure we have defined,
and therefore create a schema out of it and hold the data of a row.
The <a href="https://hackage.haskell.org/package/beam-0.3.0.0/docs/Database-Beam-Schema.html#g:2">hackage page</a>
goes deeper into the definition.</p>
<div class="highlight"><pre><span></span><span class="kr">type</span> <span class="kt">User</span> <span class="ow">=</span> <span class="kt">UserT</span> <span class="kt">Identity</span>
</pre></div>
<p>This defines a type alias. A userT with Identity is simply a user.
In this case we are filling the <code>f</code> with Identity, <a href="http://hackage.haskell.org/package/base-4.11.1.0/docs/Data-Functor-Identity.html">a container</a>
that exposes on function <code>runIdentity</code> which removes the container and
does nothing with the content.
This is usefull for the case where we want to have the user as a result from
the database.</p>
<div class="highlight"><pre><span></span><span class="kr">deriving</span> <span class="kr">instance</span> <span class="kt">Show</span> <span class="p">(</span><span class="kt">PrimaryKey</span> <span class="kt">UserT</span> <span class="kt">Identity</span><span class="p">)</span>
<span class="kr">deriving</span> <span class="kr">instance</span> <span class="kt">Show</span> <span class="kt">User</span>
</pre></div>
<p>Implement the <code>show</code> function automatically for the identity case and for the
primary key which will be introduced below.</p>
<div class="highlight"><pre><span></span><span class="kr">instance</span> <span class="kt">Table</span> <span class="kt">UserT</span> <span class="kr">where</span>
<span class="kr">data</span> <span class="kt">PrimaryKey</span> <span class="kt">UserT</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">UserId</span> <span class="p">(</span><span class="kt">Columnar</span> <span class="n">f</span> <span class="kt">Int</span><span class="p">)</span> <span class="kr">deriving</span> <span class="kt">Generic</span>
<span class="n">primaryKey</span> <span class="ow">=</span> <span class="kt">UserId</span> <span class="o">.</span> <span class="p">(</span><span class="n">_id</span> <span class="ow">::</span> <span class="kt">UserT</span> <span class="n">f</span> <span class="ow">-></span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Int</span><span class="p">)</span>
<span class="kr">type</span> <span class="kt">UserId</span> <span class="ow">=</span> <span class="kt">PrimaryKey</span> <span class="kt">UserT</span> <span class="kt">Identity</span> <span class="c1">-- For convenience</span>
</pre></div>
<p>Here we make UserT an instance of a Table.
To do this we must specify a PrimaryKey, of which we define the type in the data
line.
The next line tells which function must be used to access the primary key.
In other words we implement the primaryKey function here by saying <code>_id</code>
must be used for it.
We use a type annotation to indicate which <code>_id</code> function is used.
Leaving that out results in an ambigious type error.</p>
<div class="highlight"><pre><span></span><span class="kr">instance</span> <span class="kt">Beamable</span> <span class="kt">UserT</span>
<span class="kr">instance</span> <span class="kt">Beamable</span> <span class="p">(</span><span class="kt">PrimaryKey</span> <span class="kt">UserT</span><span class="p">)</span>
</pre></div>
<p><code>Beamable</code> provides several <a href="http://hackage.haskell.org/package/beam-core-0.7.2.2/docs/Database-Beam-Schema.html#t:Beamable">‘introspection routines’</a>.
We require it to create a <code>Database</code> out of the table.
The database will be described below.</p>
<h2 id="message-table">Message table</h2>
<div class="highlight"><pre><span></span><span class="kr">data</span> <span class="kt">MessageT</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">Message</span>
<span class="p">{</span> <span class="n">_id</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Int</span>
<span class="p">,</span> <span class="n">_from</span> <span class="ow">::</span> <span class="kt">PrimaryKey</span> <span class="kt">UserT</span> <span class="n">f</span>
<span class="p">,</span> <span class="n">_content</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span>
<span class="p">}</span>
<span class="kr">deriving</span> <span class="kt">Generic</span>
<span class="kr">type</span> <span class="kt">Message</span> <span class="ow">=</span> <span class="kt">MessageT</span> <span class="kt">Identity</span>
<span class="kr">deriving</span> <span class="kr">instance</span> <span class="kt">Show</span> <span class="p">(</span><span class="kt">PrimaryKey</span> <span class="kt">MessageT</span> <span class="kt">Identity</span><span class="p">)</span>
<span class="kr">deriving</span> <span class="kr">instance</span> <span class="kt">Show</span> <span class="kt">Message</span>
<span class="kr">instance</span> <span class="kt">Table</span> <span class="kt">MessageT</span> <span class="kr">where</span>
<span class="kr">data</span> <span class="kt">PrimaryKey</span> <span class="kt">MessageT</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">MessageId</span> <span class="p">(</span><span class="kt">Columnar</span> <span class="n">f</span> <span class="kt">Int</span><span class="p">)</span> <span class="kr">deriving</span> <span class="kt">Generic</span>
<span class="n">primaryKey</span> <span class="ow">=</span> <span class="kt">MessageId</span> <span class="o">.</span> <span class="p">(</span><span class="n">_id</span> <span class="ow">::</span> <span class="kt">MessageT</span> <span class="n">f</span> <span class="ow">-></span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Int</span><span class="p">)</span>
<span class="kr">type</span> <span class="kt">MessageId</span> <span class="ow">=</span> <span class="kt">PrimaryKey</span> <span class="kt">MessageT</span> <span class="kt">Identity</span> <span class="c1">-- For convenience</span>
<span class="kr">instance</span> <span class="kt">Beamable</span> <span class="kt">MessageT</span>
<span class="kr">instance</span> <span class="kt">Beamable</span> <span class="p">(</span><span class="kt">PrimaryKey</span> <span class="kt">MessageT</span><span class="p">)</span>
</pre></div>
<p>The boiler plate is similar to that of User, the only new concept is the
from field, which points at the user table with the primary key.
Beam can make joins on this with the <span class="caps">DSL</span>.</p>
<h2 id="database">Database</h2>
<div class="highlight"><pre><span></span><span class="kr">data</span> <span class="kt">AwesomeDb</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">AwesomeDb</span>
<span class="p">{</span> <span class="n">_users</span> <span class="ow">::</span> <span class="n">f</span> <span class="p">(</span><span class="kt">TableEntity</span> <span class="kt">UserT</span><span class="p">)</span>
<span class="p">,</span> <span class="n">_messages</span> <span class="ow">::</span> <span class="n">f</span> <span class="p">(</span><span class="kt">TableEntity</span> <span class="kt">MessageT</span><span class="p">)</span> <span class="p">}</span>
<span class="kr">deriving</span> <span class="kt">Generic</span>
</pre></div>
<p>This type defines the entire database.
Again it provides a ‘hole’ with the <code>f</code>.</p>
<div class="highlight"><pre><span></span><span class="nf">connectionString</span> <span class="ow">::</span> <span class="kt">BS</span><span class="o">.</span><span class="kt">ByteString</span>
<span class="nf">connectionString</span> <span class="ow">=</span> <span class="s">"dbname=awesome_db"</span>
</pre></div>
<p>The <code>connectionString</code> is required to connect to the database.
One probably doesn’t want to hard code it,
but for this guide hard coding is good enough.</p>
<div class="highlight"><pre><span></span><span class="kr">instance</span> <span class="kt">Database</span> <span class="n">be</span> <span class="kt">AwesomeDb</span>
</pre></div>
<p>Here we create a beam database out of the AwesomeDB type.
the <code>be</code> hole is reserved for a back end, which we don’t specify.</p>
<div class="highlight"><pre><span></span><span class="nf">awesomeDB</span> <span class="ow">::</span> <span class="kt">DatabaseSettings</span> <span class="n">be</span> <span class="kt">AwesomeDb</span>
<span class="nf">awesomeDB</span> <span class="ow">=</span> <span class="n">defaultDbSettings</span>
</pre></div>
<p>The implementation of AwesomeDB just uses the default database settings.
All structural information is already provided at type level.</p>
<h1 id="using-structure">Using structure</h1>
<p>Now we have a database structure defined we can use it
in <code>Lib.hs</code>.
We have already seen most of this source file in the previous <a href="https://jappieklooster.nl/pragmatic-haskell-ii-io-webservant.html">blog post</a>,
the new version can be seen in <a href="#dbhs">the sources</a>.</p>
<p>The functionality is still the same except now we’re using a database as
backend rather than a file.
Just like with the previous post, the example shows both how to insert,
as well as retrieve data.
Let’s inspect the new changes.</p>
<div class="highlight"><pre><span></span><span class="nf">messages</span> <span class="ow">::</span> <span class="kt">Connection</span> <span class="ow">-></span> <span class="kt">Message</span> <span class="ow">-></span> <span class="kt">Handler</span> <span class="p">[</span><span class="kt">Message</span><span class="p">]</span>
<span class="nf">messages</span> <span class="n">conn</span> <span class="n">message</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">messages</span> <span class="ow"><-</span> <span class="n">liftIO</span> <span class="o">$</span>
</pre></div>
<p>Messages will hold the resulting messages we’re to querying from the database.
<code>liftIO</code> allows functions within the <code>IO</code> context (eg, interact with the world).</p>
<div class="highlight"><pre><span></span> <span class="kt">PgBeam</span><span class="o">.</span><span class="n">runBeamPostgres</span> <span class="n">conn</span> <span class="o">$</span> <span class="kr">do</span>
</pre></div>
<p>Run the beam Monad with help of a connection.
In other words, everything within this do block is a query for the database,
and we’re explicitly using postgres to solve ambiguity.</p>
<div class="highlight"><pre><span></span> <span class="kr">let</span> <span class="n">user</span> <span class="ow">=</span> <span class="n">from</span> <span class="n">message</span>
</pre></div>
<p>Retrieve the user from message for convenience.</p>
<div class="highlight"><pre><span></span> <span class="p">[</span><span class="n">user</span><span class="p">]</span> <span class="ow"><-</span> <span class="n">runInsertReturningList</span> <span class="p">(</span><span class="kt">DB</span><span class="o">.</span><span class="n">_users</span> <span class="kt">DB</span><span class="o">.</span><span class="n">awesomeDB</span><span class="p">)</span> <span class="o">$</span> <span class="kt">Beam</span><span class="o">.</span><span class="n">insertExpressions</span> <span class="p">[</span><span class="kt">DB</span><span class="o">.</span><span class="kt">User</span><span class="p">{</span>
<span class="kt">DB</span><span class="o">.</span><span class="n">_userId</span> <span class="ow">=</span> <span class="kt">Beam</span><span class="o">.</span><span class="n">default_</span><span class="p">,</span>
<span class="kt">DB</span><span class="o">.</span><span class="n">_name</span> <span class="ow">=</span> <span class="kt">Beam</span><span class="o">.</span><span class="n">val_</span> <span class="p">(</span><span class="n">pack</span> <span class="o">$</span> <span class="n">name</span> <span class="o">$</span> <span class="n">user</span> <span class="p">),</span>
<span class="kt">DB</span><span class="o">.</span><span class="n">_email</span> <span class="ow">=</span> <span class="kt">Beam</span><span class="o">.</span><span class="n">val_</span> <span class="p">(</span><span class="n">pack</span> <span class="o">$</span> <span class="n">email</span> <span class="o">$</span> <span class="n">user</span> <span class="p">)</span>
<span class="p">}]</span>
</pre></div>
<p>the <code>runInsertReturningList</code> function call is quite complex.
The first argument defines in which table we’re using, we want to insert
something into the users table.
The second argument is a list of expressions. To get the expressions we use the
<code>Beam.insertExpressions</code> function.
This is how we insert an item, in this case we only want to insert one user.
We use the User constructor defined earlier in the <code>DB.hs</code> module to obtain a user.
The fields are populated with values or a special default value.</p>
<p>Note that although this function is complex, if we do anything wrong we get a
type error.
Our code will not compile unless we do it right.
This is one of the strengths of beam.</p>
<div class="highlight"><pre><span></span> <span class="kr">_</span> <span class="ow"><-</span> <span class="n">runInsertReturningList</span> <span class="p">(</span><span class="kt">DB</span><span class="o">.</span><span class="n">_messages</span> <span class="kt">DB</span><span class="o">.</span><span class="n">awesomeDB</span><span class="p">)</span> <span class="o">$</span> <span class="kt">Beam</span><span class="o">.</span><span class="n">insertExpressions</span> <span class="o">$</span> <span class="p">[</span><span class="kt">DB</span><span class="o">.</span><span class="kt">Message</span><span class="p">{</span>
<span class="kt">DB</span><span class="o">.</span><span class="n">_messageId</span> <span class="ow">=</span> <span class="kt">Beam</span><span class="o">.</span><span class="n">default_</span><span class="p">,</span>
<span class="kt">DB</span><span class="o">.</span><span class="n">_from</span> <span class="ow">=</span> <span class="kt">Beam</span><span class="o">.</span><span class="n">val_</span> <span class="p">(</span><span class="kt">Beam</span><span class="o">.</span><span class="n">pk</span> <span class="n">user</span><span class="p">),</span>
<span class="kt">DB</span><span class="o">.</span><span class="n">_content</span> <span class="ow">=</span> <span class="kt">Beam</span><span class="o">.</span><span class="n">val_</span> <span class="p">(</span><span class="n">pack</span> <span class="o">$</span> <span class="n">content</span> <span class="n">message</span><span class="p">)</span>
<span class="p">}]</span>
</pre></div>
<p>These lines insert the message into the db,
linking it up with the newly inserted
user trough the pk.</p>
<div class="highlight"><pre><span></span> <span class="kt">Beam</span><span class="o">.</span><span class="n">runSelectReturningList</span> <span class="o">$</span> <span class="kt">Beam</span><span class="o">.</span><span class="n">select</span> <span class="o">$</span> <span class="kr">do</span>
<span class="n">usr</span> <span class="ow"><-</span> <span class="p">(</span><span class="kt">Beam</span><span class="o">.</span><span class="n">all_</span> <span class="p">(</span><span class="kt">DB</span><span class="o">.</span><span class="n">_users</span> <span class="kt">DB</span><span class="o">.</span><span class="n">awesomeDB</span><span class="p">))</span>
<span class="n">msg</span> <span class="ow"><-</span> <span class="kt">Beam</span><span class="o">.</span><span class="n">oneToMany_</span> <span class="p">(</span><span class="kt">DB</span><span class="o">.</span><span class="n">_messages</span> <span class="kt">DB</span><span class="o">.</span><span class="n">awesomeDB</span><span class="p">)</span> <span class="kt">DB</span><span class="o">.</span><span class="n">_from</span> <span class="n">usr</span>
<span class="n">pure</span> <span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">usr</span><span class="p">)</span>
</pre></div>
<p>This query gets the resulting messages and their respective users joined together.</p>
<div class="highlight"><pre><span></span> <span class="n">pure</span> <span class="o">$</span>
<span class="n">fmap</span> <span class="p">(</span>
<span class="nf">\</span><span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">usr</span><span class="p">)</span> <span class="ow">-></span> <span class="kt">Message</span>
<span class="p">(</span><span class="kt">User</span>
<span class="p">(</span><span class="n">unpack</span> <span class="o">$</span> <span class="kt">DB</span><span class="o">.</span><span class="n">_name</span> <span class="n">usr</span><span class="p">)</span>
<span class="p">(</span><span class="n">unpack</span> <span class="o">$</span> <span class="kt">DB</span><span class="o">.</span><span class="n">_email</span> <span class="n">usr</span><span class="p">))</span>
<span class="p">(</span><span class="n">unpack</span> <span class="o">$</span> <span class="kt">DB</span><span class="o">.</span><span class="n">_content</span> <span class="n">msg</span><span class="p">)</span>
<span class="p">)</span> <span class="n">messages</span>
</pre></div>
<p>here we convert the database user and database message, to the ‘<span class="caps">API</span>’ user and
‘<span class="caps">API</span>’ messages.
The reason we need to do this is because our database data structure does not
implement <code>toJSON</code>.
Also the database structure has extra information such as the primary key which
we may want to hide from <span class="caps">API</span> clients.</p>
<h1 id="execute">Execute!</h1>
<p>To run the program we use:</p>
<div class="highlight"><pre><span></span> stack build
stack <span class="nb">exec</span> webservice
</pre></div>
<p>To test it a simple curl request was made:</p>
<div class="highlight"><pre><span></span> curl --header <span class="s2">"Content-Type: application/json"</span> -v --data <span class="s1">'{"from":{"email":"d","name":"xyz"}, "content": "does it word?"}'</span> http://127.0.0.1:6868/message/
</pre></div>
<p>We can inspect the database with postgres</p>
<div class="highlight"><pre><span></span> psql <span class="s2">"dbname=awesome_db"</span>
<span class="se">\d</span>t
<span class="k">select</span> * from messages<span class="p">;</span>
</pre></div>
<h1 id="conclusion">Conclusion</h1>
<p>We have looked at the beam library in this post and it’s interaction with
postgres.
Although the example is simple, there is quite a bit of boilerplate involved,
but once setup it provides a complete type safe <span class="caps">DSL</span> to the database.
With the database and web server in place nothing is stopping the reader from
making his next major project in Haskell!
We hereby conclude our Haskell safari successfully. </p>
<h1 id="complete-sources">Complete sources</h1>
<p>The complete sources can be found on <a href="https://github.com/jappeace/awesome-project-name/tree/beam-postgre-no-migrate">github</a>, and below.</p>
<h2 id="dbhs">Db.hs</h2>
<div class="highlight"><pre><span></span><span class="cm">{-# LANGUAGE DeriveGeneric #-}</span>
<span class="cm">{-# LANGUAGE FlexibleInstances #-}</span>
<span class="cm">{-# LANGUAGE MultiParamTypeClasses #-}</span>
<span class="cm">{-# LANGUAGE OverloadedStrings #-}</span>
<span class="cm">{-# LANGUAGE StandaloneDeriving #-}</span>
<span class="cm">{-# LANGUAGE TypeFamilies #-}</span>
<span class="cm">{-# LANGUAGE DuplicateRecordFields #-}</span>
<span class="c1">-- | db structure and source of truth</span>
<span class="kr">module</span> <span class="nn">DB</span> <span class="kr">where</span>
<span class="kr">import</span> <span class="k">qualified</span> <span class="nn">Data.ByteString</span> <span class="k">as</span> <span class="n">BS</span>
<span class="kr">import</span> <span class="k">qualified</span> <span class="nn">Data.Text</span> <span class="k">as</span> <span class="n">Text</span>
<span class="kr">import</span> <span class="nn">Database.Beam</span>
<span class="kr">data</span> <span class="kt">UserT</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">User</span>
<span class="p">{</span> <span class="n">_id</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Int</span>
<span class="p">,</span> <span class="n">_name</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span>
<span class="p">,</span> <span class="n">_email</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span>
<span class="p">}</span>
<span class="kr">deriving</span> <span class="kt">Generic</span>
<span class="kr">type</span> <span class="kt">User</span> <span class="ow">=</span> <span class="kt">UserT</span> <span class="kt">Identity</span>
<span class="kr">deriving</span> <span class="kr">instance</span> <span class="kt">Show</span> <span class="kt">UserId</span>
<span class="kr">deriving</span> <span class="kr">instance</span> <span class="kt">Show</span> <span class="kt">User</span>
<span class="kr">instance</span> <span class="kt">Table</span> <span class="kt">UserT</span> <span class="kr">where</span>
<span class="kr">data</span> <span class="kt">PrimaryKey</span> <span class="kt">UserT</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">UserId</span> <span class="p">(</span><span class="kt">Columnar</span> <span class="n">f</span> <span class="kt">Int</span><span class="p">)</span> <span class="kr">deriving</span> <span class="kt">Generic</span>
<span class="n">primaryKey</span> <span class="ow">=</span> <span class="kt">UserId</span> <span class="o">.</span> <span class="p">(</span><span class="n">_id</span> <span class="ow">::</span> <span class="kt">UserT</span> <span class="n">f</span> <span class="ow">-></span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Int</span><span class="p">)</span>
<span class="kr">type</span> <span class="kt">UserId</span> <span class="ow">=</span> <span class="kt">PrimaryKey</span> <span class="kt">UserT</span> <span class="kt">Identity</span> <span class="c1">-- For convenience</span>
<span class="kr">instance</span> <span class="kt">Beamable</span> <span class="kt">UserT</span>
<span class="kr">instance</span> <span class="kt">Beamable</span> <span class="p">(</span><span class="kt">PrimaryKey</span> <span class="kt">UserT</span><span class="p">)</span>
<span class="kr">data</span> <span class="kt">MessageT</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">Message</span>
<span class="p">{</span> <span class="n">_id</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Int</span>
<span class="p">,</span> <span class="n">_from</span> <span class="ow">::</span> <span class="kt">PrimaryKey</span> <span class="kt">UserT</span> <span class="n">f</span>
<span class="p">,</span> <span class="n">_content</span> <span class="ow">::</span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Text</span><span class="o">.</span><span class="kt">Text</span>
<span class="p">}</span>
<span class="kr">deriving</span> <span class="kt">Generic</span>
<span class="kr">type</span> <span class="kt">Message</span> <span class="ow">=</span> <span class="kt">MessageT</span> <span class="kt">Identity</span>
<span class="kr">deriving</span> <span class="kr">instance</span> <span class="kt">Show</span> <span class="p">(</span><span class="kt">PrimaryKey</span> <span class="kt">MessageT</span> <span class="kt">Identity</span><span class="p">)</span>
<span class="kr">deriving</span> <span class="kr">instance</span> <span class="kt">Show</span> <span class="kt">Message</span>
<span class="kr">instance</span> <span class="kt">Table</span> <span class="kt">MessageT</span> <span class="kr">where</span>
<span class="kr">data</span> <span class="kt">PrimaryKey</span> <span class="kt">MessageT</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">MessageId</span> <span class="p">(</span><span class="kt">Columnar</span> <span class="n">f</span> <span class="kt">Int</span><span class="p">)</span> <span class="kr">deriving</span> <span class="kt">Generic</span>
<span class="n">primaryKey</span> <span class="ow">=</span> <span class="kt">MessageId</span> <span class="o">.</span> <span class="p">(</span><span class="n">_id</span> <span class="ow">::</span> <span class="kt">MessageT</span> <span class="n">f</span> <span class="ow">-></span> <span class="kt">C</span> <span class="n">f</span> <span class="kt">Int</span><span class="p">)</span>
<span class="kr">type</span> <span class="kt">MessageId</span> <span class="ow">=</span> <span class="kt">PrimaryKey</span> <span class="kt">MessageT</span> <span class="kt">Identity</span> <span class="c1">-- For convenience</span>
<span class="kr">instance</span> <span class="kt">Beamable</span> <span class="kt">MessageT</span>
<span class="kr">instance</span> <span class="kt">Beamable</span> <span class="p">(</span><span class="kt">PrimaryKey</span> <span class="kt">MessageT</span><span class="p">)</span>
<span class="kr">data</span> <span class="kt">AwesomeDb</span> <span class="n">f</span> <span class="ow">=</span> <span class="kt">AwesomeDb</span>
<span class="p">{</span> <span class="n">_ausers</span> <span class="ow">::</span> <span class="n">f</span> <span class="p">(</span><span class="kt">TableEntity</span> <span class="kt">UserT</span><span class="p">)</span>
<span class="p">,</span> <span class="n">_messages</span> <span class="ow">::</span> <span class="n">f</span> <span class="p">(</span><span class="kt">TableEntity</span> <span class="kt">MessageT</span><span class="p">)</span> <span class="p">}</span>
<span class="kr">deriving</span> <span class="kt">Generic</span>
<span class="nf">connectionString</span> <span class="ow">::</span> <span class="kt">BS</span><span class="o">.</span><span class="kt">ByteString</span>
<span class="nf">connectionString</span> <span class="ow">=</span> <span class="s">"dbname=awesome_db"</span>
<span class="kr">instance</span> <span class="kt">Database</span> <span class="n">be</span> <span class="kt">AwesomeDb</span>
<span class="nf">awesomeDB</span> <span class="ow">::</span> <span class="kt">DatabaseSettings</span> <span class="n">be</span> <span class="kt">AwesomeDb</span>
<span class="nf">awesomeDB</span> <span class="ow">=</span> <span class="n">defaultDbSettings</span>
</pre></div>
<h2 id="libhs">Lib.hs</h2>
<div class="highlight"><pre><span></span><span class="cm">{-# LANGUAGE DataKinds #-}</span>
<span class="cm">{-# LANGUAGE TypeOperators #-}</span>
<span class="cm">{-# LANGUAGE DeriveGeneric #-}</span>
<span class="kr">module</span> <span class="nn">Lib</span>
<span class="p">(</span> <span class="nf">webAppEntry</span>
<span class="p">)</span> <span class="kr">where</span>
<span class="kr">import</span> <span class="nn">Servant</span>
<span class="kr">import</span> <span class="nn">Control.Monad.IO.Class</span><span class="p">(</span><span class="n">liftIO</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Data.ByteString.Lazy</span> <span class="k">as</span> <span class="n">LBS</span> <span class="p">(</span><span class="n">writeFile</span><span class="p">,</span> <span class="n">readFile</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Data.Aeson</span><span class="p">(</span><span class="kt">ToJSON</span><span class="p">,</span> <span class="kt">FromJSON</span><span class="p">,</span> <span class="n">encode</span><span class="p">,</span> <span class="n">decode</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">GHC.Generics</span><span class="p">(</span><span class="kt">Generic</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Network.Wai</span><span class="p">(</span><span class="kt">Application</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Network.Wai.Handler.Warp</span><span class="p">(</span><span class="n">run</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Database.PostgreSQL.Simple</span> <span class="p">(</span><span class="kt">Connection</span><span class="p">)</span>
<span class="kr">import</span> <span class="k">qualified</span> <span class="nn">DB</span> <span class="k">as</span> <span class="n">DB</span>
<span class="kr">import</span> <span class="nn">Database.Beam.Backend.SQL.BeamExtensions</span> <span class="p">(</span><span class="nf">runInsertReturningList</span><span class="p">)</span>
<span class="kr">import</span> <span class="k">qualified</span> <span class="nn">Database.Beam</span> <span class="k">as</span> <span class="n">Beam</span>
<span class="kr">import</span> <span class="k">qualified</span> <span class="nn">Database.Beam.Postgres</span> <span class="k">as</span> <span class="n">PgBeam</span>
<span class="kr">import</span> <span class="nn">Data.Text</span><span class="p">(</span><span class="n">pack</span><span class="p">,</span> <span class="n">unpack</span><span class="p">)</span>
<span class="kr">type</span> <span class="kt">UserAPI</span> <span class="ow">=</span> <span class="s">"users"</span> <span class="kt">:></span> <span class="kt">Get</span> <span class="kt">'[JSON]</span> <span class="p">[</span><span class="kt">User</span><span class="p">]</span>
<span class="kt">:<|></span> <span class="s">"message"</span> <span class="kt">:></span> <span class="kt">ReqBody</span> <span class="kt">'[JSON]</span> <span class="kt">Message</span> <span class="kt">:></span> <span class="kt">Post</span> <span class="kt">'[JSON]</span> <span class="p">[</span><span class="kt">Message</span><span class="p">]</span>
<span class="kr">data</span> <span class="kt">Message</span> <span class="ow">=</span> <span class="kt">Message</span> <span class="p">{</span>
<span class="n">from</span> <span class="ow">::</span> <span class="kt">User</span><span class="p">,</span>
<span class="n">content</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">}</span> <span class="kr">deriving</span> <span class="p">(</span><span class="kt">Eq</span><span class="p">,</span> <span class="kt">Show</span><span class="p">,</span> <span class="kt">Generic</span><span class="p">)</span>
<span class="kr">instance</span> <span class="kt">ToJSON</span> <span class="kt">Message</span>
<span class="kr">instance</span> <span class="kt">FromJSON</span> <span class="kt">Message</span>
<span class="kr">data</span> <span class="kt">User</span> <span class="ow">=</span> <span class="kt">User</span>
<span class="p">{</span> <span class="n">name</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">,</span> <span class="n">email</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">}</span> <span class="kr">deriving</span> <span class="p">(</span><span class="kt">Eq</span><span class="p">,</span> <span class="kt">Show</span><span class="p">,</span> <span class="kt">Generic</span><span class="p">)</span>
<span class="kr">instance</span> <span class="kt">ToJSON</span> <span class="kt">User</span>
<span class="kr">instance</span> <span class="kt">FromJSON</span> <span class="kt">User</span>
<span class="nf">users</span> <span class="ow">::</span> <span class="p">[</span><span class="kt">User</span><span class="p">]</span>
<span class="nf">users</span> <span class="ow">=</span>
<span class="p">[</span> <span class="kt">User</span> <span class="s">"Isaac Newton"</span> <span class="s">"isaac@newton.co.uk"</span>
<span class="p">,</span> <span class="kt">User</span> <span class="s">"Albert Einstein"</span> <span class="s">"ae@mc2.org"</span>
<span class="p">]</span>
<span class="nf">messages</span> <span class="ow">::</span> <span class="kt">Connection</span> <span class="ow">-></span> <span class="kt">Message</span> <span class="ow">-></span> <span class="kt">Handler</span> <span class="p">[</span><span class="kt">Message</span><span class="p">]</span>
<span class="nf">messages</span> <span class="n">conn</span> <span class="n">message</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">messages</span> <span class="ow"><-</span> <span class="n">liftIO</span> <span class="o">$</span>
<span class="kt">PgBeam</span><span class="o">.</span><span class="n">runBeamPostgres</span> <span class="n">conn</span> <span class="o">$</span> <span class="kr">do</span>
<span class="kr">let</span> <span class="n">user</span> <span class="ow">=</span> <span class="n">from</span> <span class="n">message</span>
<span class="p">[</span><span class="n">user</span><span class="p">]</span> <span class="ow"><-</span> <span class="n">runInsertReturningList</span> <span class="p">(</span><span class="kt">DB</span><span class="o">.</span><span class="n">_ausers</span> <span class="kt">DB</span><span class="o">.</span><span class="n">awesomeDB</span><span class="p">)</span> <span class="o">$</span>
<span class="kt">Beam</span><span class="o">.</span><span class="n">insertExpressions</span> <span class="p">[</span><span class="kt">DB</span><span class="o">.</span><span class="kt">User</span>
<span class="kt">Beam</span><span class="o">.</span><span class="n">default_</span>
<span class="p">(</span><span class="kt">Beam</span><span class="o">.</span><span class="n">val_</span> <span class="p">(</span><span class="n">pack</span> <span class="o">$</span> <span class="n">name</span> <span class="o">$</span> <span class="n">user</span> <span class="p">))</span>
<span class="p">(</span><span class="kt">Beam</span><span class="o">.</span><span class="n">val_</span> <span class="p">(</span><span class="n">pack</span> <span class="o">$</span> <span class="n">email</span> <span class="o">$</span> <span class="n">user</span> <span class="p">))</span>
<span class="p">]</span>
<span class="kr">_</span> <span class="ow"><-</span> <span class="n">runInsertReturningList</span> <span class="p">(</span><span class="kt">DB</span><span class="o">.</span><span class="n">_messages</span> <span class="kt">DB</span><span class="o">.</span><span class="n">awesomeDB</span><span class="p">)</span> <span class="o">$</span>
<span class="kt">Beam</span><span class="o">.</span><span class="n">insertExpressions</span>
<span class="p">[</span><span class="kt">DB</span><span class="o">.</span><span class="kt">Message</span>
<span class="kt">Beam</span><span class="o">.</span><span class="n">default_</span>
<span class="p">(</span><span class="kt">Beam</span><span class="o">.</span><span class="n">val_</span> <span class="p">(</span><span class="kt">Beam</span><span class="o">.</span><span class="n">pk</span> <span class="n">user</span><span class="p">))</span>
<span class="p">(</span><span class="kt">Beam</span><span class="o">.</span><span class="n">val_</span> <span class="p">(</span><span class="n">pack</span> <span class="o">$</span> <span class="n">content</span> <span class="n">message</span><span class="p">))</span>
<span class="p">]</span>
<span class="kt">Beam</span><span class="o">.</span><span class="n">runSelectReturningList</span> <span class="o">$</span> <span class="kt">Beam</span><span class="o">.</span><span class="n">select</span> <span class="o">$</span> <span class="kr">do</span>
<span class="n">usr</span> <span class="ow"><-</span> <span class="p">(</span><span class="kt">Beam</span><span class="o">.</span><span class="n">all_</span> <span class="p">(</span><span class="kt">DB</span><span class="o">.</span><span class="n">_ausers</span> <span class="kt">DB</span><span class="o">.</span><span class="n">awesomeDB</span><span class="p">))</span>
<span class="n">msg</span> <span class="ow"><-</span> <span class="kt">Beam</span><span class="o">.</span><span class="n">oneToMany_</span> <span class="p">(</span><span class="kt">DB</span><span class="o">.</span><span class="n">_messages</span> <span class="kt">DB</span><span class="o">.</span><span class="n">awesomeDB</span><span class="p">)</span> <span class="kt">DB</span><span class="o">.</span><span class="n">_from</span> <span class="n">usr</span>
<span class="n">pure</span> <span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">usr</span><span class="p">)</span>
<span class="n">pure</span> <span class="o">$</span>
<span class="n">fmap</span> <span class="p">(</span>
<span class="nf">\</span><span class="p">(</span><span class="n">msg</span><span class="p">,</span> <span class="n">usr</span><span class="p">)</span> <span class="ow">-></span> <span class="kt">Message</span>
<span class="p">(</span><span class="kt">User</span>
<span class="p">(</span><span class="n">unpack</span> <span class="o">$</span> <span class="kt">DB</span><span class="o">.</span><span class="n">_name</span> <span class="n">usr</span><span class="p">)</span>
<span class="p">(</span><span class="n">unpack</span> <span class="o">$</span> <span class="kt">DB</span><span class="o">.</span><span class="n">_email</span> <span class="n">usr</span><span class="p">))</span>
<span class="p">(</span><span class="n">unpack</span> <span class="o">$</span> <span class="kt">DB</span><span class="o">.</span><span class="n">_content</span> <span class="n">msg</span><span class="p">)</span>
<span class="p">)</span> <span class="n">messages</span>
<span class="nf">server</span> <span class="ow">::</span> <span class="kt">Connection</span> <span class="ow">-></span> <span class="kt">Server</span> <span class="kt">UserAPI</span>
<span class="nf">server</span> <span class="n">conn</span><span class="ow">=</span> <span class="p">(</span><span class="n">pure</span> <span class="n">users</span><span class="p">)</span> <span class="kt">:<|></span> <span class="p">(</span><span class="n">messages</span> <span class="n">conn</span><span class="p">)</span>
<span class="nf">userAPI</span> <span class="ow">::</span> <span class="kt">Proxy</span> <span class="kt">UserAPI</span>
<span class="nf">userAPI</span> <span class="ow">=</span> <span class="kt">Proxy</span>
<span class="nf">app</span> <span class="ow">::</span> <span class="kt">Connection</span> <span class="ow">-></span> <span class="kt">Application</span>
<span class="nf">app</span> <span class="n">conn</span> <span class="ow">=</span> <span class="n">serve</span> <span class="n">userAPI</span> <span class="p">(</span><span class="n">server</span> <span class="n">conn</span><span class="p">)</span>
<span class="nf">webAppEntry</span> <span class="ow">::</span> <span class="kt">Connection</span> <span class="ow">-></span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="nf">webAppEntry</span> <span class="n">conn</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">run</span> <span class="mi">6868</span> <span class="p">(</span><span class="n">app</span> <span class="n">conn</span><span class="p">)</span>
</pre></div>
<h2 id="data_modelsql">data_model.sql</h2>
<div class="highlight"><pre><span></span><span class="k">DROP</span> <span class="k">TABLE</span> <span class="n">ausers</span> <span class="k">cascade</span><span class="p">;</span>
<span class="k">DROP</span> <span class="k">TABLE</span> <span class="n">messages</span> <span class="k">cascade</span><span class="p">;</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">ausers</span> <span class="p">(</span>
<span class="n">id</span> <span class="nb">serial</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span>
<span class="ss">"name"</span> <span class="nb">varchar</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">email</span> <span class="nb">varchar</span> <span class="k">NULL</span>
<span class="p">);</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">messages</span> <span class="p">(</span>
<span class="n">id</span> <span class="nb">serial</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span>
<span class="n">from__id</span> <span class="nb">int</span> <span class="k">REFERENCES</span> <span class="n">ausers</span><span class="p">(</span><span class="n">id</span><span class="p">),</span>
<span class="n">content</span> <span class="nb">varchar</span> <span class="k">NULL</span>
<span class="p">);</span>
</pre></div>Pragmatic Haskell II: IO Webservant2018-06-27T22:00:00+02:002019-10-24T18:34:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2018-06-27:/pragmatic-haskell-ii-io-webservant.html<ol>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-simple-servant-web-server.html">Pragmatic Haskell: Simple servant web server</a></li>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-ii-io-webservant.html">Pragmatic Haskell <span class="caps">II</span>: <span class="caps">IO</span> Webservant</a></li>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-iii-beam-postgres-db.html">Pragmatic Haskell <span class="caps">III</span>: Beam Postgres <span class="caps">DB</span></a></li>
</ol>
<p>Most Haskell language guides will leave <span class="caps">IO</span>
<a href="http://www.seas.upenn.edu/%7Ecis194/spring13/lectures/08-IO.html">until</a>
<a href="http://learnyouahaskell.com/input-and-output">later</a>.
This guide is different,
this guide is about <em>using</em> Haskell.
Our focus is different: We build first, then learn trough <a href="https://medium.com/the-polymath-project/programming-for-personal-growth-64052e407894">delight</a>.</p>
<p><img alt="Fancy intro image" src="/images/2018/io-webserver.svg"></p>
<p>The <a href="https://jappieklooster.nl/pragmatic-haskell-simple-servant-web-server.html">previous blog …</a></p><ol>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-simple-servant-web-server.html">Pragmatic Haskell: Simple servant web server</a></li>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-ii-io-webservant.html">Pragmatic Haskell <span class="caps">II</span>: <span class="caps">IO</span> Webservant</a></li>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-iii-beam-postgres-db.html">Pragmatic Haskell <span class="caps">III</span>: Beam Postgres <span class="caps">DB</span></a></li>
</ol>
<p>Most Haskell language guides will leave <span class="caps">IO</span>
<a href="http://www.seas.upenn.edu/%7Ecis194/spring13/lectures/08-IO.html">until</a>
<a href="http://learnyouahaskell.com/input-and-output">later</a>.
This guide is different,
this guide is about <em>using</em> Haskell.
Our focus is different: We build first, then learn trough <a href="https://medium.com/the-polymath-project/programming-for-personal-growth-64052e407894">delight</a>.</p>
<p><img alt="Fancy intro image" src="/images/2018/io-webserver.svg"></p>
<p>The <a href="https://jappieklooster.nl/pragmatic-haskell-simple-servant-web-server.html">previous blog</a>
post explained how to get going with a simple minimalist servant
web server.
In this blog post the simple web server will get an extra <span class="caps">REST</span> endpoint that can
do <span class="caps">IO</span> actions.
This is an important part of pragmatic Haskell programming.
Without <span class="caps">IO</span> our program can do nothing.
Programmers are not theorists, therefore we need <span class="caps">IO</span>.</p>
<h1 id="preparation">Preparation</h1>
<p>To keep things simple, the code assumes a file exists.
Create one with an empty <span class="caps">JSON</span> array in the project root:</p>
<div class="highlight"><pre><span></span><span class="nb">echo</span> <span class="s2">"[]"</span> > messages.txt
</pre></div>
<p>Bytestrings are a convenient way of opening files and putting the results into
<code>aeson</code>, the <span class="caps">JSON</span> library.
Dealing with bytestrings requires another dependency:</p>
<div class="highlight"><pre><span></span><span class="l l-Scalar l-Scalar-Plain">dependencies</span><span class="p p-Indicator">:</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">base >= 4.7 && < 5</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">servant-server</span> <span class="c1"># http server</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">aeson</span> <span class="c1"># json</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">wai</span> <span class="c1"># web application (interface)</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">warp</span> <span class="c1"># web application implementation</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">bytestring</span>
</pre></div>
<h1 id="a-lot-of-code">A lot of code</h1>
<p>These are the <s>magic spells</s> changes which add an endpoint,
explained in detail below:</p>
<div class="highlight"><pre><span></span><span class="cm">{-# LANGUAGE DataKinds #-}</span>
<span class="cm">{-# LANGUAGE TypeOperators #-}</span>
<span class="cm">{-# LANGUAGE DeriveGeneric #-}</span>
<span class="kr">module</span> <span class="nn">Lib</span>
<span class="p">(</span> <span class="nf">webAppEntry</span>
<span class="p">)</span> <span class="kr">where</span>
<span class="kr">import</span> <span class="nn">Servant</span>
<span class="kr">import</span> <span class="nn">Control.Monad.IO.Class</span><span class="p">(</span><span class="n">liftIO</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Data.ByteString.Lazy</span> <span class="k">as</span> <span class="n">LBS</span> <span class="p">(</span><span class="n">writeFile</span><span class="p">,</span> <span class="n">readFile</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Data.Aeson</span><span class="p">(</span><span class="kt">ToJSON</span><span class="p">,</span> <span class="kt">FromJSON</span><span class="p">,</span> <span class="n">encode</span><span class="p">,</span> <span class="n">decode</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">GHC.Generics</span><span class="p">(</span><span class="kt">Generic</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Network.Wai</span><span class="p">(</span><span class="kt">Application</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Network.Wai.Handler.Warp</span><span class="p">(</span><span class="n">run</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Data.Maybe</span> <span class="p">(</span><span class="nf">fromMaybe</span><span class="p">)</span>
<span class="kr">type</span> <span class="kt">UserAPI</span> <span class="ow">=</span> <span class="s">"users"</span> <span class="kt">:></span> <span class="kt">Get</span> <span class="kt">'[JSON]</span> <span class="p">[</span><span class="kt">User</span><span class="p">]</span>
<span class="kt">:<|></span> <span class="s">"message"</span> <span class="kt">:></span> <span class="kt">ReqBody</span> <span class="kt">'[JSON]</span> <span class="kt">Message</span> <span class="kt">:></span> <span class="kt">Post</span> <span class="kt">'[JSON]</span> <span class="p">[</span><span class="kt">Message</span><span class="p">]</span>
<span class="kr">data</span> <span class="kt">Message</span> <span class="ow">=</span> <span class="kt">Message</span> <span class="p">{</span>
<span class="n">from</span> <span class="ow">::</span> <span class="kt">User</span><span class="p">,</span>
<span class="n">content</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">}</span> <span class="kr">deriving</span> <span class="p">(</span><span class="kt">Eq</span><span class="p">,</span> <span class="kt">Show</span><span class="p">,</span> <span class="kt">Generic</span><span class="p">)</span>
<span class="kr">instance</span> <span class="kt">ToJSON</span> <span class="kt">Message</span>
<span class="kr">instance</span> <span class="kt">FromJSON</span> <span class="kt">Message</span>
<span class="kr">data</span> <span class="kt">User</span> <span class="ow">=</span> <span class="kt">User</span>
<span class="p">{</span> <span class="n">name</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">,</span> <span class="n">email</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">}</span> <span class="kr">deriving</span> <span class="p">(</span><span class="kt">Eq</span><span class="p">,</span> <span class="kt">Show</span><span class="p">,</span> <span class="kt">Generic</span><span class="p">)</span>
<span class="kr">instance</span> <span class="kt">ToJSON</span> <span class="kt">User</span>
<span class="kr">instance</span> <span class="kt">FromJSON</span> <span class="kt">User</span>
<span class="nf">users</span> <span class="ow">::</span> <span class="p">[</span><span class="kt">User</span><span class="p">]</span>
<span class="nf">users</span> <span class="ow">=</span>
<span class="p">[</span> <span class="kt">User</span> <span class="s">"Isaac Newton"</span> <span class="s">"isaac@newton.co.uk"</span>
<span class="p">,</span> <span class="kt">User</span> <span class="s">"Albert Einstein"</span> <span class="s">"ae@mc2.org"</span>
<span class="p">]</span>
<span class="nf">messageFile</span> <span class="ow">::</span> <span class="kt">FilePath</span>
<span class="nf">messageFile</span> <span class="ow">=</span> <span class="s">"messages.txt"</span>
<span class="nf">messages</span> <span class="ow">::</span> <span class="kt">Message</span> <span class="ow">-></span> <span class="kt">Handler</span> <span class="p">[</span><span class="kt">Message</span><span class="p">]</span>
<span class="nf">messages</span> <span class="n">message</span> <span class="ow">=</span> <span class="kr">do</span>
<span class="n">result</span> <span class="ow"><-</span> <span class="n">liftIO</span> <span class="o">$</span> <span class="kt">LBS</span><span class="o">.</span><span class="n">readFile</span> <span class="n">messageFile</span>
<span class="kr">case</span> <span class="n">decode</span> <span class="n">result</span> <span class="kr">of</span>
<span class="kt">Nothing</span> <span class="ow">-></span> <span class="n">pure</span> <span class="kt">[]</span>
<span class="kt">Just</span> <span class="n">x</span> <span class="ow">-></span> <span class="kr">do</span>
<span class="kr">let</span> <span class="n">contents</span> <span class="ow">=</span> <span class="n">x</span> <span class="o">++</span> <span class="p">[</span><span class="n">message</span><span class="p">]</span>
<span class="n">liftIO</span> <span class="o">$</span> <span class="kt">LBS</span><span class="o">.</span><span class="n">writeFile</span> <span class="n">messageFile</span> <span class="p">(</span><span class="n">encode</span> <span class="n">contents</span><span class="p">)</span>
<span class="n">return</span> <span class="n">contents</span>
<span class="nf">server</span> <span class="ow">::</span> <span class="kt">Server</span> <span class="kt">UserAPI</span>
<span class="nf">server</span> <span class="ow">=</span> <span class="p">(</span><span class="n">pure</span> <span class="n">users</span><span class="p">)</span> <span class="kt">:<|></span> <span class="n">messages</span>
<span class="nf">userAPI</span> <span class="ow">::</span> <span class="kt">Proxy</span> <span class="kt">UserAPI</span>
<span class="nf">userAPI</span> <span class="ow">=</span> <span class="kt">Proxy</span>
<span class="nf">app</span> <span class="ow">::</span> <span class="kt">Application</span>
<span class="nf">app</span> <span class="ow">=</span> <span class="n">serve</span> <span class="n">userAPI</span> <span class="n">server</span>
<span class="nf">webAppEntry</span> <span class="ow">::</span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="nf">webAppEntry</span> <span class="ow">=</span> <span class="n">run</span> <span class="mi">6868</span> <span class="n">app</span>
</pre></div>
<p>This will setup another endpoint for messages.
The new endpoint will accept a post request under “/message”.
It will write the message to a file and then it will return the contents of
that file.</p>
<h1 id="line-by-line-inspection">Line by line inspection</h1>
<div class="highlight"><pre><span></span><span class="kr">type</span> <span class="kt">UserAPI</span> <span class="ow">=</span> <span class="s">"users"</span> <span class="kt">:></span> <span class="kt">Get</span> <span class="kt">'[JSON]</span> <span class="p">[</span><span class="kt">User</span><span class="p">]</span>
<span class="kt">:<|></span> <span class="s">"message"</span> <span class="kt">:></span> <span class="kt">ReqBody</span> <span class="kt">'[JSON]</span> <span class="kt">Message</span> <span class="kt">:></span> <span class="kt">Post</span> <span class="kt">'[JSON]</span> <span class="p">[</span><span class="kt">Message</span><span class="p">]</span>
</pre></div>
<p>The
<a href="http://hackage.haskell.org/package/servant-0.14/docs/Servant-API-Alternative.html#t::-60--124--62-"><code>:<|></code> operator</a>
is used to add an extra endpoint.
It combines the two endpoints into one.
In these lines, we are constructing something akin to a jump table.
The decleration of this operator is surprisingly simple:</p>
<div class="highlight"><pre><span></span><span class="kr">data</span> <span class="n">a</span> <span class="kt">:<|></span> <span class="n">b</span> <span class="ow">=</span> <span class="n">a</span> <span class="kt">:<|></span> <span class="n">b</span>
</pre></div>
<p>Left sign of equality is used for type, right side for data construction.
Skimming over this, like true Haskellers we will ignore the
<a href="http://hackage.haskell.org/package/servant-0.14/docs/src/Servant-API-Alternative.html#%3A%3C%7C%3E">inner workings</a> .</p>
<p>The extra end point “message” is similar in structure to the existing “user”
endpoint.
If read like a sentence “message” is a <span class="caps">POST</span> only endpoint,
which accepts a Message <span class="caps">JSON</span> body, and it returns a list of Messages in <span class="caps">JSON</span>.</p>
<h2 id="message-data">Message data</h2>
<div class="highlight"><pre><span></span><span class="kr">data</span> <span class="kt">Message</span> <span class="ow">=</span> <span class="kt">Message</span> <span class="p">{</span>
<span class="n">from</span> <span class="ow">::</span> <span class="kt">User</span><span class="p">,</span>
<span class="n">content</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">}</span> <span class="kr">deriving</span> <span class="p">(</span><span class="kt">Eq</span><span class="p">,</span> <span class="kt">Show</span><span class="p">,</span> <span class="kt">Generic</span><span class="p">)</span>
<span class="kr">instance</span> <span class="kt">ToJSON</span> <span class="kt">Message</span>
<span class="kr">instance</span> <span class="kt">FromJSON</span> <span class="kt">Message</span>
</pre></div>
<p>This is Message, apparently it’s from a <code>User</code> and has some content <code>String</code>.
Aside from using another data type inside an existing data type,
no new concepts are introduced.</p>
<h2 id="file-path">File path</h2>
<div class="highlight"><pre><span></span><span class="nf">messageFile</span> <span class="ow">::</span> <span class="kt">FilePath</span>
</pre></div>
<p>The type <code>FilePath</code> is just an alias for a <code>String</code>: <code>type FilePath = String</code>.
In other words we can use them interchangeably.
<code>FilePath</code> acts as documentation.</p>
<div class="highlight"><pre><span></span><span class="nf">messageFile</span> <span class="ow">=</span> <span class="s">"messages.txt"</span>
</pre></div>
<p>This is where we define what file name is used.</p>
<h2 id="message-handler">Message handler</h2>
<div class="highlight"><pre><span></span><span class="nf">messages</span> <span class="ow">::</span> <span class="kt">Message</span> <span class="ow">-></span> <span class="kt">Handler</span> <span class="p">[</span><span class="kt">Message</span><span class="p">]</span>
</pre></div>
<p>We define a Handler (which is an servant api endpoint).
It requires a message, then it will return a list of messages
within a <code>Handler</code>.</p>
<h3 id="do-notation">Do notation</h3>
<div class="highlight"><pre><span></span><span class="nf">messages</span> <span class="n">message</span> <span class="ow">=</span> <span class="kr">do</span>
</pre></div>
<p>The do keyword allows us to code with <a href="https://en.wikibooks.org/wiki/Haskell/do_notation">do notation</a>.
This allows us to do assignments with <code><-</code> (not assignment, but close enough).
This only works when the result container is a <a href="https://wiki.haskell.org/Monad">Monad</a>.
<em>How</em> monads work is a mystery, but usage is simple: Use do notation.</p>
<h3 id="liftio">LiftIO</h3>
<div class="highlight"><pre><span></span> result <span class="o"><-</span> liftIO <span class="o">$</span> LBS.readFile messageFile
</pre></div>
<p>Here we read the message file as a lazy bytestring and put it into the result.
One may wonder why we are using liftIO, if it’s deleted we get this:</p>
<div class="highlight"><pre><span></span>/home/jappie/projects/haskell/awesome-project-name/src/Lib.hs:46:13: error:
• Couldnt match <span class="nb">type</span> ‘IO’ with ‘Handler’
Expected type: Handler Data.ByteString.Lazy.Internal.ByteString
Actual type: IO Data.ByteString.Lazy.Internal.ByteString
• In a stmt of a <span class="s1">'do'</span> block: result <- LBS.readFile messageFile
In the expression:
<span class="k">do</span> result <- LBS.readFile messageFile
<span class="k">case</span> decode result of
Nothing -> pure <span class="o">[]</span>
Just x -> <span class="k">do</span> ...
In an equation <span class="k">for</span> ‘messages’:
messages <span class="nv">message</span>
<span class="o">=</span> <span class="k">do</span> result <- LBS.readFile messageFile
<span class="k">case</span> decode result of
Nothing -> pure ...
Just x -> ...
<span class="p">|</span>
<span class="m">46</span> <span class="p">|</span> result <- LBS.readFile messageFile
<span class="p">|</span> ^^^^^^^^^^^^^^^^^^^^^^^^
</pre></div>
<p>The <code>LBS.readfile</code> function has a return type of <code>IO</code>.
However the return type of <code>messages</code> is <code>Handler</code>.
Therefore the compiler says that it expects <code>Handler</code>, but the actual type is <code>IO</code>.
Handler implements the <code>MonadIO</code> typeclass however, which allows
<code>IO</code> by calling the <code>liftIO</code> function.
The <code>liftIO</code> function simply tells the Handler container to execute some
function within the <code>IO</code> container.</p>
<p>The dollar sign can be replaced with an open parentheses,
which is closed at the end of the line. This is equivalent for example:</p>
<div class="highlight"><pre><span></span><span class="nf">liftIO</span> <span class="p">(</span><span class="kt">LBS</span><span class="o">.</span><span class="n">readFile</span> <span class="n">messageFile</span><span class="p">)</span>
</pre></div>
<p>As said before <code><-</code> is (basically) used for assignment in do notation,
using <code><-</code> is a good way to get rid of a monad container.
<code>result</code> now contains the contents of messageFile,
the <span class="caps">IO</span> is being evaluated and removed by the <code><-</code> operator.
In Haskell a lot of wrapping and unwrapping is done. </p>
<h3 id="case-of">Case .. of</h3>
<div class="highlight"><pre><span></span> <span class="kr">case</span> <span class="n">decode</span> <span class="n">result</span> <span class="kr">of</span>
</pre></div>
<p>Here we’re decoding the contents of <code>result</code>.
decoding <span class="caps">JSON</span> may not succeed and therefore the library authors of <code>aeson</code>
made
<a href="http://hackage.haskell.org/package/aeson-1.4.0.0/docs/Data-Aeson.html#v:decode">decode</a>
return a maybe container:</p>
<div class="highlight"><pre><span></span><span class="nf">decode</span> <span class="ow">::</span> <span class="kt">FromJSON</span> <span class="n">a</span> <span class="ow">=></span> <span class="kt">ByteString</span> <span class="ow">-></span> <span class="kt">Maybe</span> <span class="n">a</span>
</pre></div>
<p>In this signature, <code>a</code> can be anything as long as it implements FromJSON.
We fulfill this condition with generic and <code>instance FromJSON Message</code>.
We give as bytestring the <code>result</code> to decode, in return it gives us <code>Maybe a</code>.
The compiler deduces that <code>a</code> in this case is <code>[Message]</code>.</p>
<p>The return value will have content if decoding succeeded (<code>Just</code>),
or it won’t if it fails (<code>Nothing</code>).
To get rid of the container we pattern match it.
This can be thought of as a switch case statement in other languages.</p>
<h4 id="nothing">Nothing</h4>
<div class="highlight"><pre><span></span> <span class="kt">Nothing</span> <span class="ow">-></span> <span class="n">pure</span> <span class="kt">[]</span>
</pre></div>
<p>In this case decoding fails.
An empty list is returned to the client.
We still must wrap this list in a <code>Handler</code>, pure is used for that.
Note that <code>pure == return</code>.
These functions both exist for <a href="https://stackoverflow.com/questions/32788082/difference-between-return-and-pure">historical reasons</a>.</p>
<h4 id="just-a">Just a</h4>
<div class="highlight"><pre><span></span> <span class="kt">Just</span> <span class="n">x</span> <span class="ow">-></span> <span class="kr">do</span>
</pre></div>
<p>In this case there is success,
the result is taken and put into x,
after which another do block starts.</p>
<h3 id="let">Let</h3>
<div class="highlight"><pre><span></span> let contents = x ++ [message]
</pre></div>
<p>Unlike the <code><-</code> operator, let does not do any unwrapping.
We can see what happens if we replace the let binding by
<code>contents <- x ++ [message]</code>,
the errors are:</p>
<div class="highlight"><pre><span></span>/home/jappie/projects/haskell/awesome-project-name/src/Lib.hs:50:19: error:
• Couldnt match <span class="nb">type</span> ‘<span class="o">[]</span>’ with ‘Handler’
Expected type: Handler Message
Actual type: <span class="o">[</span>Message<span class="o">]</span>
• In a stmt of a <span class="s1">'do'</span> block: contents <- x ++ <span class="o">[</span>message<span class="o">]</span>
In the expression:
<span class="k">do</span> contents <- x ++ <span class="o">[</span>message<span class="o">]</span>
liftIO $ LBS.writeFile messageFile <span class="o">(</span>encode contents<span class="o">)</span>
<span class="k">return</span> contents
In a <span class="k">case</span> alternative:
Just x
-> <span class="k">do</span> contents <- x ++ <span class="o">[</span>message<span class="o">]</span>
liftIO $ LBS.writeFile messageFile <span class="o">(</span>encode contents<span class="o">)</span>
<span class="k">return</span> contents
<span class="p">|</span>
<span class="m">50</span> <span class="p">|</span> contents <- x ++ <span class="o">[</span>message<span class="o">]</span>
<span class="p">|</span> ^^^^^^^^^^^^^^
/home/jappie/projects/haskell/awesome-project-name/src/Lib.hs:52:7: error:
• Couldnt match <span class="nb">type</span> ‘Message’ with ‘<span class="o">[</span>Message<span class="o">]</span>’
Expected type: Handler <span class="o">[</span>Message<span class="o">]</span>
Actual type: Handler Message
• In a stmt of a <span class="s1">'do'</span> block: <span class="k">return</span> contents
In the expression:
<span class="k">do</span> contents <- x ++ <span class="o">[</span>message<span class="o">]</span>
liftIO $ LBS.writeFile messageFile <span class="o">(</span>encode contents<span class="o">)</span>
<span class="k">return</span> contents
In a <span class="k">case</span> alternative:
Just x
-> <span class="k">do</span> contents <- x ++ <span class="o">[</span>message<span class="o">]</span>
liftIO $ LBS.writeFile messageFile <span class="o">(</span>encode contents<span class="o">)</span>
<span class="k">return</span> contents
<span class="p">|</span>
<span class="m">52</span> <span class="p">|</span> <span class="k">return</span> contents
<span class="p">|</span> ^^^^^^^^^^^^^^^
</pre></div>
<p>In the first error the compiler says that a list is not a <code>Handler</code> container.
Which we expect because the return type of this function is <code>Handler [Message]</code>.</p>
<p>The second error assumes <code>contents</code> is of the correct type,
since <code><-</code> unwraps <code>contents</code> would be of type <code>Message</code>.
The return type does not fit in this case either,
we would get <code>Handler Message</code> instead of <code>Handler [Message]</code>.</p>
<h3 id="write">Write</h3>
<div class="highlight"><pre><span></span> <span class="n">liftIO</span> <span class="o">$</span> <span class="kt">LBS</span><span class="o">.</span><span class="n">writeFile</span> <span class="n">messageFile</span> <span class="p">(</span><span class="n">encode</span> <span class="n">contents</span><span class="p">)</span>
</pre></div>
<p>We encode the contents,
then we write it to the message file in an <span class="caps">IO</span> effect.
Type safety ensures encoding always succeeds.</p>
<h3 id="return">Return</h3>
<div class="highlight"><pre><span></span> <span class="n">return</span> <span class="n">contents</span>
</pre></div>
<p>Finally we wrap the contents in a <code>Handler</code> type.</p>
<h2 id="adding-the-handler-to-the-routes-map">Adding the handler to the routes map</h2>
<div class="highlight"><pre><span></span><span class="nf">server</span> <span class="ow">::</span> <span class="kt">Server</span> <span class="kt">UserAPI</span>
<span class="nf">server</span> <span class="ow">=</span> <span class="p">(</span><span class="n">pure</span> <span class="n">users</span><span class="p">)</span> <span class="kt">:<|></span> <span class="n">messages</span>
</pre></div>
<p>This is the implementation of the <code>UserAPI</code> type described before.
We use the same operator to also add the messages handler into the server.
We don’t need to put <code>messages</code> in a container with pure because the functions’
return type is already a <code>Handler</code>.</p>
<p>Something worth pointing out is that this construction is in order,
if we pull it out of order we would get a type level.
Changing the line into:</p>
<div class="highlight"><pre><span></span><span class="nf">server</span> <span class="ow">=</span> <span class="n">messages</span> <span class="kt">:<|></span> <span class="p">(</span><span class="n">pure</span> <span class="n">users</span><span class="p">)</span>
</pre></div>
<p>Will cause an error:</p>
<div class="highlight"><pre><span></span>/home/jappie/projects/haskell/awesome-project-name/src/Lib.hs:55:11: error:
• Could not match <span class="nb">type</span> ‘<span class="o">[</span>User<span class="o">]</span>’ with ‘Handler <span class="o">[</span>Message<span class="o">]</span>’
Expected type: Server UserAPI
Actual type: <span class="o">(</span>Message -> Handler <span class="o">[</span>Message<span class="o">])</span>
:<<span class="p">|</span>> <span class="o">(</span>Message -> <span class="o">[</span>User<span class="o">])</span>
• In the expression: messages :<<span class="p">|</span>> <span class="o">(</span>pure users<span class="o">)</span>
In an equation <span class="k">for</span> ‘server’: <span class="nv">server</span> <span class="o">=</span> messages :<<span class="p">|</span>> <span class="o">(</span>pure users<span class="o">)</span>
<span class="p">|</span>
<span class="m">55</span> <span class="p">|</span> <span class="nv">server</span> <span class="o">=</span> messages :<<span class="p">|</span>> <span class="o">(</span>pure users<span class="o">)</span>
<span class="p">|</span> ^^^^^^^^^^^^^^^^^^^^^^^^^^
</pre></div>
<h1 id="execute-it">Execute it!</h1>
<div class="highlight"><pre><span></span>curl --header <span class="s2">"Content-Type: application/json"</span> <span class="se">\ </span>
--request POST <span class="se">\</span>
--data <span class="s1">'{"from":{"email":"d","name":"xyz"}, "content": "does it word?"}'</span> <span class="se">\</span>
http://localhost:6868/message
</pre></div>
<p>Worked on this machine…</p>
<p><img alt="Worked on my machine, lol, now we use IO everything is over" src="/images/2018/machine-on-fire.svg"></p>
<p>Perhaps that’s why the theorists avoid <span class="caps">IO</span> 🤔.</p>
<h1 id="in-conclusion">In conclusion</h1>
<p>Without going into much theory, we dealt with <code>IO</code>.
For example we saw that haskell is about dealing with containers,
to put certain functions in <code>IO</code> rather than the return type, one uses <code>liftIO</code>.
<code>do</code> notation was also encountered, which makes working with monads easier.
Now we can affect the world with our programs trough <span class="caps">IO</span>!</p>
<p>The complete code can be found <a href="https://github.com/jappeace/awesome-project-name/tree/message-servant">here</a>.
In the future we shall attach this simple web server to a database.</p>Pragmatic Haskell: Simple servant web server2018-06-25T15:25:00+02:002018-06-25T15:25:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2018-06-25:/pragmatic-haskell-simple-servant-web-server.html<ol>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-simple-servant-web-server.html">Pragmatic Haskell: Simple servant web server</a></li>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-ii-io-webservant.html">Pragmatic Haskell <span class="caps">II</span>: <span class="caps">IO</span> Webservant</a></li>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-iii-beam-postgres-db.html">Pragmatic Haskell <span class="caps">III</span>: Beam Postgres <span class="caps">DB</span></a></li>
</ol>
<p>There are many <a href="https://github.com/bitemyapp/learnhaskell">guides available</a>
for learning Haskell.
Setting up a something simple like a web server isn’t so
straight forward.
Perhaps choosing one of the <a href="https://wiki.haskell.org/Web/Frameworks">14 libraries</a>
is a bit much …</p><ol>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-simple-servant-web-server.html">Pragmatic Haskell: Simple servant web server</a></li>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-ii-io-webservant.html">Pragmatic Haskell <span class="caps">II</span>: <span class="caps">IO</span> Webservant</a></li>
<li><a href="https://jappieklooster.nl/pragmatic-haskell-iii-beam-postgres-db.html">Pragmatic Haskell <span class="caps">III</span>: Beam Postgres <span class="caps">DB</span></a></li>
</ol>
<p>There are many <a href="https://github.com/bitemyapp/learnhaskell">guides available</a>
for learning Haskell.
Setting up a something simple like a web server isn’t so
straight forward.
Perhaps choosing one of the <a href="https://wiki.haskell.org/Web/Frameworks">14 libraries</a>
is a bit much.</p>
<p><img alt="Type level hell: Haskell sucks" src="/images/2018/haskell-sucks.jpg"></p>
<p>This guide will give opinionated web server start.
This guide assumes no experience with Haskell,
and will get you up to speed with a (<span class="caps">REST</span>) web server called <a href="http://haskell-servant.readthedocs.io/en/stable/">Servant</a>.
Servant is a good choice as it can describe both a server and client <span class="caps">API</span>.
In the future this guide may be used as a foundation to create something
more meaningful than just a very basic <span class="caps">REST</span> <span class="caps">API</span>,
this will provide a good starting point however.
Basic <span class="caps">UNIX</span> (command line) skills are assumed.</p>
<h1 id="from-nothing-start-with-build-tools">From nothing, start with build tools</h1>
<p>Install stack:</p>
<div class="highlight"><pre><span></span>curl -sSL https://get.haskellstack.org/ <span class="p">|</span> sh
</pre></div>
<p>Only attempt shortly to install it trough a package manager.
There are other Haskell build tools, they will be more difficult in use.
There is also the possibility for fully reproducible builds at a system level
(nix).
Which is out of the scope of this guide.</p>
<p>Now setup a new project:</p>
<div class="highlight"><pre><span></span>stack new awesome-project-name
<span class="nb">cd</span> awesome-project-name
</pre></div>
<h1 id="hello-world-with-stack">Hello world with stack</h1>
<p>Appreciate what happens when this is build:</p>
<div class="highlight"><pre><span></span>stack build <span class="o">&&</span> stack <span class="nb">exec</span> awesome-project-name-exe
</pre></div>
<p>This should build successfully and output <code>someFunc</code>.
Open up <code>src/Lib.hs</code> with one’s favorite editor.
This contains a few lines, created by stack:</p>
<div class="highlight"><pre><span></span><span class="kr">module</span> <span class="nn">Lib</span>
<span class="p">(</span> <span class="nf">someFunc</span>
<span class="p">)</span> <span class="kr">where</span>
<span class="nf">someFunc</span> <span class="ow">::</span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="nf">someFunc</span> <span class="ow">=</span> <span class="n">putStrLn</span> <span class="s">"someFunc"</span>
</pre></div>
<p>This is where the <code>someFunc</code> output came from when the program was ran.
Change it to something a bit more appropriate, and rename the function too:</p>
<div class="highlight"><pre><span></span><span class="kr">module</span> <span class="nn">Lib</span>
<span class="p">(</span> <span class="nf">webAppEntry</span>
<span class="p">)</span> <span class="kr">where</span>
<span class="nf">webAppEntry</span> <span class="ow">::</span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="nf">webAppEntry</span> <span class="ow">=</span> <span class="n">putStrLn</span> <span class="s">"This is the beginning of my greetings to world"</span>
</pre></div>
<p>Does it compile?</p>
<div class="highlight"><pre><span></span>stack build <span class="o">&&</span> stack <span class="nb">exec</span> awesome-project-name-exe
/home/jappie/projects/haskell/awesome-project-name/app/Main.hs:6:8: error: Variable not in scope: someFunc :: IO <span class="o">()</span>
<span class="se">\|</span>
<span class="m">6</span> <span class="se">\|</span> <span class="nv">main</span> <span class="o">=</span> someFunc
<span class="se">\|</span> ^^^^^^^^
</pre></div>
<p>It does not compile.
There is an app folder where by default all the executable reside
(which is where the error occurs),
and a <code>src</code> folder where the library code lives (the modified file is in there).
One can future proving themselves by putting as much code in the library as is reasonable.</p>
<p>Fix the error in <code>app/Main.hs</code>:</p>
<div class="highlight"><pre><span></span><span class="kr">module</span> <span class="nn">Main</span> <span class="kr">where</span>
<span class="kr">import</span> <span class="nn">Lib</span>
<span class="nf">main</span> <span class="ow">::</span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="nf">main</span> <span class="ow">=</span> <span class="n">webAppEntry</span>
</pre></div>
<p>It builds!
Functions can be renamed, simple compile errors can be solved, and strings
can be changed. Progress!</p>
<h1 id="servant-your-first-dependencies">Servant: Your first dependencies</h1>
<p>For the impatient, there is a minimal example already <a href="https://github.com/haskell-servant/example-servant-minimal">available</a>
by the library author.
This guide will explain how to get there step by step.
In <code>./package.yaml</code>, on line 22 there is a <code>dependencies</code> key,
add <code>servant-server</code>, <code>aeson</code>, <code>wai</code> and <code>warp</code> to it like this:</p>
<div class="highlight"><pre><span></span><span class="l l-Scalar l-Scalar-Plain">dependencies</span><span class="p p-Indicator">:</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">base >= 4.7 && < 5</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">servant-server</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">aeson</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">wai</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">warp</span>
</pre></div>
<p>It may seem strange to immediately add four new dependencies,
however this is because Haskell libraries are setup to be flexible.
Even small projects grow quickly to have into the twenties of dependencies.
Code reuse is not <a href="https://www.youtube.com/watch?v=Jn3kdTaa69U">a myth</a>.</p>
<p><code>servant-server</code> is the <a href="http://haskell-servant.readthedocs.io/en/stable/">servant web server</a>.
<a href="http://hackage.haskell.org/package/aeson"><code>aeson</code></a>
is for <span class="caps">JSON</span> parsing and producing.
<a href="http://hackage.haskell.org/package/wai"><code>wai</code></a> is a web application interface and
<a href="http://hackage.haskell.org/package/warp"><code>warp</code></a> uses <code>wai</code>
to implement a web application (it binds to the port).</p>
<p>Ensure that that this is done at the root of the yaml file (no indentation).
Stack provides a way of specifying dependencies of either the executable or
library.
If its done on line 22, the root of the yaml file,
it will be a dependency for everything in the project.</p>
<h1 id="a-minimal-servant">A minimal servant</h1>
<p>A good start is going to servants’ <a href="http://hackage.haskell.org/package/servant">Hackage</a>
page,
which linked to a <a href="http://haskell-servant.readthedocs.io/en/stable/tutorial/index.html">tutorial</a>.
Servant does <span class="caps">API</span> definition <a href="http://haskell-servant.readthedocs.io/en/stable/tutorial/ApiType.html">at type level</a>.</p>
<p>If it’s unknown to the reader what a type is, think of it as describing the
shape of a function.
Functions of different shapes don’t fit together, and won’t compile.
What servant allows us to do is define this shape for a <span class="caps">REST</span> <span class="caps">API</span>.
To gain a deeper understanding of this a concrete example will be inspected
line by line.
First all lines are listed for a minimal servant (<code>Lib.hs</code>) server:</p>
<div class="highlight"><pre><span></span><span class="cm">{-# LANGUAGE DataKinds #-}</span>
<span class="cm">{-# LANGUAGE TypeOperators #-}</span>
<span class="cm">{-# LANGUAGE DeriveGeneric #-}</span>
<span class="kr">module</span> <span class="nn">Lib</span>
<span class="p">(</span> <span class="nf">webAppEntry</span>
<span class="p">)</span> <span class="kr">where</span>
<span class="kr">import</span> <span class="nn">Servant</span><span class="p">(</span><span class="n">serve</span><span class="p">,</span> <span class="kt">Proxy</span><span class="p">(</span><span class="o">..</span><span class="p">),</span> <span class="kt">Server</span><span class="p">,</span> <span class="kt">JSON</span><span class="p">,</span> <span class="kt">Get</span><span class="p">,</span> <span class="p">(</span><span class="kt">:></span><span class="p">))</span>
<span class="kr">import</span> <span class="nn">Data.Aeson</span><span class="p">(</span><span class="kt">ToJSON</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">GHC.Generics</span><span class="p">(</span><span class="kt">Generic</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Network.Wai</span><span class="p">(</span><span class="kt">Application</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Network.Wai.Handler.Warp</span><span class="p">(</span><span class="n">run</span><span class="p">)</span>
<span class="kr">type</span> <span class="kt">UserAPI</span> <span class="ow">=</span> <span class="s">"users"</span> <span class="kt">:></span> <span class="kt">Get</span> <span class="kt">'[JSON]</span> <span class="p">[</span><span class="kt">User</span><span class="p">]</span>
<span class="kr">data</span> <span class="kt">User</span> <span class="ow">=</span> <span class="kt">User</span>
<span class="p">{</span> <span class="n">name</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">,</span> <span class="n">email</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">}</span> <span class="kr">deriving</span> <span class="p">(</span><span class="kt">Eq</span><span class="p">,</span> <span class="kt">Show</span><span class="p">,</span> <span class="kt">Generic</span><span class="p">)</span>
<span class="kr">instance</span> <span class="kt">ToJSON</span> <span class="kt">User</span>
<span class="nf">users</span> <span class="ow">::</span> <span class="p">[</span><span class="kt">User</span><span class="p">]</span>
<span class="nf">users</span> <span class="ow">=</span>
<span class="p">[</span> <span class="kt">User</span> <span class="s">"Isaac Newton"</span> <span class="s">"isaac@newton.co.uk"</span>
<span class="p">,</span> <span class="kt">User</span> <span class="s">"Albert Einstein"</span> <span class="s">"ae@mc2.org"</span>
<span class="p">]</span>
<span class="nf">server</span> <span class="ow">::</span> <span class="kt">Server</span> <span class="kt">UserAPI</span>
<span class="nf">server</span> <span class="ow">=</span> <span class="n">return</span> <span class="n">users</span>
<span class="nf">userAPI</span> <span class="ow">::</span> <span class="kt">Proxy</span> <span class="kt">UserAPI</span>
<span class="nf">userAPI</span> <span class="ow">=</span> <span class="kt">Proxy</span>
<span class="nf">app</span> <span class="ow">::</span> <span class="kt">Application</span>
<span class="nf">app</span> <span class="ow">=</span> <span class="n">serve</span> <span class="n">userAPI</span> <span class="n">server</span>
<span class="nf">webAppEntry</span> <span class="ow">::</span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="nf">webAppEntry</span> <span class="ow">=</span> <span class="n">run</span> <span class="mi">6868</span> <span class="n">app</span>
</pre></div>
<h2 id="language-extensions">Language extensions</h2>
<p>The first three lines are languages extensions,
Haskell behaves different for this module according to these.
data kinds Can be temporary deleted to see what happens:</p>
<div class="highlight"><pre><span></span>/home/jappie/projects/haskell/awesome-project-name/src/Lib.hs:14:16: error:
Illegal type: ‘<span class="s2">"users"</span>’ Perhaps you intended to use DataKinds
<span class="p">|</span>
<span class="m">14</span> <span class="p">|</span> <span class="nb">type</span> <span class="nv">UserAPI</span> <span class="o">=</span> <span class="s2">"users"</span> :> Get <span class="s1">'[JSON] [User]</span>
<span class="s1"> | ^^^^^^^</span>
<span class="s1">/home/jappie/projects/haskell/awesome-project-name/src/Lib.hs:14:31: error:</span>
<span class="s1"> Illegal type: ‘'</span><span class="o">[</span>JSON<span class="o">]</span>’ Perhaps you intended to use DataKinds
<span class="p">|</span>
<span class="m">14</span> <span class="p">|</span> <span class="nb">type</span> <span class="nv">UserAPI</span> <span class="o">=</span> <span class="s2">"users"</span> :> Get <span class="err">'</span><span class="o">[</span>JSON<span class="o">]</span> <span class="o">[</span>User<span class="o">]</span>
<span class="p">|</span> ^^^^^^^
</pre></div>
<p>Data kinds is needed to insert data into a type.
A string being data in this case, it is unclear what <code>'[JSON]</code> is,
probably also something data.
Temporary breaking a program to see what <span class="caps">GHC</span> will say is an
effective way of learning more about Haskell.</p>
<p>If <code>TypeOperators</code> is disabled, <span class="caps">GHC</span> says it doesn’t like <code>:></code> in the <code>UserAPI</code> line.
Apparently <code>:></code> is a type operator.
Apparently types can have operators.</p>
<p>If <code>DeriveGeneric</code> is disabled, <span class="caps">GHC</span> says it needs to derive
<a href="https://wiki.haskell.org/GHC.Generics">generic</a>
in the data definition of User. Generic is required for serialization
(in our case <span class="caps">JSON</span> conversion).</p>
<h2 id="modules">Modules</h2>
<div class="highlight"><pre><span></span><span class="kr">module</span> <span class="nn">Lib</span>
<span class="p">(</span> <span class="nf">webAppEntry</span>
<span class="p">)</span> <span class="kr">where</span>
<span class="kr">import</span> <span class="nn">Servant</span><span class="p">(</span><span class="n">serve</span><span class="p">,</span> <span class="kt">Proxy</span><span class="p">(</span><span class="o">..</span><span class="p">),</span> <span class="kt">Server</span><span class="p">,</span> <span class="kt">JSON</span><span class="p">,</span> <span class="kt">Get</span><span class="p">,</span> <span class="p">(</span><span class="kt">:></span><span class="p">))</span>
<span class="kr">import</span> <span class="nn">Data.Aeson</span><span class="p">(</span><span class="kt">ToJSON</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">GHC.Generics</span><span class="p">(</span><span class="kt">Generic</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Network.Wai</span><span class="p">(</span><span class="kt">Application</span><span class="p">)</span>
<span class="kr">import</span> <span class="nn">Network.Wai.Handler.Warp</span><span class="p">(</span><span class="n">run</span><span class="p">)</span>
</pre></div>
<p>Moving onward, there is the module definition that stack generated,
modules are just namespaces, or similar to python modules.
Nothing really special about those.
Then there are many imports which pull functions into the module namespace.</p>
<h2 id="type-level-rest-api">Type level <span class="caps">REST</span> <span class="caps">API</span></h2>
<div class="highlight"><pre><span></span><span class="kr">type</span> <span class="kt">UserAPI</span> <span class="ow">=</span> <span class="s">"users"</span> <span class="kt">:></span> <span class="kt">Get</span> <span class="kt">'[JSON]</span> <span class="p">[</span><span class="kt">User</span><span class="p">]</span>
</pre></div>
<p>This line defines the UserAPI type, which will serve as the <span class="caps">REST</span> endpoint.
The image at the beginning of the post was about this line.
Perhaps reading it as a sentence will give us some insight,
without worrying about how it fits together:
It’s a Get request, mounted below <code>/user</code>, returning something <span class="caps">JSON</span> and of
shape/type User.
Conveniently what a <code>User</code> is will be discussed in the next section.</p>
<h2 id="domain-data">Domain data</h2>
<div class="highlight"><pre><span></span><span class="kr">data</span> <span class="kt">User</span> <span class="ow">=</span> <span class="kt">User</span>
<span class="p">{</span> <span class="n">name</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">,</span> <span class="n">email</span> <span class="ow">::</span> <span class="kt">String</span>
<span class="p">}</span> <span class="kr">deriving</span> <span class="p">(</span><span class="kt">Eq</span><span class="p">,</span> <span class="kt">Show</span><span class="p">,</span> <span class="kt">Generic</span><span class="p">)</span>
<span class="kr">instance</span> <span class="kt">ToJSON</span> <span class="kt">User</span>
</pre></div>
<p>User is just a data structure consisting of two strings:
Email and name.
This declaration method is called <a href="http://learnyouahaskell.com/making-our-own-types-and-typeclasses#record-syntax">record syntax</a>.
This data structure derives
<a href="https://hackage.haskell.org/package/base-4.9.1.0/docs/Text-Show.html">Show</a>,
<a href="http://hackage.haskell.org/package/base-4.11.1.0/docs/Data-Eq.html">Eq</a>
and Generic.
Deriving means that <span class="caps">GHC</span> will generate function implementations for this
data structure. If one calls <code>show</code> on a User, it will know what to do
(show is toString in Haskell).
<code>instance ToJSON User</code> allows the User to be converted to <span class="caps">JSON</span>
(implementation is provided by generic).</p>
<h2 id="functions">Functions</h2>
<p>Done with data, time for code!</p>
<div class="highlight"><pre><span></span><span class="nf">users</span> <span class="ow">::</span> <span class="p">[</span><span class="kt">User</span><span class="p">]</span>
</pre></div>
<p>Specifies a function that will always return a list of Users.
There are no arguments to this function.
It can be assumed the list is always the same.
This is how immutable constants are specified.</p>
<div class="highlight"><pre><span></span><span class="nf">users</span> <span class="ow">=</span>
<span class="p">[</span> <span class="kt">User</span> <span class="s">"Isaac Newton"</span> <span class="s">"isaac@newton.co.uk"</span>
<span class="p">,</span> <span class="kt">User</span> <span class="s">"Albert Einstein"</span> <span class="s">"ae@mc2.org"</span>
<span class="p">]</span>
</pre></div>
<p>This is the implementation of the before defined function.
There are apparently two users in this list, one Isaac, and another Einstein.
Note that positional arguments are used to create the Users.</p>
<h2 id="servant-server">Servant server</h2>
<div class="highlight"><pre><span></span><span class="nf">server</span> <span class="ow">::</span> <span class="kt">Server</span> <span class="kt">UserAPI</span>
</pre></div>
<p><code>server :: Server UserAPI</code> says that there is something called a Server which
has a UserAPI.
A UserAPI is known, it is defined above.
A <a href="http://hackage.haskell.org/package/servant-server-0.14/docs/Servant-Server.html#t:Server"><code>Server</code></a>
is defined in servant.
The type signature is rather complicated:
<code>type Server api = ServerT api Handler</code>, looking at the definition of <code>ServerT</code>
introduces a lot of complexity: <code>type ServerT api (m :: * -> *) :: *</code>.</p>
<p>There are some clues that can be derived (such as that <code>m</code>),
but it’s not that important to make something work.
Therefore this guide ignores it.
Note that ignoring scary looking things is an important Haskell technique.
If one is interested, help can be found <a href="https://groups.google.com/forum/#!forum/haskell-servant">here</a>,
just in case ❤.</p>
<div class="highlight"><pre><span></span><span class="nf">server</span> <span class="ow">=</span> <span class="n">return</span> <span class="n">users</span>
</pre></div>
<p>The implementation is very simple however.
The reader should be cautious, to think that return is a keyword.
It’s a function.
What both return does is wrap a value into a container.
For example an element can be wrapped in a list:
<code>return 2 == [2]</code>.
That’s all one needs to know for now
(the interested reader may look at
<a href="https://wiki.haskell.org/Monad#Monad_class">monads</a>).</p>
<h2 id="proxy">Proxy</h2>
<div class="highlight"><pre><span></span><span class="nf">userAPI</span> <span class="ow">::</span> <span class="kt">Proxy</span> <span class="kt">UserAPI</span>
<span class="nf">userAPI</span> <span class="ow">=</span> <span class="kt">Proxy</span>
</pre></div>
<p>This is just some type <a href="http://hackage.haskell.org/package/base-4.11.1.0/docs/Data-Proxy.html">level magic</a>.
Library author needed type information for a function,
but they didn’t need a value.
Proxy does that.
It’s useful if you store data at type level,
for example with the datakinds language extension, which was seen earlier.</p>
<h2 id="application">Application</h2>
<div class="highlight"><pre><span></span><span class="nf">app</span> <span class="ow">::</span> <span class="kt">Application</span>
<span class="nf">app</span> <span class="ow">=</span> <span class="n">serve</span> <span class="n">userAPI</span> <span class="n">server</span>
</pre></div>
<p>This combines the proxy and server.
A serve function takes a Proxi <span class="caps">API</span>, Server <span class="caps">API</span> and returns an application.
If type Application is inspected one can appreciate what serve does for us better:</p>
<div class="highlight"><pre><span></span><span class="kr">type</span> <span class="kt">Application</span> <span class="ow">=</span> <span class="kt">Request</span> <span class="ow">-></span> <span class="p">(</span><span class="kt">Response</span> <span class="ow">-></span> <span class="kt">IO</span> <span class="kt">ResponseReceived</span><span class="p">)</span> <span class="ow">-></span> <span class="kt">IO</span> <span class="kt">ResponseReceived</span>
</pre></div>
<p>The arrows indicate function arguments.
An application receives a request, then a callback which expects a <code>Response</code>
to produce an <span class="caps">IO</span> action which gives the result <code>ResponseReceived</code>.
However to return this function must also return a type <code>ResponseReceived</code>
wrapped in <span class="caps">IO</span>.
It may be the case that the only way to obtain this response received is to call
that callback.
The freedom to do whatever one wants is meanwhile granted with the <span class="caps">IO</span> return type.
To compile that <code>ResponseReceived</code> has to be obtained however.</p>
<h2 id="running-it">Running it!</h2>
<div class="highlight"><pre><span></span><span class="nf">webAppEntry</span> <span class="ow">::</span> <span class="kt">IO</span> <span class="nb">()</span>
<span class="nf">webAppEntry</span> <span class="ow">=</span> <span class="n">run</span> <span class="mi">6868</span> <span class="n">app</span>
</pre></div>
<p>Our initial function!
Rather than saying hello world the app is ran on port 6868 (best port).
Now build and run it in one terminal, and in another curl it:</p>
<div class="highlight"><pre><span></span>stack build <span class="o">&&</span> stack <span class="nb">exec</span> awesome-project-name-exe <span class="p">&</span>
curl localhost:6868/users
> <span class="o">[{</span><span class="s2">"email"</span>:<span class="s2">"isaac@newton.co.uk"</span>,<span class="s2">"name"</span>:<span class="s2">"Isaac Newton"</span><span class="o">}</span>,<span class="o">{</span><span class="s2">"email"</span>:<span class="s2">"ae@mc2.org"</span>,<span class="s2">"name"</span>:<span class="s2">"Albert Einstein"</span><span class="o">}]</span>
</pre></div>
<p><img alt="Good job!" src="./images/2018/good-job.svg"></p>
<h1 id="in-conclusion">In conclusion</h1>
<p>A lot of concepts have been treated within this blog post while also moving
towards something productive.
The reader can now start a new project and add arbitrary dependencies.
He knows what language extensions are and how to see them in use.
Type level magic has been encountered, and wisely was ignored.
In future this post will build on top of this work
to extend the <span class="caps">API</span> and do something with something within the handlers.
However this post has grown to big already.</p>
<p>The complete code can be found <a href="https://github.com/jappeace/awesome-project-name/tree/simple-servent-setup">here</a>.</p>Elm on fire! Shaders in elm2018-06-23T12:00:00+02:002018-06-23T12:00:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2018-06-23:/elm-on-fire-shaders-in-elm.html<p><img alt="Elm on fire" src="/images/2018/elm-fire.svg"></p>
<p>Shaders have long been on the list of possible subject to study for Jappie.
The potential of both creating <a href="https://www.vertexshaderart.com/">beautiful art</a>
as well as doing parallel processing seem incredible valuable capabilities to have.
This post comments on the effort of porting a
<a href="https://github.com/ethanhjennings/webgl-fire-particles">JavaScript WebGL fire</a>
to an <a href="https://github.com/jappeace/elmgl-fire">elm implementation</a>.
Elm …</p><p><img alt="Elm on fire" src="/images/2018/elm-fire.svg"></p>
<p>Shaders have long been on the list of possible subject to study for Jappie.
The potential of both creating <a href="https://www.vertexshaderart.com/">beautiful art</a>
as well as doing parallel processing seem incredible valuable capabilities to have.
This post comments on the effort of porting a
<a href="https://github.com/ethanhjennings/webgl-fire-particles">JavaScript WebGL fire</a>
to an <a href="https://github.com/jappeace/elmgl-fire">elm implementation</a>.
Elm was chosen as target language because it is opinionated, easy and type safe.
In this post we explore how to get started in elm with shaders,
and then move on trying to port the fire project,
finally performance is increased as much as possible.</p>
<h1 id="in-the-beginning-there-was-nothing">In the beginning there was nothing.</h1>
<p>There are some
<a href="https://github.com/elm-community/webgl/tree/master/examples">example shader</a>
setups for elm.
The `crate’ was <a href="https://github.com/jappeace/elmgl-fire/commit/fb735158f328789a7c30ae4088b8cffcc4be1fd2">copied over</a>
resulting into having a fully 3d crate!
This is not exactly the output desired, a crate is not a fire (obviously),
but now there is a skeleton for the
<a href="https://guide.elm-lang.org/architecture/">elm architecture</a>
and some example shaders to play with.</p>
<p><img alt="A crate in gl" src="/images/2018/crate.jpg"></p>
<p>From here on there are two possible paths to continue,
one can try and completely understand what the shaders do and how they
work,
or one can just copy over the shader code from the
<a href="https://github.com/ethanhjennings/webgl-fire-particles">JavaScript</a> project and see if
we can make that work.
Although
<a href="https://github.com/jappeace/elmgl-fire/commit/96f3dd293ad72f8b199d7958500f0f14ea2ed013">initial work</a>
was started on the former approach,
the latter approach won out because the topic of ‘shader’ is just too large.
There is a lot of math involved.
Although this is an exercise of exploration and learning,
trying to understand it all is a massive scope creep.</p>
<h1 id="unbreak-rendering">Unbreak rendering</h1>
<p>After copying over the shader logic from the fire project,
everything broke.
This was not surprising as the crate project was 3d, whereas the fire project
was 2d.
Luckily Elm has strongly typed input for the shaders.
Therefore solving these mismatches was relatively easy.
We could just follow the compile errors.
After <a href="https://github.com/jappeace/elmgl-fire/commit/668f714294b4423ae51e8857bf7d9e8dafa4ba8c">that</a>,
the example program was essentially gutted, only the basic
architecture and <span class="caps">API</span> calls were left in tact.
Elm forces this architecture upon us, there is no choice in this.
The result of this effort is shown below.</p>
<p><img alt="Something in gl" src="/images/2018/gl-something.jpg"></p>
<p>It does not look like much of anything, however, this is counted as progress.
Not having a blank screen is good.
The next thing to do was fixing the colors.
This happened by porting the <a href="https://github.com/jappeace/elmgl-fire/commit/dbe4c308dcc24f0af8ea6b8f85991c1d83354002">hue code</a>,
there was no elm implementation for this particular kind of Hue representation.
Because a white background and the hue produces light blue, we added a
<a href="https://github.com/jappeace/elmgl-fire/commit/dbe4c308dcc24f0af8ea6b8f85991c1d83354002#diff-3e16369f543b857a1fea048cf77b7315R120">black background</a>
which mixes into an orange.
Now we had the right color, however transparency was also broken.
Transparency was quite interesting because my initial fix involved changing the
shader.
However the the <a href="https://github.com/jappeace/elmgl-fire/commit/dbe4c308dcc24f0af8ea6b8f85991c1d83354002#diff-3e16369f543b857a1fea048cf77b7315R136">right (<span class="caps">API</span>) option</a>
was eventually found
<a href="https://github.com/jappeace/elmgl-fire/commit/bc9f5d3eecbdc47c0ef0685a005c2af03e1ccd5c">that solved this issue</a>.
With all of this in place we get a single circle with the right color!</p>
<p><img alt="Single circle!" src="/images/2018/gl-reddot.jpg"></p>
<p>Baby steps. Graphics take time.</p>
<h1 id="random-spheres">Random spheres</h1>
<p>This is not impressive at all. However, in life one may find that arity changes
everything.
A single dot on it’s own is just a single dot, but if we randomly place it all
over the screen we get something nice to look at (live <a href="/raw-html/2018/random-spheres.html">here</a>):</p>
<video controls loop video controls autoplay>
<source src="/images/2018/spheres.webm" type="video/webm">
Your browser does not support the video tag.
</video>
<p>Aside from random creation this doesn’t bring us much closer to the goal of fire.
However some more work was done on it because Jappie thought it looked beautiful.
Performance was increased by converting a particle immediately into it’s
WebGL representation.</p>
<h1 id="movement">Movement</h1>
<p>To do movement we dropped some changes from the random sphere case.
The idea of not doing an update loop at all was temporary put aside,
because using an update loop is closer to the JavaScript original.
Keeping it would make porting easier.</p>
<p>Doing movement is simply adding velocity times time to position every frame.
That’s it. The simplex noise part of the JavaScript code was also implemented
for variation in movement.</p>
<p>It turns out however that the result is somewhat unimpressive.
Yes it looks like fire, but after about 20 seconds the memory is full,
garbage collection kicks in and the program grinds to an halt.
Here is an example (live <a href="/raw-html/2018/slow-fire.html">here</a>,
may grind your computer to a halt):</p>
<video controls loop video controls autoplay>
<source src="/images/2018/slow-fire.webm" type="video/webm">
Your browser does not support the video tag.
</video>
<h2 id="speed">Speed</h2>
<p>The problem is that aside from creating particles and sending them to a <span class="caps">GPU</span>,
All existing particles must every cycle be updated with the new location.
We may observer however that the path of the particles after creation is
entirely deterministic.
Why don’t we let the shaders do this?
The idea being that we create particles with an initial position, timestamp and
velocity.
Then let the shaders calculate the position for whatever the current timestamp is.</p>
<p>When trying to implement this, it was found out that the elm gl <span class="caps">API</span> was used
in a inefficient way.
The realization came that using an entity per quad doesn’t allow us to share
the uniform across all quads.
These entities are analogues to WebGL ‘programs’</p>
<p>The architecture was redesigned to take into account multiple particles per elm
entity.
Rather than tracking lists of entities,
lists of tuples of vertices are now being tracked:
<code>List (Vertex, Vertex, Vertex)</code>.
It would’ve been preferable to use <code>Mesh Vertex</code> as type, but this type does not
support appending in the elm shader <span class="caps">API</span>.</p>
<p>This approach seemed to work much better, in fact this is probably how one
should use this <span class="caps">API</span>.
It was possible to render 500 particles now and the computer didn’t lock up
(at all):</p>
<p>live <a href="/raw-html/2018/fast-fire.html">here</a>
<video controls loop video controls autoplay>
<source src="/images/2018/fast-fire.webm" type="video/webm">
Your browser does not support the video tag.
</video></p>
<p>It’s still not very good, as the original JavaScript implementation
was able to do up to 3000 particles per seconds quite comfortably
(with a much better frame rate)…
There is not a lot of things done in the this implementation on the <span class="caps">CPU</span> side,
and still the <span class="caps">CPU</span> intense JavaScript implementation is faster.
Perhaps this is just a limit of using elm.</p>
<h2 id="more-speed">More speed?</h2>
<p>After thinking about the problem for some time another idea came to mind.
To increase speed, the amount of information send to the <span class="caps">GL</span> pipeline can be
reduced.
Every frame sends this Mesh collection to the <span class="caps">GPU</span> trough a buffer,
if this buffer can be decreased in size, speed would increase.
It would also lighten the load on garbage collection, as less objects need to
be created.
The suspicion is that elm is slow just because of garbage collection.
We can do this rather trivially by representing each particle as a single vertex,
with a position and size.
Then we just use a shader to reconstruct the vertices into quads (squares).
The vertex shader would move the vertex first,
then another (unkown) shader would do reconstruction,
then the fragment shader would do drawing.
Easy as π.</p>
<p>Stack overflow <a href="https://stackoverflow.com/questions/5821152/opengl-add-vertices-with-vertex-shader">suggests</a>
that we can use a geometry shader for this.
Unfortunately the elm <span class="caps">GL</span> <span class="caps">API</span> doesn’t support this,
it only has a slot for vertex, and fragment shader in the <a href="http://package.elm-lang.org/packages/elm-community/webgl/2.0.5/WebGL#entity">entity function</a>.
Jappie briefly got excited about adding this shader type to the elm <span class="caps">API</span>,
however he discovered that WebGL doesn’t support this type of <a href="https://stackoverflow.com/questions/8641119/webgl-geometry-shader-equivalent">shader at all</a>.
From this point it’s unclear how to increase speed.
Changing WebGL itself is borderline impossible (it would take at least years).</p>
<h1 id="in-conclusion">In conclusion</h1>
<p>Upon discovery of the WebGL <span class="caps">API</span> for elm Jappie was quite excited about using that.
However after using it, and finding the rather large performance difference
the excitement has been tempered.
Still a lot was learned from doing this project,
elm is a good entry point for graphics development,
type-safety makes the complexity quite manageable.
In fact the idea for using geometry shaders would not had been realized at all
in a faster language.</p>
<p>However in future a faster language will be used.
Not being able to get everything from a machine is quite frustrating.
This therefore will exclude any use of WebGL.
A major drive for using WebGL in the first place was to share the result online,
however this reasoning is quite flawed in that a video of the result can just
be made.
After all we don’t even include the live results here because of concern that it
will freeze the readers’ computer.</p>
<p>To all who are interested in graphics elm is recommended to start with.
Especially if they are already familiar with elm or the react/redux architecture.
Type safety on shader level is really nice, especially when you do things wrong
structurally and the compiler exactly tells you where you need to repair things
(as happened during this project with the entities).
The price one pays is execution speed in return for development speed.</p>Fun with stack: Haskell dependency management2018-05-13T16:00:00+02:002018-05-13T16:00:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2018-05-13:/fun-with-stack-haskell-dependency-management.html<p><img alt="Haskel stack logo" src="/images/2018/haskell-stack-logo.svg"></p>
<p>Working at Daisee, Jappie uses a lot of Haskell programming.
Although Haskell is obviously as amazing as the <a href="https://www.reddit.com/r/haskell/comments/zpff3/larry_wall_you_should_probably_know_about_it/">stereotype asserts</a>,
the tooling can be a bit challenging.
In this blogpost we explore these challenges.</p>
<p>One’s understanding start with the fact that there is not one unified Haskell
package manager …</p><p><img alt="Haskel stack logo" src="/images/2018/haskell-stack-logo.svg"></p>
<p>Working at Daisee, Jappie uses a lot of Haskell programming.
Although Haskell is obviously as amazing as the <a href="https://www.reddit.com/r/haskell/comments/zpff3/larry_wall_you_should_probably_know_about_it/">stereotype asserts</a>,
the tooling can be a bit challenging.
In this blogpost we explore these challenges.</p>
<p>One’s understanding start with the fact that there is not one unified Haskell
package manager.
Well there is, but only experts use <a href="https://www.haskell.org/cabal/">cabal</a>
directly.
The issue is that often dependencies don’t quite match from the
<a href="https://hackage.haskell.org/">Hackage</a> repository,
which cabal uses as repository.
Loose version bounds within cabal configurations,
cause one developer to develop against version <code>1.1</code> and another against
version <code>1.2</code>.</p>
<p>This is were <a href="https://www.stackage.org/">stack</a> stepped in.
Stack guarantees that at some point in time all the provided packages in their
repository will build together.
They do this trough package curation and providing reproducible builds at least
for <a href="https://github.com/commercialhaskell/stack/blob/master/doc/GUIDE.md#what-makes-stack-special">Haskell packages</a>.
This is hosted on their alternative to
<a href="https://hackage.haskell.org/">Hackage</a>, called
<a href="https://www.stackage.org/">Stackage</a>.</p>
<p>Because of curation and being more restrictive, the stackage repository doesn’t
host as much software as hackage does.
In certain cases we may still need to configure stack in such a way that
it pulls in a package or two from hackage.</p>
<h1 id="system-dependencies">System dependencies</h1>
<p>Haskell is not an operating system.
There is no Haskell Linux kernel or Haskell std lib.
Therefore many Haskell libraries are dependent upon C code libraries.
Stack does not include interaction with these system libraries as part of it’s
reproducible build contract.</p>
<p>The user experiences this once stack starts requesting ‘devel’ packages,
for example:
Stack will request ‘ssl’ headers, and recommends the user to install a devel
system package from his distribution package manager.
It is up to the user to find the name of this system package for his
respective distribution,
in case of fedora it’s: “openssl-devel”, and in case of Ubuntu it’s “libssl-dev”.
But gentoo wouldn’t even have devel packages, all packages in case of gentoo are
devel packages.</p>
<h2 id="enter-nix">Enter nix</h2>
<p>Nix is also a package manager like stack and cabal are,
however, it’s not focused on any specific language.
Nix is about reproducible builds, eg guaranteeing some build on a developer
machine is exactly the same as on the user machine.</p>
<p>A developer may for example have had the ssl library installed globally,
but forgot to mark it as a dependency in his software package.
A build tool would happily build that on his machine,
but now when a user downloads this code and tries to build the package it will
fail.
Nix <a href="https://nixos.org/~eelco/pubs/phd-thesis.pdf">solves this</a>.</p>
<p>In principle nix is completely independent from the Haskell ecosystem.
Build around this package manager is even an entire <a href="https://nixos.org/">distribution</a>.
But because it provides system dependencies <em>and</em> haskell dependency management,
and does so in a reproducible manner at system level,
it has become very popular among the correctness loving Haskell programers.</p>
<p>However it may be pointed out that although what nix does is impressive compared
to stack it’s harder to use.
The langauge nix langauge is peculiar, for example it has both a <code>;</code>,
and being reduced to a single expression.
It has no type system but does full correctness checking before exucting.
But it has a massive repository of <a href="https://github.com/NixOS/nixpkgs">examples</a>.</p>
<h2 id="stack-and-nix">Stack and nix</h2>
<p>Stack provides the ability to specify system dependencies trough nix.
Which is very useful as you now can let any developer regardless of Linux
distribution, get on board with your stack project,
without having to find out which specific system packages need to be installed.
Is this combination the solution? Easy to use and being fully reproducible?
Alas no this setup also has issues.</p>
<p>It has an opt in for the isolated builds nix provides,
so stack would first look if a package is specified in the nix configuration,
if not it would fall back on system libraries.
This is rather strange, because you enable the plugin for reproducible builds,
but the default behavior is to look for system libraries
(eg be not reproducible).
There is an opt out for this behavior (<code>pure:true</code>), but it seems more sensible
to make this an opt in.</p>
<p>The reason why this behavior is undesired is because if one programmer starts
using <code>http-client-openssl</code>, and happens to have <code>openssl-devel</code> installed on
his system, then his colleague try to build the project, his build will fail.
If <code>pure:true</code> was enabled the first developer would’ve noticed the issue
and added it into the nix configuration.</p>
<p>However the way stack tells the user about the missing dependency
is also peculiar.
In case of the pure configuration, if stack can’t find the library,
it will ask the user to install the dependency on his system.
Which is of course pointless because one needs to add it to the nix configuration
<img alt="Stack error message" src="/images/2018/stack-error.jpg">
Although the error message was intended to be helpful,
in this particular case it ends up sending the user in the wrong direction.</p>
<h1 id="git-and-stack">Git and stack</h1>
<p>Stack’s power lies in that it makes it easy to do the common things.
For example creating a library for github is trivial.
Stack helps by allowing users to specify a ‘github’ field in the <code>package.yaml</code>
file,
which will generate an url to both the source and the issue tracker on github.</p>
<div class="highlight"><pre><span></span><span class="l l-Scalar l-Scalar-Plain">name</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">voicebase</span>
<span class="l l-Scalar l-Scalar-Plain">version</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">0.1.0.0</span>
<span class="l l-Scalar l-Scalar-Plain">license</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">BSD3</span>
<span class="l l-Scalar l-Scalar-Plain">author</span><span class="p p-Indicator">:</span> <span class="s">"Jappie</span><span class="nv"> </span><span class="s">Klooster"</span>
<span class="l l-Scalar l-Scalar-Plain">maintainer</span><span class="p p-Indicator">:</span> <span class="s">"jappie.klooster@daisee.com"</span>
<span class="l l-Scalar l-Scalar-Plain">copyright</span><span class="p p-Indicator">:</span> <span class="s">"Daisee"</span>
<span class="l l-Scalar l-Scalar-Plain">github</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">blah/blah</span>
</pre></div>
<p>Now although github is extremely popular in opensource it’s of course not the
only choice,
One may put it on a different host, such as
<a href="https://bitbucket.org/daisee/voicebase/src/master/">bitbucket</a>,
or host their own source code with <a href="https://gitlab.com">gitlab</a>.</p>
<p>There is no doubt that stack supports specifying an alternative,
however it is unclear how to replace the github field properly with alternatives.
The assumption was that the <code>gihtub</code> field was just a shorthand to fill in some
other fields.
It is hard to find what it does because it’s an odly specific question,
but it <a href="https://stackoverflow.com/questions/40332040/what-goes-in-a-stack-package-yaml-file?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa">turns out</a>
that <a href="https://github.com/sol/hpack">hpack</a>
is used to parse the <code>package.yaml</code> file.
Perhaps the <code>git</code> field is used to replace <code>github</code>?
It is unclear.
It’s also ironic in that we can immediatly observe the difference strong
types make in documentation ability,
this yaml file is not strongly typed.</p>
<h2 id="commit-dependencies">Commit dependencies</h2>
<p>In certain situations one may need to change a library.
The typical way of doing this is to make a github fork, and then specify in the
dependency manager, that a specific (git) url needs to be used on a specific
commit hash.
For example: If one tries to get the <code>56e833</code> commit from the
<a href="https://bitbucket.org/daisee/voicebase/src/master/">voicebase library</a>,
you should record the hash and url somewhere.</p>
<p>If we google for the query <code>stack.yaml git dependency</code>,
you just need to know that you configure this in <code>stack.yaml</code>, which is completely unrelated to <code>package.yaml</code>
google points you to
an outdated doc page which fails to work.
A helping <a href="https://stackoverflow.com/questions/43789271/stack-yaml-not-pulling-in-dependency-from-github?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa">stackoverflow</a>
gives the reason why it doesn’t work, the syntax recently has changed and
Google hasn’t updated yet.
One can hardly blame stacks maintainers for this of course.
But it still impacts a users’ experience.</p>
<p><img alt="Outdated docs" src="/images/2018/stack-old-docs.jpg"></p>
<h1 id="conclusion">Conclusion</h1>
<p>Dependency management in Haskell is complicated.
Even if one is able to become productive in the langauge,
any of the problems described here could still make it difficult enough for them
to give up on the system they want to build.
Learning about functors and applicative is fun,
learning what specific syntax to use to make stack pull in a git repository is not.</p>Reddit poster plugin for pelican2018-05-07T09:00:00+02:002018-05-07T09:00:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2018-05-07:/reddit-poster-plugin-for-pelican.html<p>Yesterday the <a href="https://github.com/getpelican/pelican-plugins/pull/1026">Reddit Poster</a>
plugin for <a href="http://docs.getpelican.com/en/stable/">pelican</a> was finished.
This is an initial step towards providing Reddit integration with pelican.</p>
<p>What this plugin does is look a predefined list of subreddits names in <a href="https://raw.githubusercontent.com/jappeace/jappeaceApplication/master/content/reddit-poster.md">an article</a>,
then it posts the article to all those subreddits.
Aside from the subreddits in the …</p><p>Yesterday the <a href="https://github.com/getpelican/pelican-plugins/pull/1026">Reddit Poster</a>
plugin for <a href="http://docs.getpelican.com/en/stable/">pelican</a> was finished.
This is an initial step towards providing Reddit integration with pelican.</p>
<p>What this plugin does is look a predefined list of subreddits names in <a href="https://raw.githubusercontent.com/jappeace/jappeaceApplication/master/content/reddit-poster.md">an article</a>,
then it posts the article to all those subreddits.
Aside from the subreddits in the list, there is also a ‘collection’ subreddit
where all articles posted too.</p>
<p>Another goal in the future may be the ability to fetch comments from reddit
at compile time and display them below posts.
However the decision was reached to reduce the scope of this project to just
posting first, as there was not enough time available to make fetching work.
It is hoped however that many people now will start reacting to these posts
so that it may be useful to implement this ability too.
That will however be done in a separate plugin.</p>
<p>This project has been lingering in Jappie’s <span class="caps">TODO</span> list for years now.
It is a great relief to finally make some progress in it.
To advertise this website now the only thing that needs to be done is
specifying the list of subs in the relevant article.</p>Starting at Daisee2018-04-11T20:30:00+02:002018-04-11T20:30:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2018-04-11:/starting-at-daisee.html<p>Recently quite a few changes have happened within Jappies’ life.
First of all, he is no longer at <a href="https://www.openlearning.com/">OpenLearning</a>.
This relationship ended at the end of February.
March was used for job searching.
To help with this a proper <a href="https://jappieklooster.nl/pages/portfolio.html">portfolio</a>
was constructed.
Last week there were three interviews with Daisee …</p><p>Recently quite a few changes have happened within Jappies’ life.
First of all, he is no longer at <a href="https://www.openlearning.com/">OpenLearning</a>.
This relationship ended at the end of February.
March was used for job searching.
To help with this a proper <a href="https://jappieklooster.nl/pages/portfolio.html">portfolio</a>
was constructed.
Last week there were three interviews with Daisee,
and Saturday Jappie accepted the offer.</p>
<h1 id="bio">Bio</h1>
<p>For this new job Jappie was asked to write a short bio,
which he thought would be a nice addition to this blog post as it explains his
current situation quite well:</p>
<blockquote>
<p>Jappie joins <a href="https://daisee.com">Daisee</a> as Software Engineer,
he is quite excited about working with Haskell and practical <span class="caps">AI</span>.</p>
<p>In his previous role Jappie was a Full stack developer at OpenLearning.
Building various systems,
from spam detection to remote code execution within containers.
Before that he just graduated from his MSc in <span class="caps">AI</span> from Utrecht University,
he chose to travel rather than taking up an offer for a PhD position <br>
because he didn’t felt much for a career in academia.
The decision for doing a masters in <span class="caps">AI</span> was made during his time in China
where he did an internship for his bachelor degree in Software Engineering.</p>
<p>Arriving here on a `working holiday’, Jappie likes it here and hopes he can
stay longer.
The weather is better, the meetups are interesting,
and it’s quite satisfying to build up a life starting from almost nothing.
Jappie enjoys building things,
from programming project to virtual kingdoms in computer games.</p>
</blockquote>
<h1 id="expectations">Expectations</h1>
<p>Daisee and Jappie found each other at the functional programming meetup.
They announced it as doing work with <span class="caps">AI</span>,
and Jappie happens to be interested in both functional programming as well as <span class="caps">AI</span>.</p>
<p>During the interview it was explained that initial tasks however were of a more
practical nature.
Such as devops and implementing a better <span class="caps">UI</span>.
so there isn’t much <span class="caps">AI</span> work to begin with.
However Jappie expects that after a while there would be more tasks in either
of that.</p>
<h1 id="first-day">First day</h1>
<p>After finishing his first day Jappie looks back with satisfaction.
The <span class="caps">CTO</span> takes a much more active role than at his previous company,
which is pleasant because now it feels like there is direction.
His colleagues seem nice too, all day people were coming over to introduce
themselves to Jappie.
His desk neighbor is also nice and very knowledge-able.
Apparently a hardened Haskell veteran and a fellow Gentoo user/victim (rip Gentoo).</p>
<p>Then there was the coffee machine.
At his previous job the <span class="caps">CEO</span> didn’t like coffee, so didn’t invest in it.
In the new job however there was a full blown espresso machine.
This machine is both a blessing and a curse,
it’s a blessing in that now one can drink all he desires,
but a curse in that addiction will follow.
Not to speak about the impact it has upon ones’ sleep cycle.</p>
<p>Thinking about the introductions, some of his colleagues had very impressive
track records indeed.
For example, one of the founders apparently led a merger between two big
companies.
At the time it Jappie didn’t quite realize how difficult doing such a thing is.
Jappie thinks it’s a good thing he’s surrounded by veterans.
He can learn a lot from these people, just by being around them.</p>Flask, docker and the backend2018-03-11T00:00:00+01:002018-03-11T00:00:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2018-03-11:/flask-docker-and-the-backend.html<p>Jappie Klooster is working with friends on a react native app.
It was attempted to do this completely without server-side with help of Firebase.
This post describes the thought process behind not using Firebase for
<em>everything</em>, and setting up a custom backend instead.</p>
<p>The first major issue, for this use …</p><p>Jappie Klooster is working with friends on a react native app.
It was attempted to do this completely without server-side with help of Firebase.
This post describes the thought process behind not using Firebase for
<em>everything</em>, and setting up a custom backend instead.</p>
<p>The first major issue, for this use case specifically, is the type of database
provided within Firebase.
It’s a NoSQL database that promises to sync between all devices.
They probably mean syncing lazily, it’s implied in the docs,
but there is no definitive statement around that.
NoSQL is the idea that <a href="http://www.monitis.com/blog/cc-in-review-the-key-differences-between-sql-and-nosql-dbs/">data doesn’t need relationships</a>.
For the core business logic of our app, we need relationships between data,
and we need a lot of them.
Technically one can represent these within NoSQL, but now you need to solve
problems which most <span class="caps">SQL</span> databases have already done for you.</p>
<p>Another <em>big</em> disadvantage of Firebase is that it’s proprietary.
Once a large chunk of the business logic, and data resides inside the Firebase
ecosystem, it’s hard to get out.
This means it’s relatively easy for the owner of the proprietary system, Google,
to crank up the price.
Neither Jappie nor his collaborators wanted to </p>
<h1 id="language-choice">Language choice</h1>
<p>The primary question imposed for deciding which language to use was
“How fast do you want it?”.
<em>Fast</em>.
Therefore a host of options was eliminated, such as Haskell and Rust.
In fact because we wanted a pan-team familiar choice,
only two real contenders were left. NodeJS and Python.
The choice of Python was made because it is a much more simple
language than JavaScript.
Another advantage of choosing either for these languages would be that our
editors are already setup to handle them.</p>
<h1 id="docker">Docker</h1>
<p>Many members of the team stressed the importance of containerization.
Jappie, being the only one that had worked intimately with this technology
was quite happy with this development.
What was rather surprising was the suggestion that the team ought to run
these containers by hand on rented <span class="caps">VPS</span>’es.
Is docker not supposed to make orchestration more easy?
Do there not exist many tools that build on top of docker to provide variances
in orchestration? Such as docker-compose, docker swarm, appengine and Kubernetes?
All these questions entered into Jappie’s mind,
but he felt arguing in this case would be difficult wihtout an example.
Once the team would start working with the containers they would see for
themselves how easy orchestration is.
The primary building block was in place anyway (docker), moving to another
orchestration methodology will be easy.</p>
<p>It was here that Jappie decided to pick up this task.
He knew about docker from previous experience already, and his other major task
had just finished.
He started with a little hello world in python served through <a href="http://flask.pocoo.org/">flask</a>.
The he put this in a docker container by writing a <code>Dockerfile</code>, and finally
he wrote a little docker-compose script to attach an idle Postgres container
which would be need later.
All this was done in the space of about an hour or two.
Setting up docker isn’t difficult.
Then it was time for the next step, connecting flask to Postgres.</p>
<h1 id="flask">Flask</h1>
<p>Flask is a micro framework.
It gives some Rest support, and that’s about it.
They sometimes provide some glue scripts that connect some projects to other
projects, but there is no main monolithic architecture behind it.
In stark contrast to <a href="https://www.djangoproject.com/">Django</a>,
which has already done most things for you.</p>
<p>This micro framework setup, gives the project a big advantage in that they can
have independent versions, and gives the capability to swap out some projects
for others quite easily.
The big disadvantage, as Jappie encountered, is that its fairly challenging to get
into.
One may even say it’s hard to get started.
The reason for this is that gluing the <span class="caps">ORM</span> to the framework has to be done by
hand.
This is of course because people may want to choose a NoSQL storage mechanism
rather than an <span class="caps">SQL</span>.
Jappie wanted to have an <span class="caps">ORM</span> to handle migrations and as a method of
describing the model in.</p>
<p>The thing that ended up working for Jappie was this
<a href="https://github.com/davidism/basic_flask">bootstrapping script</a> which was about
5 years old and of course outdated.
But Jappie managed to get <a href="https://github.com/jappeace/basic_flask">it running</a>,
by googling errors, and stripping features.
This should make it easier for future projects to start with flask in a
docker environment.
A pull request was made to merge it back into the original project.</p>
<h1 id="deploying">Deploying</h1>
<p>Once the docker-compose setup was completed considerations went out how to
deploy this live.
Being able to setup a live instance early has the advantage that the team can
test the entire (rather complex) infrastructure.
AppEngine was chosen as method of deployment as it could save in costs.
AppEngine essentially allows client code to be shared
across Kubernetes clusters (one assumes),
making it cheaper for small apps to use as the
overhead of running Kubernetes itself does not need to be taken in account.</p>
<p>Since Jappie had never worked with AppEngine before he chose to do the
<a href="https://cloud.google.com/python/getting-started/hello-world">hello world</a>
tutorial first.
Note that he had to choose the flexible environment as the standard one only
provides python 2.7, which at the time of writing has a bit over
<a href="https://pythonclock.org/">two years</a> left until deprecated.
Using python 2.7 is therefore a really bad choice.
Even though google <a href="https://cloud.google.com/appengine/docs/the-appengine-environments">says</a>
python 2.7 is the standard,
Jappie urges his peers to take this claim with suspicion.
Which the <a href="https://wiki.python.org/moin/Python2orPython3">python wiki</a> backs up:</p>
<blockquote>
<p>Python 2.x is legacy, Python 3.x is the present and future of the language</p>
</blockquote>
<p>After the hello world was successfully put online
(in python 3 with the flexible environment),
it was time to try setting up the entire system.
For this a managed Postgres needed to be setup on Google Cloud and connected to AppEngine.</p>
<p>Google however seems to favor more popular technologies,
and the primary docs assume you want to use <a href="https://cloud.google.com/python/getting-started/using-cloud-sql">MySQL</a>.
Postgres was the preferred choice for the team however.
It is more
<a href="https://www.quora.com/What-are-pros-and-cons-of-PostgreSQL-and-MySQL-With-respect-to-reliability-speed-scalability-and-features">standard compliant and has better data types</a>.
An Uber post however claims there are some interesting advantages
to <a href="https://eng.uber.com/mysql-migration/">MySQL</a> if you manage the architecture
yourself.
This is done by google however in this use case,
besides because an <span class="caps">ORM</span> is used, switching database technology is relatively easy.</p>
<p>The search engine doesn’t show the docs for Postgres with flask in AppEngine.
These docs were found in the <a href="https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/flexible">example project</a>.
The source file had a link to
<a href="https://cloud.google.com/appengine/docs/flexible/python/using-cloud-sql-postgres">using postgres with AppEngine</a>.
To test if the Postgres instance was running
this <a href="https://cloud.google.com/sql/docs/postgres/connect-admin-proxy">guide</a>
was used.
Jappie was uncertain why a cloud proxy was recommended this much,
in his memory he hadn’t set this up last time with a Django project.
However with that in place one can create a
<a href="https://medium.com/@mohammedhammoud/postgresql-create-user-create-database-grant-privileges-access-aabb2507c0aa">specialized user</a>
for manipulating the database (rather than the root Postgres account).</p>
<p>Going trough those tutorials the useful commands were collected into a
makefile. With this in place he had no longer need to refer to them
and could just <code>cat</code> the makefile to see how to deploy for example:</p>
<div class="highlight"><pre><span></span><span class="nf">deploy</span><span class="o">:</span>
<span class="nv">SQLALCHEMY_DATABASE_URI</span><span class="o">=</span>postgresql+psycopg2://USER:PASWORD@/DB?host<span class="o">=</span>localhost python ./manage.py db upgrade <span class="o">||</span> <span class="o">(</span><span class="nb">echo</span> <span class="s2">"make sure the proxy is running"</span> <span class="o">&&</span> <span class="nb">false</span><span class="o">)</span>
gcloud app deploy
<span class="nf">describe</span><span class="o">:</span>
gcloud sql instances describe DB_NAME
<span class="nf">proxy</span><span class="o">:</span>
cloud_sql_proxy -instances<span class="o">=</span><span class="nv">INSTANCE_PATH</span><span class="o">=</span>tcp:5432
<span class="nf">connect</span><span class="o">:</span>
<span class="nb">echo</span> <span class="s2">"warning, requires you to setup the proxy in another terminal"</span>
<span class="c"> # psql "host=127.0.0.1 sslmode=disable user=postgres"</span>
psql <span class="s2">"host=127.0.0.1 sslmode=disable user=USER dbname=DB password=PASSWORD"</span>
</pre></div>
<p>Now any of the team could deploy with <code>make deploy</code>. Note that the credentials
where retracted out of security reasons.</p>
<h1 id="conclusion">Conclusion</h1>
<p>There is a lot involved to setting up a backend.
It took about two days going from knowing almost nothing about flask and
AppEngine, to having a hello world.
Not even taking into consideration all the previous knowledge Jappie already
had acquired with docker, Postgres and python itself.
Note that at this point we haven’t even made anything of interest.
This post was about setting up infrastructure…
It is to hope that this story may help some other hackers,
so that they can save a few days.</p>New new website style changes2018-03-08T00:00:00+01:002018-03-08T00:00:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2018-03-08:/new-new-website-style-changes.html<p>Due to circumstances in his life, Jappie Klooster decided it’s
time for some more website changes!
These are all intended to simply reduce the amount of time it costs maintain
this website.
This should allow Jappie Klooster to update it more regularly as the friction
for updates is decreased …</p><p>Due to circumstances in his life, Jappie Klooster decided it’s
time for some more website changes!
These are all intended to simply reduce the amount of time it costs maintain
this website.
This should allow Jappie Klooster to update it more regularly as the friction
for updates is decreased.</p>
<h1 id="brutalist-style">Brutalist style</h1>
<p>Jappie Klooster became unhappy with the style, although it entertains him endlessly,
it may give the wrong idea to a third party.
To untrained eyes it may look like he attempted to make it look good
(which wasn’t the intention at all).
So rather than trying to make it beautiful, Jappie decided to go in full brutalism mode.
In essence all style except the most practical ones will be stripped.
For reference:</p>
<h3 id="old">old</h3>
<p><img alt="old" src="/images/2018/old-theme-reference.jpg">
As the reader see the old style was rather dark themed.
It used the linux kernel compilation program as a reference, as discussed
<a href="https://jappieklooster.nl/website-launch.html">in another post</a>.</p>
<h3 id="new">new</h3>
<p><img alt="new" src="/images/2018/new-theme-reference.jpg">
In the new style Jappie just started deleting html and <span class="caps">CSS</span> untill he came
accross this, it’s not as brutalist as initially promised,
but looks a lot more attractive and was a lot more simple to accomplish.</p>
<p>Jappie intentionally put the image of the new style in here, even though it’s the
current style. It is assumed that this style is not permanent either.</p>
<h1 id="getting-rid-of-org-mode">Getting rid of org mode</h1>
<p>The believe is still held that org-mode is a superior standard compared to markdown.
It can do so many things, such as inline code execution and pipe the result to
the document itself, as having full reference support.
These are just some of the many great features of org-mode.
However org-mode has issues.
The fact that it’s entirely confined to Emacs for <span class="caps">HTML</span> rendering is a massive problem.
Emacs needs to be booted for each article just to do the <span class="caps">HTML</span> rendering.
There is no easy way Jappie Klooster knows of to keep this process active as a daemon somehow.
Technically Emacs can do this, but it is meant to do communication with <span class="caps">XORG</span>,
not to act as a script execution platform.</p>
<p>In the future Jappie Klooster may do investigation to make this work fast,
however for now the easiest way of increasing building times for this website is to just
use the markdown export feature of org-mode.
The reason he chose org-mode in the first place as a viable alternative is that it
doesn’t try to lock you into it.
He now executes this option, because there is simply no time to investigate the
preferred route.</p>Plan prediction2017-12-21T12:00:00+01:002017-12-21T12:00:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2017-12-21:/plan-prediction.html<p>For the raster project a main selling point will be the automatic prediction
of future scheduling.
There are two major schools of thoughts to go with that I know of.
Constraint satisfaction solving and data driven approaches (use statistics).</p>
<p>First of all the most straight forward approach is to use …</p><p>For the raster project a main selling point will be the automatic prediction
of future scheduling.
There are two major schools of thoughts to go with that I know of.
Constraint satisfaction solving and data driven approaches (use statistics).</p>
<p>First of all the most straight forward approach is to use something like
<a href="https://www.OptaPlanner.org/">OptaPlanner</a>.
In this approach constraints are laid upon the problem, such as 2 cooks
need to at least work on Saturday, but preferably 3.
This single constraint has two different part, the hard minimum, and the ‘soft’
preference.
With these kind of rules in place, the OptaPlanner can use heuristics to figure
out who to plan when.
You could add as many constraints as you want, such as have less preference for
when employees asked free or let students not work during the day.</p>
<p>This is a very precise approach and almost surely will work,
as other systems have <a href="https://www.youtube.com/watch?v=sOWC4qrXxFk&index=5&list=PLJY69IMbAdq0uKPnjtWXZ2x7KE1eWg3ns">demostrated</a> <a href="https://github.com/kiegroup/optashift-employee-rostering">already</a>.
However the issue with this approach is the actual specification of rules.
This is quite difficult to do, so for a fact I can’t push this onto the company
owners.
What I could do is create some kind off helping <span class="caps">GUI</span> around it.
Or simply use generic predefined rules and if owners want something more
sophisticated they should contact me.
On top of that the OptaPlanner system I know of is written in Java,
which is kind off problematic since I now need to develop some
micro-service architecture around it.</p>
<p>Here is started considering the second approach.
The idea I had is that I use the already entered data by the owner to create a
statistical model of how he likes to make his plannings.
Then we use this model to make future predictions.
This requires zero configuration by the owner,
and also doesn’t have the language interaction problem as it’s just some calculations.
What I imagine is just adding a predict button to the current roster screen
that uses the statistical model to magically create a roster.</p>
<p>Now the method I thought of is simple, it’s an adaptation of <a href="https://en.wikipedia.org/wiki/Q-learning">q-learning</a>,
which uses a lookup table for getting action probabilities, if successful
probability is increased, otherwise decreased.
I’m doing something more simple, here is the gist of it:
We count per day how many job types are planned in on average over <em>n</em> weeks.
Then we plan in that amount of job types, for remaining fractions we flip a
coin.
If someone has asked free we remove him from the potential pool of workers.
If there are not enough workers for a job type we just throw a warning for that
date and let the owner figure it out.</p>
<p>The beauty of this technique is that it is simple,
but it takes a lot of work out of planning in people.
If this is well received there is tons of room for extension.
For example naive Bayesian networks can be used to take into account
‘experience’.
If we add a reservation datasource somehow prediction could become even more
accurate by using it as another influencing factor.
Whilst the basic interface remains the same, press the ‘predict’ button and
magically you get a roster.</p>
<p>In conclusion I will attempt a statistical approach rather than OptaPlanner,
because it’s more simple.
In the future I can still choose to experiment with OptaPlanner.
It would be much more interesting to use that, once I know how to register
demand accurately,
for example the amount of reservations for an evening would be a good indicator
for restaurants.</p>Jakartian Christmas2017-12-19T21:00:00+01:002017-12-19T21:00:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2017-12-19:/jakartian-christmas.html<p>Unfortunately Jesiska couldn’t make it to Australia in time for Christmas.
Therefore I decided basically last week to go to Jakarta instead!
I personally don’t care much about Christmas, but I thought the office would
be closed during this time.
(Many companies close in the Netherlands for sure …</p><p>Unfortunately Jesiska couldn’t make it to Australia in time for Christmas.
Therefore I decided basically last week to go to Jakarta instead!
I personally don’t care much about Christmas, but I thought the office would
be closed during this time.
(Many companies close in the Netherlands for sure during this time),
and I was just very eager to see Jesiska again.</p>
<p>I’ve been neglecting this blog a little bit lately, I’ve just been busy,
well actually work is a huge time sink.. it sucks.
Then I’ve also been trying to increase my income with various project I will
discuss next.
Although I’ve been neglecting the most important one which is the financial one.
It just seems so boring every time I think about it.
I need to do this soon because my bonus interest of my savings account will stop
at the new year.
This holiday conveniently overlaps with that.</p>
<p>The <a href="http://raster.click">raster</a> project core is up and running. Basically anyone can sign up to it
and take advantage of it for free! Because I haven’t setup payment yet!
Because many payment providers ask tax details! Which is difficult to do in
Australia!..
I need to go back to the Netherlands sometime to finalize this part.
Meanwhile I’m letting my father test it so I can get some feedback.
He was genuinely quite excited about the system.
This is quite a help because there are just details I don’t think about,
such as making sure a week always begins on Monday (rather than it just being
whatever day your on), or making the grid clickable so it’s quicker to plan in. etc.</p>
<p>In the weekend I was hacking with some colleagues on a little app idea we had.
This is totally different than the raster project, and I get to work together
with some other people.
Which is nice cause most of my projects tend to be alone undertakings.
Not sure if this going to become anything, although it is ambitious,
but we’re keeping it secret for now.
What we will do is try and share technical ideas and contribute to the
open source community.</p>
<p>I’ve also been considering applying for permanent residency. I looked into that
and apparently to get enough points I need to take an English test.
Seems a bit ridicules considering that I’ve done my masters in English,
but that’s not the worst part.
The English test would cost 300$, and there is no guarantee that I get an high
enough score (although I can’t imagine why I wouldn’t, except for my dyslexia).</p>
<p>What I want most of all is too see Jesiska again.
This situation sucks even with all this stuff happening, it’s pointless without Jesiska.</p>Jappie lives with kangaroos2017-11-06T00:00:00+01:002017-11-06T00:00:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2017-11-06:/jappie-lives-with-kangaroos.html<p>I’ve been living in Australia for well over a month now and I really enjoy it!
The first two weeks were quite a wild ride, looking for my first job and finding
a permanent place to stay.</p>
<p>I had 3 interviews appointments prepared before arriving.
I finished them in …</p><p>I’ve been living in Australia for well over a month now and I really enjoy it!
The first two weeks were quite a wild ride, looking for my first job and finding
a permanent place to stay.</p>
<p>I had 3 interviews appointments prepared before arriving.
I finished them in the first week and managed to get 3 offers.
I just choice the one that looked most interesting and also happened to pay the most!</p>
<p>Finding a house was a whole other level of insanity.
I think I’ve been in contact with a scammer, that offered an apartment in the
middle of the business district for 900$ per month (900$ per week would be more realistic).
All I had to do is deposit some money in a bank account and he would surely sent
the keys.
In another situation the rent agent wanted me to deposit money into his account
for the deposit without signing a contract! What are these people thinking?
Rental prices are trough the roof too, I’m paying 330$ per week right now..
That’s about the same amount what I’d pay in the Netherlands per month!</p>
<p>Things seem to have settled more down now.
I have this pattern of a morning training, going to work, go to home and put
food in the microwave which I prepared over the weekend.</p>
<p>I quite enjoy the job too, although it’s a lot less programming than what I
expected.
A lot of project management is involved too, and <span class="caps">QA</span> is super slow.</p>
<p>Meanwhile I’ve been working on a little side project to try and increase my
income.
It’s <a href="http://raster.click">http://raster.click</a>, a roster system aimed at restaurants.
This is basically re-doing an idea I had when I was younger,
but this time I set it up as a service rather than selling the source.
This allows me to keep working on it if it becomes adopted by a reasonable
amount of people.</p>
<p>Finally I’ve been trying to come up with a way to invest all this money I’m
earning. The banks are much nicer in Australia because you get actual interest.
However It’s too low in my opinion.
I’ve had several ideas ranging from making an automated trader to start mining
crypto currencies.</p>
<p>Sadly Jesiska hasn’t arrived yet. I’m not even sure if she will before Christmas.
It seems to take ages and while I enjoy living like this,
it’s not the same without Jesiska.
I actually wish I would’ve stayed longer in Indonesia at this point.
Oh well we’ll see what happens next.</p>Work work work, ya ya ya2017-10-02T00:00:00+02:002021-08-07T18:17:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2017-10-02:/work-work-work-ya-ya-ya.html<p>Last Wednesday I started my first job at <a href="https://www.openlearning.com/">Open Learning</a>.
It has been interesting,
I know for a fact I contributed almost nothing
to the company up to this point.
This is to be expected according to reddit since it’s the first week.
I did ask a lot of …</p><p>Last Wednesday I started my first job at <a href="https://www.openlearning.com/">Open Learning</a>.
It has been interesting,
I know for a fact I contributed almost nothing
to the company up to this point.
This is to be expected according to reddit since it’s the first week.
I did ask a lot of questions since I was able to start programming for real yesterday.</p>
<p>The first today I just spend on getting familiar with the platform and the last
day I dabbled around a little in react (hello world!).
The reason I have for saying I did little contributions is because I just took up
more time of other employees asking questions then producing much.
Even the <span class="caps">UX</span> work hasn’t been integrated yet because the designer was busy.</p>
<p>What I didn’t expect however was having a good time over there.
I enjoy talking with the various people and the tasks aren’t that bad.
There is also a pantry, free fruit and some tea I take advantage of.
For lunch we usually go out eating but I’m considering taking lunch as I wanna
safe some money especially this first month.</p>
<p>I noticed a lot of discussions where about food, actually the people at the
office seem genuinely really interested in food.
Which is good because I like trying new foods.
Unfortunately Australia doesn’t seem to have much of local cuisine, but they’ll
happily import dishes from neighbouring countries (which are delicious).</p>
<p>What’s even better is that they allow you to work from home.
At this point I can’t really do that yet because I’m a newby and ask a lot of
questions, but at some point I also maybe able to do that.
Once per week having no travel times would be very good!</p>
<p>Aside from asking to many questions (in my experience) the only other big issue
I have is leaving times.
I’m not sure how they decide upon that.
Its always kind off awkward for me to leave the place,
but on the other hand I also don’t want to do (free) overwork.
I think I’m gonna ask about these things because they maybe an issue later.</p>
<p>The place is nice, the things they’re doing could definitely help
society and I enjoy the work (not so much the <span class="caps">UX</span> testing part,
but it was necessary).</p>What do you think about jakarta2017-09-28T00:00:00+02:002017-09-28T00:00:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2017-09-28:/what-do-you-think-about-jakarta.html<p>Now I’ve finally arrived in Australia, its time to reflect upon my
expectations I had for Jakarta.
Actually.. While being in Jakarta I got asked this question quite frequently:
“What do you think about jakarta?” and I just stood there with no answer.
This post is an attempt to …</p><p>Now I’ve finally arrived in Australia, its time to reflect upon my
expectations I had for Jakarta.
Actually.. While being in Jakarta I got asked this question quite frequently:
“What do you think about jakarta?” and I just stood there with no answer.
This post is an attempt to answer that question,
and at the same time I address the <a href="./journey-oceania">expectations post</a> I made before.</p>
<blockquote>
<p>For Indonesia I’m expecting the population to be better at English then the
local Chinese where in China.</p>
</blockquote>
<p>People definitely where better at English than in China,
at least in my experience.
I had several taxi drivers that could speak some English and even the locals
know or try some words.
In China only students ever tried to speak English,
other people just didn’t want to or couldn’t.
I suspect this is in part due to cultural difference.
However for China the great Chinese firewall and systematic cultural isolation
(that is in part by design) helps to marginalize exposure to English for sure.</p>
<p>In China learning English is something which parents are willing to pay good
money for, but Indonesians are willing to go <em>much</em> further with international
schools. All day to day conversation happens in English in these schools
and the entire education system of Singapore is blatantly copied.
This turns out in little children walking around malls speaking in an almost
perfect British accent with each other.
This was such a weird experience to hear perfect British by a 10 year old’s with
Bahasa (Indonesian) all around you.</p>
<p>The schools are a little weird, apparently there coincide 3 different major
systems, Indonesian, Chinese and Singapore(/American).
However once I thought about this a little I realized we basically have the same
in the Netherlands, for example the ‘free schools’ or ‘open schools’.</p>
<blockquote>
<p>My grandfather actually told me many Indonesians would be capable of speaking
Dutch. I seriously doubt that</p>
</blockquote>
<p>I was totally right by the way about nobody being able to speak Dutch,
I only ever met one person who said something in Dutch to me and it was done
in a way where the words didn’t make much sense.
They said ‘nice to meet you’ in dutch, without introducing themselves first.</p>
<blockquote>
<p>The weather will be hot in Indonesia, at least in Jakarta and there will be lots
of rain.</p>
</blockquote>
<p>The weather wasn’t that hot actually. Most of the time I was quite comfortable
in the Jakarta heat.
There was also practically no rain, although one time there was a thunderstorm,
which killed the electricity at the place I slept.
This caused me to sleep in the heat which is a terrible experience.
After this I realized that people in the slums sleep trough this every day.
I suddenly felt very bad for those poor souls living there.
They probably never get a good night rest.</p>
<h1 id="surprises">Surprises</h1>
<p>What really surprised me though was the food.
If you’re a fan of strong flavoured foods Jakarta is a great place to go!
Another thing I didn’t expect was the insane amount of malls.
They just casually build three malls next to each other, and then fill it up with
schools, fitness clubs and travel agencies, aside from the stuff you’d expect
such as shops and restaurants.
A reason for the popularity of malls may be the security that keeps out street
sellers and the <span class="caps">AC</span> which makes walking around there much more pleasant.</p>
<p>Another really nice thing was the thousand island trip.
So this is basically a boat ride of about an hour to one of the islands on the
coast of Jakarta.
However it’s really similar to Aruba for about a tenth of the
cost to get there.</p>
<p>Okay so here we’re kind off getting out of Jakarta, but what I really really
recommend is the night safari. You’re being taken around the zoo in a little
train, although it goes a little to fast to experience well,
and can’t take pictures for sure. However it also includes a theme park and
taking pictures with baby animals
(need to pay a little for the animals but it includes the opportunity).</p>
<p>What’s really strange to see is that some people, have as job, guiding traffic
but they’re not employed by government.
They just sort off start guiding traffic on their own and then live of the
tips.
A similar kind of people do the same for parking, they’re just living parking
meters,
but they also help with getting in and out and making sure you park straight.
As a consequence all cars are parked really nicely.</p>
<p>Finally another peculiar thing is that there are <strong>so</strong> many people self employed,
it’s ridiculous if you think about it.
Little shops everywhere, the traffic people, parking people etc. All self employed.
I think this is a major difference from the developed world,
where you generally start working for a boss because it just pays that much
better. But with those low wages, it’s just that more attractive to start for yourself.</p>
<h1 id="so-what-do-i-think-about-jakarta">So what do I think about Jakarta?</h1>
<p>I think that Jakarta is like an uncut gem.
It already has some really attractive attributes,
but until issues such as the crazy traffic and drinkable tap water are
addressed it isn’t as good as it could be.
Still it was an amazing experience to be there and I’d definitely come back for
the food alone.</p>Beginning of eternal summer2017-08-16T16:00:00+02:002017-08-16T16:00:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2017-08-16:/beginning-of-eternal-summer.html<p>Right now I’m in the airport waiting to go to Jakarta.
It’s really weird what I’m doing I think, few people actually travel alone by
plane, but in my case I’m just sort of emigrating and finding a job in a
foreign country.</p>
<p>I want to …</p><p>Right now I’m in the airport waiting to go to Jakarta.
It’s really weird what I’m doing I think, few people actually travel alone by
plane, but in my case I’m just sort of emigrating and finding a job in a
foreign country.</p>
<p>I want to try and do an eternal summer. What this entails is basically
following the pattern of migration birds, once winter come you move south,
where it’s warmer. In my case, when dutch winter comes, I’m going all the way
south to Australia, where winter just is about to end and summer is about to
begin and when Australian winter comes I go back again.
This probably won’t work if I get a good job in Sydney.
However getting a programmer job right now looks like a big hurdle.</p>
<p>I listened to freakonomics that told me most of the population is over
educated, which means that people with master degrees are competing for entry
level positions, and people with bachelors are again competing for even more
lower level positions. <span class="caps">IE</span>, it’s crowded at the top, which pushes competition
downwards.
This seriously disturbed me, because I started doing this too.
For example, I started replying to junior positions because it looked like
nobody reacted to my other letters.
I must have applied to over 50 positions right now.
Being a foreigner also doesn’t help I think. At this point I would love to do
the low level trainee-ship (Java 101) they offered me in the Netherlands except
then in Sydney.
I still hope this can be done, like at this point the economy is growing again
right?
People need more programmers right?
Nope, I think the job market tends to lag behind economic numbers.</p>
<p>However when I arrive in Indonesia I’ll be doing a holiday first.
Not only that I’m expecting to work a little on the web shop system I’m building,
and maybe even do some applying.
Of course just in the evenings, when I’m not doing cool traveling stuff.</p>
<p>I’m also thinking about tracking the stuff I’m doing in Indonesia holiday a
little on the blog in a journal kind off style,
I know nobody is reading this but <strong>I</strong> like writing this.
Directly encoding images etc, would also be nice.</p>
<p>Maybe I’ll also add my Chinese journeys to the blog. I at least have the journal
of my second voyage in the bag. which I don’t have yet digitally.
In the passed however I usually started full of enthusiasm with recording but
after a while I got bored of it.
Perhaps I should take it more casually and just spend a limited amount of time
on it.
Actually that maybe a good idea because then I’ll only record interesting stuff.
And it help with motivation because I’ll think “Oh these 20 minutes are over quickly”.</p>Journey to Oceania2017-08-07T22:00:00+02:002017-08-07T22:00:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2017-08-07:/journey-to-oceania.html<p>Wednesday 2017-08-16 I’ll leave towards Oceania.
First I will go for a month to Indonesia, having a little holiday and meeting
my almost girlfriend.
Then I will move on to Australia to do a “working holiday”,
where I hope to do my profession, as a software engineer or work …</p><p>Wednesday 2017-08-16 I’ll leave towards Oceania.
First I will go for a month to Indonesia, having a little holiday and meeting
my almost girlfriend.
Then I will move on to Australia to do a “working holiday”,
where I hope to do my profession, as a software engineer or work in <span class="caps">AI</span>
(think either data mining or machine learning).</p>
<p>This is the second major journey I undertake, which in my case
means being away for more than three months. This time I’m expecting to stay
away for at least six months, depending on if I can get good work and a visa.
Perhaps I’ll write in a later blog post about the first journey I undertook to
China, where I had a great time and which motivated me to leave again.
In this blog post however I’ll record my expectations for this journey,
so that one day I can look back and see what a fool I was.</p>
<p>For Indonesia I’m expecting the population to be better at English then the
local Chinese where in China.
A Chinese man can say with quite some confidence he’s not going to encounter
many foreigners in his life, in fact I had people tell me this when I was there,
Which is in stark contrast to Dutch mentality where you expect to encounter many
foreigners and thus you’d have to learn at least English and either German or
French (all three also is common).
I also expect Indonesians to be willing to learn more foreign tongues,
because they don’t have a common tongue such as the Chinese have with
Mandarin (in fact there are thousand of languages in Indonesia),
which means they’re expecting to meet people that don’t know their native
language, and thus English will be used as lingua franca.
especially on Bali island everyone will know English because of tourism and
also in Jakarta, but to a lesser extend.</p>
<p>My grandfather actually told me many Indonesians would be capable of speaking
Dutch. I seriously doubt that, and my (almost) girlfriend has confirmed this is
only true for older Indonesians.
My Indonesian girlfriend has also warned me life is slower in Indonesia
than in China (which was already pretty laid back).
By this I think she means that people aren’t too strict with appointments and
that we can expect to wait for stuff to happen.</p>
<p>The weather will be hot in Indonesia, at least in Jakarta and there will be lots
of rain. I know this because we talk often about the weather and I’m pretty
sure it rains there more.
Another environmental thing is that there will be lots of traffic in Jakarta,
and especially many scooters and motorbikes. Probably similarly to the amount of
<a href="https://en.wikipedia.org/wiki/Guilin">Guilin</a>, actually the climate maybe also similar to Guilin summer times.</p>
<p>Australia is a whole other beast of course. Everyone will speak English there
and I’ll probably communicate quite easily.
My grandfather thought I’d have some problems with Australian English,
but I doubt that.
I mean perhaps I have to get used to it for a week, but that pales in comparison
to the efforts I had to take for learning the heavy accents of mainland Chinese people.</p>
<p>I do expect Australians to be incredibly polite and somewhat more laid back than
Dutch people, however still ‘faster life’ than in China and Indonesia.
By politeness I mean, trying to keep a kind off distance from strangers and
embezzling your words so they sound ‘better’.
This politeness of course breaks down between friends.</p>
<p>I’m expecting Sydney to be dominated by cars. Like cars to the extreme.
Whereas in the Netherlands we use bicycles, you can get anywhere by car in
Sydney.
I’m expecting Sydney to be about the size of <a href="https://en.wikipedia.org/wiki/Xiamen">Xiamen</a>.
Although much less dense (because of the cars needing their space).
Sydney will probably have about the same weather as the Netherlands,
laying a lot farther south than Indonesia, although with the seasons mirrored of course.</p>
<p>My friends also told me Australia is a very deadly place, where everything
tries to kill you.
I think that’s highly exaggerated, mostly because people tend to put as much
distance between themselves and nature as possible, especially in cities.</p>
<p>It’s a little short but there is it, a quick overview of my expectations for
these two countries.
Probably full of stereo types, but this is why I do the journey.</p>
<blockquote>
<p>Travel is fatal to prejudice, bigotry, and narrow-mindedness,
and many of our people need it sorely on these accounts.
Broad, wholesome, charitable views of men and things cannot be acquired by
vegetating in one little corner of the earth all one’s lifetime.
– Mark Twain</p>
</blockquote>Making money with foss2017-05-22T00:00:00+02:002020-05-11T10:33:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2017-05-22:/making-money-with-foss.html<p>In this blog post I will discuss how to make money with open source software.
Why do you care?
Making money of this model is hard,
yet you got to pay the bills.
Even though the free software variant is <a href="https://www.gnu.org/philosophy/free-sw.en.html">consumer friendly</a>.
(note that these consumers maybe businesses too).
This …</p><p>In this blog post I will discuss how to make money with open source software.
Why do you care?
Making money of this model is hard,
yet you got to pay the bills.
Even though the free software variant is <a href="https://www.gnu.org/philosophy/free-sw.en.html">consumer friendly</a>.
(note that these consumers maybe businesses too).
This is a big problem.
Lot’s of projects are maintained in people’s unpaid free time,
next to their day jobs.
So they don’t really get the love they deserve.
Can we fix this?</p>
<p>I would like to do this myself full time.
Lest I became <a href="https://www.reddit.com/r/financialindependence/">financially independent</a>, I couldn’t.
So in here I will list the options I found trough researching the subject.
As both an overview to myself and to the interested reader.</p>
<h1 id="google-results">Google results</h1>
<p>In my naive approach I searched for <a href="https://www.google.nl/search?q=making+money+with+foss&ie=utf-8&oe=utf-8&client=firefox-b&gfe_rd=cr&ei=50YjWYn_CdHU8geKob64BQ">“Making money with foss”</a>.
The first site that came up was <a href="http://www.infoworld.com/article/2612393/open-source-software/greed-is-good--9-open-source-secrets-to-making-money.html">info world</a>.
This site was full of warning flags,
for example having a
click bait title and using a pagnation.
After skimming the article it wasn’t about making money of an open source project,
but potential reasons for open sourcing (part of) your code.
It was aimed at mid to high level management,
non technical texts are of no interest to me,
mainly because they usually have an agenda.</p>
<p>A <a href="http://www.fosslc.org/drupal/node/131">second article</a> linked to <a href="http://carlodaffara.conecta.it/?p=90&cpage=1#comment-50">another page</a>
with a better list, but I couldn’t find it.
The third <a href="http://www.cio.com/article/3178621/open-source-tools/how-to-make-money-from-open-source-software.html">article</a> was just a summery of a video and wasn’t very focused.
Okay so my search terms were bad.
Time to be more specific.</p>
<p>I searched for <a href="https://www.google.nl/search?q=making+money+with+foss&ie=utf-8&oe=utf-8&client=firefox-b&gfe_rd=cr&ei=50YjWYn_CdHU8geKob64BQ#q=foss+business+models">“foss business models”</a>, you know you did something right when
scholarly articles start being presented.
There was an entire <a href="https://en.wikipedia.org/wiki/Business_models_for_open-source_software">Wikipedia page</a> on the subject.
I guess more people are really interested in this.
So does this blog post die in the cradle?
I say no, I can still give my opinions on that page.
Another thing is that discussing this page on my blog will force me to closely
look at the possibilities.
Besides the fact that there is a (quite big) wiki on it only shows how much
people care about this subject.</p>
<h1 id="the-business-models">The Business models</h1>
<p>The core problem open source software faces is not having a monopoly on
distribution that copyright provides.
Therefore we can’t sell it directly:
Anyone can start distributing the software without asking anything for it,
without running a loss because distribution is
(practically) free for <a href="https://stratechery.com/2015/aggregation-theory/">software on the internet</a>.
Giving up this monopoly is a necessary requirement for open source,
called the <a href="https://www.gnu.org/philosophy/free-sw.en.html">second and thirth freedoms</a>.
This is also done with the <span class="caps">BSD</span> like licenses,
so we can consider it a part of open source.
Therefore we need to find a way of making money by working around this restriction.</p>
<h2 id="sell-stuff-besides-software">Sell stuff besides software</h2>
<p>One of the primary strategies is selling stuff besides the software.
Red Hat does this trough <a href="https://en.wikipedia.org/wiki/Red_Hat#Business_model">providing support contracts</a> for example.
This can be extended by providing certificates of expertise
(which Red Hat also does).</p>
<p>I don’t like this approach as it gives the incentive to make bad software.
If your software is hard to use, then only you can sell support.
You should make your software easy to use.</p>
<p>Another strategy is selling of merchandise, such as fan <a href="https://store.wikimedia.org/collections/accessories">T-shirts or coffee cups</a>.
Which the wikimedia foundation does.
Selling merchandise is quite clever because its much more easy for someone to
buy ‘something’, rather than just donating.
Besides its free advertisement for your project,
<em>and</em> your users now have something to identify themselves with.
So I think if you have an opensource project you should seriously consider doing this.</p>
<p>Finally a programmer can rent himself out to a project to add features or do
bug fixes.
These include <a href="https://www.Bountysource.com/">Bountysource</a> or <a href="https://en.wikipedia.org/wiki/Kickstarter">kickstarter</a>, where Bountysource is
usually fix an issue first and then get paid whereas kickstarter works on
promises.
In essence you’re offering up your own time to work on a project.
In both cases there is of course the persistent issue that its <a href="https://softwareengineering.stackexchange.com/questions/648/how-to-respond-when-you-are-asked-for-an-estimate">hard to decide</a>
how much time this will take.
With kickstarter however you can estimate it better because you’re probably
starting your own bigger project, so the overhead of estimation is smaller.
With Bountysource each bounty requires its own estimate and these are often
smaller, think of about \$100, its hard to decide to even bother with that.
Another issue with Bountysource is that you have to switch from project to
project often, which requires a lot of getting familiar with the code base
overhead.
The model of Bountysource is less then perfect to live off.
Although for existing developers of a project its a great motivation to work a
little more.
But don’t expect strangers to ‘join in’, just because of the bounties
unless they’re highly valued.
In fact you can see that few people ‘live’ off Bountysource just by the
overview of bounty hunters. Only three people at the time of writing got over
\$2000 in the last 90 days and only 9 over \$1000.</p>
<p>In the strategy of renting yourself out is going to interested parties yourself
and offer to write software.
I’ve attempted this at times with for example <a href="https://github.com/jappeace/offertex">offertex</a> and <a href="https://github.com/jappeace/schijt-je-rijk">schijt je rijk</a>.
The issue that always gets me is negotiating about price.
And of course once the work is finished, you need to find <em>more</em> work.
Therefore you won’t get better overtime.
Imitating red hat with support contracts could do that,
but starting with that on your own is hard.
Although I had other factors working against me,
living in a remote village didn’t help.
So you mileage may vary.</p>
<h2 id="donations">Donations</h2>
<p>Another powerful trick of trying to become self sustained on software is using
donations.
The big issue with this is of course getting people to donate in the first place.
A famous example of a donation scheme as that of <a href="http://www-archive.mozilla.org/press/mozilla-2004-12-15.html">mozzilla firefox 1.0 release</a>.</p>
<p>Recently another option has popped up in the form of <a href="https://www.patreon.com/">patreon</a>.
The idea is pretty simple, subscribe to a creator and every time he puts
something out (or every month) donate a small amount decided by you.
<a href="https://www.patreon.com/landley">Some</a>
<a href="https://www.patreon.com/kozec">developers</a>
<a href="https://www.patreon.com/bcachefs">have</a>
<a href="https://www.patreon.com/pippin">embraced</a>
<a href="https://www.reddit.com/r/linux/comments/5omtvg/patreons_to_support_open_source_projects_please/">this idea</a>,
although I haven’t seen anyone that earns self sustaining amounts.
It does provide a reliable income stream and is aimed at <em>individuals</em>,
and since there are <a href="https://www.patreon.com/cgpgrey">other</a> <a href="https://www.patreon.com/avasdemon">creators</a>
who have managed to get to a sustainable level.
We can expect this to happen eventually for software developers too.</p>
<p>The big advantage patreon offers over Bountysource is that,
rather than having to think about how much time creating a feature costs as
with Bountysource,
you can just continue improving the project how you think it should be done.
Of course some trust in the developer is necessary for that,
Bountysource doesn’t have that problem, since payment is done afterwards and I
assume regulated by Bountysource.
However apparently Bountysource offers a similar service <a href="https://salt.bountysource.com/teams/neovim">for projects</a>.</p>
<p>Another approach to this is to start a foundation.
Blender is developed like <a href="https://www.blender.org/foundation/">this</a>,
and if your project has any kind of traction,
do consider this.
Trusting a foundation is much more easy because
they’re regulated by law.</p>
<p>A more recent iteration of donations is using twitch.
Essentially you’re just explaining what you’re doing on a live
stream and then people can ask questions while you’re programming.
Twitch handles the payment processing for you.</p>
<h2 id="advertisements">Advertisements</h2>
<p>A trick often overlooked by most developers is advertising.
In principle free software is not against the idea of advertising.
However a problem with this is that anyone can take your software,
remove the advertisements, and redistribute the add free version.
You can prevent this from happening by offering two versions,
one with adds and one without and then ask your users to support the project
by downloading the one with adds.</p>
<p><a href="https://en.wikipedia.org/wiki/Adblock_Plus#Controversy_over_ad_filtering_and_ad_whitelisting">Addblock plus</a>
famously white listed adds as a way of generating revenue.
It may be considered hypocritical,
but remember that as free software anyone can fork it and remove this feature.
Not that you have to since there are already <a href="https://github.com/gorhill/uBlock">alternatives</a>.
Note that although Adblock Plus probably made quite a good load of money
with their white listing program,
it will probably kill the project eventually.
Because now they’re essentially running an extortion racket,
becoming nothing more than parasites and providing lower quality user experience.
I imagine the people who go out of their way to install an ad blocker are not
the kind of people who are satisfied with an ever laxer white list.</p>
<h2 id="crypto-currency">Crypto Currency</h2>
<p>There is an opportunity in crypto too.
Although this immediately sounds fishy.
I’d say that taking a couple cycles per minute to do some
mining isn’t <em>that</em> bad.
I mean how is it any different from stealing a users
attention with advertisements?
It’s probably even better because the users’ time is more valuable
than machine cycles.
No wonder that google <a href="https://www.theregister.co.uk/2018/03/14/google_cracks_down_on_crypto_with_ad_ban_from_june/">cracked down</a>
on this.
Crypto is a substitute for advertising,
Google is essentially a massive ad company.
But this has the same problem as advertisment in that forks will
occur which would take it out.</p>
<h2 id="license-tricks">License tricks</h2>
<p>The final category for making money with open source is license trickery.
So this comes from the idea that, if you are the sole copyright holder,
you can put the software under various licenses.
Note that it is a big if to be the sole copyright holder,
you need consent that any contributor is handing over their copyright.
And as we will see in the re-license case, this can be abused.
Which increases contributing barrier significantly.
But in return as project owner you get a lot more possibilities for making money.</p>
<p>Dual licenses are a practice where you offer one open license,
and another business aimed license which promises more support than the open
license or removes restrictions (such as forcing open source).
This is <a href="http://lucumr.pocoo.org/2013/7/23/licensing/#the-stricter-gpl">where <span class="caps">AGPLV3</span> shines</a>, original authors can offer large organizations an
alternative license, however downstream receivers of the code under <span class="caps">AGPL</span> cannot
do this.
<a href="https://en.wikipedia.org/wiki/MongoDB">Mongodb</a> is an example that does this and they can only do this by
asking contributors to <a href="https://www.mongodb.com/legal/contributor-agreement">hand over copyright</a>.
As a contributor you should be wary of doing this,
copy left doesn’t work if you hand over copyright.
If you work under <span class="caps">BSD</span> or <span class="caps">MIT</span> kind of licenses it doesn’t matter
(and I assume you already came to terms with this).</p>
<p>However, this does allow a company to thrive upon open source.
A company is still required to hoard in the business deals with other companies,
and to collect the copyright assignments from contributors.
It may be taken over similarly as Oracle did to Sun.
But the <span class="caps">AGPL</span> based code was already under license,
so the community can step in and take over development,
as happened with Illumos (Solaris fork) after Oracle went on its rampage.
Although a recent <a href="https://stratechery.com/2019/aws-mongodb-and-the-economic-realities-of-open-source/">article</a>
by Ben Thompson cast some doubt on the long term efficacy of this strategy.</p>
<p>This has however a darker side in potential license trolling.
Which <a href="https://lists.debian.org/debian-legal/2013/07/msg00000.html">Oracle (who else) did</a> for example with a database.
Changing from <span class="caps">BSD</span> to AGPLv3, which in case of Debian required around 100 other
dependent packages to change to AGPLv3 too.
Which of course is not going to happen.
Oracle probably did this to force users of that database to take a commercial
license instead, taking foss projects (such as Debian) as collateral damage.</p>
<h2 id="proprietary-extensions">Proprietary extensions</h2>
<p>Proprietary extensions involve releasing an open source core and add
(usually business centred) proprietary extensions.
The Wikipedia page lists several example but the one I’m personally
familiar with is the IntelliJ project.
Which is <a href="https://github.com/JetBrains/intellij-community">opensource</a>, has a <a href="https://www.jetbrains.com/idea/features/editions_comparison_matrix.html">proprietary paid extension</a>, and also a <a href="http://www.jetbrains.org/display/IJOS/Contributor+Agreement">cla</a>
(which doesn’t hand over copyright but does a similar thing,
licenses a right to copy).</p>
<p>IntelliJ is kind off open source, but many developers want to pay for things
such as <span class="caps">CSS</span> or JavaScript support.
Although I’d say any text editor can do that, such as <a href="http://spacemacs.org/">spacemacs</a>.
where IntelliJ shines is Java and Scala.
As far as I can see are the ‘supported’ features, just bells and whistles.
However they maybe valuable for a professional developer, or to a software house
to which the license fees are nothing compared to developer time.</p>
<p>Because IntelliJ is open source it allowed Google to create android studio.
This is great for the IntelliJ team because now there is another party
that is dependent on their core of which they hold all copyright.
Google may help developing the IntelliJ Java core, just to get it
to work for android developers.
This is the thing most companies are after with open source,
free programming manpower.</p>
<h2 id="delayed-open-sourcing">Delayed open sourcing</h2>
<p>This is the thing <a href="https://en.wikipedia.org/wiki/John_Carmack">John Carmack</a> famously did with <span class="caps">ID</span> tech.
After some time selling the games he would release the source of the games.
Which he did for <a href="https://github.com/id-Software/DOOM">Doom</a>, <a href="https://github.com/id-Software/Quake">Quacke</a> even <a href="https://github.com/id-Software/DOOM-3">Doom 3</a>.
This in turn led the games to be developed upon for a long time after their
release.
For example <a href="https://ioquake3.org/">ioquake3</a>, still actively develops the quake engine,
driving sales of the quake game itself because the assets aren’t freely available.</p>
<p>So he open sourced partly dated tech, because he was confident he could do better now.
And he increased sales of already released games,
because people wanted to hack on it.
I think this is pretty genius, gaining some extra juice out of old projects.
The really nice thing is that the lifespan of these games has tremendously increased.
Creating a core of die-hard followers.
Doom has been ported to essentially <a href="https://en.wikipedia.org/wiki/List_of_Doom_source_ports">everything</a>.
It’s a mystery to me why not more companies are doing this.</p>
<h2 id="re-license">Re-license</h2>
<p>If you are the sole copyright holder, you can stop distributing under the
open source license and re-license it.
Originally I didn’t want to include this option because you’re no longer
doing foss at this point.
But if your project is dying, because you can’t give it enough love.
This maybe totally valid.
It’s your project after all,
if you can make a valid business out of it by stop doing opensource
I’d give you nothing but praise.</p>
<p>Having said that,
this also opens up the opportunity to hate upon
Oracle. So lets hate upon Oracle.</p>
<p>So if we Google: <a href="https://www.google.nl/search?q=why+oracle+is+horrible&ie=utf-8&oe=utf-8&client=firefox-b&gfe_rd=cr&ei=Hi0sWcTQNOvGXqT5o7gM">Why oracle is horrible</a>, we can get some <a href="https://www.quora.com/Whats-so-bad-about-Oracle">dumb</a> <a href="https://www.quora.com/Why-do-some-people-hate-Oracle">quora</a> answers.
These are just not the point.
This <a href="https://www.reddit.com/r/linux/comments/2e2c1o/what_do_we_hate_oracle_for/">reddit thread</a>, sums it up nicely.
What is really dog kicking evil were the Solaris issues, which is discussed in
this <a href="https://www.youtube.com/watch?v=-zRN7XLCRhc#t=33m0s">this video</a>.
A little further in <a href="https://www.youtube.com/watch?v=-zRN7XLCRhc&feature=youtu.be&t=2482">the video</a> its is explained how it happened.
So what happened is that Oracle obtained all copyright from various authors by
buying <span class="caps">SUN</span> which required initially handing over copyright,
Open Solaris was closed by Oracle with a re-license.
This was only possible because Sun asked contributors to fork over copyright.
What we can learn from this is that if you contribute to free software and care
about it, <strong>never hand over copyright</strong>.
I’m happy to say however that a fork of Solaris occurred called <a href="https://wiki.illumos.org/display/illumos/illumos+Home">Illumos</a> that
seems to still be active.</p>
<h1 id="in-conclusion">In conclusion</h1>
<p>Because we are interested in making money,
this post will took us all over the place.
On the one hand we have the greedy businesses,
and on the other side the diligent developer.
Licenses were never discussed in university,
which is interesting because this is <em>the</em> method for making
money with software.
I think having discussed the overview and shown some concrete examples was a
good exercise.
I was not aware at all for example of the AGPLv3 practices which
are interesting (without passing moral judgment).</p>Tool survey2017-04-01T00:00:00+02:002017-04-01T00:00:00+02:00Jappie J. T. Kloostertag:jappieklooster.nl,2017-04-01:/tool-survey.html<p>Some time ago I made a blog post about thesis <a href="thesis-writing-tips.html">writing tips</a>.
However while writing that a large part started to be about text editing tools
and version control.
To keep the thesis writing tips post more focused I postponed writing about
that. This post treats my tools of choice …</p><p>Some time ago I made a blog post about thesis <a href="thesis-writing-tips.html">writing tips</a>.
However while writing that a large part started to be about text editing tools
and version control.
To keep the thesis writing tips post more focused I postponed writing about
that. This post treats my tools of choice.</p>
<p>All people will only ever care about a niche of tools.
The reason is that in our world, we
have too many choice to form an opinion about each of them.
Tell me for example about your favorite pick axe or transport ship,
you probably don’t have an opinion about these tools.
Which just shows how more complex our society became compared to for example the
stone age,
where everyone had a favorite type of stone.</p>
<p>Note that isn’t enough to form an opinion about a tool to care about it,
you actually have to have used it for a while,
and then switch to another quite different tool to start caring.
Only once you switch you will form a deeper ‘bond’ with the new one or old one.
I think I switch quite often.
Which brings us to another reason for this post,
I want to compare my current tools of choice with those of future me.</p>
<h1 id="operating-system">Operating System</h1>
<p>This is where things will get interesting.
The main theme of my selection has always been tweakability.
I want to be able to change things I don’t like.
This is seen most clearly from my <span class="caps">OS</span> choice:
<a href="https://gentoo.org/">Gentoo/Linux</a>,
this is a source based rolling release distribution.
Most distributions work with pre-compiled software.
However software provides often at compile time, flags that enable or disable
features.
Gentoo standardizes this and allows you to specify per software package
which options you want or don’t want.</p>
<p>The reason I chose Gentoo initially is more petty, a class mate of my was using
it, and I didn’t even understand what “source based” or “rolling release” meant.
I wanted to be at least at the same level as my classmates,
so in my stubbornness I spent about 2 weeks installing it on my own the first
time, with help of the handbook.
It took so long because I didn’t know anything really about the shell and unix,
Gentoo really forces you to learn.</p>
<p>Now I stick with Gentoo because it allows me to do customizations,
while updates won’t break these.
Gentoo’s packagemanager is very careful in preserving configuration files.
Unlike Ubuntu,
which upgraded me out of my custom alternatively installed desktop environment.
Gentoo has to be more careful, because of its inherent configurable-ness.
Whereas Ubuntu provides ease of use, which goes hand in hand with,
knowing it better than the user.</p>
<h1 id="control-everything-with-keyboard">Control everything with keyboard</h1>
<p>During my software engineering classes I got introduced to the idea that the
mouse is pretty slow to work with.
The key difference between mouse usage and keyboard only usage is that mouse
usage often involves searching the right place to click, wheareas using the
keyboard is just doing what you want to do.
By systematically making the mouse less important,
working with the computer becomes more like playing an instrument.
I use several programs that minimize mouse usage:</p>
<ol>
<li><a href="https://i3wm.org/">i3-wm</a> or its wayland successor <a href="http://swaywm.org/">sway</a>
(as soon as wayland takes off, it’ll probably be some years later)
This is the window manager that fills the screen with active window by
default, or splits it if there are multiple windows open.
It also has work spaces (basically groups of windows under a global tabbing system).</li>
<li><a href="http://www.vim.org/">vim</a>. This is a text editor with “modes”.
normal mode is navigation, and insert mode is for typing text.
If you work a lot with text its definitely worth learning this.
But beware, expect the first two weeks or so of using this program to
have lower productivity. (and basically no productivity the first day ;) )</li>
<li><a href="http://spacemacs.org/">Spacemacs</a>. This is an evil combination of Emacs and
vim. I use this for software development and writing larger texts.
whereas I use vim for quick and dirty edits (besides vim is available on
every Unix system whereas Emacs and especially Spacemacs is not).
Oh I also run this in daemon mode so that I can use i3 to move windows rather
than the pretty bad buffer navigation of Emacs.</li>
<li><a href="https://github.com/ranger/ranger">Ranger</a>. This is a file browser that works
with vim like bindings, I also have a Spacemacs plugin that does the same
inside Spacemacs.
Actually if you combine i3 + vim + ranger you have a really solid <span class="caps">IDE</span>.</li>
</ol>
<p>I haven’t found a good way to eliminate mouse usage from the browsing experience.
There are plugins that give vim like navigation,
but I never really found them compelling to use.
This maybe just personal taste,
but I think it also has to do that browsing,
is usually quite similar to searching.</p>
<h1 id="write-everything-down">Write everything down</h1>
<p>I’ve slowly over the years come to realize how unreliable my memory is and
how liberating it is to write stuff down.
No need to think about something which you’ve already written down.
I have a little idea project in which I write down random ideas
I have for programs or systems, even political once.
The reason for doing this is that the idea can then exit my mind without the
fear of forgetting it,
often I will later extend it or realize its unfeasible or impractical.</p>
<p>To this project I also added a planning file which contains some stuff I
probably should work on.
This really helps focusing and narrowing down a direction I want to go in.
They also provide a dialogue between passed me, current me, and eventually
future me.
It was a classmate who brought me up on this idea, but CGPGrey pointed out
the dialogue aspect.
The dialogue aspect also holds true for ideas,
but ideas are generally not plans yet.
Plans have some form of commitment too it, you just need to find the time.</p>
<h2 id="version-control">Version control</h2>
<p>I mentioned “projects” before, by this I mean I have the files under version
control.
I use <a href="https://git-scm.com/">git</a> for version control.</p>
<p>Wait, I’ll clarify the concept of version control first.
Think of it as Ctrl+Z on steroids managed by a dedicated program.
It can track changes over multiple files and directories, and with each
change you want to save you can add a message.
It can also send these changes to other machines or ‘locations’.
There are actually a bunch of options for this
<a href="https://subversion.apache.org/">svn</a>,
<a href="http://www.nongnu.org/cvs/">cvs</a>,
<a href="http://bazaar.canonical.com/en/">bazaar</a> and many more.
At this point its safe to say however that git crushed all competition,
the reason for this is that the architecture of git compared to other systems
is dead simple.
Also <a href="https://github.com/">github</a> helped a lot.</p>
<p>However I don’t just use git for just programming projects, no.
I use also it for important configuration files, my todo list, my idea project
and of course my thesis.
Basically most information I put into the computer that can be represented
as plain text without to much trouble, I put in git.
The reason for this is that it has a much nicer history than for example dropbox,
and provides advanced merge mechanisms.
Besides with git I don’t need to have to trust some external service,
since its decentralized.</p>
<p>Actually git has slowly become a sort of diary of mine.
For example when I delete items from the planning or ideas they’re not truly
gone, just stored in a previous commit (=ctrl+z log entry),
with a message why the change happened.
I think future me, or historians if I ever become that important,
can track most of my life at this point with help of git.
In the back of my mind I’ve actually developed an opinion that git usage
is a basic computer skill and should be thought on schools,
since it could make collaborative work on documents much easier.
Although I know this won’t happen in the near future,
since it would also require the rejection of binary formats for text processing
all together.
Which is something harder to ask than replacing with an open format,
with which entire <a href="https://www.documentfoundation.org/">foundations</a>
have trouble pushing upon people and organizations.</p>
<h1 id="discovery">Discovery</h1>
<p>The source of finding new tools is an important thing to discuss.
I think I had three major sources over the years:</p>
<ol>
<li>School, mandatory content</li>
<li>School, classmates</li>
<li>Reddit</li>
</ol>
<p>The first two sources have essentially become closed since I started writing
the thesis and no longer physically are at the school.
However I’m expecting a fourth source in the form of colleagues once I start working.</p>
<p>I think mandatory school content is generally the worst quality.
First of all it lags behind the inovation curve.
Secondly most people already know how to use that since its mandatory so you’ll
have most competition on that area if you’re an expert.
Finally it doesn’t really give you much of opinions on the matter at all.
Giving dumb reasons for learning the tool “you have to”,
or “this is industry standard” rather than giving a comparison between
alternatives.
However I did find that the school of applied science (Windesheim),
usually was even further behind than the University of Utrecht.
But maybe it was because I was following a master course rather than a bachelor.
Note that I’m not really talking about the theory,
but the <em>tools</em> with which you can put theory into practice.</p>
<p>Classmates are a much better source, in fact one of the best I think because
they can tell you their opinion, and opinions are <em>so</em> important.
Even if you disagree with them, it forces you to defend your position
and reconsider, which makes your choice stronger.
Or if you fail you may flip and gain a new experience and form new opinions.</p>
<p>I have mixed feelings about reddit, its a major source of distraction,
but it has a lot of gems (not even hidden, I mean reddit has search functionality!).
I often subscribe to sub-reddits of tools I’m interested in using to learn more
about them and to lurk on those valuable opinions.
But the hivemind, seriously fuck that politically correct piece of shit…
<sup><sup>I love you</sup></sup>.
Our relation is complicated.</p>
<h1 id="paranoia-security">Paranoia (security)</h1>
<p>I encountered the abysmal state of computer security first in Windesheim.
I won’t talk about the state of computer security because I don’t care.
What they did on Windesheim, was: They just taught is to hack so that we
would know how to defend.
I hated it.
You may say, oh isn’t hacking super exciting?
No its not.
Its not really constructive,
you just need to think in terms of possibilities and try out everything.
Rather than programming, where you just try out something that comes to
mind and then later review to see how to improve (or stop once satisfied).
Then this gets mixed with the idea that paranoia is <em>good</em>.
I know from my experience that under certain circumstances I have paranoid tendencies,
I don’t need to promote that in my life.
Finally its never enough. Being secure is always a moving target unless you can
formally proof you’re secured, which you can’t because that’s as <a href="https://stackoverflow.com/questions/476959/why-cant-programs-be-proven">hard as
programming</a>.</p>
<p>Since that class, which is now almost 3 to 4 years ago,
I had several concerns in the back of my mind.
My weak set of just 3 passwords used rationally.
If one got cracked I would lose a third of my online accounts.
If my email account got cracked I would lose everything.
And the fact my identity could easily be faked.
So I addressed these concerns recently (about 3 months ago).</p>
<p>To start with the most important issue, passwords.
For this I use something called a ‘password manager’ which called
<a href="http://keepass.info/">Keepass</a>,
actually I use the <a href="https://keepassxc.org">qt based one</a>
(works better with Linux).
I use this one because its opensource, and has a <span class="caps">GUI</span> and an android client.
The reason for using this is complicated and explained very well
<a href="https://www.youtube.com/watch?v=3NjQ9b3pgIg">here</a>
but basically its more safe than memorizing several and it also happens to be
more convenient.
I use <a href="https://syncthing.net/">syncthing</a> to share the database with my phone
and my server, I don’t wanna lose that.
I just have a huge password, I didn’t want to do the keyfile, because I’m afraid
to lose that.</p>
<p>I recommend everyone to use a password manager.
Its easier to use,
it can autofill passwords in <a href="https://github.com/pfn/keepasshttp">your browser</a>
and you just have to remember the master password.
It allows you to have much less “stuff” in your mind and its more secure.
I even put <span class="caps">WIFI</span> passwords in there.
<a href="https://wiki.gnome.org/Projects/NetworkManager">Network manager</a>
also allows viewing of <span class="caps">WIFI</span> passwords,
at least in <a href="https://www.kde.org/plasma-desktop">plasma</a>,
but since I’m almost always in i3 its just easier to have in Keepass.
Besides now I have the <span class="caps">WIFI</span> passwords on my phone too.</p>
<p>For authentication (making it difficult to steal my identity),
I went trough the effort of setting up
<a href="https://www.gnupg.org/">pgp</a> for email and git.
For email I use the <a href="https://www.mozilla.org/en-US/thunderbird/">Thunderbird</a>
extension
<a href="https://www.enigmail.net/index.php/en/">enigmail</a>.
All my git commits and emails are signed now.
To configure signing in git I followed
<a href="https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work">this</a>.
<a href="https://stackoverflow.com/questions/10161198/is-there-a-way-to-autosign-commits-in-git-with-a-gpg-key">This</a>
stackoverflow question explains how to sign automatically.</p>
<p>Linus Torvalds
<a href="http://git.661346.n2.nabble.com/GPG-signing-for-git-commit-td2582986.html">believes</a>
auto signing is unnecessary.
I disagree, it makes it much harder to steal my identity,
without me needing to think about it.</p>
<p>For email I know its a good thing to sign,
spoofing email is <a href="https://lifehacker.com/how-spammers-spoof-your-email-address-and-how-to-prote-1579478914">incredibly easy</a>
and by signing it,
spoofing my email becomes difficult.
Assuming the recipient checks it, which most people don’t,
but if nobody sets this up nobody will ever start using it.
So I setup email signing more as a matter of principle I guess.
Oh and btw, if everyone actually did this,
the life of spammers would become significantly harder,
since spoofing is no longer an option.
So shame on you if you don’t do this.</p>
<p>I also setup a <span class="caps">VPN</span> on the server with <a href="https://openvpn.net/">openvpn</a>,
mainly in case I want to go to china again,
I’ll just have that around.
It allows you to visit google there.
The <span class="caps">VPN</span> client is basically a use flag for network manager.</p>
<p>Using a <span class="caps">VPN</span> by default won’t help you with security much.
Maybe if you use not encrypted <span class="caps">WIFI</span>,
but even then if you visit https websites the connection is encrypted anyway.
Using a service for it is especially dumb,
because now that service provider has always access to your packets.
If you want to do a <span class="caps">VPN</span> you need to setup your own
<a href="http://www.comparevps.com/"><span class="caps">VPS</span></a>,
and then setup your own <a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-an-openvpn-server-on-ubuntu-16-04"><span class="caps">VPS</span></a>.
I actually host this website and the <span class="caps">VPN</span> on the same <span class="caps">VPS</span> at
<a href="https://galaxyhostplus.com">Galaxy host plus</a>.
with the cheapest offering <code>VPS OpenVZ Hosting - OVZ - 2016 - LC1</code>.</p>
<h1 id="other-random-stuff">Other random stuff</h1>
<p>I don’t know what to say about most of these.</p>
<ul>
<li><a href="https://github.com/cgag/hostblock">Hostblock</a>
for disabling reddit, youtube and facebook while working on stuff.
This is a huge productivity boost</li>
<li><a href="https://www.mozilla.org/en-US/firefox/new/?scene=2">Firefox</a> for browsing.
I think google has enough opurtinity to spy upon me.
They won’t get my browser.</li>
<li><a href="https://help.gnome.org/users/gnome-terminal/stable/">gnome terminal</a>
as terminal emulator, wait I have a reason for this.
It’ll rewrap text on screenresizing. <a href="https://konsole.kde.org/">Konsole</a>
won’t do that, but konsole renders
<a href="https://github.com/tonsky/FiraCode">firacode</a> litagures correctly.</li>
<li><a href="https://github.com/Valloric/YouCompleteMe">You complete me</a> to give both
Spacemacs and vim better contextual awareness for various languages</li>
<li><a href="http://orgmode.org/">org mode</a> to write any document in.</li>
<li><a href="https://fbreader.org/">Fbreader</a> for reading ebooks.</li>
<li><a href="http://qpdf.sourceforge.net/">qpdf</a> for pdf files.</li>
</ul>Optimus time!2017-01-29T00:00:00+01:002017-01-29T00:00:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2017-01-29:/optimus-time.html<p>Using gentoo is a bliss most of the time.
The package manager portage is one of the most advanced managers that I’ve ever
used.
Last week I encountered how much better it is than for example apt, when my old
laptops power supply burned out and I had to …</p><p>Using gentoo is a bliss most of the time.
The package manager portage is one of the most advanced managers that I’ve ever
used.
Last week I encountered how much better it is than for example apt, when my old
laptops power supply burned out and I had to resort to a xubuntu machine.
Apt has no capability for example to run multiple install processes at the
same time, whereas portage just goes with that.
Apt also has no propper versioning system, which makes python 2 and python 3
seperate packages, but portage can use slots for that.
This has the advantage that the search tool results become a lot less cluttered.</p>
<p>However using gentoo has also its disadvantages.
It can demand a lot of time once you start tweaking with it.
Yesterday I spent about 2 hours figuring out why my graphics card was slow.
3 hours trying to use the card in a dedicated manner (
rather than hot switching which the optimus technology does, promises to do, but
doesn’t do).
Which failed because documentation is often to technical and its hard to debug,
because they sort of expect you know how to debug.
I ended up with the galium driver which upon kvm from vmware??
That didn’t sound right, and indeed this was the software render.</p>
<p>Then I tried using the bumlbebee and primus driver from the main portage
tree and non masked primus variant instead of the bumblebee overlays ones.
This maked primus run work again. (it was borke for about one year, and I used
optirun -b primus instead which worked).
using the mainline gentoo tree also revealed that apperanlty primus requires
nvidia to have a use flag enabled.
Its the thing I don’t like about useflags, how poorly they are documentated
I think the idea is that normally portage will tell you what use flags you need,
but not for edge cases (such as using an overlay).</p>
<p>So at this time I was being hope full again because I got some results,
therefore I did one last ditch attempt to try and use a dedicated one.
I rebooted and X11 ended up in a black screen with a nonworking keyboard.
Thanks X for taking con troll over everything (wayland still worked).</p>
<p>If your keyboard fails you’re in big trouble,
X is sort of optional to fix things, but without a keyboard you can’t even
access the backup tty’s linux comes with (alt+{f1,f2,…}).</p>
<p>So now I had to make a new live usb, because this new laptop has no cd player,
and I only had live <span class="caps">CD</span>’s.
With the live [usb|cd] the plan is to fix your system so you can boot at least
into a tty.
So in my situation it just involved disabling the display manager
(which disables the keyboard).
In case of systemd you just remove the symlink
`/etc/systemd/system/display-manager.service`.
So after I did taht I spend over 4 hours figuring out what was wrong with the
display manager.
Apparently this new bumbleed clashes with X11,
I used the `/var/log/emerge.log` to figure this out
(I kindoff forgot all the stuff I changed and for some reason journalctl
recorded nothing of sddm.
Also deleting the xorg.conf was a bad idea. it’ll also produce a black screen.
However at this point I had about 5 different configuration of xorg so I just
coppied the working one in there.
I also found out that if you specify the wrong name in auto login of sddm
you will also get a black screen but with active keyboard so you can just
login to another tty to do more debugging.</p>
<p>So this time the great optimus technlogy (which I really start to hate),
has taking 9 hours.
This is on top of the first time when I learned about bumblebee in my early
gentoo/steam experience (I guess another 8 hours at least).
And that one time optimus suddenly broke (perhaps 4 hours untill I found the
optirun -b <span class="caps">PRIME</span> trick, prime is required btw otherwise steam won’t accept it).</p>
<p>I don’t know, I’m salty at both optimus and gentoo to a lesser extend.
I mean all these problems were just configuration problems I would’ve
encountered too on other distro’s I guess.
But gentoo makes it so easy to go out of your way and play with the configurations.</p>
<p>Perhaps next time I should just go with an <span class="caps">AMD</span> laptop, I hear they are making
tons of progress with their opensource drivers.</p>Deployment and todo’s2016-12-27T12:00:00+01:002016-12-27T12:00:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2016-12-27:/deployment-and-todos.html<p>Yesterday I deployed the website, to my terror it wasn’t responsive
on mobile devices (adapt to screensize).
So with this post I fixed that, by replacing the
basic font-size specifications from small to 1vw (everything else was em).
The rest of the site already used percentages.
The site needs …</p><p>Yesterday I deployed the website, to my terror it wasn’t responsive
on mobile devices (adapt to screensize).
So with this post I fixed that, by replacing the
basic font-size specifications from small to 1vw (everything else was em).
The rest of the site already used percentages.
The site needs to be responsive because its ironic to have a <span class="caps">CGA</span> based theme
being responsive.</p>
<p>So to deploy I found a vps partner in <a href="https://galaxyhostplus.com/">Galaxy Host Plus</a>.
They offer a vps package for about 2 euro 50 in the month I get <span class="caps">512MB</span> of ram,
<span class="caps">50GB</span> of disk space <span class="caps">500GB</span> of bandwidth and an <span class="caps">IP</span> address.
This is a pretty good deal I think, took me about 2 hours to find (you can get a
lot more expensive for less good specs, it should become cheaper every year).
The domain name I had already registered at <a href="https://www.starthosting.nl/">Starhosting</a>.
Its 11 euro per year.
So to make this site (self) sustainable I need to make at least
$$11/12+2.5=3.42$$ euro per month.
Although this is not completely fair since I’ll be using the vps for other stuff
as well, such as Syncthing and private git (not for source code but just personal
stuff, such as undeveloped ideas).</p>
<p>To setup this website it took about 2 days to figure out both pelican and styling
the theme.
I could have done this a lot faster with wordpress or something but it
wouldn’t have made me as happy as this theme and going full static website mode.
Full static website mode scales a lot better than wordpress and is cheaper in
maintenance (less band with, less memory used because no <span class="caps">PHP</span> or Apache).
Besides, I already knew how to <span class="caps">CSS</span>, <span class="caps">HTML</span>, and more pure <span class="caps">JS</span> than would be
considered healthy (yes I’ve done some native <span class="caps">JS</span> development, no libraries).
Actually now I think about it I would’ve probably gotten lost in wordpress and
for me it wouldn’t have been faster since I really wanted this theme.</p>
<p>Lets talk about why I wanted this theme.
First of all it fits me, I’m the kind of person who doesn’t care much for
designs.
Secondly I suspect its more memorable then a standard fashionable website with
bootstrap.
Thirdly I made it from the ground up (not the html structure,
I sort off borrowed that, since I didn’t know how to pelican),
which means its quite easy for me to change it.
Fourthly I think there is a select group of people who like this theme,
and those people probably also like what I have to say.
Therefore the theme is practical.</p>
<h1 id="todo-list">Todo list</h1>
<ul>
<li>+Scale mobile+
This is done now.</li>
<li><span class="caps">HTTPS</span>
With lets encrypt around I probably should do this to please the security</li>
<li>Add advertisements
This is until my Patreon finally takes off</li>
<li>Comment system. I have an idea for this, basically to make a reddit bot to
auto create a thread for me and then link this that per post in this website
with javascript. Shouldn’t be to hard, but probably a weekend of work
(If anyone wants to help please contact me)</li>
</ul>Website Launch2016-12-25T12:04:00+01:002016-12-25T12:04:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2016-12-25:/website-launch.html<p>So this is the post where I’m launching this website.
I finished the css mostly as a beautifull <span class="caps">CGA</span> theme.
In this post I’ll discuss my decision making process.</p>
<h1 id="why-make-a-site">Why make a site?</h1>
<p>As an extension of my youtube account I will use this website,
mostly for subjects …</p><p>So this is the post where I’m launching this website.
I finished the css mostly as a beautifull <span class="caps">CGA</span> theme.
In this post I’ll discuss my decision making process.</p>
<h1 id="why-make-a-site">Why make a site?</h1>
<p>As an extension of my youtube account I will use this website,
mostly for subjects I can’t make videos for.
Either because of lack of time or because a video format just isn’t suited for
it (for example when talking about a subject involving a bunch of shell
commands, this site can be a place to put those commands).
It can also be for themes that need to be well referenced
(controversial opinions for example).</p>
<p>The main purpose of this is to make money.
I want to adopt a digital nomad like lifestyle (or something similar),
and maintaining a website seems like a good addition.
Currently I’m running this at a loss of course.
Therefore I included advertisements on release and setup a patreon page.
I need to make this work as soon as possible.
However I also hate advertisements,
therefore I make a promise to delete them as soon as patreon can cover the
costs of running the website.
By which I mean hosting costs (vps in my case) and domain name costs.</p>
<h1 id="site-setup">Site setup</h1>
<p>I use pelican to create this site.
It is a script for parsing text based content formats and generate html from
them, so that you don’t have to redefine the menu every blog post for example.
So what happens it that you write your content in markdown,
then you run the script which generates the static html for you.
I tried setting up org, but had some problems so I’ll use markdown first,
I think I can mix formats anyway.</p>
<p>You can specify the structure of the html in the theme and of course the css.
Note that even if the html is static you can still embed JavaScript,
so you’re not committed to just static on each page.
This is why I chose this tool.
No commitments for the future and a relative simple setup.
I’m thinking of later embedding a comment system (probably reddit), and
macgyvering together a vote system, but this has to wait.</p>
<h2 id="the-theme">The theme</h2>
<p>The <span class="caps">CGA</span> theme was an intentional choice.
My target audience is mainly people with interests in technical things,
and I’m assuming that these kind of people will like such a theme either out
of nostalgia or pragmatism.</p>
<p>I got the inspiration from the Gentoo website they launched I think 2 years
ago as an April fools joke, here is a screenshot:</p>
<p><img alt="gentoo cga" src="/images/2016/gentoo-cga.png"></p>
<!-- screenshot -->
<p>(btw, if anyone has the original html implementation
please contact me, I want to re-host it).
I really loved the idea of just hosting a website like that.
I had started working on a theme, using the kernel config as a style guide:</p>
<p><img alt="kernel config" src="/images/2016/linux-config.png">
<!-- screenshot --></p>
<p>However I’ve been remarkably busy since that time
(grad school, China internship etc),
so I hadn’t come around finishing it.
Untill now since I started writing my thesis,
I’m realizing that soon I need to work,
and I don’t really want to work for someone else.</p>
<p>I like making <span class="caps">CSS</span> designs based of images, and I’m one of those oldtimers that
doesn’t use anything fancy like less.js or bootstrap.
So you can see here the result of my work.
I may add some more art work in the future but I’m intending to use this theme
for quite a while.</p>
<p>Which brings us to another advantage of this theme, it will age very well.
Unlike the newest fashion websites where people put huge images at the beginning
of their posts and where you have to scroll endlessly to read something is this
kind of theme tried and tested.
I see the modern website industry becoming more like the fashion industry
where practicality has to make way for beauty.
I refuse to partake in such an “arms race”,
redesigning my website every year or having to choose a picture every post
seems just like a chore to me.
If I want to add pictures to a post I’ll do it because I want to,
not because its fashionable.</p>
<h2 id="no-cdns-fb-links-or-google-analytics">No <span class="caps">CDN</span>’s <span class="caps">FB</span> links or google analytics</h2>
<p>External requests to other domains significantly decrease page loading times.
Don’t believe me? Well you can measure it yourself with your browser, it has
an build-in network monitor (right click -> inspect element -> network, now press
ctr+f5 and see what takes the longest to load, you can see most things go in parallel,
but you should pay special attention to the things that go in sequence).
Many page load speed improvement website actually recommend a <span class="caps">CDN</span>,
which is stupid because laying the http connection to a <span class="caps">CDN</span> is much slower
then just using the existing connection.</p>
<p>But <span class="caps">CDN</span>’s aren’t meant to increase pageload,
they’re meant to decrease serverload by moving parts of the
static content delivery from the dynamic server to dedicated servers,
freeing up server time and thus decreasing server load.
But my entire website is static so why would I bother with that?
server load is not going to be my bottleneck any time soon.</p>
<p>When I browse the internet I use umatrix to block things such as google
analytics and whatever the facebook requests do.
Mainly out of privacy concerns but also because this speeds every website
significantly, especially if you have a poor internet connection.
Besides I don’t imagine my target audience uses facebook that much,
correct me if I’m wrong.</p>
<h3 id="update-i-was-totally-wrong-about-cdns-at-least-for-fonts">Update: I was totally wrong about <span class="caps">CDN</span>’s (at least for fonts)</h3>
<p>I did some testing, and it turns out there are significant speed improvements
for using a <span class="caps">CDN</span>.
Ok you win this one, fonts.google.com.
But I’m still not going to add those like buttons.</p>
<p>Please do not that if you test this stuff that you have to press ctrl+f5 to tell
the browser not to use a cache.</p>
<p>See the difference in results below:</p>
<p>Testing myself (from the Netherlands)
<img alt="doing delivery from website (testing myself)" src="/images/2016/cdn-own-before.png">
<img alt="doing delivery from cdn (testing myself)" src="/images/2016/cdn-own-after.png"></p>
<p>Testing with a test service configured from new york:
<img alt="doing delivery from website (testing with service from newyork)" src="/images/2016/cdn-service-before.png">
<img alt="doing delivery from cdn (testing with service form newyork)" src="/images/2016/cdn-service-after.png"></p>
<p>From hong kong I got a ridiculous increase, from 3 seconds down to about 40ms.
I don’t have a screenshot of that though (and to lazy to revert).</p>Second post2016-12-24T19:51:00+01:002016-12-24T19:51:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2016-12-24:/second-post.html<p>Yeah, I needed two actually to make sure everything looks good.</p>
<p>So I guess I can type lots of information here.</p>
<p>I wonder how long it will continue..</p>
<p>Quite long acutually.</p>
<p>Well, that’s fine. I’ll just think off more spaced out words!</p>
<p>Damn I need to type so …</p><p>Yeah, I needed two actually to make sure everything looks good.</p>
<p>So I guess I can type lots of information here.</p>
<p>I wonder how long it will continue..</p>
<p>Quite long acutually.</p>
<p>Well, that’s fine. I’ll just think off more spaced out words!</p>
<p>Damn I need to type so much before it will crop.</p>
<p>Heheheh, this is cropped.</p>
<p>Ok I guess this happens on word count. Or character, doesn’t really matter,
at least not vertical space. Which I wondered and why I typed so much.</p>First post2016-12-24T19:43:00+01:002016-12-24T19:43:00+01:00Jappie J. T. Kloostertag:jappieklooster.nl,2016-12-24:/first-post.html<p>So I’m trying to setup this website and to do this I need some filler text.
Some people would make prommises about putting something up every week or so.
But I don’t wanna do that because I hate chores, so this may as well be a huge
waste …</p><p>So I’m trying to setup this website and to do this I need some filler text.
Some people would make prommises about putting something up every week or so.
But I don’t wanna do that because I hate chores, so this may as well be a huge
waste of time.</p>
<h1 id="why-setup-a-site">Why setup a site?</h1>
<p>Because I’m oppinionated people.</p>
<h1 id="you-have-some-spell-error-over-hre">You have some spell error over hre</h1>
<p>Use the comment section to help fix any spell errors</p>
<h2 id="there-is-no-comment-section">There is no comment section</h2>
<p>Exactly. We must conclude that there are no spelling errors eiher.</p>