Around 6 years ago I was experimenting and pushing the practical boundaries of what the JVM could handle with regards to heap size. We had Java 8 at the time, and allocating hundreds of GB of heap space was risky. Lots of time and effort was spend in many IT teams around the world just trying to keep the GC happy with this setup. Some, often myself included, would recommend we aim for an architecture with several smaller JVMs, each dealing with a more manageable heap size. But such a setup does sometimes bring with it a lot of complexity. While a web application might easily scale horizontally like that, not all application have that luxury. ...

Consider Amazon, the e-commerce bits of the company. While it might have started out as an online bookshop, it evolved into a generalized online marketplace. So what was critical in the success of Amazon? The internet would probably be a good answer to that question, among many other things. But how much of the overall software and business development efforts at Amazon do you think relate to core internet activities? Sure, it’s got a big online presence, which will require a lot of internet hardware, internet people and teams, ops, etc.. ...

Many years ago, in my late teens, I used to attend The Gathering. It’s a fantastic combination of people, passion, dedication and fun mixed together for a few days each Easter. If you’re not familiar with computer culture this whole thing might seem very alien, and, for many, somewhat a waste of time. But for those of us whom are familiar with this domain, and especially those of us who had some experience with the demoscene, it brings back a feeling of accomplishment. ...

Since late 2011 I’ve been working on distributed computing platforms, where software that runs across more than one machine is running as what we’d call a cluster. Now, as I focus more on blockchain and DLT, a lot of the approaches to computing are similar. A lightweight version of the challenges with working on distributed systems can be found when working on software that uses multiple threads to do concurrent computing. Things can quickly get very ugly if you do not know how to approach the challenge. ...

We’ve long had content delivery networks (CDN), that help speed up and reduce latency for your web visitors. This works by making sure that the CDN has servers that are physically closer and on a better network connection (fewer hops) compared to where your visitor is based. You maintain multiple copies of the same content spread out around the world, just to ensure you stay close to where your users are. ...

Some time ago I read “The (De-) evolution of My Laptop Battery“. At the time I had an old (2008 model I believe) MacBook Pro, on it’s first (or second) battery, and that battery was already heavily degraded. Because of that I saw no need to start logging the current battery capacity, but I made a note of doing that as soon as I got myself a new laptop. So, little over a year ago, when I retired by old and faithful for a brand new MacBook Pro, one of the first things I did was to set up a cron job that every half hour stores the current battery stats to a local file when turned on. ...

In 2010 I wrote a blog post called “Node.js, is this the next big thing?” I came to think about that again this week when I randomly watched Yevgeniy (Jim) Brikman’s video about the Play Framework at LinkedIn. Combine this with the progress of Scala.js and ClojureScript and I think we can conclude that node.js was very much an indicator of direction with simplification, unification and a return to an event driven model. All for good reasons. Look at job postings and you’ll find plenty of companies looking for the full-stack engineer. It’s difficult to be a full-stack engineer, especially a good one, if each tier in the stack is very different. It makes both economical and technical sense to have our tools reach across both server-side and client-side, while still maintaining a healthy separation of concerns with a 3-tier (or N-tier if N >= 3) system. ...

Over the last few months I’ve been learning Clojure. Best way to learn I find is to use what you learn for something you find useful or interesting. This morning, while browsing Hacker News, I found myself reading about Vitter’s reservoir sampling algorithm, with an example developed in Ruby. After spending a little bit of time thinking over the reasoning for why this algo would work as claimed, I decided to give it a quick spin in Clojure. I’ve published my experiment on GitHub if you’re interested in a different take on the algo implementation, with some Incanter inspired visual confirmation of it’s claim. ...

Having some spare time over the Easter weekend I thought I’d spend a little time trying out a few things, including: Java 8 Direct memory allocation with sun.misc.Unsafe So I made a hash-map implementation that stores map entries on memory allocated outside the typical JVM heap: BinaryOffheapHashMap First of all: The GC process in Java is a lifesaver, allowing us all to focus on writing functionality rather than obsessing over managing the mundane task of allocating and freeing memory. Yet, it doesn’t mean you shouldn’t be memory-aware. Failing to understand the memory impact of your code can lead to horrible performance, as you’re simply creating too much garbage. You can potentially try to avoid an ill-timed GC pause by having a bigger heap, but you’re just postponing the inevitable. Then there’s the case when you want to store huge amounts of data inside your JVM app. The lifecycle of this data is well understood and not part of any fancy distributed/parallel algorithm. You push your data in, then get it back out, potentially deleting it at some point. Isn’t it a bit wasteful to allocate a 200G heap, use it mainly for this purpose, and have the GC process scan through all that? For GC, the good die young, and it doesn’t scale that well with increasing heap space. ...

For the TLDRs: This blog post presents the reasoning behing a project called ResourcePriorityBlockingQueue which is a blocking queue implementation that: Allows you to assign priority on tasks, just as with a PriorityBlockingQueue. Tasks may belong to task groups where each group may have a different priority. In that case, tasks are prioritized by group first, then task priority second. For a particular instance of a ResourcePriorityBlockingQueue you give it implementations of a ResourcePrioritizer and a ResourceAllocator, which further defines to which resources a particular task is made available. Focus as been put on making the code highly concurrent with as little synchronized code as possible. Lock free code is used where applicable, with efficient use of internal data structures. This gives ResourcePriorityBlockingQueue a performance characteristic which is comparable to that of a pure PriorityBlockingQueue when used in a similar fashion. It’s available on GitHub, so feel free to poke around. Any constructive feedback is always welcome. ...