Busy wait and queue performance
For the last month or so I've been rather busy developing a trading platform for algorithmic trading, connected to LMAX (you should also check out their very interesting Disruptor framework). It's a rather comprehensive solution with both risk management, position management, back testing, data management on tick level, etc built in. Everything that connects to and communicates with the exchange is event based, basically on a topic based setup. So one event producer may have zero or more event consumers.
I've really enjoyed finally having a proper concurrent project to work on, poking around the Java concurrency source code in order to find a few tips and tricks as to how I can squeeze out a bit more performance. It's not that the trading I'll be doing is that high frequent per say, it's more about eliminating latency. Now, of course, most of the computational power will be needed in the actual business logic as that's where the heavy lifting happens. But since the event system that glues everything together is so omnipresent, having it perform well is also very important. Analyzing the spacing between each tick for presumably one of their busiest instruments, EUR/USD, indicates that the most frequent observation is between 1 and 10 milliseconds, so that gives me a rough estimate as to how quick the consumers need to be. If they can't keep up on average the events will begin queuing up, which is very bad.
For the event system itself, delivery of one event from a producer to a consumer is on average 170 microseconds, so that basically leaves the consumer with almost 100% of the time (given that the exchange connected producers are very-to-fairly lightweight). I'm not really sure where to do further optimizations along my critical paths, so I might need to go crazy and do manual memory management or something (with the Unsafe class), but I doubt I'll need it. I think most of the quick win gains currently is at the network/IO level.
Anyway, I digress. Because the point is that while performance testing various parts I got some conflicting evidence as to which concurrent queue implementation to use. So I decided to do a somewhat more comprehensive test while still running it in an isolated environment. There are three factors at play here: Do we have a slow or quick producer, slow or quick consumer, and finally are we using busy wait loops when waiting for events? A slow producer or consumer would typically be slow if it needs to wait for IO, so that's simulated here with a Thread.sleep(1) call.
From JavaDoc we know:
- A
ConcurrentLinkedQueueis an appropriate choice when many threads will share access to a common collection. This queue does not permit null elements. ArrayBlockingQueueis a classic "bounded buffer", in which a fixed-sized array holds elements inserted by producers and extracted by consumers. This class supports an optional fairness policy for ordering waiting producer and consumer threadsLinkedBlockingQueuetypically have higher throughput than array-based queues but less predictable performance in most concurrent applications.
The most helpful utility since Evernote: Spool
I've been a relatively heavy user of Evernote for a while now. I don't mean heavy as in using all of its features or storing massive notes or anything, simply that my everyday personal habits depend heavily on it. Having a notetaking client that's available from my laptop and my phone, with transparent multi-way synching and frictionless integration and sharing between apps (at least on my phone where it matters the most) is immensely helpful.
In addition to all the small TODO's I store there (like movies to watch, books to read) it was also very helpful in organizing all the little things I needed to fix while writing my dissertation. Another huge use case for me was when I browsed the web on my phone. Sadly, very few webpages give me a good mobile experience. So typically several times per day, say during my lunch or dinner, I would find a webpage I wanted to read, often from Twitter. But given the useless mobile experience I would save it on Evernote for reading later when I got back to my laptop.
In here lies an opportunity, and that's exactly what Spool solves for me. I started using Spool a few days ago, and so far the experience has been excellent. Sure, I had some issues in the beginning, but hey, this is beta software and I'm happy to help out with testing. I don't think I'm using Spool exactly as they targeted it. When discovering their app during TCDisrupt I got the impression the focus was mainly on finding webpages (with or without videos) on your computer, and then store them for offline reading on your mobile device. Absolutely a good use case, and something I for sure will use it for. But mainly I use it for two reasons at the moment:
- I find a webpage on my phone that I want to read, but the mobile experience is horrible. Solution: Sync it to Spool, and within moments you get the same webpage but optimized for my phone. This is where I think my use differs from how they presented it.
- Given that I need to take the tube to and from work everyday, their offline storage is brilliant. London tubes don't have phone coverage, so now I can make good use of my offline time. This of course is very much part of how they presented it.
The only thing I need to find time to explore a bit more is their browser/computer experience. I know it's there and it looks good, just haven't had time to look into it. Sometimes I store a webpage on Spool rather than Evernote since I might find time to read it on my phone. But then I get home and might as well read it on my laptop. In this case I miss Evernote, so I would love for some form of integration between these two apps. But that's a minor thing compared to the benefits it has already given me 
Technical analysis and its dependency on volatility, the report
Some time ago I posted about the topic of my dissertation. There were some interest in getting access to the report when it was completed, and I'm glad to say that it's finally done.
So if you want to go straight to the details, here's the link: The profitability of technical analysis in a high frequency setting and its dependency on volatility.
If on the other hand you don't feel like reading 11000 words or so, here's the 30000 ft summary:
There is a strong (significant) positive relationship between returns from these strategies and volatility. If you're a prop trader you might be doing a face palm right now. But remember that academia have spent at least 30 years or so building a framework around the assumption that markets are efficient. Technical analysis shouldn't in that context be able to produce any abnormal returns, so we need to play at that field when testing these things. I am however delighted to see an increase in fields like behavioral and emotional finance as that surely is the only decent way forward in distancing academia from its extreme views.
So what does that relationship imply? Well, since volatility is fairly easy to "predict" as it is rather persistent, this also allows us to know when you should deploy or not to deploy technical analysis. That allows us to vastly improve our trading results compared to either always using technical analysis, or worse, purely rely on a long only buy-and-hold based strategy.
To produce these results one minute OHLC bars was used, analyzing the 15 biggest (in terms of trading volume) companies currently part of the S&P 100. If you're interested in how this was analyzed feel free to contact me. Most of the software was Java based and custom made by me, deployed across a cluster of Linux servers. With over 12 years of data doing daily parameter optimization and back testing, that amounts to about 30 TB of analyzed data.