Apache Ignite and Apache Spark helping scalp traders win big
Life can be slow in Porto – a coastal city in northwest Portugal known for its magnificent bridges and port wine. But the data there is BIG and it moves FAST. At least that’s the case with data Manuel Mourato works with.
Manuel is a Big Data engineer at Nomad Tech. His company is a leading provider of IoT solutions for the railway industry and is developing a Big Data solution for the ingestion, processing and storage of train fleets data in near real-time.
But in his spare time, Manuel is working on a distributed application with in-memory capabilities that collects, processes, stores, classifies and visualizes different equities based on their current and historic prices – aiding stock traders with investment decisions in real-time.
He started the project last year, inspired by friends working in financial markets who told him they needed continuous access to real-time data for their micro-trades. These investments involve small amounts of money invested for very short periods of time – from seconds to minutes – and usually during the most volatile periods of intra-day market trades. Scalp traders buy and sell hundreds of very small equities daily and the best equipped can make a hefty sum by the closing bell.
“Having those investors in mind, it is quite hard to make predictions without in-memory computing (IMC) technologies,” Manuel said. “Just one minute of data-points, for a large number of equities, is too fast and diverse for one person to review without such tools. The solution I’m developing may help to tackle this problem by giving investors a way to check classified equities based on their prices in real time (min to min).”
IMC tech will give these traders data much faster and in greater amounts than it would normally be possible, he added.
“This solution will enable them to take data from share prices and then run it against an algorithm that looks at data for that minute – and then predicts the price the next minute. This data will guide their investments.”
And what’s under the hood of Manuel’s distributed application? Apache® Ignite™ and Apache® Spark™ of course!
In-Memory Computing Summit Europe
Manuel will explain everything during his session at the In-Memory Computing Summit Europe, happening June 25-26 in London. His presentation is titled, “When One Minute Can Cost You a Million: Predicting Share Prices in Real-Time with Apache Spark and Apache Ignite.”
Due to its technical nature, he recommends that attendees have some background in data engineering and architecting, or experience as investors in the stock market.
What to expect?
“You will first learn how to set up a set of Kafka producers and brokers, either by using Docker containers or separate instances, to consume price data from multiple equities via one or more APIs, on a minute to minute basis, in a fault tolerant fashion,” Manuel said.
“The processing phase will come next, in which we will look at the Spark Streaming module as a way to perform parallel in-memory computations on the ingested data and thus optimizing performance, preparing data for storage on an Apache Ignite grid comprised of one or more cached tables, via the Ignite-Spark module, thus providing sub second, SQL like syntax querying access to equity prices.”
Manuel said he'll then demonstrate how all of your tables can be persisted to HDFS, followed by a cache cleanup to manage memory usage.
“At the same time, every five minutes, another Spark application shall perform a classification using its Machine Learning module -- for all equities based on current and historic prices, to determine if investors should invest or not.”
The grand finale
“End users will then finally be able to visualize these conclusions with minute-to-minute updating Tableau charts connected directly to Apache Ignite, which you will learn how to set up as well,” Manuel said.
His session takes place on Monday, June 25 at 11 a.m. at the In-Memory Computing Summit Europe.
Get the latest new and updates by following the conference on Twitter @IMCSummit. And attend the conference for just £20! A limited number of £20 tickets are being raffled off each week. Enter to win a full conference pass for only £20 (you'll save hundreds if you win!). Enter here. Good luck!