October 2014
Tom Guinan: Advantage Futures devotes significant resources to provide superior support for traders so they can focus on trading. In the old days, it was more out-trade and open outcry support. Today, trading is predominantly electronic. The Advantage IT Department has grown from three to seventeen full-time employees. Our team includes network engineers, front-end support specialists, server support specialists, i-Series support specialists as well as second and third shifts providing 24 / 6 coverage. Having such complete IT support differentiates us from most of our competitors.
Question: For years traders have been concerned with speed. Is speed still relevant and if so, how is Advantage addressing it?
Patrick Mead: Speed will always be relevant in a first-in setting like trading. Bandwidth is often confused with “speed” on computer networks where “latency” is the true measure of how quickly information moves from point A to point B. Real bandwidth is far more abundant than it was 10 to 15 years ago when it comes to network hardware capabilities. We have 10, 40 and 100 gigabit options despite some services requiring only 100 meg or 1 gig. Network latency is a significant factor to a speed sensitive audience. At Advantage, we meet regularly with the top network hardware manufacturers to stay current with what is available or what is in the works in terms of hardware and latency improvements. We then discuss internally whether this new technology can effectively provide an edge to our clients.
Chase LaPlaca: Recently we installed switches in DC3 that port to port are the fastest switches on the market. If your server connects to that switch, you are going to get the fastest throughput from a Layer 2 standpoint.1 From a WAN standpoint, we are looking at some other technologies, meaning data center to data center, in an attempt to remain as fast as possible there as well.
Tom Guinan: We strive to stay ahead of the curve and in doing so we talk with vendors and service providers about various offerings and new technology. Obviously, some big names in networking technology are Cisco and Juniper. Currently, the fastest edge switch available is the Cisco NX 3548. Cisco did a great job with really fast throughput. We implemented them on the CME edge so that our clients could benefit by being a few nanoseconds faster. The other part of the conversation is how we keep the big transport parts, the Core devices fast and reliable. Historically, Advantage employed Cisco devices as Core and we continue to evaluate if Cisco is still the best alternative for Core devices in the Advantage network. While we have seventeen full-time employees, including in-house network engineers, we also engage consultants that work with other industry participants. Consultants might work with the exchanges as well as competitors and provide valuable perspective on what is going on in our industry. In addition, we have had numerous meetings with representatives from Cisco and Juniper in order to provide our clients with a fast and reliable network. The best solution could be a multi-breed network.
Patrick Mead: A multi-vendor solution is often times the best option. For a very long time Cisco ruled the roost in our industry, but there are other vendors involved now so mixing the network and utilizing each vendor’s strong points is a solid choice. We are confident that compatibility is not an issue while using multiple vendors and if incompatibility were to come up, it can be easily addressed. Most of our carriers are using Juniper gear which is directly connected to our Cisco equipment with no compatibility problems. Stability has also been no different when dealing with Cisco to Cisco connections or Juniper to Cisco connections.
Question: Are traders better off going with a third party provider to improve our systems or are we better off developing it ourselves?
Pinkesh Patel: One of the major factors we look at is the style of trading. Are they automated, point and click or looking at going to DMA? We try to determine if the client has the resources and time to develop software. Once we have this information, we can make recommendations to our clients. For example, we may suggest utilizing a third party vendor who is already certified with the exchange. With this solution the client is likely to save time, money and resources because the vendor is already approved by the exchange. After that we just need the client to provide us a risk module for testing which some vendors provide.
Question: How does Advantage help clients distinguish what is available in the market place versus building your own?
Neil Machchhar: This is one of the major questions posed by traders. There is such a wide variety of trading styles ranging from point and click trading to hybrid automated trading to full out algorithmic trading. It really boils down to the type of trading the person is trying to accomplish. The first couple of things to determine are if the trader is latency sensitive and is a proximity solution going to be beneficial. If the trader is not latency dependent, is the trader looking for an off the shelf solution? Are they a developer? Can they develop their own system or can the trader find a developer who can build the system? If they cannot, there are off-the-shelf solutions available which provide the capability to do these things without the need to write code. Some providers will offer canned algorithms or canned automated solutions which do not require developers. As we consult with the client and better understand their needs, we will be able to provide guidance as to which solution may make the most sense.
Tom Guinan: We try to determine what resources the client has available and how adequately the client will be able to handle all of the things involved in creating their own programs. We help compare that to the value they get in spending those resources on creating something versus all of the improvements that our ISV’s have made available in trading software. At Advantage, we have clients trading on fifteen front ends. We also have clients colocating servers at our datacenters. Some colocated servers run licensed software and some colocated servers run proprietary software.
There exists a wide range of services available which we review with clients and then we support them with whatever choice they make. Part of our value is highlighting some of the factors that clients should consider to make an educated decision.
Question: With the threats we hear about from North Korea, China and Russia, what does Advantage do to protect trade data and client information?
Patrick Mead: It was once good enough to simply filter unwanted internet traffic by way of IP address and port. Once a new security practice becomes more common, the intruders find a way to defeat these measures. They are now able to sneak malicious code through these simple filters undetected by the firewall IP and port rules. The new breed of firewalls can dig into the contents of a packet to detect this type of activity on the transmission and application
level, thus thwarting the attempt. We deploy such machines in our offices and will soon be rolling out those same machines into our datacenters. Other measures are to simply deny certain systems access to the internet or by using separate hardware away from the core network. Advantage uses both of these methods as well for systems that meet that criteria.
Chase LaPlaca: Patrick and I come from a security background; our firewall rules are very tight. We meticulously put in rules to keep it as safe as possible.
Tom Guinan: The rules applied to firewall security limit the access that anybody coming onto our system has so that you really cannot penetrate without an extremely sophisticated effort.
Question: Does Advantage have an advisory relationship with a security firm?
Tom Guinan: Yes, we engage more than one security consultant. We have conducted thorough reviews of both our network and network security. These are fairly extensive undertakings. We keep them on an advisory basis as they are familiar with our systems. The Advantage IT Team has people dedicated to monitoring network security all day, every day.
Neil Machchhar: A lot of IT firms offer technology solutions, colocation and other tech services and all have access to the same gear. Everyone is going to have a fast switch and fast connection. The advantage of having tech services in-house vs. outsourcing is the execution. You get superior response time by having these services available in-house from well qualified professionals who have experience in network security and trading technology. This is a value add for traders that should be considered when deciding technology needs. Another value add is intimate knowledge of the client’s set-up and technology needs. Some firms choose to outsource to save money but that does not encompass the same value of having it in house.
Tom Guinan: We leverage our department and get economies of scale by supporting thousands of Advantage clients. Even though there are seventeen team members, which is a large team by industry standards, they manage and enhance our network for our entire client base. We believe this is a real value for our clients. If there is a network issue, a consultant may take 15-20 minutes to re-familiarize with an account. Advantage’s network engineers are looking at it all day, all week, as are the front end engineers. They are familiar with what you trade, where your server is located and
who you are. All of our services are built to help you maximize your uptime and provide a fast, reliable network.
Question: Is there a way for traders to see if they are fully optimizing switches and the performance of their system?
Patrick Mead: We do not monitor at the application level. The software developer is usually the best judge of how quickly his firm’s trades were entered into the market and the acknowledgment received. Many developers create their own tools for measuring these transactions. If we get too involved in the monitoring of local and transit interfaces, we will hinder the process and potentially add latency. If a client baselines their order entry speed and sees a change, then we would dig deeper. Trade entry appears very simple at face-value but there are many variables in the price data
and order entry process that make it challenging to diagnose. At this point we begin a top-down troubleshooting process that may involve digging deep if the network appears to be a culprit. On a day-to-day basis we prefer to stay out of the way of trade flow and not act intrusively.
Question: Is there a way to best analyze how often I am hung out there in the market? Sometimes it is unclear whether it is the market itself, network connection or if I am running at optimal speed?
Chase LaPlaca: You would have to do tapping along the way, and we do not currently have a way to measure that without being intrusive. There are factors we can look at from a network level which provide hints as to whether it is a network issue or not. We also employ switch health tools to see if information is getting dropped at the switch or at your box or not at all. If a client comes in with latency questions or concerns about their front end, our network team looks at the devices involved to see if there are any drops or anything indicating that there might be a problem on the network gear. This also applies to the servers.
Tom Guinan: Basically, the question is if a trade gets dropped, how do you tell whether it is market conditions or a network issue. A network issue is usually detectable from measurements that are constantly being monitored.
Patrick Mead: If the network does appear suspect, the most effective way to determine if information was on the wire or lost in transit is a bit intrusive. We install a network tap on the questionable line. Once installed these are passive devices that do not impede data-flow but the installation process forces a machine off-line for a time. This is the best way to determine if a particular link passed the information along or did not. We do monitor link status on a
day-to-day basis. This tells us if the link is stable or not. Monitoring at the application layer from the network, however, would interfere with price and order signaling. This type of monitoring is process intensive and counterproductive to a speed sensitive financial trading environment. This type of monitoring is where the tap would come into play.
Neil Machchhar: Everything on our network is load balanced based on the number of traders using them. We actively look through the log files to see whether or not there is any negative impact to adding traders. For example, we closely monitor our shared servers in DC3 as well as the hardware resources that are being utilized. The same goes for our TT gateways. We install new gateways as needed. Typically, with our TT infrastructure, there are price servers and order servers. Price servers are pulling in prices and the order servers are sending out orders. Our redundant price servers and multiple order routers are load balanced to target optimal utilization.
Tom Guinan: Part of our commitment to technology is to separate the order servers and the price servers so that they are load balanced. While it is a substantially larger financial commitment to buy the servers, it is much more scalable to have separate price servers and order servers.
Neil Machchhar: Load balancing is a manual process, but it is definitely something that must be done, especially with TT.
Question: Does Advantage have any resources to help program on the TT ADL?
Neil Machchhar: We do not have in-house resources, but we have relationships with people who have done this for years and are former TT employees.
Question: Regarding the different front ends, is there a comparison on what is the fastest available?
Tom Guinan: We have fifteen different providers and stay agnostic. I do not believe vendors can provide a true comparison as there are many variables that may be beyond their control. For example, a TT environment hosted at Advantage may be faster than a TT environment hosted elsewhere. CQG and CTS are hosted by the service providers and may have speed statistics or you can test them yourself to see which performs better and gets you more fills.
Neil Machchhar: These off-the-shelf solutions are going to be rather comparable in terms of speed. When you get into the automation part of it they are more latency sensitive, some might perform better than others. We do not have any type of comparison sheet or analysis; however, we can provide internal latencies data/ information.
Question: What do you see as some of the trends on the front end or server side?
Neil Machchhar: From a front end perspective, TT has a major overhaul in the works. From our standpoint, it will reduce the amount of our management as the trading platform will be hosted by TT. They are improving their applications by upgrading to current technology instead of the legacy systems that have been in place since electronic trading first went live.
Tom Guinan: As our ISV’s in technology evolve, our roll gets modified but we will assist our clients in every possible way. That is one point we want to strongly emphasize. For example, as TT announced the plans for their new front end, Advantage hosted a forum for Advantage clients to hear from TT. They originally planned to roll out the new software in 2014, but I would now estimate it will not be widely adopted until late 2015. So what does that mean for our clients? It means that you can still rely on Advantage Futures for your technology needs. If TT’s new rollout gets delayed and we are supporting legacy TT, we will continue to do the same job we have been doing for over eleven years. If their application gets modified or another vendor comes along with a solution that you prefer, we will do our best to support it.
Neil Machchhar: As these companies develop and evolve, obviously, the race to zero remains a big question. A lot of companies are moving to new, faster technologies such as FPGA (Field Programmable Gate Array). Firms are offering FPGA solutions which are faster since the trade logic is processed directly on the FPGA card and does not need to communicate and process through other internal server hardware such as hard drive, processor, memory, etc. In terms of speed, they are getting faster.
Question: Advantage Futures has network engineers, server and front end specialists. Are there other less obvious roles on your IT support team?
Tom Guinan: Yes, for example, we have three employees dedicated to supporting statement creation technology. How does this impact the trader? Everybody knows the importance of getting a statement every day. Advantage has a team dedicated to making sure this technology is working and that accurate statements go out on time every day. This is good for traders and a strength for Advantage. Advantage clients can call on our staff to help with their technology needs without engaging a 3rd party consultant. This is a differentiator that we have from other firms. We in-house the technology support needs, whether it is statement creation, network engineers, server experts, etc. The only thing we do not do is code, because clients would not want Advantage’s coder to know what they are writing. That is obviously proprietary information.
Closing
Tom Guinan: The evolution in the race to zero does continue. Regulators and certainly exchanges are trying to figure out the best way to encourage good behavior and discourage bad behavior. Also, nobody knows if exchanges or regulators are going to put a governor on messages to slow down high frequency traders or how the trading ecosystem is going to evolve. Whatever unfolds, Advantage will work with exchanges, ISV’s and regulators to best support our clients. We will make sure we do everything we can to help inform our clients of any changes to the trading system.
We will continue to provide a healthy trading environment that conforms to whatever rules are in place.
