Commercial-Off-The-Shell Enterprise Real Time Computing (COTS ERTC) -- Part 4: Network Requirements

Enterprise applications rarely work in isolation nowadays. Instead they usually cooperate with one another in a network environment.
Due to Ethernet's uniqueness, this topic will only discuss the TCP/IP suite over Ethernet. The networks in Figure 4.1 are used for the discussion.

Figure 4.1 Networks

1. Network Latency
The network latency from a host in LAN1 such as Host11 to a host in LAN2 such as Host21 is the cumulative latency of all the following component:

• the TCP/IP stack latency on Host11. It includes all OSI layers except the 7th application layer (you have full control at the application layer). At least the 2 lower layers are implemented in the NIC firmware. All other layers are either included in the OS or also included in the NIC firmware.
• the switch latency in LAN1. The switch is a Layer 2 device. Originally an unintelligent Layer 1 hub or repeater was used. I will compare the latency difference later on.
• the routers latencies between LAN1 and LAN2. The routers are actually Layer 3 switches.
• the switch latency in LAN2. Again the switch is a Layer 2 device.
• the TCP/IP stack latency on Host21. Again it includes all OSI layers except the 7th application layer.
• the propagation delay from Host11 to Host21. It is the time it takes for signals to travel in the physical copper cables or fibers. Signals in fibers or copper cables can travel at about 2/3 of the fundamental speed of light.
  If the communication distance is short e.g. in a LAN, the propagation latency can be negligible compared to other larger latencies.
  On the other hand if the the communication distance is long, it can't be ignored. For example the straight distance from San Francisco to New York is about 4,125 kilometers and the one-way signal travel needs about 19.6ms that is the lower bound of your overall latency optimization.

Network latency can also be defined as the round-trip latency between Host11 and Host12. However since our discussion focuses on latency and jitter analysis of all involving components, the above one-way definition is sufficient.

The data unit in our discussion is a packet. For example we can set it anywhere from the minimum Ethernet frame size (it is about 64 bytes) to the maximum MTU (it is about 1500 bytes) and compare the difference.
When you analyze your application level latency, the data unit is an application level request, which has complications when interacting with the underlying Maximum Segment Size (MSS) of TCP and the MTU of Ethernet frames (later sections will explain more details).

The prioritized latency and jitter analysis mentioned in Part 1 still applies here. For example upgrading to a faster switch in San Francisco doesn't make sense because the minimum latency to New York is bounded by the 19.6ms propagation delay and the latency of switches is usually in low micro-seconds.

The following sections will discuss the latency and jitter in all involving components.
2. Non-Determinism in Ethernet
Ethernet, as standardized by IEEE 802.3, is implemented at Layer 1 and 2 of the OSI model. Its latency comes from the processing delays at  Layer 1 and 2. The latency has jitter because of its media access control (MAC) protocol - CSMA / CD at Layer 2.

With CSMA / CD, each host detects if another host is transmitting in the shared medium before it tries to transmit its own data (Carrier Sense). When a host detects a carrier, its Carrier Sense is turned on and it will defer transmission until determining the medium is free. This deference is unfortunately not predicable. If two hosts happen to transmit simultaneously (Multiple Access), a collision occurs and all frames are destroyed.

If we replace the switch in LAN1 / LAN2 with a hub / repeater, all four hosts will be in one collision domain. The more hosts trying to send data, the more collisions and the more non-deterministic.
Hosts sharing a half duplex connection are also in the same collision domain.For example even there are only Host11 and Host12 communicating to each other using a half duplex connection in LAN1, collisions still happen when Host11 and Host12 tries to send data to each other concurrently.

Although redesign of MAC protocol can solve the collision problem, it is not compatible to the existing ubiquitous Ethernet deployments and thus is appropriate and not a COTS solution.
If we can avoid collision, Ethernet will be much deterministic. This is what the switch in LAN1 / LAN2 is supposed to do. With the switch each host is guaranteed the exclusive use of the medium in both directions and is thus in a different collision domain from other hosts.
For example Host11 can communicate with Host12 while Host13 is communicating with Host14 without causing any collisions or frame drops.

3. Non-Determinism in TCP/IP
IP is at Layer 3; TCP and UDP are at Layer 4. The biggest jitter at these upper layers actually comes from the non-determinism of the underlying CSMA / CD.
For example if the underlying frame was dropped due to collision, the upper packet will be lost if it is UPD and will be retransmitted if it is TCP thanks to TCP's time-out mechanism.

A side note for TCP's time-out mechanism: if the network is slow, a low time-out value can lead to false retransmission. This is another reason why RTC applications prefer such fast networks as LAN.

You should also be aware of the following points
3.1 Non-Deterministic Routing Paths
A router has an additional Layer 3 compared to a switch. The more routers, the more potential communication paths between LAN1 and LAN2, and the more non-deterministic. A long communication path also leads to more frame drops due to data corruption. So a RTC system is usually enclosed in a LAN; otherwise make sure your routing path is deterministic.This is why many algorithmic trading systems especially high-frequency trading system, collocate with exchanges.

If the communication path is predicable, the determinism of a network with switches and routers will depend on that of those switches and routers. Modern switches and routers are actually very fast and can show very low jitter if managed properly, which will be discussed in Section 4.
.
3.2 Packet Header Overhead and Fragmentation
Each packet at IP, UDP and TCP layers has a header besides the real payload. The head size for IP, UDP and TCP is 20, 8 and 20, respectively.
Actually the Ethernet frame at Lay 2 also has a 20 bytes overhead. When an IP packet is greater than the Ethernet MTU (it is about 1500 bytes), IP needs to fragment and reassemble its packet.
TCP also fragments its stream into packets based on the MSS, the available window size and some other factors.

On one hand, more fragmentation and reassembly means more extra computing and higher latency. On the other hand, smaller packets than the MTU or MSS mean more significant header overhead and lower throughput.
In order to reduce jitter, your application level requests shouldn't have too variant sizes (different sized requests also mean different transmission delay). You should make your requests as large as possible before the latency misses your target. If possible, please decide the request size based on MTU.

3.3 TCP Flow Control
In order to avoid TCP packet drops from buffer overflows, TCP employs a sliding window mechanism to control its data flow.
So slow processing at the receiver host requires the sender host to send out smaller packets or even have to wait if the window size becomes 0. In order to reduce this jitter, make sure both ends can process network data fast and predictably.

If the network is slow, the sender may have to wait for the latest window size from the receiver. A fast network such as LAN can reduce this waiting latency. 

3.4 Nagle's Algorithm
Because of the 20 bytes header overhead in TCP, Nagle's algorithm coalesces a number of small outgoing messages into one packet and sends it out.
This algorithm is usually unacceptable to RTC systems because they need immediate response for each of their requests. This is especially true when the RTC requests are small.

RTC systems usually take the following two counter-measures:

• Enclose each request in one TCP packet;
• Turn off this algorithm by using the TCP_NODELAY socket option.

3.5 Delayed Acknowledgment
Again because of 20 bytes header overhead in TCP, a pure TCP response packet will have too much overhead. So this mechanism delays the TCP layer response by waiting about one or two hundred milliseconds, and hopes to send back the response along with the upper application layer response using just one TCP packet instead of two.

Unfortunately a one or two hundred milliseconds deadlock can occur if this mechanism interacts with the Nagle's algorithm and your application's response doesn't synchronize with the TCP layer's e.g. an application request was sent out in two or more TCP packets. Please refer to this resource for a detailed analysis.

In order to avoid this jitter, you need to take one or more of the following measures:

• disable the Nagle's algorithm;
• disable or configure the delayed acknowledgment mechanism. Please refer to this resource on Windows; and TCP_QUICKACK on Linux;
• Make sure your application level request is enclosed in one TCP packet.

3.6 UDP with Application Level Retransmission and Flow Controls
TCP implements its reliability at the cost of more computing work and header overhead than UDP. If TCP's reliability is more than what you want, you can use UDP and implement your simple reliability by yourself at the application layer. This approach should give your lower latency and jitter than using TCP.

4. Latency and Jitter of a Switch / Router
A router has an additional Layer 3 function. Its processing latency in a well-planed network will be predicable through a warm-up period that should cache all routing paths in the router. So we will focus on switches only.
A modern switch basically has two packet forwarding methods:

• Store and forward. A whole frame is buffered before being forwarded. A checksum is usually performed on each frame.
• Cut through. The switch reads only up to the frame's hardware address before starting to forward it. There is no error checking.

Because a cut-through switch doesn't have the additional buffering step, it has even shorter latency. However a cut-through switch will fall back to store and forward if its outgoing (egress) port is busy at the time a frame arrives. So we will focus on store-and-forward switches.

The one-way port-to-port latency of a switch is the cumulative delays of the following components:

• Layer 1 processing at both ports including signal modulation and data framing;
• Switch Fabric Latency. A switch has an internally shared high-bandwidth fabric that is much faster than any of its ports.
• Store and forward latency;
• Queuing latency. It occurs when different ingress ports are sending frames to the same egress port concurrently. Since only one frame can be transmitted at a time from the egress port, the other frames must be queued for sequential transmission in a FIFO manner. This phenomenon is called head-of-line blocking. Due to the FIFO behavior, the latency of a frame in the queue is unpredictable.
  So each port of a switch has an outgoing queue, which along with the switch fabric actually gives the impression of simultaneous paths among its multiple ports. 

Detailed analysis can be found from these two resources: Switched Ethernet Latency Analysis and Latency on a Switched Ethernet Network.

Both analyses show the queuing latency is usually the largest and a switch's jitter comes from it due to its unpredictable head-of-line blocking. .
In order to reduce the queuing latency and jitter, you need to take one or more of the following measures:

• Plan your network traffic properly including a predicable many-to-one mapping from ingress ports to egress ports; a smaller many-side value if a higher predictability is needed; and more importantly, no egress port should be oversubscribed.
  For example if you egress port is of 10GbE, you can only have a maximum of 10 1GbE ingress ports concurrently sending data to it; otherwise frame loss will occur. If you need some 1GbE port to have a higher predictability, you need to reduce the number of 1GbE ingress ports concurrently sending data to the 10 GbE egress port. 
• Because a switch's queuing jitter can cause avalanche effect at subsequent  network hops, this is another reason why you need to reduce the number of routers in your networks;
• Apply Virutal Lan (VLAN) or Priority values to different traffic. Make sure your RTC frames are in a separate VLAN or have the highest priority. This is similar to the different priority levels for RTC threads we discussed in Part 3. 

Finally in this section, even a hub / repeater has lower latency than a switch thanks to its simple Layer 1 processing, a RTC system should not use it unless necessary such as port mirroring because it creates jitter.

5. Gigabit Ethernet
Ethernet evolved from 10BASE-T to 100BASE-T to the current widely deployed 1GbE or Gigabit Ethernet. Now even 10GbE has often been seen in backbone networks and high-end servers as a cheaper alternative to the appropriate and expensive high-speed interconnects such as Fiber Channel and InfinitBand.
A modern cut-through 10GbE switch such as BLADE'S RackSwitch G8124 can has an average port-to-port latency in 680 nanoseconds (the average is on latencies of several different packet sizes from the minimum 64 bytes to the maximum 1518 bytes).

Although this 680 nanoseconds latency is at the same magnitude as the main memory, the load on CPU increases linearly, without some kind of TCP offloading or OS kernel bypassing, as a function of packets processed, with the usual rule of thumb being that each bit per second of bandwidth consumes about a HZ of the CPU clock. For example 10Gbs of network traffic consumes about 10GHz of CPU that is much higher than the 3.33GHz of Intel's latest processor Core i7.
As more of the host CPU is consumed by the network load, both CPU utilization and host send / receive latency and jitter become significant issues.
RDMA over TCP/IP or iWARP has come up as a rescue. Basically an iWARP NIC or R-NIC allows a server to read/write data directly between its user memory space and the user memory space of another R-NIC-enabled host on the network without any involvement of the host operating systems.

API's have been implemented for different platforms including the OpenFabrics Enterprise Distribution (OFED) by the OpenFabrics Alliance for Linux operating system, and the Winsock Direct protocol for Microsoft Windows.

Commercial-Off-The-Shell Enterprise Real Time Computing (COTS ERTC) -- Part 5: Java Requirements

Because programs designed using standard Java (either Java SE or Java EE or Java ME) exhibit non-deterministic timing of execution, standard Java is not widely used in RTC, especially in HRT.
However due to standard Java's popularity in many industries, extending it with RTC functions is very attractive. The Real-Time Specification for Java (RTSJ) is designed to seamlessly augment standard Java with RTC functions.

In this topic I will describe the key areas where standard Java creates jitters and how the two populate RTSJ implementations - Oracle's Java RTS and IBM's WebSphere RT - reduce them.
This is also a good time to compare RTSJ with C/C++ if they both run on the same underlying RT OS.

But before we start, here are few things about RTSJ:
Its implementations usually need a RT OS. Actually it is a must for HRT. In order to lower the learning curve, RTSJ doesn't introduce any new Java syntax, instead provides several new classes. Both Java RTS and WebSphere RT support Java SE 5 or later.

1. Threads, Synchronizations and Memory Management Models
Because Java threads have a one-to-one relationship with the underlying OS native threads (e.g. Java threads are implemented using the native POSIX threads on Solaris and stock Linux), most of Java threads' functions are delegated to OS native threads as we discussed in Part 3.

1.1 Regular Java Threads vs Java Real-Time Threads
Regular Java threads created by class java.lang.Thread (JLT) are not suitable for RTC due to the following factors:

• JLT only has 10 priority levels, which is not enough for COTS ERTC.
• JLT maps to OS non-real-time threads. For example, JLT maps to the SCHED_OTHER scheduling policy on Linux that creates jitter due to its dynamic priority adjustment.
• JLT has the "priority inversion" problem because java.lang.Object.wait() doesn't implement the "priority inheritance" logic.
• JLT doesn't provide priority-based synchronization because java.lang.Object.notify() is not required to wake up the highest-priority thread; and the "unlock" action in the "synchronized" statement is not required to choose the highest-priority thread to be the next lock owner.
• JLT is subject to unpredictable pauses induced by GC because JLT allocates objects in the standard Java heap.

RTSJ provides a new thread type javax.realtime.RealtimeThread (RTT) to fix the above problems:

• RTT must have at least 28 priority levels. For example Java RTS has a default of 60 on Solaris and 49 on stock Linux;
• RTT maps to OS level real-time threads. RTT maps to the SCHED_FIFO scheduling policy on stock Linux which provides fixed-priority scheduling based on FIFO queues (another stock Linux RT scheduling policy "SCHED_RR" doesn't meet RTT requirements due to its round-robin feature). 
• RTSJ requires java.lang.Object.wait() implement the "priority inheritance" logic. On Linux RT and Solaris, it can delegate to the underlying POSIX threading services.
• RTSJ requires java.lang.Object.notify() wake up the highest-priority thread, and the "unlock" action in the "synchronized" statement choose the highest-priority thread to be the next lock owner. On Linux RT and Solaris, they can again delegate to the underlying POSIX threading services.
• RTSJ provides a subclass of RTT called javax.realtime.NoHeapRealtimeThread (NHRT) which is protected from GC-induced jitter (see next section for details). NHRT is intended for HRT.

Because RTT exposes many of the underlying OS native threading functions to Java, RTSJ can compete squarely with C/C++ in the above areas.

1.2 New Memory Management Models
Because NHRT is not affected by GC pauses, it is allowed neither to use the standard garbage-collected heap nor to manipulate references to the heap.
Besides the standard heap, RTSJ provides two memory management models that RTT and NHRT can use to allocate memory on a more predicable basis:

• Immortal Memory.
  It is not garbage-collected. Its primary use is to avoid dynamic allocation by statically allocating all the needed memory ahead of time.
  It is like the malloc() / new operator in C/C++ without the corresponding free() / delete operator because once an object is allocated from it, the memory used by the object will never be reclaimed. Any unintended allocation into it is regarded as memory leak and can cause out of memory.
• Scoped Memory. It is not garbage-collected either. It is intended for objects with a known lifetime such as temporary objects created during the processing of a task. It will be entirely reclaimed at the end of the lifetime such as when the task finishes.
  Because many standard and third-party Java libraries create many temporary objects, it is impractical to use immortal memory if RTT or NHRT has to link to many libraries. In this case scoped memory is a better choice.

Although the two new models enable RTSJ to compete with C/C++ equally at the memory allocation area, they are hard to use and very error-prone because objects allocated in the two new models and the standard heap have difference GC characteristics and lifetimes, and assignment rules must be enforced.
So the recommended use of the two models is limited to programs that can't tolerate GC pauses such as in HRT.

1.3 Communication between NHRT and RTT / JLS
Because an NHRT could block on a lock held by an RTT or JLS. While the RTT or JLT holds the lock, GC could preempt it and indirectly preempt the NHRT. If this is not tolerable, you either should void lock sharing or use the following non-blocking queues for resource sharing:

• javax.realtime.WaitFreeReadQueue class for passing objects from RTT / JLT to NHRT.
• javax.realtime.WaitFreeWriteQueue class for passing objects from NHRT to RTT / JLS

Because many Java libraries use internal locking, NHRT linked to Java libraries must avoid indirect GC preemption to ensure HRT behavior.

Because the new JTSJ classes and RT functions above mentioned have a medium-level learning curve for Java SE developers, you are recommended to still use JLT with careful tunings if SRT can meet your business requirements. Those tunings included the OS level tunings as mentioned in Part 3, limiting the number of your application threads, avoiding of resource sharing and GC low-pause tuning as mentioned in Section 2 in this part.

2. Garbage Collection
Although automatic garbage collection provided by JVM greatly eases the memory management in Java compared to C/C++, it is unfortunately another big source of non-determinism. 
Although standard JVM provides different GC algorithms such as serial, parallel and concurrent to allow users to make a trade-off between pause and throughput, all algorithms involve a stop-the-world (STW) pause, in which all application threads except NHRT are stopped so that GC can run without interference. This STW behavior is only acceptable for SRT or loose HRT.

Although RTSJ doesn't define deterministic GC, a RT JVM must provide one in order to support HRT. There are basically two approaches: work-based and time-based.
Both approaches are aimed to minimizing the effect of long pause by doing incremental collection in a GC cycle. Unfortunately neither can provide HRC guarantee.

2.1 Work-Based
Because the standard STW GC behavior blankly taxes all application threads, the work-based approach has each thread do a specific amount of incremental GC work proportional to its allocation amount each time it allocates an object.
However your application is still unpredictable because the GC cost spread-out is often uneven because the allocation is often uneven and the amount of time to do a fixed amount of incremental collection work is variable as shows in Figure 5.1 courtesy of this resource:

Figure 5.1 Risks of Work-Based Garbage Collection 

2.2 Time-Based
It schedules a fixed amount of collection time in each GC cycle.
Although it spreads the GC cost equally through a GC cycle, there is no direct correlation between the allocated collection time and the reclaimed memory, and your application is still unpredictable as shown in Figure 5.2 courtesy of this resource:

Figure 5.2 Time-Based GC and Undesirable Outcomes

2.3 Oracle Java RTS 2's Approach
Java RTS uses a modified work-based approach called "Henriksson's GC" or Real-Time GC (RTGC). This RTGC can be configured to run as one or more RTTs that run at a priority lower than critical threads (NHRT and critical RTT) and higher than non-RT threads (JLS) and maybe non-critical threads as well (non-critical RTT) so that critical threads may preempt the RTGC, and RTGC may preempt non-RT threads and non-critical threads to keep up with the application memory allocation rate. This is shown in Figure 5.3 courtesy of this resource:

Figure 5.3 Henriksson's GC

The initial RTGC priority is lower than non-critical RTT but is boosted to its configurable "maximum priority" higher than non-critical RTT if remaining memory is close to another configurable "memory threshold". These two configurable parameters enable you to tune the balance between non-critical threads' deterministic and memory throughput.

Figure 5.3 shows that RTGC ensures HRT only for critical threads that should competes pretty well with C/C++,  while trying to offer SRT for non-critical threads that usually can't compete with C/C++.

2.4 IBM WebSphere RT 2's Approach
WebSphere RT's Metronome is a time-based deterministic GC. It divides a GC cycle into a series of discrete quanta, approximately 0.5ms but no more than 1ms in length, that are devoted to either GC work or application work.
Even a single GC pause has an upper bound of 1ms, it is not enough because if several quanta were devoted to GC work, the application still experience a longer pause time as we discussed in Part 1. It must also meet another parameter called "utilization" that is the percentage of time quanta allocated to an application in a given window of time continuously sliding over the application's complete run.
Figure 5.4 shows a GC cycle divided into multiple 0.5ms time slices preserving the default 70% utilization over a 10ms windows courtesy of this resource:

Figure 5.4 Metronome GC Sliding window utilization

Compared to Oracle's RTGC, Metronome provides more determinism to JLT and non-critical RTT while RTGC ensures HRT for critical RTT.  Because learning RTT takes time, Metronome is recommended to implement SRT with JLS.

2.5 Standard JVM's Approach

Both Oracle's Hotspot JVM and JRockit provide so called low-pause concurrent GC that only briefly pauses the application and runs concurrently with your application for most of the time.
Hotspot JVM allows you to specify both a target pause time and throughput while JRockit only allows you to specify a target pause time that seems to be inadequate based on our discussion in Part 1.
Such a low-pause concurrent GC can at best provide SRT.

Another very promising SRT implementation is provided by JRockit Real Time, which extends the existing mostly concurrent low-pause GC by ensuring the target pause time and limiting the total pause time within a prescribed windows (unfortunately such a windows can't be configured by users).
JRockit Real Time GC is more deterministic than the regular JRockit concurrent low-pause GC. Oracle claims it is the industry’s fastest JVM with its average response time in microseconds for most well-managed applications and low, single-digit, millisecond response with a five nines reliability guarantee. These numbers are pretty good for SRT even in the financial services industries.
 
Here is the biggest advantage of tuning GC compared to the new RTJS classes in Section 1: it is transparent to Java SE developers. In other words, it doesn't need developers to change their existing programming models or adapting to new RTC models.

Finally in this section it is often hard to pick a selection between SRT  or loose HRT coded using C/C++ and SRT coded using Java along with a deterministic low-pause concurrent GC.


3. Class Loading, Linking, Initializing and Compilation
3.1 Class Loading, Linking, Initializing (LLI)
A standard JVM loads, links and initializes classes on demand; it also unloads classes no longer referenced. Though LLI provides great flexibility on memory and CPU consumptions and the one-time cost can probably be made up if the same class is referenced multiple times, it is still another big source of non-determinism.
For example class loading usually takes at least tens of milliseconds because it usually involves disk IO. The linking and initializing are also very CPU-intensive.

You can eliminate the LLI jitter by pre-doing LLI in the application warm-up phase (it is defined in Part 3) either by calling the standard java.lang.Class.forName() or using RTSJ implementation-specific utilities.
For example, Oracle's Java RTS 2 allows you to either specify a list of classes for pre-loading and / or pre-initializing on the command line at JVM startup or use the Initialization-Time-Compilation (ITC) API during runtime (Both Java RTS and WebSphere RT can automatically generate a list of classes that were loaded or initialized by your application's execution).

Both approaches for eliminating LLI jitter are basically transparent to Java SE developers.

3.2 JIT vs Ahead-of-Time (AOT / ITC)
If you used C/C++ static compiler before, you know compiling (optimization included) is very CPU and memory intensive. So modern JVM JITs initially interpret Java methods and , for only those methods that execute frequently, later compile to native code.
Because the dynamic nature of Java makes much important information only available during runtime, JIT can generate even better code than statically compiled language like C/C++. Recompiling (deoptimization), overriding (virtual) method optimization and escape analysis are just three examples.
So if JIT can carefully balance the compiling time and optimization aggressiveness, the one-time compiling cost will be made up by the multiple later executions of the native code and the average execution time will be equal to or even shorter than the corresponding C/C++ program.

However because the compiling time is up to JVM and there is execution time variation between interpreted code and native code, the dynamic JIT is another big source of non-determinism.
The only solution to eliminate JIT jitter is to use some kind of static compiling such as AOT in WebSphere RT and ITC in Java RTS.

However AOT/ITC also has some disadvantages.
Firstly, the platform portability is comprised by AOT. ITC is a bit better due to its initialization-time compiling instead of AOT's development-time compiling.
Secondly AOT/ITC-compiled code, though faster than interpreted code, can be much slower than JIT-compiled or C/C++ code because AOT/ITC can only make few conservative optimizations with little information available at hand.

The debate of  JIT vs AOT/ITC is still involving. Readers are recommenced to this resource to gain more insight.  I personally recommend JIT because you can still achieve SRT and even loose HRT with it. With such fine tunings as warm-up and background JIT compiling, your Java programs can compete toe-to-toe with C/C++ programs.

4. Timers and Clocks
If RTC needs nanosecond resolution and your system can only provide milliseconds at beat, it will cause jitter.
RTSJ provides new classes javax.realtime.Clock and javax.realtime.HighResolutionTime to expose high-resolution clocks in underlying hardware and OS as discussed in Part 2 and 3.
You should also use the following standard Java methods if you need nanosecond resolution:
  java.lang.Object.wait(long timeout, int nanos); 
  java.lang.Thread .sleep(long millis, int nanos);

You shouldn't use standard Java's java.util.Date or java.lang.System.currentTimeMillis() due to their low resolution and synchronization with the world clock (this synchronization looks like jitter when this world clock is updated).

In summary RTSJ can compete toe-to-toe with C/C++ in most areas for building COTS ERTC applications. The only concern is the learning curve of its functions extensions to standard Java. However the curve is not steep and often necessary for RTC.
If you had bias against using Java to build RTC systems, hopefully you will have a second thought after this discussion.

Commercial-Off-The-Shell Enterprise Real Time Computing (COTS ERTC) -- Part 3: Operating System (OS) Requirements

Since OS is between hardware and Java program language, it provides your COTS ERTC applications with more enabling functions. The latency and jitter requirements to OS are much higher than to hardware.

Again such general purpose OS as Windows NT, Linux and Solaris are designed primarily for high throughput at the cost of poor latency. Generally there are two trends for OS to support COTS ERTC.
One is to stick with the general purpose OS with the help of fine tunings through which SRT is usually achievable. Those OS's stick with the high-throughput design goal. Windows NT belongs to this category () (Although there are indeed quite many commercial efforts to extending NT for RTC functions, they are appropriate and hence doesn't belong to COTS ERTC).

The other trend is to add RTC function to the OS so that both throughput and RTC workloads can be handled and even HRT can also be implemented at the cost of slight lower throughput.
More and more RTC features have been added to the Linux mainline kernel since version 2.6 through a RT patch (We will hereafter use "Stock Linux" for general purpose Linux, "Linux RT" for Linux with the RT patch). Red Hat Enterprise MGR and SUSE Linux Enterprise Real Time (SLERT) are representative.
Oracle / Sun also has significant RTC features in its Solaris itself for long.
Actually both Linux and Solaris are POSIX compliant including the real-time and thread extensions. So they both belong to this category (There are also many other efforts to extending stock Linux for RTC functions such as RTLinux, RTAI. They are more or less like a dual-kernel approach. Because they are appropriate and never got into the mainline Linux kernel, they don't belong to COTS ERTC) .

1. Preemptable Kernel
Multi-threaded programs are known to programmers for long. However OS fully preempting user-space threads doesn't necessarily mean its kernel is also fully preemptable. Actually different OS's provide different degrees of preemption. Obviously low-degree preemption means high latency and jitter.

Figure 2 in part 2 shows the OS scheduler takes "interval response time" to preempt an interrupted thread(usually a low-priority thread) with a preempting thread (usually a high-priority thread). The shorter the interval response time, the more preemptable.

Whenever the processor receives an interrupt, it calls an interrupt handler, a.k.a. an interrupt service routine (ISR) to service the interrupt.
Preemption latency is the time needed for the scheduler to determine which thread should run and the time for the new thread to be dispatched.
Context switch is the kernel saves the state of the interrupted thread or process, loads the context for the preempting thread or process, and begins execution.
I will focus on ISR and preemption latency because different OS's employ different strategies.

1.1 ISR
On Linux RT and Windows NT, ISR is divided into two parts: the First-Level Interrupt Handler (FLIH) (or Upper Half on Linux RT) and the Second-Level Interrupt Handler (SLIH) (or Lower Half or Bottom Half on Linux RT; Deferred Procedure Call (DPC) on Windows NT).
FLIH quickly services the interrupt or records platform-specific critical information which is only available at the time of the interrupt, and schedules the execution of SLIH for further long-lived interrupt handling.
Because FLIH typically masks interrupts at the same or lower level until it completes, it affects preemption and causes jitter. So to reduce jitter and to reduce the potential for losing data from masked interrupts, OS should minimize the execution time of FLIH, moving as much as possible to SLIH.

SLIH asynchronously completes long interrupt processing tasks in a kernel thread scheduled by FLIH. Because it is implemented in a thread, the user can assign a priority to it and the scheduler can dispatch it along with other threads.
For example, if your RT application thread has a higher priority than SLIH, only FLIH interrupts your RT application thread and SLIH will not run until your RT application thread has done.
Because the ISR in Figure 2 only effectively represents FLIH, the whole interval response time was cut short.

On Solaris ISR is a whole and implemented in a kernel thread. Because such a thread has higher priority than all non-ISR threads including RT ones, it makes kernel less preemptable and causes much larger jitter to your RT application threads than the previous approach.

Windows NT has additional jitter caused by DPCs being scheduled into a FIFO queue. So if your high-priority DPC is put behind a low-priority one, the high-priority DPC can't be executed until its prior low-priority one is done.

1.2 Preemption Latency
Traditionally when a low-priority thread calls a kernel function through a system call, it can't be preempted even by a high-priority thread until the system call returns. This is again primarily due to high-throughput consideration (The more interrupts, the more overhead and the lower throughput).
This is the situation for stock Linux Kernel 2.5 or prior that has many lengthy kernel code paths protected by spin locks or even  by so called Bigger Kernel Lock (BKL is basically a kernel-wide or global lock).

Changing BKL to localized spin locks is the first step toward preemption. But a spin lock is typically not preemptable because if it is preempted, the preempting thread can also try to spin-lock the same resource, which causes deadlock.

To make kernel more preemptable is to break down a lengthy code path into a number of shorter code paths, between which preemption points are created which is stock Linux kernel 2.6 or later has enabled. SRT can be achieved at best in this case.

The extreme approach is to convert all spin locks to sleepy mutexes so that your kernel code is preemptable at any point which is what Linux RT has enabled. HRT needs this capability.

However because Linux should be able to handle both throughput and RTC workloads, a better and practical approach may be to use adaptable locks which are spin locks for short-running code paths and are mutexes for long-running code paths based on statistics.
Actually SLERT 11 provides such adaptable locks.

Windows NT has been fully preemptable from the very beginning.

2. Priority-Based Scheduling
The scheduler in a general purpose OS is designed to maximize overall throughput and to assure fairness for all time-share threads / processes. To provide equitable behavior and ensure all time-share threads / processes can eventually be executed, the scheduler adjusts thread priorities dynamically so that the priorities for resource-intensive threads are lowered automatically while the priorities for IO-intensive threads are boosted automatically. In other words, even you initially assigned a high priority level to a time-share thread, it will not starve other threads.

This is not desirable for RT threads which always need to run before any low-priority thread in order to minimize latency at the cost of lower throughput of other threads.
Besides the traditional time-slice and dynamic-priority threads, Windows NT, Solaris and stock Linux all provide RT threads which have fixed priorities and always run before TS and other low-priority threads.
In other words, the scheduler will not adjust those RT threads' priority and they will not be preempted by TS or other lower-priority threads unless they wait, sleep or yield.

Both stock Linux and Solaris provide two scheduling policies for RT thread.  One is Round-Robin which is similar to the TS thread scheduling; the other is FIFO where the prior RT thread runs to complete before the late RT thread with the same priority level.

The priority level range for RT threads can't to be too small. Otherwise your RT thread scheduling flexibility will be severely constrained.
Windows NT includes 32 priority levels of which 16 are reserved for the operating system and
real-time processes. This range is really too tight.

Stock Linux RT priority class provides 99 fixed priority levels ranging from 1 to 99 (0 is left for non-RT threads).
The following RT thread priority mapping table was extracted from Red Hat Enterprise MRG tuning guide:

Priority	Threads	Description
1	Low priority kernel threads	Priority 1 is usually reserved for those tasks that need to be just above SCHED_OTHER
2 - 69	Available for use	Range used for typical application priorities
70 - 79	Soft IRQs
80	NFS	RPC, Locking and Authentication threads for NFS
81 - 89	Hard IRQs	Dedicated interrupt processing threads for each IRQ in the system
90 - 98	Available for use	For use only by very high priority application threads
99	Watchdogs and migration	System threads that must run at the highest priority

Although an important feature for RT thread scheduling is to schedule your RT application threads to be higher than kernel threads, it can possibly cause the system to hang and other unpredictable behavior such as blocked network trafic and blocked swapping if crucial kernel threads are prevented from running as needed (now you should have more feeling how your RT thread is scheduled at the cost of lower overall system throughput).
So if your RT application thread is higher than kernel threads, make sure they don't runaway and you also should allocate some time for kernel threads.
For example, your RT thread doesn't run too long or it runs periodically based on a RT timer or it is driven by external periodic RT events or you have multiple CPUs at least one of which is dedicated to kernel threads.

3. Priority Inheritance
Priority Inversion occurs when a high-priority thread blocks on a resource that is held by a low-priority thread, and a medium-priority thread preempts the low-priority thread and runs before the high-priority thread, which causes jitter for the high-priority thread.
Priority inheritance fixes the priority inversion problem by temporarily enabling the low-priority thread to inherit the priority of high-priority thread so that the formerly low-priority thread can continue to run to finish without being preempted by the medium-priority thread. The inheriting thread restores its original low priority when it has released the lock.

Both Solaris and Linux RT support priority inheritance. Unfortunately Windows NT doesn't support it.
If possible, try to avoid a high-priority thread from sharing the same resource as a low-priority thread. Obviously this appears to be more important to Windows NT.

4. High-Resolution Timers
Section 1.5 in part 2 mentioned the need for high-resolution timers which are backed by high-resolution clocks on most modern hardware. The OS just takes advantage of hardware timers by providing you with different system calls for high-resolution timers besides the traditional system call for regular timers.

For example both Solaris and Linux support system call "timer_create" and "timer_settime" with clock type "CLOCK_HIGHRES" on Solaris or CLOCK_REALTIME / CLOCK_MONOTONIC on Linux (you need to enable a kernel parameter "CONFIG_HIGH_RES_TIMERS" available on 2.6.21 and later on X86) to access high-resolution timers.

5. CPU Shielding
Windows NT, Solaris and stock Linux all support CPU shielding which allows you to bind different processors / cores to different interrupts and threads including both kernel and user space ones. The bound CPU is shielded from unbound interrupts and threads.

For example, you bind your high-priority application thread to one CPU while other CPUs take care of other threads including kernel thread, and interrupts including NMI and SMI so that you are confident that your high-priority application thread has low latency and is very predicable.
This means more to Solaris because its ISR is implemented in a thread whose priority is higher than any non-ISR thread including your RT application thread.

6. Others
6.1 Memory Pinning
Windows NT, Solaris and stock Linux all allow you to pin your high-priority thread to physical memory to avoid being swapped to high-latency disks.
Due to the mechanism in disks, disk IO access latency is in milli-seconds, which is at least one order of magnitude slower than memory access. So OS swapping is a major contributor to latency.

6.2 Early Binding
The late binding of dynamic libraries in OS can induce unpredictable jitter to your RT application thread. To avoid jitter, Both Linux and Solaris provides for early binding of dynamic libraries through an environment vairable LD_BIND_NOW.
Windows NT doesn't seems to support such early binding. To counter-attach this, you can warm up (it is hereafter either the program's start-up phase or an initialization phase before the time-critical execution) your application before asking it to execute time-critical code.

6.3 Locking Improvement
Stock Linux use so called "Futex" to avoid system calls for un-contended locks. Solaris uses a similar mechanism called "adaptive lock".

7 COTS ERTC scenarios with OS
Even an OS provides both through-put and RTC functions, the RTC functions are at the cost of slight throughput degradation. Actually many observations show only a minority of workloads truly need tight HRT. Accordingly users should always first try OS without the RTC functions enabled.

For example on Windows NT and stock Linux, if your low latency requirements can be met through such tunings as using RT threads, CPU shielding, memory pinning, priority inversion avoidance, HR timers, application warm-up, and early bind and preemption kernel configuration on stock Linux, don't try Linux RT. Actually many SRT can be achieved using Windows NT or stock Linux

If you need high predictability or tight HRT, you have to use Linux RT such as MRG and SLERT, or Solaris.

Commercial-Off-The-Shell Enterprise Real Time Computing (COTS ERTC) -- Part 2: Hardware requirements

Because hardware is at the bottom of a COTS ERTC application stack, we usually focus on throughput instead of latency in order to reduce higher level latency based on our discussion in Section 1.2 in part 1. But we still want to reduce hardware jitter.
Because hardware contributes the least to your COTS ERTC application's latency and jitter compared to other parts, you usually put less effort into hardware for your application jitter optimization.

In this part, I will talk about the latency incurred by computer hardware. I will also talk about how to reduce latency and jitters using such high-performance co-processors as GPU and FPGA.

1. Latency by Computer Hardware
1.1 Interrupt Latency
System timer, power management and such IOs as keyboard, mouse, disk and NIC use interrupts to notify the processor of the need for attention.
Software can also generate an interrupt to notify the processor of the need for a change in execution.

All modern hardware and OS use interrupts to implement thread preemption as shown in Figure 2.

                                            Figure 2 Thread Preemption Through Interrupt

Whenever there is a higher level interrupt, the OS scheduler preempts the low priority thread with the high priority thread. This preemption takes time i.e. "Interval Response Time" in Figure 2. This white paper shows a typical 25us interval response time on modern hardware.
RTC needs to bound this "Interval Response Time" which consists of several components as shown in Figure 2. Only "Interrupt Latency" relates to hardware. All others relate to device driver and OS and will be covered in the next topic.

Interrupt latency is the time required for the interrupt to propagate through the hardware from its source to the processor's interrupt pin.Because it is hardware line level latency, it is usually negligible compared to other components on modern hardware platforms.

Because most interrupts come from IO devices which usually have small time-sensitive data buffer, Interrupts have the highest scheduling priorities meaning they can interrupt any application and OS kernel threads including your RT ones. This has two implications.On one hand for a high priority event such as market data feed from an external source, its interrupt of all low priority threads and switching to the corresponding RT thread is really what you want.
On the other hand your really don't want your RT thread to be interrupted by such unnecessary IOs as keyboard or mouse. But your RT thread was still interrupted by such IOs.
However if the interrupt latency (more precisely, the interval response time) is bounded, your RT application thread behavior in both scenarios is still predicable.

As mentioned in the previous topic, COTS processors are usually designed for high throughput at the cost of high interrupt latency.
However this "high interrupt latency" is only significant for tight HRT. For SRT,
it is negligible compared to larger latencies by other parts in the stack and also thanks to the constant performance improvement in modern computer hardware.

1.2 System Management Interrupt (SMI)
Only X86 processors have those SMI's. When they happens, they suspend the CPU's normal execution and switch it into system management mode (SMM).
Because they usually run inside BIOS firmware, they are transparent to OS and you can't intercept them.  They are the highest priority interrupt in the system even higher than the NMI. Because they can last for hundreds of microseconds, they can cause unacceptable jitter for RTC, especially for HRT.

However they are not that bad if you OS supports ACPI because ACPI took the ownership of power management from SMM. Both Windows and Linux support ACPI (so you can configure your RT application thread higher than the OS ACPI thread).

This resource also lists other measures to mitigate SMI's impact.

IBM also optimized its LS21(Model 7971) and HS21 XM (Model 7995) to reduce SMI jitter for non-fatal SMI by deferring non-fatal SMI's functions to low-priority OS threads (When such a fatal SMI as a memory or chipset error happens, its function can't be deferred because the function has to be used to fix the error).

1.3 DMA Bus Mastering
This happens on the currently pervasive PCI bus. When a device uses the DMA, other devices that want to use the DMA have to wait until the previous device is done, which can cause many micro-seconds jitter based on this resource.

1.4 Max Performance vs Max Power Saving
ACPI can put CPU and other devices into some low-power-consumption or even power-off state after a period of inactivity, it will cause jitter when they have to be woken up.

ACPI support at least two extreme power schemes.
One is "Max Performance" which keeps CPU and other devices active all the time.
The other is "Max Power Saving" which power off CPU and other devices.
The "Max Performance" is preferred for RTC.

1.5 High Resolution Timer
The default system timer usually has a resolution of 10ms on most platforms. Although you can usually lower it to 1ms by configuring your OS, a high-resolution system timer can result in too many interrupt overheads which will severally lower throughput.

However even 1ms is still too coarse for RTC because RTC needs high resolution timer for its periodic or one-time task accurate scheduling and nano-sleep function (after all RTC applications have time or deadline constraints). At least microsecond resolution is required for RTC to keep jitter lower.
A hardware-based timer is needed also because it can generate interrupts as needed even you need a nano-second resolution e.g. the "dynamic tick" or even tickless implementation on Linux RT.

UltraSPARC systems can provide nanosecond timers while most modern X86 systems can only provide as good as microsecond times.

1.6 Cache
Because processors are orders of magnitude faster than the main memory e.g. modern processors have nanosecond execution time per instruction while main memory has microsecond access latency, caches are used to bridge the access latency gap. However this unavoidably creates jitter.

For example if your RTC application can completely fit into L1 cache or L2 cache, its latency will be very low and predicable. Otherwise the main memory access will cause jitter. But this is probably still acceptable to SRT or loose HRT based on your business rules.

1.7 Instruction Level Parallelism
Modern processors use such instruction level parallelism as out-of-order execution, pipeline and supercalar to improve your thread (CPU) throughput. However this again unavoidably create unpredictable temporal behavior for your thread instructions. 
But because they are at instruction level, they are probably still acceptable to SRT or loose HRT based on your business rules.

1.8  Multi-Processor / Core
Because higher processor clock rate leads to lower latency, the traditional approach of increasing processor performance is increasing clock rate based on Moore's Law. However such an approach eventually ran into power consumption and expensive cooling issues.
Multi-processor systems are one solution that can definitely improve throughput. But the interconnect between processors unavoidably introduce additional latency and jitter.
Multi-core processors may be a even better solution for RTC that can achieve both high throughput and low latency thanks to the shared high-speed bus among cores and the shared memory model.

A very important application of multi-processor / core in RTC is the so called "CPU shielding" (it also has few other names such as cpu binding or interrupt binding or thread binding or fine-grained processor control). "CPU shielding" is implemented in OS.

Here are some examples.
You can bind low priority interrupts to one CPU and dedicate another CPU to your RTC thread.
If you have multiple RTC threads that need to run simultaneously, you also have to have CPUs.
In case of Java, you need to bind at least one CPU to the concurrent GC or RTGC or the background JIT compiler so that it can truly run concurrently with your RT threads.

1.9 NUMA
The multiple CPUs mentioned in 1.8 usually have equal access to the shared main memory. This unfortunately also causes contentions when several processors attempt to address the same memory besides the benefits.

NUMA attempts to address this problem by providing separate memory for each processor. For problems involving spread data (common for servers and similar applications), NUMA can improve the performance over a single shared memory by a factor of roughly the number of processors (or separate memory banks).
However if not all data ends up confined to a single task, which means that more than one processor may require the same data, NUMA has to move data between memory banks, which causes jitter.


Both current X86 and UltraSPARC processors can provide good functions in the above areas. They can be used as the underlying platforms for COTS ERTC applications.

2. Co-Processors
The shift to multi-processor / core processors forces application developers to adopt a parallel programming model to exploit CPU performance, which proves to be very error prone.
Traditionally HPC uses clusters consisting of COTS multi-core servers to run data and compute intensive applications. In order to cut down such complex computation to RTC level, hundreds, even thousands of COTS multi-core servers along with high-bandwidth interconnect need to be deployed, which not only creates maintenance headache, but also consumes quite a lot of power. Even the interconnect usually has low latency, it still incurs latency.

CPU-based systems augmented with hardware accelerators as co-processors are emerging as a even better solution to the Moore's Law dilemma. This has opened up opportunities for accelerators like Graphics Processing Units (GPUs), FPGAs, and other accelerator technologies to advance HPC to previously unattainable RTC levels.


2.1 General-Purpose Computing on Graphics Processing Units (GPGPU)
Because traditional CPU design focuses on general purpose (both high volume and low volume; both management task and ALU processing etc), the number of cores and the vector size in SIMD are both small.
Because GPU's specialization in 2D or 3D graphics rendering acceleration using its highly pipeline parallel structures, it can have hundreds of  processor cores each of which can handle hundreds of independent threads. Its SIMD's vector is much longer than CPU's. Also the internal interconnect among cores has much higher bandwidth than the external interconnect in traditional clusters.

GPGPU is the technique of using a GPU to perform computation in applications traditionally handled by the CPU, which is made possible by the addition of programmable pipeline stages - shaders and higher precision arithmetic to the rendering pipelines, which allows software developers to use stream processing on non-graphics data.

A modern GPGPU itself is a cluster of hundreds of cores capable of handling tens of thousands of threads, which can be hundreds of times faster than a transitional cluster made of hundreds of processors.

OpenCL is an GPU programming framework that is supposed by all major GPU vendors. It supports both task and data level parallelism. For data level parallelism, users only need to partition data properly and are not responsible to manage threads, which is much less error-prone compared to the multi-core parallel programming.

GPU interfaces with a computer using PCIe, which can cause latency problem for high-volume and data intensive computing. This is why GPGPU recommend to use more threads to hide latency.

2.2 Field-programmable gate array (FPGA)
FPGA is an integrated circuit designed to be configured by the customer or designer after manufacturing — hence "field-programmable".
Because it uses highly paralleled hardware to implement your application logic traditionally implemented in software along with CPU or GPU,  it provides line speed latency, which is even shorter than GPU.
The FPGA architecture provides the flexibility to create a massive array of application-specific ALUs that enable both instruction and data-level parallelism.
Because data flows between operators, there are no inefficiencies like processor cache misses; FPGA data can be streamed between operators.

FPGA interfaces with a computer using either PCIe or processor bus such as Intel's FSB or QPI or AMD's HyperTransport. The later option makes FPGA just like another processor, which doesn't have cache coherency issue and also enjoys high bandwidth and low latency.

Although the C-to-FGPA compilation toolkit from Implulse enables a developer to use C instead of HDL to design application logic, the learning curve is still high. The developer still needs to know some basic hardware design knowledge and also compilation parallel skills such as loop unrolling and instruction pipeline.

Commercial-Off-The-Shell Enterprise Real Time Computing (COTS ERTC) -- Part 1: Introduction

This five-part series covers how to use real time computing (RTC) to process enterprise workloads using commercial off the shell (COTS) hardware and software based on my experience and researches.
By "enterprise workloads" and "commercial off the shell", I mean the discussion is not about the traditional RTC tailored to a few specific application tasks for bare metal or embedded devices. So you know how COTS ERTC came out.

An enterprise application stack typically consists of, counting from the bottom, the underlying hardware (micro-processors and / or co-processors) and OS platforms,  the network,  the program language (I will focus on Java) and the application itself.
Here are the five parts for this series:

• Part 1: Introduction
• Part 2: Hardware (Microprocessor and Coprocessor) requirements
• Part 3: Operating System (OS) requirements 
• Part 4: Network requirements 
• Part 5: Java requirements

In this introduction, I will clarify the following importance concepts: throughput vs latency; hard real time (HRT) vs soft real time (SRT); proprietary vs COTS.

1. Throughput and Latency
An application's performance can have quite a few metrics. However throughput and latency (or response time) are the two most important aspects for most enterprise workloads and for most developers.
Throughput is the amount of useful work accomplished over a period of time.
Latency is the time needed to accomplish some amount of useful work.

In my above definitions, I use such fuzzy words as "a period" and "some amount". You should quantify them based on your business logic. They are mathematically inverse to each other.

By "useful work", it means there are also overheads that can't be avoided completely during the period. Usually the more interruptions such as sending users feedback during the period, the more overheads. So in order to have a better throughput, the application needs to have few interruptions or longer latency.
On the other hard, in order to have lower latency, you have to process less amount of work before sending out a response, which means lower throughput. Be careful not to lower the amount of work too much otherwise you may well end up nothing but too frequently sending out responses, which is practically not doing any useful work.
The relationship between throughput and latency can also be shown in Figure 1 where you can see lower latency (more overhead) means lower throughput.

Figure 1 useful work and overhead over a period

1.1 Is it possible to improve both throughput and lower latency at the same time?
It depends.
It is not possible if you don't put more resources into your application.
It is very achievable if you put more resources into your application such as upgrading your hardware or deploying more hardware or employing more efficient algorithms.

Here is a theoretical example.
Suppose your application needs to process 1,000,000 transactions. The user needs to get a feedback after some number of transactions have been processed. Each transaction processing takes 1ms and each feedback incurs 10ms overhead.
You are asked to calculate the application's throughput over a period of 1s.
Basically you have two configurations.
(1)The user doesn't need low latency or fast response
This needs your application to send out feedback every a larger number of transactions. If the batch number is 100, you can get the following result by simple calculation:
  throughput: 909 transactions
  latency: 100ms

(2)The user needs to have low latency or fast response.
This needs your application to send out feedback every a smaller number of transactions. If the batch number is 10, you can get the following result by simple calculation:
  throughput: 500 transactions
  latency: 10ms

From the above numbers, you know lower latency means lower throughput and this becomes more obvious when the per-feedback overhead becomes larger and larger.
How do you keep the 10ms latency in (2) while still enjoying the 909 transaction throughput in (1)? You have to make your per-transaction processing more efficient and / or lower the per-feedback overhead.
Suppose you can't lower the per-feedback overhead but you do know a better algorithm or be able to upgrade your hardware to cut down the per-transaction processing time. By simple calculation, you know you per-transaction processing time has to be 0.55ms or better instead of the original 1ms.

Here is a real example about JVM's garbage collection (GC).
In this case your application threads are doing useful work while the GC just incurs overhead. The typical Stop-The-World (STW) GC causes application pause time which is the main contributor to latency. The more memory to collect, the more pause time, the lower throughput.
In order to reduce pause time, the GC has to collect memory more frequently or concurrently, which unfortunately incurs more overheads such as thread context switching and bookkeeping.
If you want very low pause time, the GC may have to collect memory way too frequently, which is practically equal to STW.


In order to have both good throughput and latency, modern JVMs employ more advanced GC algorithms such as parallel GC, concurrent GC and dynamic adjustment based on adaptive algorithms and also require multi-core processors.

1.2 End user level latency relies on underlying throughput
In the above first example, if the user defines latency as the time taken to process all 1,000,000 transactions, what strategy will the underlying system employ to get a better latency?
Obviously the underlying system can process 1,000,000 transaction faster using throughput than latency (the throughput and latency here are of the underlying system, not the end user level).

In the above second example, the latency consists of your application thread time plus the GC pause time in the application window. Let's focus on the GC pause time. Obviously in order to minimize the GC pause time, the GC should use throughput collectors such as STW parallel collector instead of latency collectors such as concurrent Mark Sweep collector.

This last point is very important because it help you understand the following points:

• Improving infrastructure level throughput can not only improve overall system throughput but also provide users with low latency;
• Higher level low latency doesn't means the underlying systems must also provide low latency. This is why people focus more on High Performance Computing (HPC) than on RTC (see Section 3 for details);
• You should also not be surprised when you hear a vendor boasting of his product capable of low latency and high throughput.

2. Hard Real Time (HRT) and Soft Real Time (SRT)
The widely recognized RTC definition comes from Donald Gillies in RTC FAQ: 

"A real-time system is one in which the correctness of the computations not 

only depends upon the logical correctness of the computation but also upon 

the time at which the result is produced. If the timing constraints of the  

system are not met, system failure is said to have occurred."

In other words, RTC must be deterministic to guarantee it performs within the required time frame or deadline. It is not about throughput, it is about latency. It is not about how fast the response is, it is about its latency being predicable.
No system can provide absolutely constant latency. The variance in latency is called Jitter.

Practically RTC can be classified as either HRT or SRT:
An HRT system must meet all its deadlines or its jitter is always bounded; otherwise it has catastrophic consequences.
An SRT system can still function correctly if it misses its deadlines occasionally or its jitter sometimes overshoot the boundary. The occasional misses usually happen in worst case scenarios.

2.1 Predictability or determinism is subjective and driven by your business requirements
It really depends on your business requirements how to setup a boundary, how tight the boundary is and whether the boundary can be broken occasionally.

For example, suppose your mean latency is 10ms and the jitter is 5ms. If you boundary from 5ms to 15ms is acceptable, you get an HRT system.
However if the above boundary is not acceptable and a new tighter boundary such as from 8ms to 12ms is required, you can still get an SRT system if your business really don't need HRT and you can predict the jitter only occasionally overshoots the boundary.

2.2 It still matters how fast your response is
Although RTC is not about how fast the response is, RTC systems usually have low latency and are very fast. Actually lower latency also makes your system more predicable because the lower latency usually means less jitter.

For example, if the options pricing in financial services takes several hours, even days, nobody will think it is real-time pricing and it has to be used as offline pricing.
Also probably nobody in the financial services industry would think a 1s latency is real-time and wants to use it in his/her high frequency trading.
However on the other hand, very low latency is much harder and more expensive to achieve or even impossible without using proprietary hardware and software.

Hereafter we will use "tight HRT" to mean HRT with very low latency requirement and "loose HRT" to mean HRT with lenient latency requirement.
Although they are again very fuzzy and depend on your business requirements, they will not affect our discussion.

2.3 RTC is not for everyone
Based on the above analysis in section 1, we know RTC achieves its low latency at the cost of poor throughput or requires more resources to achieve both lower latency and high throughput. So RTC is just not for everyone and should be targeted at those applications that need it in your enterprise. Also if SRT can meet your needs, don't use HRT because HRT has much higher requirements on all parts in the application stack than SRT does.

2.4 RTC is still strange to most developers.
Because RTC can trace its root from embedded devices in the tel-comm, auto and other control industries, most enterprise developers are not familiar with it.
Another reason is most hardware and OS platforms and application designs deployed in enterprises focus on throughput. Their algorithms are optimized for average cases instead of the worst cases.  But HRT needs to reduce jitter in wost cases so that a predicable boundary can be defined.
Recalling 1.2, you should further understand why HRT is much harder to implement than SRT.

3. Proprietary and COTS
Most developers heard of RTC from such RTOS as QNX and VxWorks. They are only used in embedded systems which have little memory, limited external connection, are tailored to a few specific tasks, and are limited to a small user base and fewer vendors. So they are proprietary niche products not good for enterprise applications.
But those traditional RTC systems really shine and are unbeatable when there is a need for very predicable timing behavior or HRT. Actually theoretically only HRT is RTC.

In the past several decades, both hardware and software have experienced rapid advancements in both functionality and performance, and many have been commoditized. Such COTS systems as X86 microprocessors, GPGPU / FPGA, Linux, Windows, Java and C# have become more powerful both on throughput and latency while still being cheap and easy to use.
Unfortunately those general purpose COTS products are traditionally designed with throughput in mind. However due to their huge deployments, large user base and wide industry support, RTC on top of COTS or COTS ERTC is very attractive.
The most important signature of a COTS ERTC system is RTC functions are added without changing existing COTS system's functions so that a COTS ERTC system can support both traditional and RTC enterprise workloads.

A very bright spot in COTS ERTC is to implement an SRT system. Compared to traditional proprietary RTC or HRT, It is easy to build and can handle most of ERTC workloads while still maintaining good throughput. Such an SRT system is so appealing that it should be the first choice for an enterprise.

Another bright spot is High Performance Computing (HPC) is often employed to make high-volume computing tasks qualified as RTC based on COTS ERTC clusters or GPU/FGPA (see the above Section 1.2). 

Using traditional single processor / core, such complex tasks as options valuation and risk analytics in financial services often take many hours even days. HPC can cut down the computation time to sub-second or even less on average, which basically qualify them as SRT.

When vendors say they can provide financial services industries with real-time products, they really usually mean SRT based on COTS ERTC instead of HRT.
Here are more examples:

• "best execution" and transparency regulated in the US's RegNMS and European's MiFID.
  RTC can definitely reduce inconsistent trading times that can result in server fines.
• FTSE 100 Index which is calculated in real time and published every sub-second. (Remember not long ago it was 15s, which is not RTC due to long latency)
• Reuters Data Feed Direct (RDF-D) is an ultra-low latency real-time feed handler that has an average latency (so SRT) of less than one millisecond during periods of high traffic.
• Algorithmic Trading, especially High Frequency Trading (HFT) needs to execute a trade in low milliseconds or even sub-millisecond in order to beat competitors

Because RTC achieves low latency at the cost of throughput, it is very import for those COTS ERTC to keep the original functionality while providing additional RTC. After all RTC is not for everyone and an enterprise probably needs both RTC and non-RTC functionality to co-exist.
Fortunately such COTS ERTC building components are commercially available.

4. Prioritized Approach
The overall application latency is the cumulative delays of all parts in the application stack. If any part creates jitter, your application will experience jitter.
So in order to develop a COTS ERTC system, all parts in the application stack should ideally support RTC. However you should prioritize latency and jitter analyses.
For example if your Java GC has a 500ms jitter while the OS scheduling has a much smaller 5µs jitter, the GC jitter tuning should take precedence because any scheduling jitter improvement is negligible.

Since the bottom parts usually have lower latency and create less jitter than the top ones, the top parts usually have higher requirements on latency and jitter than the bottom parts, especially for HRT.
For example if you just need SRT, you can directly tackle your Java and application logic without putting too much effort on modern fast hardware or OS.

The following parts in this series will analyze latency and jitters for all parts in the application stack and find ways to reduce them.