TimesTen -- an Oracle Database Saviour for Performance-Critical Applications?

Performance-critical applications such as those used in trading and telecoms often need low latency in sub-milliseconds or even microseconds and high throughput.
If such an application uses a disk-based database for persistence (or the system of record), you probably will have to miss your low latency requirement because disk-based accesses incur latency in many milliseconds and often seconds.

In order to achieve low latency and high throughput, there should be at least 3 requirements to be met:

1. Uses memory-based accesses. The access rate for the commonly wildly used computer DRAM is in microseconds;
2. Collocates your application logic with your data. The traditional multi-tiered approaches incurs network IO latency.
3. A fast and predictable JIT compiler if you use Java. Oracle's JRockit is a good choice.

I know 2 products which can meet the above requirements and worth a try.
One is to adopt Gigaspaces' space-based programming paradigm.
The other is to employ an in-memory database (IMDB) such as Oracle's TimesTen along with your traditional database such as Oracle.

I will focus on TimesTen here and compare it with Gigaspaces.
As mentioned above, using Oracle database alone in a performance-critical application is not practical. Accordingly Oracle just doesn't have the same popularity in such applications as in others.
Fortunately if we use Oracle along with its cache companion - TimesTen, it is a very attractive option.

TimesTen is an in-memory relational database from Oracle. Because it caches all data in memory and has no disk access, TimesTen can achieve breakthrough performance such as 5-microsecond reads and 15-microsecond updates based on its white paper "Using Oracle In-Memory Database Cache to Accelerate the Oracle Database".
TimesTen is targeted to run in your application tier, close to applications, and optionally in process with applications. A TimesTen database may be used as the database of record, and/or as a cache to an
Oracle database.

As a cache to Oracle database, TimesTen can be embedded into your application tier and only cache the performance-critical subset of your Oracle database while still leaving your Oracle database for non-performance-critical applications. This architecture is shown in Figure 3 in the above white paper:

In the above figure, you can only cache the subset of an Oracle database that needs real-time processing such as stock trading data. The cached data can be either read-only, read-mostly or read-write. In case of data modification, the "Cache Agent" will take care of the data synchronization between TimeTen and Oracle.
You can also scale out your application tier horizontally by deploying several TimeTimes together to form a so called cache grid.

Here are my TimesTen comparisons to Gigaspaces.
Commons to Both:

• cache data along with your application logic;
• supports multiple topologies such as partitioned, replicated or both;
• supports ACID properties often provided by traditional relational databases; 
• supports asynchronous operations for better performance such as transaction write-behind and asynchronous replication;
• besides as a cache bus, they can also be used as a message bus.

Pros and Cons on TimesTen and Gigaspaces:

• Gigaspaces can cache all your data in a computing cloud such as Amazon's EC2 while TimesTen only practically caches subset of your whole enterprise data;
• Gigaspaces has more flexible deployment topologies than TimesTen. For example, Gigaspaces can adjust your deployment based on your SLA while TimesTen's deployment tends to be static;
• Gigaspaces supports application level optimistic locking while TimeTen only provides 2 traditional database's isolation levels - read-commit and serializable;
• Gigaspaces intrinsically supports parallel processing such as map-reduce paradigm through its master-worker pattern and thread executor framework while using TimesTen you have to build up such functions by yourself.  For example, if you want to do a search in your TimesTen cache grid (suppose you partitioned your data in the grid), you have to code by yourself to submit tasks to grid memebers and finally aggregate the each individual searching result;
  
• It is easier to adopt TimesTen than Gigaspaces because TimesTen intrinsically supports the popular JDBC API while Gigaspaces's intrinsic JavaSpaces API is still not that popular; 
• TimesTen is budget and deployment friendly because it only needs to cache a subset of your whole enterprise data. Most often a large amount of your enterprise data don't need real-time accesses and accordingly caching them may be wasting your valuable resources.
• TimesTen has closer integration with Oracle than Gigaspaces does. For example, TimesTen supports the same flexible locks as in Oracle and materialized views;
• TimesTen seems to have a bit better performance. For example, TimesTen can achieve 5-microsecond reads and 15-microsecond updates while Gigaspaces can achieve sub-millisecond latency at best. This may be because TimesTen was implemented by C/C++ and Gigaspaces was implemented Java (JVM has pause).

    

Daylight Saving Time (DST) and Timezone Handling in Java, JDBC and Oracle

The following problem has been puzzling me for 2 days.
Because MISO (Midwest ISO. A power market for several Mid West regions) doesn't support DST, it sent us data at hour 2 on Mar 14,2010 which is the DST beginning date (the hour 2 is supposed to be skipped if the ISO supports DST).

The date values in question are stored in an Oracle column called endDate which is of Oracle's date type without time zone information. In other words, you interpret such a date type's components (year,month,day,hour,minute,second and millisecond) in your local timezone.
If your data are across timezone, you should use Oracle's "timestamp with timezone" or "timestamp with local timezone". Such a date type has an additional timezone component based on which you interpret other components.

This additional timezone component is the key to understand the difference between Oracle's date type and Java's date type.
Because Java's date type represents the specified number of milliseconds since the standard base time known as "the epoch", namely January 1, 1970, 00:00:00 GMT. (Don't miss this GMT timezone!)
When you try to load an Oracle date value without a timezone into a Java date object, you will face difficulty without using any timezone in Java.

We used Hibernate's TimestampType to map this DB column to a Java Date type.
I used the following code to extract the hour in the endDate:
    Calendar cal = Calendar.getInstance();    //(1)

  int he;                                   //(2)
       
  cal.setTime(getEndDate());                //(3)
  he = cal.get(Calendar.HOUR_OF_DAY);       //(4)

The code is running in east coast which is currently in DST (it is August 2010). The hour in line (4) returns 1 for the hour 2 in question and the getEndDate()'s toString() also shows hour 1 instead of hour 2 or hour 3.

Getting to know why this happened is quite involving and confusing mainly due to the DST switch. We explain it in three steps.

First we need to know how Hibernate and JDBC driver retrieves you endDate value to a Java date object.
Hibernate's TimestampType just calls the following ResultSet's method to get the endDate.

   java.sql.Timestamp getTimestamp(String columnLabel);  

Because our DB column doesn't have timezone, what timezone will be used in the returned Timestamp (it extends date type) in the above method (still remember my previous statement "Java's date type represents the specified number of milliseconds since the standard base time known as "the epoch", namely January 1, 1970, 00:00:00 GMT. ")?
Astute readers may recall there is any similar method in ResultSet that allows you to provide a timezone through a Calendar:

   java.sql.Timestamp getTimestamp(int columnIndex, Calendar cal) ;

This method is exactly to handle a DB column without a timezone so that you can interpret the DB date times in a specified timezone based on your business logic.

The Java document for the first method doesn't say what time zone is associated with the returned timestamp value. This is unfortunate and different JDBC drivers may do different things. Later I will present what I found based on my testings.
Actually Java's date doesn't care about any time zone per se; it only remembers the the number of milliseconds since the "epoch". It is Java's calendar that incorporates a timezone (and also a locale).
Remember the preferred way to create a date object is to create a calendar first then call its getTime() to return the date. Calendar's getTime() creates a date by calculating the number of milliseconds since the "epoch".
On the other hand, you can assign a date value to a calendar whose time zone may be different from the original time zone that created the date.
For example, you created hour 1 in EST(GMT-5), then you assign this time to a calendar whose time zone is CST(GMT-6). The assigned calendar will returns a date whose hour is 0.
We can also infer that in order to return the same hour, the 2 time zones assigned to the 2 calendars must be the same.

Secondly, Java has different Calendar creation call syntax.
They behave differently for different timezones on Mar 14,2010(or any other DST beginning dates) even they all refer to the same region.
Take the New York region for example. Suppose our code runs in New York, the following 3 calls all return time zones in the New York region:
  Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("GMT-5:00")); //EST. No DST support      (1)
  Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("GMT-4:00")); //EDT. No DST support      (2)
  Calendar cal = Calendar.getInstance(); //default to "America/New_York". Supports DST  (3)

With (1), you can set the calendar's hour to 1,2 and 3 etc, even hour 2 doesn't exist and it actually corresponds to hour 3 in EDT.
With (2), you can also set the calendar's hour to 1,2 and 3 etc, even neither hour 1 or 2 exists and they actually correspond to hour 0 and 1 in EST, respectively.
With (3), you can set the calender to all 24 hours except 2. Specifically hour 1 corresponds to hour 1 in EST and hour 3 and later correspond to hours in EDT. When you try to set hour 2, Java actually changes to hour 3 in EDT because call syntax (3) supports DST and hour 2 doesn't exist.
(It is easy to understand if you just think the time zones in call syntax (1) and (2) are some first-class time zones and the time zone in (3) can be either GMT-5 or GMT-4 depending on the hour).

Finally we connect the dots together and shows you why line (4) got hour 1. 
When Hibernate uses the getTimestamp() without a calendar, the Oracle JDBC driver uses GMT-5 for hour 1 and GMT-4 for other hours to create a calendar and eventually returns a timestamp (I am not sure whether this is standard practice). So the hour 2 in the endDate is kept as hour 2 which actually corresponds to hour 1 in EST.
Because Line (1) is the Calendar call syntax (3) which supports DST, line (4) returns the actually hour 1 in EST.

Actually our application needs to return whatever hour MISO sent us without any DST offsetting. Based on the above analysis of Date and Calendar, line (1) must specify the same timezone as the one used to create the endDate by the JDBC driver. Unfortunately this is a guess game if you DB column doesn't have timezone information.

There are 2 solutions.
One is to design a DB column with time zone information.
The other is to extend Hibernate's TimestampType by explicitly specifying a GMT timezone such as your local timezone based on the raw GMT offset (Such Timezones ignore DST schedules).

Lastly the toString() from a Date object is based on your local timezone which may confuse you when your intended timezone is different.

Quartz CronTrigger Scheduling Policy - Fixed-Rate or Fixed-Delay?

As you know the Java Timer class allows you to schedule task either as Fixed-Rate or Fixed-Delay.

In fixed-delay execution, each execution is scheduled relative to the actual execution time of the previous execution. If an execution is delayed for any reason (such as garbage collection or other background activity), subsequent executions will be delayed as well. In the long run, the frequency of execution will generally be slightly lower than the reciprocal of the specified period (assuming the system clock underlying Object.wait(long) is accurate).
In fixed-rate execution, each execution is scheduled relative to the scheduled execution time of the initial execution. If an execution is delayed for any reason (such as garbage collection or other background activity), two or more executions will occur in rapid succession to "catch up." In the long run, the frequency of execution will be exactly the reciprocal of the specified period (assuming the system clock underlying Object.wait(long) is accurate). 

Quartz supports Unix/Linux-like cron scheduling with its CronTrigger class which has a fixed-rate like execution policy.
For example if you configured your job to run every 5 seconds in a thread pool with more than 1 worker thread, the job will be kicked off every five 5 seconds.
If the previous job execution has not be done yet, the new job execution will pick up an available worker thread in the thread pool. This is different from Java's Timer which only has one single worker thread.

Quartz doesn't support Fixed-delay as of version 1.6. Actually you can find the details by digging into its source code.
The key classes and interfaces are its QuartzScdedulerThread, JobStore and CronTirgger.
In method run() of QuartzScdedulerThrea, you can find it calls triggerFired() before calling runInThread() to actually run the job.
triggerFired() will eventually lead to a call to triggered() in CronTrigger where nextFireTime is calculated.
So you know no matter how long your job run in runInThread(), the next execution time is always correctly calculated from its last execution time by method triggerFired().

Default Round Mode when Setting a BigDecimal Scale

If you are not familiar with BigDecimal and want to set its scale to, say 2, you might just call myBigDecimal.setScale(2) assuming the BigDecimal will round excess decimals automatically for us. Unforrtunately BigDecima's default rounding mode is not rounding anything. So if your big decimal is 1.234, you will get a "Rounding necessary" exception.

Almost all developers I worked with assumed the setScale() will do some automatically rounding. They didn't realize it until they got a the above run time exception.
Since it is so common(based on what I experienced), why shouldn't it be default to a rounding mode such as ROUND_HAVE_UP?

Master-Local -- the most common topology in IMDG

IMDG (In-Memory-Data-Grid) functions like an in-memory database to the front end users. In order to be scalable, IMDG has to provide partitioned topology besides the replicated one.

If we bundle an application locally with a data partition together and the application only processes its local data, we form what the GigaSpaces calls a PU(Processing Unit).
Because PU doesn't need to communicate with others, we can achieve linear scalability by deploying multiple PUs.

However such a tightly couple application rarely exists practically.
On one hand IMDG basically partitions data evenly without caring about too much of your application. After all IMDG, as the delegate of a system of record (the database), should be able to server all applications.
On the other hand your application should implement your specific business logic and you may have different applications for different business requirements.

So your universal data and your specific business requirements are just conflicting with each other.
To solve this dilemma, a Master-Local topology is often employed where the master cache (or Data-only Processing Unit or EDG in GigaSpaces) itself can be either replicated, partitioned or both; the local cache (or embedded space in GigaSpaces), caches data from, and synchronizes any update with, the master on demand. The communication between local and master is usually a single network hop and extreme low latency can be achieved using switches.
Here is a diagram from GigaSpaces where the master is replicated:

If your local cache is read-only, the performance is the best.
Oracle Coherent also calls the above topology as Near Cache.

Going back to GigaSpaces' "XAP Order Management Tutorial" in its XAP 7.0.0 "Quick Start Guide" section. As its comment mentions, the AccountData comes from IMDG which is simulated by AccountDataLoader in the example. In practice it should come from an EDG which connects to the local embedded space in the order runtime processing unit.

Wife just Got a Sweet Offer from a Dream Hospital

I really feel happy and be proud of my dear wife, lili.
She just got an offer from one of the best US hospitals in the US.
Before this, she got quite a few interview opportunities. However none of them went further either because we don't want to leave the tri-state areas or she just barely missed.

She has been interested in such medical research and clinical training position in Radiation Oncology for several years.
She alway feels excited whenever she thinks she will be a future medical physicist and assist doctors to cure cancer patients.
So she has the great motive to have learned almost all courses required for this position and got very good GPA (average 3.85) while she has been doing 3-year long post-doc researches.

She well deserves this great offer.
Congratulations!

Parallel Computing

In the past several months I have been familiarized myself with parallel computing products such as the  CylicBarrier and CountDownLatch in Java 6, the Fork-Join in the upcoming Java 7, Map-Reduce and Google File System from Google, Hadoop from Yahoo, GridGains, In-Memory Data Grid (IMDG) such as GigaSpaces and Coherent, and Oracle Real Application Cluster (RAC).

I am really impressed by IMDG, which brings the data closer to, or be together with, your application logic so that you can achieve high throughput and also sometimes low latency.
By reading some white papers from Gigaspaces, I just feel the tier structure for enterprise applications are just leading me to the wrong direction, though many companies still follow it in order to have "scalable" performance.
If we can enjoy all of data bus, message bus and parallel computing bus together with in-memory access latency using GigaSpaces' so-called SBA, why do we still bother using multi-tier structure and frequently waiting for network and database related IO?

Will Oracle still be dominate in the next several years?

RTSJ or Standard Java with Fast JVM for Low Latency Applicaitons?

For low latency applications as often required by financial services and telecommunication industries, there seems to be 2 trends. One is to use Real Time Specification for Java (JTSJ); the other is to stick with standard Java helped with some fast JVM.

My problem with RTSJ is it is not transparent to standard Java due to its extensions such as RealTimeThread, NoHeapRealTimeThread, Scoped and ImmortalHeap that are required in order to achieve Soft or Hard real time goal. It also require you to have a real time OS.
Because standard Java has a much larger user base than RTSJ does, RTSJ's incompatibility is the biggest showstopper for its wide application.
RTSJ mainly fixes 3 problems that make traditional Java indeterministic:
  GC's pause;
  JVM's adaptive optimization;
  JVM's dynamic class loading.

On the other hand, JVM has being improved a lot each year. JIT compilers also become more and more powerful with such features as dynamic and adaptive optimizations, concurrent and parallel GC algorithms and dynamic instruction scheduling.

For example, you can improve both throughput and response time with the concurrent and parallel GC in Oracle/Sun's hotspot JVM compared to the traditional serial GC.
With its latest G1 which is available in Java 6 update 14 and later,  you can even achieve soft real-time latency.

Oracle / Bea's JVM JRockit is especially exciting. Oracle claims it is the industry’s fastest JVM with its average response time in microseconds for most well-managed applications and low, single-digit, millisecond response with a five nines reliability guarantee.

I am just wondering which one has more momentum and which one has more applications in the finance services industries.

My article "Java Memory Model from a Programmer's Point-of-View" was published on DZone

You probably feel difficult to read Java Memory Model (JMM for short) from either Java Language Specification Third Edition (JLSv3 for short) or Java Virtual Machine Specification Second Edition (JVMSv2 for short) due to several reasons such as academic jargons and lack of basic understanding of modern microprocessors' out-of-order completion mechanism.
After all only JVM implementers must understand all of JLSv3 and JVMSv2.

So this article only teaches you the necessary parts of JMM that are needed for a programmer to make safe multi-threaded code. Here is the URL:
JAVA MEMORY MODEL FROM A PROGRAMER’S POINT-OF-VIEW

However I do have a question regarding the "Causality Loop" mentioned in the article and JMM: what microprocesses are supporting it?
I actually sent an email request to professor Sarita V. Adve because his paper "Memory Models: A Case for Rethinking Parallel Languages and Hardware" mentioned the same thing in Figure 4's case (a).

Here is my question:

My understanding is when core 1 speculatively executes "Y = 1", it puts the new value in a temporary store (core 2 will not see it) until it detects the predicate in the "if" statement becomes true; I have similar argument for core 2.  So neither core 1 nor core 2 will make its speculative write operation visible to the other core.

hibernate.jdbc.batch_versioned_data can't be set to TRUE for Oracle JDBC driver

Duo to Oracle's popularity, we originally assumed it should be safe to turn on this flag for Oracle JDBC drivers until we saw a Unit testing exception.
Basically the Unit test tried to update a POJO in an optimistic way. It should have failed because the same POJO was just updated by another user before it. However Hibernate (3.3) just silently returned without resulting in any database update.
I traced the code to Hibernate's method checkBatched() in class Expectations$BasicExpections. The rowCounts Oracle returned turns out to be always -2 (Statement.SUCCESS_NO_INFO) for all its version 9i,10g and 11g JDBC drivers.

This returned value "-2" was finally verified by Oracle's JDBC Developer's Guide titled "Update Counts in the Oracle Implementation of Standard Batching" in Chapter 23 "Performance Extensions".
Basically it says:

• For a prepared statement batch, it is not possible to know the number of rows affected in the database by each individual statement in the batch. Therefore, all array elements have a value of -2. According to the JDBC 2.0 specification, a value of -2 indicates that the operation was successful but the number of rows affected is unknown.
  
• For a generic statement batch, the array contains the actual update counts indicating the number of rows affected by each operation. The actual update counts can be provided only in the case of generic statements in the Oracle implementation of standard batching.
  
• For a callable statement batch, the server always returns the value 1 as the update count, irrespective of the number rows affected by each operation.

The different returned values for different statements can be explained by the following Oracle implementation details:

In Oracle JDBC applications, update batching is intended for use with prepared statements that are being processed repeatedly with different sets of bind values.

The Oracle implementation of standard update batching does not implement true batching for generic statements and callable statements. Even though Oracle JDBC supports the use of standard batching for Statement and CallableStatement objects, you are unlikely to see performance improvement.

But I still have a hard time to believe that the back-end powerful Oracle database even couldn't know  the number of rows affected by each preparedStatement in a batch.

I am equally disappointed with the way Hibernate is handling Statement.SUCCESS_NO_INFO(-2).
This is how it happens: Hibernate uses preparedStatement for batch updates. Because the where clause in the update sql used a staled version number, Oracle just didn't update anything and returned successfully. Finally Hibernate still interprets Statement.SUCCESS_NO_INFO(-2) just as an successful update instead of throwing any optimistic exception. As you know this is not acceptable.

I know Hibernate is in a dilemma in this case because when Oracle returns Statement.SUCCESS_NO_INFO(-2) it either means some rows were updated or no row was udpated.
But I still like Hibernate to throw some exception to remind users of the updating ambiguity.

The good news is batch inserting is still safe as because returning Statement.SUCCESS_NO_INFO(-2) must mean a row was successfully inserted into the database.
So you may have to create a separate datasource for batch inserting only.