Enterprise Architecture

Architecture for Financial Applications – Rethinking Object Oriented Paradigm

Multi-Tier Application Architecture isn’t a new concept to anyone who has done any sort of enterprise development to the point nobody ask about this during technical interviews anymore.
At minimum, there’re always three basic tiers whether you’re building web application or client server application:
a. Presentation
b. Application – Business logic
c. Data source
For financial applications, where do you put your calculations? That’s a matter of debate (but it shouldn’t).
I can’t tell you how many times I have seen applications gets built using the standard cookie cutter: DAO to load records from database into entities in Application tier, fashionably complying to every standard practice using OR mapper such as hibernate, Repositories and DAO with Spring and IoC container for every object. I’m not sure if people do this because they feel the need to comply with Golden OO design paradigm. Or too afraid to deviate from “Best Practice”. This pattern simply don’t apply to all scenario. Not just “edge cases”.
For starter,
a. What kind of calculation are you running? Derivatives risk and pricing? VAR? Stressing? Time series analysis, covariance calculations, factor construction in portfolio optimization? Theses are computationally intensive. Quant libraries generally in c++, Python,  Java. And typically load distributed and calculations, thus, done in “Application Tiers”.
Or are you running simple pnl updates, aggregate position level pnl/return/risk to book level? funding or trade allocation? Reconciliation? These are simple mathematics (no quant Library) : key concatenation/matching, simple aggregations. This brings us to next point.
b. Data volume, performance, and proximity to data source. If your calculations sources or operate on a lot of data, unless nature of calculation complex. Or that it requires quant libraries. There’s probably very little reason why these should be done in Application Tier. Databases are extremely good at keys concatenation /matching, aggregation and simple arithmetic. If data already in database, you’re processing more than a few thousand rows, performance gains can be realised by running these calculations in database/SQL BEFORE you move data from database tier to application tier. Even if you have multiple data sources (even message bus or non-SQL sources) : One can always build simple data feeds, consolidate into single database. Downside to this approach is, SQL not portable across different database vendors.
c. Support
If calculations done in SQL, this means production trouble shooting can be done without a debugger. What this further means is that Level One support don’t need bother developers. More importantly, fixes can be simple SQL patches – no need recompile and redeploy, which adds to the risk.
d. Simplicity, Agile, and Maintainability
Let’s keep things simple. You’re adding complexity everytime you add a bean, entity, DAO – especially you have to maintain these mundane pieces of code manually. Imagine the amount of work you need to do if new fields need to be added? Worse if additions are to be performed on multiple entities/DAO (Although we no longer need updates hbm files anymore thanks very much). Speaking of which, #dotnet is full of Agile magic. Linq-to-SQL [completely] automates the process of generating entities & DAO: https://www.youtube.com/watch?v=bsncc8dYIgY
You don’t even need hand code the entities (Unlike Java/Hibernate). Linq-to-SQL, however, does not support updates. However, one can simply delete old entity/DAO files, then drag-drop to re-create the entities/DAO in seconds. This said, the achilles heel of Linq-to-SQL is: It supports Microsoft SQL Server only! (Ouch!!!)
#msdev are blessed with Linq-to-SQL. It doesn’t follow that Linq-to-SQL is the shortest path for all scenario. Most financial applications deals with data in Tabula format – #yesSQL!. And most financial calculations are simple additions/subtract and multiplications which can be done in SQL layer. Another Agile Magic that #dotnet has (And Java has not) is DataTable – there’s nothing simpler than loading a DataTable with a DataAdapter and bind to Grid on front-end. Example: http://www.dotnetperls.com/sqldataadapter
Many times, there’s just no real need for creation of entities just to comply with OO paradigm. Just because you don’t “bean” your reports and put them in nice little coffins mean it’s Dirty coding. In fact, by having fewer source files, your code is cleaner.
In fact, emphasis of Agile development should not be about Kanban board, stand-up meetings and micro-managing developers by incompetent or non-technical Project Managers (Titles should be more appropriately renamed to Project Administrative Assistance). Instead, emphasis and focus should be exploration and utilization of available Open Source and Commercial tooling that actually do actual work for you, and do things as simply as possible leveraging such technologies.
Don’t Over-Engineer and Happy Coding!

Oh… apparently last time someone think about this was back in 2010: http://blog.jot.fm/2010/08/26/ten-things-i-hate-about-object-oriented-programming/

Advertisements

Reverse Engineering Data-flow in a Data Platform with Thousands of tables?

Ever been tasked to inherit, or migrate an existing (legacy) Data Platform? There are numerous Open Source (Hadoop/Sqoop, Schedulix …) and Commercial tools (BMC Control-M, Appliedalgo.com, stonebranch …etc) which can help you operate the Data Platform – typically gives you multitude of platform services:
• Job Scheduling
• ETL
• Load Balancing & Grid Computing
• Data Dictionary / Catalogue
• Execution tracking (track/persist job parameters & output)

Typical large scale application has hundreds to thousands of input data files, queries and intermediate/output data tables.
DataPlatform_DataflowMapping

Mentioned Open Source and Commercial packages facilitates operation of Data Platform. Tools which helps generates ERD diagrams typically relies on PK-FK relationships being defined – but of course more often than not this is not the case. Example? Here’s how you can Drag-drop tables in a Microsoft SQL Server onto a Canvas to create ERD – https://www.youtube.com/watch?v=BNx1TYItQn4
DataPlatform_DataflowMapping_Who

If you’re tasked to inherit or migrate such Data Platform, first order of business is to manually map out data flow. Why? To put in a fix, or enhancement, you’d first need to understand data flow before any work can commence.

And, that’s a very expensive, time consuming proposition.

There’re different ways to tackle the problem. Here’s one (Not-so-Smart) option:
• Manually review database queries and stored procedures
• Manually review application source code and extract from it embedded SQL statements

Adding to complexity,
• Dynamic SQL
• Object Relational Mapper (ORM)

The more practical approach would be to employ a SQL Profiler. Capture SQL Statements executed, and trace the flow manually. Even then, this typically requires experienced developers to get the job done (Which isn’t helping when you want to keep the cost down & delivery lead time as short as possible). As such undertaking is inherently risky – as you can’t really estimate how long it’ll take to map out the flow until you do.

There’s one command line utility MsSqlDataflowMapper (Free) from appliedalgo.com which can help. Basically, MsSqlDataflowMapper takes SQL Profiler trace file as input (xml), analyze captured SQL Statements. Look for INSERT’s and UPDATE’s. Then automatically dump data flow to a flow chart (HTML 5). Behind the scene, it uses SimpleFlowDiagramLib from Gridwizard to plot the flow chart – https://gridwizard.wordpress.com/2015/03/31/simpleflowdiagramlib-simple-c-library-to-serialize-graph-to-xml-and-vice-versa/

Limitation?
• Microsoft SQL Server only (To get around this, you can build your own tool capture SQL statements against Oracle/Sybase/MySQL…etc, analyze it, look up INSERT’s and UPDATE’s, then route result to SimpleFlowDiagramLib to plot the flow chart)
MsSqlDataflowMapper operates on table-level. It identify source/destination tables in process of mapping out the flow. However, it doesn’t provide field-level source information (a particular field in output table comes from which source tables?)
• The tool does NOT automatically *group* related tables into different Regions in diagram (This requires a lot more Intelligence in construction of the tool – as we all know, parsing SQL is actually a very complex task! https://gridwizard.wordpress.com/2014/11/08/looking-for-a-sql-parser-for-c-dotnet). At the end of the day, it still takes skilled developer to Make Sense of the flow.

Happy Coding!

Java and dotnet Interop

This article is about Java-dotnet Interop. We’ll explore what options we have for different scenario where interop is required.

First, when we say “Java-dotnet Interop”, there are two possibilities:

1. Java -to- dotnet communications

2. dotnet -to-Java communications

Secondly, we assume, if you’re developing in Java, you’d run it on Linux (Or simply put, if your application written in Java, why would it run on Windows?)

Given above, what are our options?

 

1. Socket

Anand Manikiam has written a piece on this subject, http://www.codeproject.com/Articles/11602/Java-and-Net-interop-using-Sockets

The pros for this approach are:

a. No middle-ware

b. Fast

The cons are:

a. Resiliency

b. Casting complex object/classes from byte[]?

c. Message security? Encryption? Anti-tampering? DOS? If not implemented this be Intranet application only.

 

2. Web Services

I’ve written an article of consuming Java-ws from dotnet:

https://gridwizard.wordpress.com/2014/12/26/java-ws-and-dotnet-interop-example/

You will also find plenty of discussions on consuming WCF-from-Java:

http://www.codeproject.com/Articles/777036/Consuming-WCF-Service-in-Java-Client

The pros for this approach are:

a. No middle-ware

b. Higher level of compatibility with code coded in more languages (C++/SOAP, Python, R …etc)

The cons are:

a. Less fast than socket

b. Resiliency

c. Message security? Encryption? Anti-tampering? DOS? If not implemented this be Intranet application only.

d. Slower than Socket! (Web Services overhead)

 

3. Message Bus

RabbitMQ (http://www.rabbitmq.com) is all about Messaging. If you’re developing real time applications, RabbitMQ offers high performance battle tested communication platforms and it as an API for just about any language on the planet. C++, dotnet, Java, Perl, Python…

Pros are:

a. Resiliency – producers and consumers can die and crash at any moment.

b. Performance

cons:

a. You need install Middleware, and if you’re a software vendor, you’d need bundle installation of RabbitMQ with your application

 

4. Commercial Tools

Depending on what you’re building, if what you’re trying to build is a computing grid, then there are commercial tools which allows you to run jobs on basically any platform, coded in any language.

Appliedalgo.com for instances supports:

a. Scheduling, conditional job chaining and Workload Automation

b. Grid Computing – nodes/slaves on any platform/language

c. Automatic persistence of run history, parameters, input and results

(Even configure cell level validations by “IsNumber”, or use of user specified Regular Expression)

d. GUI for you to track run parameters, input and results

However, such tools inevitably introduces execution overhead. So depending on whether you’re …

a. Executing high number of light weight jobs –> Probably should not use any tool besides a Message bus such as RabbitMQ

b. Executing medium number of medium weight jobs –> Best application of Workload Automation Data Platforms such as Appliedalgo.com

c. Executing low number of heavy weight jobs –> Best custom coded, persistence via BCP (There’s no other way for million rows or #bigdata processing)

 
But this would not be a viable option for instance if you’re building a hotel booking system with web tier built in ASP.NET and backend in Java with Java-ws

Happy Coding!

 

 

Multi tiering for Financial Applications

Multi-Tier Application Architecture isn’t a new concept to anyone who has done any sort of enterprise development to the point nobody ask about this during technical interviews anymore.
At minimum, there’re always three basic tiers whether you’re building web application or client server application:
a. Presentation
b. Application – Business logic
c. Data source
For financial applications, where do you put your calculations? That’s a matter of debate (but it shouldn’t).
I can’t tell you how many times I have seen applications gets built using the standard cookie cutter: DAO to load records from database into entities in Application tier, fashionably complying to every standard practice using OR mapper such as hibernate, Repositories and DAO with Spring. I’m not sure if people do this to learn the different technologies? To comply with Golden OO design paradigm. Or too afraid to deviate from “Best Practice”. This pattern simply don’t apply to all scenario. Not just “edge cases”.
For starter,
a. What kind of calculation are you running? Derivatives risk and pricing? VAR? Stressing? Time series analysis, covariance calculations, factor construction in portfolio optimization? Theses are computationally intensive. Quant libraries generally in c++, Python,  Java. And typically load distributed and calculations, thus, done in “Application Tiers”.
Or are you running simple pnl updates, aggregate position level pnl/return/risk to book level? funding or trade allocation? Reconciliation? These are simple mathematics (no quant Library) : key concatenation/matching, simple aggregations. This brings us to next point.
b. Data volume, performance, and proximity to data source. If your calculations sources or operate on a lot of data, unless nature of calculation complex. Or that it requires quant libraries. There’s probably very little reason why these should be done in Application Tier. Databases are extremely good at keys concatenation /matching, aggregation and simple arithmetic. If data already in database, you’re processing more than a few thousand rows, performance gains can be realised by running these calculations in database/SQL. Even if you have multiple data sources (even message bus or non-SQL sources) : One can always build simple data feeds, consolidate into single database. Downside to this approach is, SQL not portable across different database vendors.
c. Support
If calculations done in SQL, this means production trouble shooting can be done without a debugger. What this further means is that Level One support don’t need bother developers. More importantly, fixes can be simple SQL patches – no need recompile and redeploy, which adds to the risk.
d. Simplicity, Agile, and Maintainability
Let’s keep things simple. You’re adding complexity if you are doing simple maths in application tier, everytime you add a bean, entity, dao.
Happy Coding!

Web vs Mobile vs Client-Server application?

What are your options when developing a new application? Both js and WPF/Winform support MVVM, real time data binding. You can build modern UI with MVVM framework in both js and WPF/Winform. You can now debug/compile/unit test both javascript with gruntjs as you can do with dotnet (from Visual Studio, Cruise Control/TFS/NUnit) or Java. How’d you choose?

Client Server

Pros

  • Unified, stable API. API/framework evolves (and become obsolete) less quickly in comparison to js frameworks. For instance, client can be built in dotnet/WPF or Winform for instance, while backend can be implemented in dotnet, Java, or C++ on Windows or Linux box.
  • Better organisation of code from maintenance/support perspective – dotnet/Java supports inheritance, Javascript doesn’t.
  • One less layer to code/develop, more Agile – In comparison to web based application, data need be wired to clientside/js, that’s additional dev lead-time
  • Not having to test against different browser
  • Access to Operating System API whereas js is sandboxed by browser. For example access to camera/mic.
  • Security – proprietary logic can be obfuscated with anti-tampering technologies.

Cons

  • Tied down to particular platform, or operating system

Web

Pros

  • Accessible from different platform/OS – mobile/tablet/workstations
  • Responsive  design – layout adaptable to different client devices, for example Bootstrap

Cons

  • Lack of unified standards, for example you’d need to choose between angular or nodejs (MVVM)
  • Testing against different browsers (At least Chrome/Firefox/IE) & clients (Desktop, Android devices & iPhone with various screen sizes)
  • Framework/libraries evolves very quickly, version change may be risky unless thoroughly tested
  • Browser updates may break your application (For example http vs https, security around websocket calls)
  • Very often there’ll be back-end development in dotnet, Java or C++ and unavoidably, you’d need to hire back-end developers
  • Everyone can scrutinize, hack, attack, *borrow* your Client side javascript

It’s quite obvious that for most applications, there will always be a back-end built in dotnet, Java or C++. The question really is, what would you choose to build the UI. There’s a lot of subjective preferences around this but personally, here’s my humble opinion on the subject.

1. Backend

Backend in dotnet or Java. Unless you’re writing device drivers, root kits, or highly optimized algo trading pipelines with DMA (Direct Market Access), there’s little reasons to do in C++. The small additions of complexity in syntax and availability of talents adds to the cost of choosing C++ for backend implementation. This add, you should be aware that you can implement *Backend* with node.js now. You can open database connection from js on node.js, you can even interface with RabbitMQ from js on node.js. Some market data vendors even begin to expose API for node.js – https://github.com/bloomberg/blpapi-node

So, truth is, you can implement backend in js – question is would you? (I wouldn’t – I prefer a proper OO language with availability of commercial & open source utility libraries. Math.NET for example.)

2. UI

Is this an end user applications? What’s intended Target Audience? Do you want to pay for the additional cost of developing UI in js?

  • If this is UI for a back-end server (And it has a UI), for example, workload automation/computing grid or EDI messaging infrastruture (more in manufacturing), then building UI in WPF/Winform/Swing/AWT/SWT saves you development cost and headaches in testing against different platforms and is thus the preferred option. Bear in mind while most workstations are Windows, on server side you can have Linux or Windows boxes. Further, your staff/clients may still prefer a responsive web interface so they can check application status/health on the run from mobile devices. One example is batch processing/grid system from Appliedalgo (https://appliedalgo.com). They have a client on Windows (https://www.youtube.com/watch?v=QdM9S0Bsc0A), also a responsive web interface (https://www.youtube.com/watch?v=dtYUona1omo)
  • If this is an enterprise application which requires complex data visualization/manipulation which can only be done on Workstation (not Mobile devices), for example, real time financial derivative risk management system (Example, Derivation.co.uk), then building UI in WPF/Winform (Most Workstations are Windows) saves you development cost and headaches in testing against different platforms and is thus the preferred option.
  • If this is an enterprise application for sales staff always on the run, example SAP REX (http://www.sap.com/pc/tech/mobile/software/industry-apps/retail-visit-management-app/index.html), then building in web or mobile application allows your application accessible from mobile devices, with responsive layout, and is the preferred option. If, for instance, compliance requires that all presentation (Investment Proposition to be specific) to clients be recorded (both audio/video), then you may even need build UI native Andriod or IOS. One such example CRM module from Axisoft.net.
  • If this is a web portal, for example, hotel booking system (hoteljen.com) or e-commerce portal (Amazon.com), then building in web application allows your application accessible to widest array of audience with responsive layout, and is thus preferred option.

 

Realtime Analytics: Just how many CPU/Processors would you need if your Market Data Service feeding thousands of ticks per second?

The Problem Statement

Ok, here’s the problem statement. You are building the application infrastructure for a group of 100 traders. The process starts with market data feed. A PriceServer stands between your corporate network and external market data vendors, Bloomberg, Reuters, Morningstar, Xgnite…etc, pumping in price ticks at rate of 3k per sec. CalcServer’s sitting in the back picks up the ticks, run some calculations, then publish results to trading/risk management screens.

Question is,

Just how many CPU would you need in your server farm if your Market Data Service feeding thousands of ticks per second?

 

Architecture

Different Loading Conditions - Different Tools

The problem we have at hand falls under CASE 3 or CASE 4 where updates frequency is high and distribution latency permitted is low, whether calculations is short, or lengthy.

Consider this hypothetical architecture. No scheduler, no load balancer, no caching or persistence support, minimal logging (As those found in tools such as appliedalgo.com, BMC control-m, or Autosys). Analogy is Space Shuttle construction, eliminate any components or weights you don’t need.

PriceServer, CalcServer and SimpleTradingScreen all feeds directly from RabbitMQ. The intention is to minimize latency/overhead cost along execution path starting from PriceServer, to CalcServer, then finally SimpleTradingScreen.

CalcFarm Architecture

How many processors we need in our CalcFarm? The maths is actually not that complicated. PriceServer is feeding the message bus (RabbitMQ) at rate of 3k ticks per second. That’s 1/3 ms per update. If CalcServer farm fails to process (Dequeue from RabbitMQ, run calc, then publish result back to RabbitMQ) a single ticks in 1/3 ms, then it’d lag behind PriceServer. If CalcServer farm takes 1 ms to process one tick, then it’s three times slower than PriceServer. Essentially, every additional 1ms in processing of published ticks requires additional 3 CPU’s in CalcServer farm in order that CalcServer farm stay in pace with PriceServer tick rate of 3k ticks/sec.

Imagine your calculations on average takes 100 ms to execute. That’s equivalent to 300 additional CPU’s. Or 37.5 servers each with 8 processors.

What you are doing here is essentially two things:

(a) To balance data velocity supported by each components along the overall execution chain

(b)  minimize latency along execution path between two points: PriceServer and SimpleTradingScreen.

 

Source Code? 

Want to experiment with this hands on? We’ve uploaded the source code to Git. Everything is built in .NET 4.5/C#.  Note that you need to install RabbitMQ first.

https://github.com/gridwizard/CalcFarm.git

Youtubehttp://youtu.be/FfwxRcZiEFw

One aspect worthy of special attention is that serialization/deserialization is high traffic area.  This happens when you queue or dequeue to/from RabbitMQ. General consensus is that “System.Runtime.Serialization.DataContractSerializer” and “System.Xml.Serialization.XmlSerializer” are relatively slow. For demo purpose, we’ve chosen “NetSerializer” – a free serializer from Codeproject.com. Source from Git: https://github.com/tomba/netserializer

REF:
(a) http://maxondev.com/serialization-performance-comparison-c-net-formats-frameworks-xmldatacontractserializer-xmlserializer-binaryformatter-json-newtonsoft-servicestack-text/
(b) http://www.codeproject.com/Articles/351538/NetSerializer-A-Fast-Simple-Serializer-for-NET

 

Start CalcServer Farm?

STEP 1. RabbitMQ

Please download+install RabbitMQ, make sure the service is running.

STEP 2. Start CalcServer

From command prompt, navigate to \CalcServer\bin\Release\

Simply type CalcServer. There’re only three settings in app.config:

<appSettings>

    <add key=”QueueUrl” value=”localhost”/>  – Point at RabbitMQ

    <add key=”DetailLog” value=”false”/> – For debugging only. If DetailLog=true, every price tick received would be logged and printed to screen. Set to false otherwise it will slow down CalcServer significantly.

    <add key=”MaxThreadPoolSize” value=”8″ /> – Maximum size of thread pool. Note, threads and concurrent instances of CalcServer on same physical machine share same set of Physical Resources, configure this wisely.

  </appSettings>

STEP 3. Start PriceServer

From command prompt, navigate to \PriceServer\bin\Release\

Simply type PriceServer. There’re only four settings in app.config:

<appSettings>

    <add key=”QueueUrl” value=”localhost”/>

    <add key=”DetailLog” value=”false”/>

    <add key=”PerSecPublishThrottle” value=”3000″/>     – Max publish rate

    <add key=”MaxCountPublishesCulmulative” value=”0″/>  – Max cumulative # publishes before stop

  </appSettings>

STEP 4. Start SimpleTradingScreen

From command prompt, navigate to \SimpleTradingScreen\bin\Release\

Simply type SimpleTradingScreen. There’re only two settings in app.config, I’m not going to repeat what’s already been covered.

<appSettings>

    <add key=”QueueUrl” value=”localhost”/>

    <add key=”DetailLog” value=”false”/>

  </appSettings>

 

How do we know that it works?

  • Tested on Intel 2.6GHz (Single Processor) with 4GB RAM (Very low end dev test machine, so if it can feed thru 3k/sec, so can you)
  • CalcServer dump performance statistics into “CalcServerStatistics”.

Basically,

  1. CountPxUpdateRecv is # ticks CalcServer has picked up from RabbitMQ.
  2. CountCalcCompleted is # calculations (dequeue+calc+publish result) been completed.
  3. Gap = CountPxUpdateRecv – CountCalcCompleted

If CalcServer is able to keep in pace with PriceServer, what you’d see is Gap stabilize.

The following result is done with PerSecPublishThrottle = 3000 (i.e. 3000 ticks per second)

Try set this to, for example, 10000. With Single instance of CalcServer running you will start seeing “Gap Widening” (i.e. Gap keeps increasing most of the time)

CalcServerStatistics

 

We’ve also written a simple NUnit test case.

It simply kicks off CalcServer and PriceServer. Let it sits for a few minutes, then examine CalcServer performance dump file \CalcServer\bin\Release\CalcServerStatistics.log

Then assert that “Average Gap” < 10k

It’s same as manually examining “CalcServerStatistics.log”, and confirm that “Gap” isn’t monotonically increasing, which is an indication that CalcServer can’t keep up with PriceServer.

CalcFarmNUnitTest

 

In addition to “Gap”, which is a measurement of throughput disparity, latency can also be measured simply by adding time stamps to messages.

latency = timestamp (PriceServer publish) – timestamp (SimpleTradingScreen recv)

Happy Coding!

SOA – Perspective from IT vendors vs Perspective from Financial Services

SOA (Service Oriented Architecture) is to re-factor different logical components into different “Services” – typically, “Services” means “Web Services” – obvious advantages?

  • Enterprise components can be efficiently re-used
  • Components developed in different languages (Java/.NET/C++/Perl…etc) can talk to each other.
  • Services can be hosted on Cloud [Optional]

Here’s IBM version of Enterprise SOA from @simonlporter (Pic from Simon’s tweets, pic url pic.twitter.com/Ygt8LPfFLW)

SOA and IBM

Typically, architecture perceived by IT vendors are very generic, with intention that structure be adaptable to biggest number of clients from different sectors and industries. For example, in above diagram:

These are generic Enterprise Services.

Businesses typically elaborate this structure and layer on top sector specific implementations. From perspective of Financial Services, in particular, asset management firms, hedge funds, investment banks, you’d see for example:

  • Market Data Service (For example aggregate across multiple data vendors Bloomberg, Reuters, Morningstar…etc)
  • Yield Curve Service
  • Deals Execution, Allocation and Audit Service (Pre-trade/post-trade analysis, Compliance…etc)
  • Position Service (Trading positions)
  • Risk Management Services (VAR calculations, Derivative pricing, sensitivity analysis. Stressing and Scenario Analysis)
  • RFQ (Request for Quotation, for structured OTC products)

Despite the differences, a well layered IT architecture in the bottom is critical in how easily businesses can layer sector-specific application services on top. Unfortunately, very often, this is not the case. Anti-patterns commonly observed:

  • One application logs to enterprise logging service, and another logs to another database with different log table schema
  • One application load balance computations using BMC Control-m (http://www.bmc.com/products/control-m-workload-automation/job-scheduling-software.html ), another to Applied Algo ETL Suite  (http://appliedalgo.com), yet another build their own.
  • Equity derivative trading desk uses home grown risk engine for derivative risk management.  Swap desks uses VBA sheets. Convertible trading desk could be using a vendor solution such as Derivation (Derivation.co.uk) – all risk system computes derivatives sensitivity Delta/Gamma/Vega/Theta… and persists EOD figures in different database tables, each with different schema.

Architectural break down typically happens during corporate mergers, or a result of ineffective IT governance – typically characterized by constantly changing or divergent chain of command.

This said, while properly layered SOA is critical for large enterprises, for small hedge funds, too much layering may not be economically feasible. It is thus a delicate balance for CIO’s and IT managers to remain vigilant so the business, its management architecture, and technical architecture evolve in lock steps.