Data & ETL
​
​
​​
-
Wall Street - I managed (with a team of 7) a global reference data repository for the second largest investment bank on Wall Street with 5 million equity products, 20+ million fixed income products and 50 million corporate action records in production databases (one per product type) with 50 global replicas. It was all master data that had to be correct 24x7 365 days a year. ​We didn't use any COTS ETL-type system such as DataStage, Informatica or Ab Initio as these were not "agile" enough to deal with the volume and variety we needed. Instead, we developed dozens of custom programs that could support hundreds of data streams carrying hundreds of gigabytes of data all across the enterprise working separately and independently but with a single overall goal, to have "the right data in the right place at the right time". Some programs consisted of a few lines and ran for a few seconds, some were composed of up to 10 sub-modules and ran for 3 full days (options expiry weekend).
​​
The greatest​ singular achievement: I fixed (using a script) 10 million historical equity records in a production database (regulatory failure) during North American production hours with no issues, failures or downtime in a system where a 1-second latency in data availability could cost millions of dollars in losses.
​
-
RBC - I rearchitected an ETL enterprise system of registered businesses to double its throughout during its 10-hour runtime using several different methods including multi-threading, SQL tuning, table cleansing, loading script refactoring etc.
​​
The greatest​ singular achievement: I fine-tuned a 2000-line multi-statement SQL script to cut the runtime from 2 hours to 40-50 minutes.
-
eCommerce - I developed an ETL process that allowed my client to migrate their client's MSSQL (national pizza chain) customer/order database with 10 years worth of data to my client's private cloud database in 3 days as opposed to 8 months using their migration program migrating the source data 1 business record at the time. The script was multithreaded to download the data but single-threaded to upload it.
​​
The greatest​ singular achievement: One of the SQL statements to insert orders into the new Oracle table was 2000 lines long because it converted all the data in the fields into XML segments.