Commercial Business to Business Energy Supplier
Two major projects were undertaken; a data warehousing system supporting in excess of 200 million records and a data migration project. This was in addition to ad hoc support and maintenance on the customer's Visual FoxPro v9 application.
Initially taking over a Finance Data warehousing and reporting project designed to store data in the order of 200 million records and 90Gb on an Oracle database, we re-factored the processing to do more on the Oracle server and significantly reduce the daily update processing time.
On taking on the project, the batch update process was taking more than 5 hours on its daily run and expected to increase as record volumes increased. We developed Oracle stored procedures to move processing to the server and reduced the batch process time to 3 hours which was not expected to increase with record volumes.
We also worked with the client to produce data views to assist in their reporting requirements and consulted with them on the design of Crystal reports formats for extracting data. The client covered the cost of development in the first 3 months of using this system.
This data migration task of significant size moved data from DBF files and Oracle to SQL server. Foxclever undertook the role of producing a set of central controlling data tables from which many other processes sourced data.
The project was essentially one of batch processing with an emphasis on speed; a tenth of a second saved on one record iteration could mean several hours saved in the overall migration process.
The majority of coding was done in Visual FoxPro v9 accessing DBF or Oracle tables and creating intermediary, consolidated DBF tables which were then used to produce CSV files for bulk loading to SQL server.
Some coding for migrating accounting data (from Sage) was done in SQL server stored procedures.