First DBS Ltd - Technical Consultancy

Home | Stories | Business Consultancy | Technical Consultancy | Health Checks | Our People | Contact Us

Big Data

Is this marketing hype or a genuine and useful new way of doing things? We believe it is a bit of both. It is true that a vastly increased data volume sometimes makes new analysis and user experiences possible. After all, a movie is really just a sequence of still pictures. However there are some common unrealistic expectations, in particular that there is somehow no need to understand data properly any more.

The data lake concept has a lot of appeal. Just throw the data in as it happens to be available and aim to make sense of it properly later, when you want to go fishing. Unfortunately there are some traps here. If some data items require special protection under GDPR there is no avoiding up front analysis to identify them. In some cases GDPR rules may completely prohibit the holding of some data in a data lake. Also the format of incoming data can change on you. If you are lucky some component of your system will fail with an informative message when this happens. If you are unlucky the output from your data lake will simply be wrong and you will never know.

Should we all rush to realtime? Batch processing is old fashioned right? Well, maybe. It all turns on the specific business requirements. For example, access to the most up-to-date data possible is certainly beneficial in the context of authorising payment card transactions. Many types of analysis focussed on understanding customer behaviour have no such need however, and these requirements can be met more simply if batch operations are included.

Logical and Physical Data Modelling

Our philosophy here is "Keep It Simple!". Any unnecessary complexity in a Data Model will cascade into much more unnecessary complexity in software designed and built to access your database. In our experience it is worth taking the trouble to get a Data Model right.

We have experience in customising large template data models, including the various industry models from IBM's Information Framework (IFW). These include the Banking Data Warehouse (BDW), Insurance Application Architecture (IAA), Telecoms Data Warehouse (TDW) and Retail Data Warehouse (RDW)

The challenge with these data models is judging where simplification is appropriate, and where complexity must be retained to meet likely future requirements.

Data Warehousing and Business Intelligence

First DBS Ltd is able to offer advice and practical assistance with the implementation of Datawarehouse and Business Intelligence systems. These systems typically present their own brand of unforeseen difficulty, and there can be great benefit in talking to somebody who has been there before.

When constructing an integrated view of data from a variety of sources, transforming the data to a common representation can be quite a challenge. Typically, discussions need to take place with representatives of the business to determine the most appropriate trade-off between the goals of consistent presentation and the availability of the fullest possible detail in reporting. The arrangements made need to support the intended analysis by business users of the system without introducing unnecessary complexity or inconsistency.

First DBS Ltd has experience of the design and implementation of Extract Transformation and Load (ETL) arrangements using ETL products such as DataStage and Informatica, where these tools are favoured by our clients. However in our experience equally good results can often be achieved at lower cost using more basic tools such as SQL, database loading tools and Unix shellscripts. The benefits of controlling a data mapping via a Graphical User Interface (GUI) are often an illusion with real world systems.

Complex transformation requirements such as spatial processing or matching using procedural algorithms often dictate a bespoke approach to portions of the ETL in any case. We have experience of such bespoke ETL development using Netezza SQL with spatial extensions, Oracle PL/SQL, PERL, Java and C.

Special techniques are applicable to constructing the Database Schemas used to support both adhoc queries and regular reporting in Data Warehousing and Business Intelligence systems. Data marts commonly use the star schema, as recommended by the data warehousing guru Ralph Kimball. This arrangement uses a central high volume "fact" table with many-to-one relationships to lower volume static "dimension" tables. It is often possible to retain an acceptable interactive response even with enormous data volumes using this approach. Many Relational Database Management System (RDBMS) products provide facilities to optimise query performance using star schemas. Pre-calculated aggregates can also be worthwhile in some situations, and it can be arranged that these are used automatically without the business user having to be concerned with the mechanisms "under the covers". First DBS Ltd can advise on the suitability of the various techniques.

The availability in recent years of database appliance products designed specifically for high volume data warehousing / business intelligence, such as Netezza, presents the possibility of dramatic performance improvement compared with using a general purpose database product. First DBS understands how to exploit the parallel architecture of such applicances by arranging the distribution of data optimally across processing units.

The appropriate overall data architecture is important to establish when an organisation first embarks on a data warehousing project. Depending on the number of data sources and analysis environments ultimately envisaged, it can be appropriate to employ a common staging area or Enterprise Data Warehouse (EDW). The number of ETL interfaces required can be reduced by this means, and with an EDW in place new data marts can be produced without repeating data source ETL work. There is no one-size-fits-all architecture, however. First DBS is able to help organisations select the data architecture best able to support its planned data warehousing project(s).

First DBS Ltd has experience of a variety of user interface and query tools applicable to Data Warehousing and Business Intelligence systems, including Business Objects, COGNOS and Tableau.

Home | Stories | Business Consultancy | Technical Consultancy | Health Checks | Our People | Contact Us