A service for pulling medical data from a variety of sources

LANIT-TERCOM developed a service that gathered medical data and displayed it in a single format such that it can be used in future studies. The client was an Israeli company offering integration and data management services to organizations running medical studies.

Primary project objectives:

• Develop a user-friendly interface for adding differently structured data sources with a variety of data transfer protocols to the system without modifying system code
• Deliver strong performance given that hospitals store large quantities of data (up to several terabytes)
• Ensure abstract and anonymized data, as medical information cannot be passed on to researchers as such without violating non-disclosure laws for personal information
• Provide management for business processes like adding new researchers or requesting access to data
• Develop the solution using open-source technology that is free for commercial use

Solution components:

• ETL (extract, transfer, load) system for extracting data and compiling it in the database
• Editor for working connections to data sources and ETL rules
• Database for storing all needed information
• Load balancer
• Web-based user interface
• Activiti framework for managing business processes

Technology used:

The business logic was written in Java using the Spring and Hibernate frameworks. PostgreSQL was used as the database, while HTML5/CSS3/AngularJS/Bootstrap worked for the user interface. Data sources are supported in PostgreSQL and Apache Impala. The solution was rolled out to the Amazon Web Services cloud.