Skip to content

Architecture ​

The Open Integration Engine (OIE) is an open source integration engine for healthcare. It receives messages from one system, filters and transforms them under rules you define, and delivers them to other systems, in the formats those systems speak (HL7, X12, EDI, DICOM, XML, among others).

Core concepts ​

Filters, transformers, channel scripts and the JavaScript connectors all run JavaScript with access to the message, the maps and Java. Each part below has a usual job, and none is limited to it.

Channels ​

A channel is an interface you configure to receive data from a source, take actions on that data (filter, transform, extract), and send it out to zero or more destinations.

Each channel has general properties including a unique ID, name, and description. Channels can also be configured with code template library links, library resources, deploy/start dependencies (to control deployment order), attachment handler settings, message storage and pruning settings, and custom metadata columns.

Data types ​

A data type is the format of the message a channel is handling: HL7 v2.x, X12, DICOM, JSON, and so on. Using the inbound data type, the engine serializes the incoming message into an internal representation that filters and transformers can work with. Using the outbound data type, it deserializes that representation into the format that leaves the channel.

The internal representation varies by data type. For example, HL7 v2.x data is parsed into an E4X XML object, JSON data becomes a JavaScript Object, and Raw data stays a string, unparsed.

Supported data types include:

  • Delimited Text
  • DICOM
  • EDI / X12
  • HL7 v2.x
  • HL7 v3.x
  • JSON
  • NCPDP Telecom
  • Raw
  • XML

See Data Types and Formats for details.

Connectors ​

A channel consists of multiple connectors:

  • Source Connector. Gets data into the channel. Every channel has exactly one.
  • Destination Connectors. Usually send the message somewhere, but need not: a Channel Writer left at its default channel, None, sends nothing, and a JavaScript Writer can run a script that only sets a map value. A channel is configured with at least one, and may have several. A given message may still reach none of them, if the source filter rejects it or destination set filtering removes them all. Each has an enabled flag and a wait for previous destination setting that decides its chain membership.

For example, you may receive data over HTTP, then write it out to a file and insert pieces of it into a database.

A connector's type decides its configuration. A TCP Listener asks for a local address and port, an HTTP Listener for a base context path, a Database Reader for a query and a polling schedule.

Every source connector also carries a common block of source settings, though not all of them apply to every type:

  • a source queue, which when on accepts the message and responds immediately rather than waiting for processing to finish. Turning it on enables a queue buffer size and changes what the response can be set to
  • a response selector, choosing which response goes back
  • batch processing, splitting one inbound payload into several messages. Offered only by types that support batching
  • max processing threads, controlling how many messages the channel processes at once

An attachment is content stored with a message but apart from its body. The channel's attachment handler runs on the inbound message before the preprocessor and pulls attachments out of it, leaving a reference in their place. It can take the entire message, content matching one or more regular expressions, the pixel data of a DICOM message, or whatever a JavaScript script or a custom Java class extracts. A script can also create, read or update attachments at other stages with addAttachment, getAttachments and updateAttachment. By default a destination puts the content back in place of the reference before it sends. See Message Lifecycle and Storage.

See Channels, and Source Connectors and Destination Connectors for what each type exposes.

Filters ​

A filter decides whether a message should proceed to the next step or not. A filter returns true (accepted) or false (filtered). Filters are comprised of multiple rules, joined by AND or OR operators.

  • If the source connector filters out a message, it will not be processed by any destination connectors.
  • If a destination connector filters out a message, other destinations may still process it.

Transformers ​

A transformer modifies a message, converts it from one format to another, and extracts pieces for later use. A transformer has an inbound data type and an outbound data type (e.g., HL7 v2.x to XML). Transformers are comprised of multiple steps, each of which modifies or extracts data.

See Filters and Transformers for details.

Response transformers ​

Every destination can have a response transformer, which runs on what the destination got back. Its steps can do anything a transformer step can, and it can also set the status the message ends up with. It is the only transformer that runs after a connector rather than before one. Unless its inbound data type is Raw, it runs only when a response came back, so most failed sends skip it; see Response transformers.

Channel scripts ​

Four special scripts are associated with each channel:

  • Deploy Script. Runs once right before a channel is deployed.
  • Preprocessor Script. Runs once for every message, after the attachment handler has optionally extracted data, but before the source filter/transformer. It can modify the incoming message.
  • Post Processor Script. Runs once for every message, after the source connector and all destinations have completed (excluding asynchronous processes like the destination queue). Has access to responses from all executed destinations and can return a custom response for the source connector to use.
  • Undeploy Script. Runs once after the channel has been stopped and its connectors undeployed.

Global scripts of the same four kinds run around these for every channel. See Global Scripts.

How a message moves ​

OIE ChannelresponseSourceSystemDestinationSystem(s)Source ConnectorPreprocessorSource Filter/TransformerDestination 1 Filter/TransformerConnector 1Response Transformer 1Destination 2 Filter/TransformerConnector 2Response Transformer 2Postprocessor

Destination 2 follows destination 1 in the diagram because it waits for the previous destination, which is the default. With that setting off, each destination starts a chain of its own and the chains run concurrently.

Neither end of that diagram has to be outside OIE. A Channel Reader source receives from Channel Writers on other channels, a Channel Writer destination hands the message to another channel on the same engine, and a script can dispatch to one through router.

A message moves through these steps:

  1. The source connector receives the message.
  2. The attachment handler, if one is configured, pulls content out of the message and leaves a reference in its place.
  3. The preprocessor script can modify the raw message before anything parses it.
  4. The source filter decides whether the message goes on, and the source transformer converts and extracts from it. A message filtered here reaches no destination.
  5. For each destination: its filter and transformer run, then its connector runs, which usually sends the message, either now or later from its queue.
  6. That destination's response transformer handles whatever comes back.
  7. The postprocessor script runs once every destination has finished, and can build the response itself.
  8. If something is waiting on the other end, the response goes back to it. A listener source such as an HTTP, TCP, or Web Service Listener returns it to the caller. A polling source such as a Database Reader or File Reader has no caller, so the response is selected and stored on the message but sent nowhere.

Four pages explain the mechanisms behind those steps:

The Glossary collects the terms these pages use.

Administration interfaces ​

  • Administrator. A Java Swing desktop client for managing channels, viewing messages, configuring settings, and monitoring the system. Connects to the server over HTTPS using a RESTful API. Requires a compatible launcher; see Desktop Administrator for the available options.
  • Web Administrator. A browser-based administrator. It is a separate application that uses the same REST API as the desktop client.
  • Command Line Interface (CLI). A terminal-based client for scripting and headless administration. Uses the same REST API as the Administrator.

Extensions ​

OIE uses a plugin architecture where connectors, data types, and other features are loaded as extensions. Extension types include:

  • Connectors. Source and destination communication protocols
  • Data Type Plugins. Message formats and the settings the engine uses to serialize and deserialize them
  • Server Plugins. Background services and event handlers
  • Transmission Mode Providers. Frame-level protocol handling (e.g., MLLP)

Extensions are loaded from the extensions directory when the engine starts. An extension can ship a desktop UI, a web UI, or both; the web administrator serves the web side. Open source extensions are available beyond those bundled with the engine. See the Plugin Guide, and openintegrationengine.org/plugins for the community catalog.