Integrations

Integration is the process of bringing together various types of software sub-systems so that they create a unified single system. Software integration can be required for a number of reasons, such as: Migrating from a legacy system to a new database system, including cloud-based data storage.

There are numerous different ways to pass data around, by choosing the correct integration tool for the job rather than making one tool fit all provides the most agility and will be where you see the effort pay off. At Data Engineers we have expertise in changing the way your integrations work so they take advantage of modern cloud based technologies. Below are some of the Azure based components which we develop on a regular basis.

Azure Function Apps

Azure Functions is a serverless solution that allows you to write less code, maintain less infrastructure, and save on costs. Instead of worrying about deploying and maintaining servers, the cloud infrastructure provides all the up-to-date resources needed to keep your applications running.

We often build systems to react to a series of critical events. Whether you're building a web API, responding to database changes, processing IoT data streams, or even managing message queues - every application needs a way to run some code as these events occur.

At Data Engineers, we have developed function apps to for all stages of the integration platform, whether just to use it as a façade to downstream systems, data exposure for 3rd party access, message handline and web application middleware.

Azure Logic Apps

Azure Logic Apps is a cloud-based platform for creating and running automated workflows that integrate your apps, data, services, and systems. With this platform, you can quickly develop highly scalable integration solutions for your enterprise and business-to-business (B2B) scenarios. As a member of Azure Integration Services, Azure Logic Apps simplifies the way that you connect legacy, modern, and cutting-edge systems across cloud, on premises, and hybrid environments.

Some of the Logic App use cases include:

• Performing a function on a schedule
• Route and process messages across cloud services
• Moving files and processing files in Azure Storage

At Data Engineers, we have a firm understanding of their use and limitations having developed with Logic Apps since their inception many years ago. We have developed both simple and complex cases where visibility of the data flow is required.

 Azure Service Bus

Azure Service Bus is a fully managed enterprise message broker with message queues and publish-subscribe topics. Service Bus is used to decouple applications and services from each other, providing the following benefits:

• Load-balancing work across competing workers
• Safely routing and transferring data and control across service and application boundaries
• Coordinating transactional work that requires a high-degree of reliability

At Data Engineers, we have re-developed point-to-point integrations to become publish-subscriber models using the Azure Service Bus as the messaging platform.

 Azure Event Grid

Azure Event Grid is a highly scalable, serverless event broker that you can use to integrate applications using events. Events are delivered by Event Grid to subscriber destinations such as applications, Azure services, or any endpoint to which Event Grid has network access. The source of those events can be other applications, SaaS services and Azure services.

With Event Grid you connect solutions using event-driven architectures. An event-driven architecture uses events to communicate occurrences in system state changes, for example, to other applications or services. You can use filters to route specific events to different endpoints, multicast to multiple endpoints, and make sure your events are reliably delivered.

At Data Engineers, we have extensive used the event grid to help push data from azure storage accounts through to snowflake.

 Azure Storage Accounts

Azure Storage is Microsoft's cloud storage solution for modern data storage scenarios. Azure Storage offers highly available, massively scalable, durable, and secure storage for a variety of data objects in the cloud. Azure Storage data objects are accessible from anywhere in the world over HTTP or HTTPS via a REST API.

At Data Engineers, we make extensive use of the following Azure Storage services:

Blog

Allows unstructured data to be stored and accessed at a massive scale in block blobs. We utilize Azure Blogs within our DevOps / DataOps processes as well as for helping transfer data to snowflake or as a external storage location for semi-structured data.

Files

Offers fully managed cloud file shares that you can access from anywhere via the industry standard Server Message Block (SMB) protocol. We utilize Azure Files when needing to share resources across multiple Azure Virtual Machines instead of having to rely up replication services.

Tables

Allow you to store structured NoSQL data in the cloud, providing a key/attribute store with a schema less design. We utilize Azure Tables as a configuration storge for application configuration and state which can change over time.

Queues

Allows for asynchronous message queueing between application components. We utilize Azure Queues when we use snowpipe within Snowflake as the triggering mechanism for data ingestion operations.

How we can help?

We can become an extension of your existing development team or work independently in conjunction with your needs doing as much or as little as you need us to do. We are able to do architectural designs through to development, peer reviewing and deployments into your environment.