File streaming: Transferring large files with ease

Transferring large files in Enterprise

Enterprises often have the need to transfer large files of few hundred megabytes from one system to another. Some popular use cases include:

  • Move daily data dump files from database or ERP (such as NetSuite) into a file storage system (such as Amazon S3)
  • Transfer files from multiple sources into one place for another app to process. For example, consolidating data files of sales and marketing into a SFTP folder, then import to Salesforce
  • Back up the company’s files and save into another file storage system

Traditional approaches to file transfer usually have a small file size limit or require large server for processing. With Workato, the File streaming mechanism allows you to transfer large files between different systems easily, without any setup.

Streaming: Transferring large files easily

Streaming is Workato’s solution to transfer large files between 2 applications. Let’s see how it works.

First, let’s take a look at the traditional approach to file transfer. The Integration Server downloads the whole file from the Source App, saves it in memory, then uploads the file to Destination App.

Traditional File Transfer Mechanism
Figure 1: Traditional File Transfer Mechanism

The disadvantage of this method is that file size will be limited by server’s memory. And because 1 server handles multiple file transfers at one time, it may only allocate memory as low as 20MB for each file. To transfer large files of few hundred megabytes, you will need to pay for a much larger server!

Workato solves this problem by cutting a large file into smaller chunks, then “streaming” 1 chunk at a time to the destination. This allows Workato server to technically transfer files of unlimited size, and you don’t need to pay that extra cost to move your data.

Workato File Streaming Mechanism
Figure 2: Workato File Streaming Mechanism

 

How to use Streaming in Workato

At the moment, we have implemented Streaming for transferring file from Amazon S3 to SFTP. The Streaming mechanism happens under the hood, so you don’t need to do any extra setup. Try out this sample recipe with files of up to 300MB.

Streaming Recipe Example
Figure 3: Workato recipe to transfer file from Amazon S3 to SFTP

We are rolling out Streaming to the rest of file triggers & actions in Workato. The upcoming ones are SFTP, Box, Dropbox, Service Now, etc. Note that in this first implementation of Streaming, we have a job run time limit of 30 minutes, so you can transfer files of up to about 300MB.

In the future iterations, we aim to remove this file size limit completely by dividing the chunks into several jobs. By then you will be able to transfer file of unlimited size between any app.

Stay tune for future product updates. And head out to our Workato community page for more automation use cases!

Workato Community Recipes