Drop folders service for content ingestion

Using a drop folder

Drop folders are used to automate the ingestion of content into the Kaltura platform. The drop dolder service constantly watches each drop folder for new content and activates automatic ingestion of new content to the specific account.

Kaltura offers multiple configuration options for setting each drop folder to a specific workflow.  Contact your Kaltura account manager to learn more about how drop folders can simplify your workflow, and activate and configure drop folders on your account.

Use the Kaltura drop folder service to:

• Automate or partially automate an ongoing content ingestion workflow.
• Support bulk ingestion of content to Kaltura, either when you set up an account and want to migrate a large amount of existing content, or for use on an ongoing basis.
• Work with content providers and operational teams who do not have access to the Kaltura Management Console (KMC) or any other content management application.

Kaltura’s drop folder service is designed to support a variety of integration requirements and business needs and has flexible configuration options. 

Drop folder locations

Hosted on the Kaltura cloud

When account-specific drop folders are located on the Kaltura cloud, they are fully hosted and managed by Kaltura. Publishers are granted FTP/SFTP Aspera access to one or more account-specific drop folders. Publishers are responsible for uploading their content to the drop folders. Kaltura recommends this option for publishers who prefer a fully-managed drop folder solution, and for publishers whose workflow and security needs are better met by this option.

Hosted on-premise by the publisher

Kaltura defines a drop folder configuration with the drop folder location and access details. The publisher locates account-specific drop folders at the publisher's premises or on any FTP/SFTP-accessed location.  A Kaltura drop folder 'watcher' constantly checks the drop folder. When content is ready to be ingested, it is downloaded from the drop folder to the Kaltura cloud. Kaltura recommends this option for publishers who already have a network location that they want Kaltura to watch and use for ingestion, and for publishers whose workflow and security needs are better met by this option.

Drop folder configuration options

KMC administrators can view and delete media files in the account drop folder on the Drop Folders Control page. The drop folder input package may be a single source file or a complete media package including multiple flavors and metadata xml. You can create a new entry based on the input package or add it to an existing entry.

Drop Folder configuration options:

Transcoding profile

The transcoding profile attached to the drop folder settings defines the transcoding settings required for content ingested from the drop folder. Consequently, the transcoding profile also determines whether media files in the drop folders are handled as a single file media source or as part of a media package. Kaltura fully transcodes a single file media source, while a media package consists of multiple bit rate transcoding flavors that are generated by a local transcoder. A media package is added from the drop folder to a single media entry as one unit.

File name patterns

By default, drop folders handle all file name patterns. A publisher can define a specific drop folder file name pattern to be handled by the drop folder (white list approach). The file name pattern should be configured when you need to apply a drop folder operation only to files of a specific type or naming structure.

A publisher can define a specific drop folder file name pattern to be ignored by the drop folder (blacklist approach). The file name pattern should be configured when you need a drop folder operation to ignore files of a specific type or naming structure. This option is useful when transferring temporary files created by transfer client applications when uploading files to the drop folder. The temporary files usually use a specific name pattern or extension. You can mark to ignore a specific name pattern or extension by the drop folder service.

Ingestion types

Content ingestion from a drop folder can be triggered in two ways:

Media files: Only media files are placed in the drop folder for ingestion. An automatic process uploads these files to Kaltura. Metadata editing is conveniently done in the KMC. This type requires additional configuration options for specific policies in the drop folder profile.

Kaltura MRSS-formatted XML: In this method, a Kaltura MRSS-formatted XML, containing both metadata and references to related media files, is placed in the drop folder. The XML may include metadata for multiple media items and references to media files related to each item, which can be located within the drop folder or on a remote location. An automatic process is triggered to create or update media entries in Kaltura, populating the metadata fields from the XML and processing the ingestion and transcoding of referenced media files. If media files are within the drop folder, ingestion starts only when all files referenced in the XML are present.

To learn more, please refer to What is bulk upload and FTP content ingestion?

Ingestion from media files - matching policies

When a drop folder is set to ingest content from media files, one of the following matching policies should be selected:

Create an entry from media

For automatic content creation, media files in the drop folder always generate a new entry in Kaltura. This is handy for users who don’t have access to the KMC or for users who are not using the KMC to upload media files. Use this when you want files uploaded automatically and metadata edited afterward.

1. Drop a media file into the drop folder. A new entry in Kaltura is automatically created. This triggers an automatic conversion in  Kaltura according to the transcoding profile settings attached to the drop folder. The title of the new entry is based on the file name.
2. Edit the new entry metadata in the KMC as needed. A process to automatically replace the media cannot be triggered from a drop folder configured to support this policy.

Match media to an existing entry

In the drop folder, media files are automatically matched to existing entries in Kaltura using a Reference ID present in the file name. Ensure the Reference ID matches an entry's metadata item. This policy suits dynamic environments needing simultaneous metadata and media handling.

1. Prepare a new entry in the KMC.
2. Set metadata for the new entry and set its Reference ID in the Edit Entry's Metadata tab. Reference ID example: MyVideoClip123
3. Drop a media file in the drop folder and include the Reference ID in the filename. For example MyVideoClip123.mp4

The MyVideoClip123.mp4 file in the drop folder automatically will be added to an entry with the “MyVideoClip123” string as a Reference ID metadata item. This triggers an automatic conversion in Kaltura according to the Transcoding Profile settings associated with the drop folder. When there is no entry in the system with a matching Reference ID, the media files remain in the drop folder until a matching entry is created.

When the matched entry is already associated with media, a process is triggered that automatically replaces the media. 

Match media when entry exists or create a new entry

When a media file in the drop folder lacks a matching Reference ID in Kaltura, the system creates a new entry. The file is then added to this entry. This policy suits scenarios where automatic media upload, post-upload metadata editing, and triggering media replacement processes from a drop folder are necessary.

Ingestion from Media Files - matching a regular expression

When a drop folder is set to match media files to an existing media entry, the file name must include the Reference ID that is used as a matching key.  By default, the file name itself is treated as a Reference ID.  

Examples: (the Reference ID is red):

MyVideoClip123.mp4  

When an account is configured to support the use of a local transcoder and ingestion of multiple bit rate media files to a single media entry, file naming convention must be defined to also include the system name of a specific flavor in the transcoding profile that this file should be matched to.

In the following examples, all media files are added to a single entry and each file is matched to  the appropriate transcoding flavor, according to its system_name definition in the transcoding  profile  (the Reference ID is red, the transcoding flavor system name is blue):

MyVideoClip123_HD.flv

MyVideoClip123_Basic-Small.flv

MyVideoClip123_Basic-Large.flv

Drop folder file management policies

For drop folders hosted on-premise, publishers can define a file deletion policy. This policy determines if files are automatically deleted by the Kaltura drop folder service after successful ingestion or manually by the publisher via the KMC.

When selecting automatic deletion, publishers must specify the number of days before files are removed, limited to successfully processed files only.

For drop folders hosted in Kaltura, files are automatically deleted shortly after successful processing. Manual deletion can be performed via the KMC or Kaltura API, though direct manual deletion from the folder is not recommended due to potential synchronization issues.

In both cases, any change to a file in the drop folder resets its handling, treating it as a new file.

Drop folder access settings

When a drop folder is managed by Kaltura and is hosted on the Kaltura cloud, Kaltura’s technical support team sets the drop folder location and provides full access and authentication information to the publisher.

When a drop folder is hosted and managed at the publisher's premises, the publisher must provide the following information, per preferred transfer protocol, to Kaltura, to set up the drop folder configuration:

Drop folders control panel

In the KMC Content menu, KMC administrators can use a control panel/table to monitor and troubleshoot media files in drop folders. Access to the Drop Folders control panel can be granted to specific administrators through the administrator's user role definition. 

The Drop Folders control panel includes the current processing status of each file in the drop folder, and an option to filter the file display for a specific drop folder, status, and creation date range. For files that completed their processing, a link to the entry editing window is available as well. 

An additional option to manually delete files from the drop folder can be granted separately only to authorized administrators. 

Drop folders table

The drop folders files table in the Drop Folder Control page lists all media and/or XML files currently available in the account’s drop folders. The current details of each file are available within the table. In the filter bar on the left side of the page, you can filter the files by:

• Search criteria
• Drop folder
• File creation date
• Drop folder File Processing status

If you need to delete files from the drop folder, click Delete from the 3 dots menu.

Drop folder files statuses

The following statuses describe the different stages and possible error cases that may apply to a specific file in the drop folder:

• Parsed from XML – A reference to the media file was parsed from the XML. The media file is expected but is not yet available in the drop folder.
• Uploading – File is currently uploading to the drop folder.
• Pending – File upload is completed - waiting for drop folder processing to start.
• Waiting for Related Files (Formerly “Matched”) – Processing of this file should continue when all related media files are available in the drop folder (i.e. other files referenced in the same XML or other flavors expected for the same entry).
• Waiting for Matched Entry (Formerly “Not Matched”) – Processing of this file should continue when all related files are available in the drop folder (for example, XML file, other media files referenced in the same XML, or other flavors expected for the same entry).
• Processing - Drop folder engine is currently processing this file.
• Downloading - The file is currently importing from the drop folder to Kaltura. (applicable only in drop folders hosted on-premise by customers)
• Done – Drop folder processing was completed for this file.
• Error – Drop folder engine processing failed.
• Download  Failed - Failed to import the file from the drop folder to Kaltura (applicable only in drop folders hosted on-premises by customers).
• Delete Failed - Failed to delete the file from the drop folder.

Unsupported file name characters

When uploading to a drop folder, it's important to note that there are unsupported characters in file names that should be avoided when uploading files. Below you can find a full list of all unsupported characters:

 For more information on the XML Custom data upload see the following:
http://www.kaltura.com/api_v3/xsdDoc/?type=dropFolderXmlBulkUpload.dropFolderXml