Straatos Cloud Archive

1. Introduction

The Straatos Cloud Archive provides a means to store- and retrieve documents and metadata in an archive. The data can be archived in the customers Azure environment, or alternatively, CumulusPro can host the archive in our data centers. This document describes how the Straatos Cloud Archive can be configured in the customers Azure cloud environment.

The archive uses an Archive Connector that connects to a SQL database, Azure AI Search and Azure Storage and uses an Azure Key vault to store all secrets- and passwords.

Via the CumulusPro BPM platform (workflow) the documents can be uploaded to the archive via our CumulusPro Archive Connector. Retrieving documents can be done via our CumulusPro Web User Interface (TaskUI) and/or via a secure REST API.

This documentation does not provide network or application-level security recommendations. Please adhere to your company's policies and recommendations regarding security, or refer to the Microsoft Azure Security frameworks and recommendations. The content here is specifically concentrated on the setup and configuration of each component, excluding general security measures.

2. Diagram of component interaction

API Management Service is optional. Alternatively, an Application Gateway or direct access to the Web App Service can be used.

3. Creating all Azure Resources

The following resources are required and have to be created in the customers Azure environment, and it is highly recommended to create all azure resources in the same Azure region (example West Europe):

Azure Resource Group
Azure Web App Service
Azure AI Search
Azure Key Vault
Azure Storage Account
Azure SQL Database
Azure API Management

3.1 Azure Resource Group

A resource group is a container that holds related resources for an Azure solution. The resource group can include all the resources for the solution, or only those resources that you want to manage as a group. Below is an example of a resource group - and you can choose your own name for this group:

3.2 Azure App Service

The Straatos Archive Connector runs on an Azure Web App Service which you can create in the Azure Portal. When creating the app service, ensure the app service uses the following configuration:

Create the Web App Service in the resource group you have created

Name	Web App Service Name (example above: webapp-cpro-archive )
Publish	Code
Runtime Stack	.Net 7 (STS)
Operating System	Windows
Region	Same region as the other resources (example above: West Europe )
Pricing Plan	The size of the Archive Connector service depends on the usage (example: the amount of documents that are stored and retrieved). We recommend starting with a Standard S2 plan and adjusting the sizing based on peak usage for optimal performance.

3.3 Azure AI Search

Azure AI Search provides secure retrieval of documents that are stored in the archive. When creating this component - please use the following recommended settings:

To ensure both the ability to write to the AI Search engine and conduct data searches, we recommend starting with, at minimum, the Standard 25GB/Partition plan (see below)

3.4 Azure Key Vault

Azure Key Vault is a cloud service that provides a secure store for secrets. You can securely store keys, passwords, certificates, and other secrets. Azure key vaults can be created and managed through the Azure portal.

The Key Vault needs to be in the same region as the other Azure resources created as part of the Archive

Access Configuration / Permission model

Networking

3.5 Azure Storage Account

The Azure storage account contains all archived documents. When creating this component - please use the following recommended settings:

3.6 Azure SQL Database

The Azure SQL Database is used to store all metadata / indexfields of all stored documents. Also, it contains the setup configuration of the Archive. You can make use of an existing database server in your own subscription. Both vCore and elastic pool databases are supported. A database should be created as follows:

Create a separate database for the archive - and give this database a convenient name (example: CPROArchive )
Create a database scheme (example: Archive )

Create a user and a secure password for this user
Connect the user to the database scheme - and give this user DBOWNER rights on the created database
Create a database connection string - used for configuration of the archive:
Example: Data Source=tcp:customer-europe.database.windows.net,1433;Initial Catalog=<databasename>;User Id=<username>;Password=<secure password>

3.7 Overview of all created Azure Components

Below is an overview of all components that you have created:

4. Configuring all Azure Resources

This chapter will describe the complete configuration of all Azure components in the customer Azure environment.

4.1 Azure App Service configuration

The complete archive setup/configuration is done in the App Service - and for this - you need the archive connector zip file (ArchiveConnector.zip).

This ZIP file will be provided by CumulusPro and please contact CumulusPro to obtain the latest version of this ZIP file (support@cumuluspro.com).

After you have received this file - please follow the following steps:

Unzip the file ArchiveConnector.zip
Deploy the ZIP file in the Azure App Service as following:
Go to the settings of your Azure Web App and open Advanced Tools
Click on "Go" and a new web browser TAB will be opened

Click Debug console > CMD

Go to directory homesitewwwroot

In this directory (homesitewwwroot), the ZIP file should be deployed
You can do this by "Drag and Drop" the ZIP file into this directory
The ZIP file will be automatically extracted into the wwwroot folder

After the ZIP file is deployed - you can edit / modify the appsettings.json file. The purpose of this appsettings.json file is to configure the complete archive setup when there is no key vault available at the customer site.

It is not recommended to configure settings directly in the appsettings.json file; instead, it is advisable to utilize key vault or app service configuration, or a combination of methods. Before deploying a new archive connector, ensure that you create a backup copy of the appsettings.json. If you choose not to set up the settings in appsettings.json, make sure all settings are commented. The section below describes how to do this.

Edit the appsettings.json file

Make sure the different sections are commented "out" by adding the underscore ( _ ) before each setting:

Note: Do this for every section- and key in the appsettings.json file

Web App Configuration

Select "Configuration" (see below)
Add a new configuration setting by clicking on "New application setting"

Create the following Web Application settings:

Name: Database:CreateUpdateDatabase
Value: true

Name: KeyVault:Name
Value: keyvault-cpro-archive

Note: please use your own created key vault name value

Name: StorageAccount:StorageName
Value: storagecproarchive

Note: please use your own created storage account name value

The value of the SearchIndex:IndexUri can be found in the Azure AI Search

Name: SearchIndex:IndexUri
Value: https://search-ai-cpro-archive.search.windows.net

Name: SearchIndex:IndexName
Value: cprosearchindex

Note: please use your own created Index Name value

Setup Managed Identity on the Web App

To enable the Managed Identity on the Web App:

Select Identity (see below)
Set Status to On.

4.2 Azure Key Vault configuration

Key vault access always uses Managed Identity and below is a description of how to set up Azure role-based access control on the Azure Key Vault

Select the Azure Key Vault and click on "Access configuration"
Select "Azure role-based access control (recommended)"

Go to Access control(IAM) and click on Add > Add role assignment.

Click on Job function roles and search for Key Vault Reader from the search bar. Select Key Vault Reader and click on the Next button.

From Members section, click on Select members and a window on the right appears to select members. Search for the Azure Web App you have created

Click on "Review + assign" button

Repeat the "Add role assignment" for the role "Key Vault Secret User"

Add the database connection string to the Key Vault

Note: In order to create any secrets in the Key Vault - be sure that you have enough user rights

Name: ConnectionStrings--ArchiveContext
Secret value: Data Source=tcp:mydb.database.windows.net,1433;Initial Catalog=the-archive-catalog;User Id=mydbaccount;Password=verysecret

The database user specified should have the following access rights on the database:

DDL writer/reader access because it will create the database upon application startup.

If needed, these access rights can be revoked after the first (successful) startup.

4.3 Azure AI Search configuration

Azure AI Search uses Managed Identity and below is a description of how to set up Azure role-based access control on the Azure AI Search

Select the AI Search and go to Access Control (IAM) and click on Add > Add role assignment.

Add the role "Search Index Data Contributor"

Add the role "Contributor" role (under tab Privileged administrator roles)

Change the "Keys" setting of the Cognitive Search resource
In the search service, click on "Keys" and select "Role-based access control"

4.4 Azure Storage Account configuration

The Azure Storage Account is responsible for storing all documents in the archive. Therefore, the following settings must be applied:

Create a new container in the Storage Account (example: cparchive)

Add the new created Azure Storage Account Container to Azure Web Application settings:

Select the Web App and click on the Configuration and add a New application setting

Name: StorageAccount:ContainerName
Value: cparchive

The Azure Storage Accounts also work with Managed Identity and you have to set the correct role rights so the Azure Web App can have access to the Azure Storage container you have just created:

Select the Storage account and go to Access Control (IAM) and click on Add > Add role assignment.

Add the role "Storage Blob Data Contributor"

4.5 Azure Service Bus configuration

NOTE: The Azure Service Bus will be configured in the CumulusPro Azure environment - so CumulusPro Professional Services will take care of the settings below

Create two new Queues in the Azure Service Bus - the "archive-result" is a mandatory queue and must have this name:

Select the Azure Service Bus and click on "Queues"
Add a new Queue named "archive-result"

Add another queue - you can use your own name:

Add a new Queue named "archivecpro"

The following queues are now created:

Add the new created Azure Servicebus queue to the Azure Web Application settings:

Select the Web App and click on the Configuration and add a New application setting

Name: ServiceBus:QueueName
Value: archivecpro

Service bus Shared Access Policy

Create a Share Access Policy on the service bus:

Select the Service Bus Namespace and click on the Shared access policies and add a New SAS Policy
Add the access policies "Send" and "Listen"

Add the "Primary Connection String" to the Key vault

Click on "sharedcpro" to open the configuration
Copy the "Primary Connection String" - needed for the Key vault setting

Select the Key vault and click on the Secrets and add a new secret by clicking on "Generate/Import"

Create a secret

Name: ServiceBus--ConnectionString
Secret value: the Primary Connection String you copied earlier

Example: Endpoint=sb://servicebuscproarchive.servicebus.windows.net/;SharedAccessKeyName=sharedcpro;SharedAccessKey=......

You will now have the following secrets in the Key Vault:

4.6 API Management Configuration

If the Azure API Management is used, then the following API calls to the archive connector need to be configured. The curl commands below do not include the body or the Authorization (Bearer Token). Highlighted parts are parameters that change between calls.

curl -X 'PUT' 
  'https://<your domain>/API/Document/<documentId>/SetLegalHold' 
  -H 'accept: */*'

curl -X 'PUT' 
  'https://<your domain>/API/Document/<documentId>/UnSetLegalHold' 
  -H 'accept: */*'

curl -X 'DELETE' 
  'https://<your domain>/API/Document/<documentId>/DeleteDocument' 
  -H 'accept: */*'

curl -X 'GET' 
  'https://<your domain>/API/File/<file>' 
  -H 'accept: */*'

curl -X 'GET' 
  'https://<your domain>/API/Index' 
  -H 'accept: application/json'

curl -X 'POST' 
  'https://<your domain>/API/Index/AddField' 
  -H 'accept: */*' 
  -H 'Content-Type: application/json-patch+json'

curl -X 'POST' 
  'https://<your domain>/API/Index/RemoveField' 
  -H 'accept: */*' 
  -H 'Content-Type: application/json-patch+json'

curl -X 'POST' 
  'https://<your domain>/API/Index/RemoveDocumentField' 
  -H 'accept: */*' 
  -H 'Content-Type: application/json-patch+json'

curl -X 'POST' 
  'https://<your domain>/API/Index/RebuildIndex' 
  -H 'accept: */*' 
  -H 'Content-Type: application/json-patch+json'

curl -X 'GET' 
  'https://<your domain>/API/Index/ListIndexes?definitionId=<definitionId>' 
  -H 'accept: application/json'

curl -X 'POST' 
  'https://<your domain>/API/Index/ActivateIndex?definitionId=<definitionId>' 
  -H 'accept: */*'

curl -X 'POST' 
  'https://<your domain>/API/Index/CancelRebuild' 
  -H 'accept: */*'

curl -X 'POST' 
  'https://<your domain>/API/Index/ResumeRebuild?definitionId=<definitionId>' 
  -H 'accept: */*'

curl -X 'GET' 
  'https://<your domain>/API/Index/IndexingLogs?limit=<limit>&order=<desc/asc>' 
  -H 'accept: application/json'

curl -X 'POST' 
  'https://<your domain>/API/Search' 
  -H 'accept: application/json' 
  -H 'Content-Type: application/json-patch+json'

5. Test your configuration

After you have configured all Azure resources - you can test if the Archive is working:

Select the Web App and click on "Advanced Tools"
Click on "Go" to open the Kudu Services

Open a Command Shell by selecting "CMD"

Go to the "LogFiles" directory

Open the "eventlog.xml" by clicking on the "pencil"

Clear this eventlog by deleting the content and saving this file

Select the Web App and click on "Overview"
Restart the Web App by clicking on "Stop" and "Start"

If there are no error messages, you can test the complete configuration by opening a browser and using the following URL:

https://webapp-cpro-archive.azurewebsites.net.azurewebsites.net/health

In case of a successful deployment, you will get the following response:

6. Straatos Configuration

This chapter describes the actions that have to be done by CumulusPro.

Create ArchiveDefinition, servicebus queues and define key vault connections

As an Organisation Admin, create an ArchiveDefinition, with an archiveconnector url pointing to the ArchiveConnector.

The following has to be done by a CumulusPro Consultant

For each archive definition you want to use, create an entry in the Straatos key vault called “<archiveName>-ServiceBusConnectionString”

The <ArchiveName> is the name provided when you created the archive in StraatosV2 under /IAM/Tenants/SetArchiveDefinition/{tenantId}.

Recommended is to use a separate servicebus for each ArchiveConnector out in the wild, but that is not a requirement, you can share a Service Bus, but beware that tokens to fetch archive data are put on the queue.
In this Service Bus, two queues should exist, one named after the archivedefinition name, but in lower case and all special characters removed (if they were used). The other called “archive-result”.

For StraatosAdapter to correctly create tokens, a Key should be created in the Key Vault called TokenKey

StraatosV2 should have Get and List access to the Keyvault configured. For Cryptographic Operations, it also needs Sign permissions.

Create your own Knowledge Base