Rebuilding the Archive Index

This article describes the purpose of, and procedure for, rebuilding the Microsoft Azure AI Search index used by the Straatos Archive.

Overview

The Straatos Archive search index is continuously updated whenever documents are added to or deleted from the archive. These changes are synchronised between the SQL database and the Azure AI Search index.

Over time, certain situations may require a full rebuild of the index:

• Configuration changes:

  • In the Straatos Archive (for example, removing index fields for specific document types). In this case, existing documents are not automatically updated in the search index—only newly added documents reflect the new configuration.

• Manual changes:

  • Made directly in the SQL database related to search data. These changes are not synchronised back to Azure AI Search.

In such scenarios, the search index may contain outdated or unnecessary data, making it larger and less efficient. A full rebuilding of the index is required to bring the search index back into a consistent state.

---

Permissions and Prerequisites

• The user must be authenticated.

• The user must have tenant administrator permissions.

• Ensure sufficient storage quota is available in Azure AI Search.

During rebulding, the existing index remains active while a new index is built in parallel. As a guideline, rebuilding requires approximately the same storage capacity as the current index. Depending on configuration changes, the required storage may be higher or lower.

---

Starting a Rebuild

The rebuild is triggered via the following API call:

curl -X 'POST' \
  'https://straatosv2-eu-api.cumuluspro.net/API/Archive/RebuildIndex/{tenantId}' \
  -H 'accept: */*' \
  -d ''

Replace {tenantId} with the appropriate tenant identifier.

---

Behaviour During Rebuilding of the index

• Users can continue to search and retrieve documents as usual.

• No documents can be added or updated in the archive while rebuild is in progress.

• Rebuilding of the index can take a significant amount of time.

As a guideline, rebuilding a 10 GB search index takes approximately 12–13 hours.

---

Monitoring the rebuild Progress

You can monitor the progress and logs of the rebuild process using the following API call:

curl -X 'GET' \
  'https://straatosv2-eu-api.cumuluspro.net/API/Archive/RebuildLogs/35' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer ey...'
Request URL

This endpoint returns the current status of the index rebuild and any errors encountered.

Example Log Output (Successful Rebuild)

[
  {
    "id": 58,
    "timestamp": "2025-12-13T09:26:48.4246839Z",
    "message": "Indexing finished, please manually remove the old search index, or resume the rebuild if it was not fully finished",
    "isError": false,
    "searchIndexDefinitionId": 20
  },
  {
    "id": 57,
    "timestamp": "2025-12-13T09:26:48.4173321Z",
    "message": "Index rebuilding finished, new index activated, index id 20",
    "isError": false,
    "searchIndexDefinitionId": 20
  },
  {
    "id": 56,
    "timestamp": "2025-12-13T09:26:48.317855Z",
    "message": "Rebuilding index done, now activating for use, index id 20",
    "isError": false,
    "searchIndexDefinitionId": 20
  },
  {
    "id": 55,
    "timestamp": "2025-12-12T20:24:34.6127758Z",
    "message": "Indexing (re)started, parallel threads: 10",
    "isError": false,
    "searchIndexDefinitionId": 20
  }
]

If the rebuild completes successfully, the new index is automatically activated. If errors occur, the new index is not activated and the existing index remains in use.

---

Listing Search Indexes

To list the search indexes known to the Straatos Archive, use the following API call:

curl -X 'GET' \
  'https://straatosv2-eu-api.cumuluspro.net/API/Archive/ListIndexes/{tenantId}' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer ey...'

This returns the indexes as recorded in the Straatos database.

This API does not validate the actual indexes present in Azure AI Search. Additional indexes may exist in Azure AI Search that were created manually or not cleaned up correctly.

Example Response

[
  {
    "id": 20,
    "indexName": "archiveprod20251212-1",
    "createdAt": "2025-12-12T20:24:33.105566Z",
    "rebuildStarted": "2025-12-12T20:24:33.1056359Z",
    "latestRebuildId": 8642440,
    "rebuildFinished": "2025-12-13T09:26:48.4020132Z",
    "active": true
  },
  {
    "id": 18,
    "indexName": "archiveprod20250505-1",
    "createdAt": "2025-05-05T16:09:51.842953Z",
    "rebuildStarted": "2025-05-05T16:09:51.8430506Z",
    "latestRebuildId": 6962182,
    "rebuildFinished": "2025-05-06T01:54:13.3186574Z",
    "active": false
  }
]

In this example, two indexes exist. The active flag and timestamps indicate which index is currently in use and which one is older.

---

Cleaning Up Old Indexes

After the new index has been tested and confirmed to work correctly, the old index must be removed manually to free up Azure AI Search storage quota.

Delete the Index in Azure AI Search

1. Open the Azure Portal.

2. Navigate to the Azure AI Search service.

3. Delete the index that is no longer active.

Remove the Index from the Straatos Database

After deleting the index in Azure AI Search, remove the corresponding entry from the Straatos SQL database.

Retrieve the existing index entries:

SELECT  * FROM [archive].[SearchIndexDefinition]

Delete the unused index entry:

DELETE FROM [archive].[SearchIndexDefinition] where [Id] = {your id}

Replace {your id} with the ID of the index you want to remove.