Define what people need to find
This reference design brings records from several enterprise systems into a custom search application. People need to find an exact identifier, narrow a broad query, and open the authoritative record. A browser interface and application API sit above a derived search index.
Assume authenticated users, source-specific permissions, changing schemas, and finite indexing delays. Agree on searchable fields, permitted previews, freshness, and source ownership. The index helps people discover records; it does not become the system of record.
Treat indexing as a maintained copy
Connectors capture additions, updates, deletions, and permission changes. Documents keep a source-scoped identifier, source link, content version, indexing time, and access metadata. Upserts use stable identifiers. Reject obsolete source versions so a delayed retry cannot restore stale content or permissions. Advance a connector checkpoint only after its writes are acknowledged; retries must not create another copy of the record.
Content timestamps may miss permission-only changes. Use an explicit permission-change feed or periodic reconciliation, and monitor its lag separately. Deleted records need versioned tombstones or a comparable removal process, retained long enough to prevent delayed updates or rebuilds from restoring them. Rerunning a full import does not necessarily remove records that disappeared from the source. Quarantine malformed records with an owner and a recoverable reason.
Give the interface a small, useful API
The interface offers a search box, relevant filters, clear source labels, and links back to original records. The backend accepts an approved request shape: query text, filters, sorting, and bounded pagination. It validates field names, fixes the returned fields, enforces timeouts and resource limits, and builds the engine query.
Exact identifiers and free-text phrases may need different matching rules. Return snippets only from permitted fields. Distinguish an empty result from a timeout or incomplete source coverage. A count represents the permitted search scope, not necessarily everything held across the enterprise.
Authorize the full response
The API derives identity and access scope from the authenticated session. Apply that scope to result retrieval, direct document fetches, previews, exports, facet counts, and suggestions. Hiding a result card after retrieval leaves other ways to expose its contents or existence.
Document-level security is one mechanism with product-specific limitations. Test every enabled response surface and combined-role policy. Disable unsupported facets, suggestions, or statistics endpoints instead of assuming they inherit safe behavior. Cached results must respect current access. If permission changes require immediate effect, consult the current permission authority or suspend the affected search scope until its index is safe to serve.
Measure whether the search is useful
Keep a reviewed set of representative queries: exact IDs, abbreviations, ambiguous terms, and legitimate no-match cases. Check expected records, ranking, filters, and access-denied cases when analyzers, synonyms, mappings, or field weights change. Better ranking cannot repair missing source data.
Measure query latency, errors, source lag, deletion lag, and reviewed search quality separately. Query text and result clicks can contain sensitive information; collect only what evaluation needs, with restricted access and retention. Feedback helps identify a confusing result, but a click alone does not establish relevance.
Plan for rebuilding and operating the index
Build a changed mapping into a separate index version. Reconcile document identities, deletions, permissions, and representative queries before switching reads. Catch up changes that arrived during rebuilding and define rollback behavior. An older index must not restore access that has since been revoked.
A custom application adds connector maintenance, interface work, API ownership, and relevance testing. It earns that cost when users need a shared workflow across sources. For a narrower need, an existing search interface or direct source search may be sufficient.