Keeping indexes in sync with their sources
Lesson 3 of 3 in Vector Stores at Professional Scale.
A vector index is a cache of yesterday's truth: the moment the source document changes, the index answers confidently from the past. Two things keep that from happening: data maintenance systems that hold the index current, and connectors into where the documents actually live — document management systems, knowledge bases, internal wikis. Bedrock Knowledge Bases ships connectors (S3, Confluence, SharePoint, Salesforce, a web crawler), and syncing is one API call: StartIngestionJob. Sync is incremental — the connector crawls what is new, modified, or deleted since the last sync (the first sync crawls everything), using the source's own change tracking.
Three freshness patterns cover the exam: scheduled refresh (EventBridge Scheduler fires StartIngestionJob nightly or hourly), event-driven updates (an S3 upload event triggers Lambda, which starts the ingestion job — freshness in minutes), and real-time change detection for custom stores (source webhooks or streams drive embed-and-upsert directly). The choice is a freshness-SLA question, not a taste question.
Scheduled refresh
EventBridge Scheduler → StartIngestionJob on a cadence. Simple, predictable, and incremental sync keeps it cheap. Right when the freshness SLA is "hours": documentation, policies, catalogs.
Event-driven updates
S3 upload/change event → Lambda → StartIngestionJob (or direct embed-and-upsert for custom stores). Freshness in minutes, cost proportional to change volume. Right when stems say "shortly after documents arrive" — and the exam guide's "real-time change detection systems" is this pattern generalized to webhooks and streams.
Real-time change detection at volume
When change arrives as a stream rather than a file drop — operational systems emitting thousands of record updates a minute — Amazon Kinesis Data Streams (or Amazon MSK for Kafka-native estates) carries the change feed: producers publish every create, update, and delete, and a consumer re-embeds and upserts the affected documents as events arrive. Same freshness goal as the S3-event pattern, built for high-volume streaming change capture — the exam's "real-time change detection systems" at scale.
Automated synchronization workflows
For fleets of sources: Step Functions orchestrates per-source ingestion jobs, verifies completion, records sync state (DynamoDB is the natural ledger), and alarms on failures. Sync that nobody watches is sync that silently stopped three weeks ago.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.