Build DeepSearch in TypeScript – Matt Pocock

This article delves into leveraging Vitest globals in the context of building and testing sophisticated AI applications, drawing insights from the “Build DeepSearch in TypeScript” course. It explores the crucial role of testing, observability, and structured architectures in developing production-ready AI solutions. The course emphasizes moving beyond basic LLM API interactions to develop robust, testable, and maintainable AI agents.

Vitest Globals

Vitest globals provide a convenient way to access Vitest’s testing functions and utilities without explicitly importing them in each test file. This simplifies test code and reduces boilerplate, making it easier to write and maintain comprehensive test suites, vital for production-ready applications.

Streamlining Tests with Vitest Globals

Vitest offers a set of global variables that are automatically available in your test files. These variables include functions like describe, it, expect, beforeEach, afterEach, beforeAll, and afterAll. By using these globals, you avoid the need to import them in every test file, which significantly reduces the amount of code you have to write. This is particularly useful when building complex AI applications, where you might have hundreds or even thousands of test cases.

Imagine trying to write tests for an AI agent that interacts with a database, caches results, and authenticates users, all using complex prompts. Without Vitest globals, each test file would require multiple import statements for basic testing functions. This adds unnecessary noise and makes the tests harder to read and maintain. By embracing Vitest globals, you can focus on the core logic of your tests, making them more concise and easier to understand.

The reduction in boilerplate also means less opportunity for errors. Typos in import statements can lead to frustrating debugging sessions. By relying on Vitest globals, you eliminate this potential source of error and improve the overall reliability of your test suite. This is especially important when dealing with AI applications, where even small errors can have significant consequences on the model’s behavior.

Benefits of Using Vitest Globals in AI Testing

When it comes to testing AI applications, the benefits of using Vitest globals extend beyond mere convenience. AI systems, especially those using Large Language Models (LLMs), can be complex and unpredictable. Effective testing is crucial to ensure reliability, accuracy, and robustness. Vitest globals make it easier to integrate testing into your development workflow and streamline the process of writing, running, and maintaining tests.

The reduced boilerplate allows developers to focus on writing meaningful tests that cover a wider range of scenarios. Testing AI applications involves verifying not only the functional correctness of the code but also the behavior of the AI model itself. This often requires writing tests that evaluate the model’s responses to different inputs, check for biases, and ensure that the model adheres to ethical guidelines.

With Vitest globals simplifying the testing process, developers can spend more time thinking about these critical aspects of AI testing. They can more easily write tests that use LLM-as-a-Judge to evaluate the quality of the model’s output, or define custom datasets to test the model’s performance on specific tasks. This results in a more thorough and effective testing process, which is essential for building trustworthy AI applications.

Potential Drawbacks of Vitest Globals

While Vitest globals offer significant advantages by streamlining test code and reducing boilerplate, there are potential drawbacks to consider. These drawbacks primarily revolve around code clarity, maintainability, and potential conflicts with other libraries or frameworks.

One common concern is that relying too heavily on globals can make code less explicit and harder to understand, especially for developers who are new to the project or to Vitest itself. When all testing functions are available without explicit import statements, it can be less clear where they are coming from and how they work. This can make it more difficult to trace the flow of execution and understand the dependencies within the test suite.

Another potential issue is the risk of naming conflicts. If other libraries or frameworks also define global variables with the same names as Vitest globals, this can lead to unexpected errors and difficult-to-debug issues. While this is relatively rare, it’s a possibility to be aware of.

To mitigate these drawbacks, it’s important to use Vitest globals judiciously and to follow best practices for code organization and documentation. For example, you can use comments to explain the purpose of each test and to clarify the role of Vitest globals. You can also use linters and code formatters to enforce consistent coding standards and to highlight any potential naming conflicts.

Drizzle Redis

The integration of Drizzle ORM and Redis plays a crucial role in building scalable and efficient AI applications. Drizzle ORM provides a type-safe way to interact with relational databases, while Redis offers a fast in-memory data store for caching frequently accessed data. This combination is essential for optimizing performance and ensuring the responsiveness of AI-powered systems.

Optimizing Data Access with Drizzle and Redis

Drizzle ORM, a modern TypeScript ORM, offers a type-safe and efficient way to interact with databases like PostgreSQL. It allows developers to define database schemas as TypeScript types and perform database operations using a fluent API. This approach eliminates the need for manual SQL queries, reducing the risk of errors and improving code maintainability.

In the context of AI applications, Drizzle ORM can be used to store and retrieve various types of data, such as user profiles, training datasets, and model parameters. Its type safety ensures that data is always accessed and manipulated correctly, preventing runtime errors and improving the overall reliability of the system. Furthermore, Drizzle’s query builder allows for complex data retrieval scenarios commonly needed for features such as user-specific AI interactions, or AI training runs filtered accodring to specific criteria.

Redis, on the other hand, is an in-memory data store that provides extremely fast read and write access. It is often used for caching frequently accessed data, such as API responses, session information, and computed results. By storing data in Redis, you can reduce the load on your database and improve the overall performance of your application.

Balancing Persistence and Speed

Using Redis for caching in conjunction with Drizzle ORM for persistent storage strikes a good balance between speed and data integrity. Imagine the “DeepSearch” application described in the course briefing. Without caching, every search query would require a database lookup. This can become a bottleneck, especially as the number of users and the complexity of the queries increase.

By caching search results in Redis, you can significantly reduce the number of database queries and improve the response time for frequently accessed search queries. When a user submits a search query, the application first checks if the results are already available in Redis. If they are, the results are retrieved from Redis and returned to the user immediately. This dramatically reduces the response time and improves the user experience.

If the results are not in Redis, the application uses Drizzle ORM to query the database. Once the results are retrieved, they are stored in Redis for future use. This ensures that subsequent requests for the same search query are served from the cache, further reducing the load on the database.

Caching search results is just one example of how Redis can be used to optimize performance in AI applications. It can also be used to cache API responses, session information, and computed results. By strategically caching frequently accessed data, you can significantly improve the performance and scalability of your AI-powered systems.

Implementing Caching Strategies with Redis

Effective caching requires a well-defined strategy that addresses issues such as cache invalidation, cache eviction, and cache consistency. Cache invalidation refers to the process of removing stale data from the cache. Cache eviction refers to the process of removing data from the cache when it is full. Cache consistency refers to the degree to which the data in the cache matches the data in the database.

There are several caching strategies that you can use to manage these issues. One common strategy is time-to-live (TTL) caching, where each entry in the cache is assigned a TTL value. After the TTL expires, the entry is automatically removed from the cache. This is a simple and effective way to ensure that the cache does not contain stale data.

Another strategy is least recently used (LRU) caching, where the cache evicts the least recently used entries when it is full. This ensures that the cache always contains the most frequently accessed data. You can also use a combination of TTL and LRU caching to optimize cache performance.

The specific caching strategy that you choose will depend on the specific requirements of your application. However, it’s important to have a well-defined strategy in place to ensure that your cache is effective and reliable. Proper configuration of the cache is critically important in applications that handle Personal Identifiable Information (PII) or Personal Health Information (PHI) so that you meet compliance thresholds and service level objectives.

Langfuse Architecture

Langfuse architecture offers a robust framework for observing and evaluating the performance of LLM applications. By providing detailed tracing and comprehensive evaluation tools like Evalite, Langfuse architecture empowers developers to understand and improve the behavior of their AI agents.

Observability in LLM Applications

Langfuse architecture emphasizes the importance of observability in LLM applications. Observability refers to the ability to understand the internal state of a system by examining its outputs. In the context of LLM applications, observability involves tracking the inputs, outputs, and intermediate steps of the LLM calls.

This data can be used to identify bottlenecks, debug errors, and optimize performance. For example, tracing the execution path of an LLM call can reveal which prompts are taking the longest to process or which tools are being used most frequently. This information can be used to optimize the prompts, reduce the number of tool calls, or improve the performance of the tools themselves.

Langfuse architecture provides tools for collecting and analyzing this data. It allows you to track the inputs, outputs, and intermediate steps of LLM calls, as well as the metadata associated with each call, such as the timestamp, the user ID, and the model version. This data can be visualized in a dashboard, allowing you to easily identify patterns and trends.

Evalite for Unit Testing AI Outputs

One of the critical components of Langfuse architecture is Evalite, a tool for unit testing AI outputs. Evalite allows you to define success criteria for your LLM calls and to automatically evaluate the outputs against those criteria. The course dedicates significant time (Days 3-5) to integrating tools like LangFuse for tracing and Evalite for unit testing AI outputs. It covers what makes good success criteria and introduces LLM-as-a-Judge and custom Datasets.

This is particularly important in AI applications, where the outputs can be unpredictable. By defining success criteria and automatically evaluating the outputs, you can ensure that your LLM calls are producing the desired results. Evalite supports various types of success criteria, such as: Exact match: The output must match a specific string. Regular expression: The output must match a regular expression. Semantic similarity: The output must be semantically similar to a reference output.

You can also use Evalite to define custom success criteria, such as checking whether the output adheres to a particular format or whether it contains specific keywords. By using Evalite, you can ensure that your LLM calls are producing high-quality outputs that meet your specific requirements. Evals are the unit test of the AI application world.

Transitioning from Vibe Checks to Objective Measurement

The shift from subjective “vibe checks” to objective measurements is a central theme in Langfuse architecture. Traditionally, developers have relied on their intuition and subjective judgment to assess the performance of LLM applications. However, this approach is not scalable or reliable.

As AI applications become more complex, it is essential to have objective means to measure their performance. Langfuse architecture provides the tools and techniques to make this transition. By defining success criteria, collecting data, and analyzing the results, you can objectively evaluate the performance of your LLM calls and identify areas for improvement.

This approach is not only more reliable but also more efficient. By focusing on objective measurements, you can quickly identify the most impactful changes and prioritize your development efforts. This allows you to iterate faster and build better AI applications.

Here’s a table summarizing the key differences between subjective “vibe checks” and objective measurements in AI development:

Ratelimiter Create

Implementing rate limiting is crucial for protecting AI APIs from abuse and ensuring fair usage. Ratelimiter create function, in conjunction with Redis, provides a robust mechanism for controlling the number of requests a user can make within a given time period. This is especially important for AI applications, which can be resource-intensive and vulnerable to malicious attacks.

Protecting APIs with Rate Limiting

Rate limiting is a technique used to control the rate at which users can access an API. It helps to prevent abuse, protect resources, and ensure fair usage. Without rate limiting, APIs can be vulnerable to various types of attacks, such as denial-of-service (DoS) attacks, brute-force attacks, and spamming.

DoS attacks aim to overwhelm the API with a flood of requests, making it unavailable to legitimate users. Brute-force attacks attempt to guess passwords or other sensitive information by repeatedly trying different combinations. Spamming involves sending unsolicited messages or requests through the API.

Rate limiting can help to mitigate these risks by limiting the number of requests that a user can make within a given time period. For example, you can set a limit of 100 requests per minute per user. If a user exceeds this limit, their subsequent requests will be rejected.

Redis-Based Rate Limiting Implementation

Using Redis for rate limiting offers several advantages, including high performance, scalability, and reliability. Redis, as an in-memory data store, can handle a large number of requests with low latency. It also supports atomic operations, which are essential for implementing rate limiting correctly.

The ratelimiter create function can be used to create a rate limiter that stores the rate limiting data in Redis. The rate limiter typically works by incrementing a counter in Redis for each request. If the counter exceeds the limit, the request is rejected. Redis also provides mechanisms for expiring the counter after a certain period of time, ensuring that the rate limiter resets periodically.

This approach is efficient and scalable, as Redis can handle a large number of concurrent requests. It is also reliable, as Redis provides data persistence and replication features. By using Redis for rate limiting, you can protect your AI APIs from abuse and ensure fair usage. This can be implemented with libraries that help perform these operations such as ioredis alongside algorithms such as the leaky bucket.

Configuring Rate Limiting Parameters

Configuring the rate limiting parameters is crucial for balancing protection and usability. You need to set the limit high enough to allow legitimate users to access the API without being unduly restricted, but low enough to protect the API from abuse.

The appropriate limit will depend on the specific requirements of your application. Factors to consider include the types of operations that the API supports, the expected traffic patterns, and the resources available.

For example, if your API supports resource-intensive operations, such as training an AI model, you may need to set a lower limit than if it only supports simple read operations. You should also consider the expected traffic patterns. If you anticipate a sudden spike in traffic, you may need to increase the limit temporarily to avoid rejecting legitimate requests.

It’s good to dynamically adjust the limits based on various factors, such as the user’s subscription level, the time of day, or the overall system load. You also can implement different rate limits for different API endpoints.

Build DeepSearch in TypeScript

Build DeepSearch in TypeScript is a comprehensive course designed to equip developers with the skills and knowledge necessary to build production-ready LLM applications. By focusing on practical, hands-on development, the course bridges the gap between proof-of-concept and deployment, emphasizing the importance of testing, observability, and structured architectures.

Key Concepts and Themes

The course emphasizes the importance of moving beyond basic LLM API interactions to develop robust, testable, and maintainable AI agents. The central premise is that building genuinely useful AI applications requires more than just hitting an LLM API and getting back stock chat responses. Instead, it advocates for applying established software engineering practices—testing, metrics, analytics—to AI development.

The course also addresses key challenges faced when developing sophisticated AI applications, such as:

• Implementing essential backend infrastructure (databases, caching, auth) specifically for AI-driven applications.
• Debugging and understanding the black box of AI agent decisions, especially with multiple tools.
• Ensuring chat persistence, reliable routing, and real-time UI updates for a seamless user experience.
• Objectively measure AI performance moving beyond subjective ‘vibe checks’ for improvements.
• Managing complex agent logic without creating brittle, monolithic prompts that are hard to maintain and optimize.

Through hands-on development of the ‘DeepSearch’ AI application, participants will learn how to overcome these challenges and build production-ready AI systems.

Hands-on Development of DeepSearch

The course is highly practical, guiding participants to build out a ‘DeepSearch’ AI application from the ground up. The project starts with a pre-built foundation using Next.js TypeScript (of course), PostgreSQL through the Drizzle ORM, and Redis for caching.

Participants will implement a Naive agent with search tools, persist conversations to a database, integrate observability platforms (Langfuse), set up evals (Evalite), and refine agent logic through task decomposition. They will also learn how to implement advanced patterns, such as the Evaluator-Optimizer loop, to improve the reliability and accuracy of the AI agent.

The hands-on development approach allows participants to gain practical experience with the technologies and techniques covered in the course. By building a real-world AI application, they will develop the skills and confidence they need to build their own production-ready AI systems.

Course Structure and Progression

The course follows a structured progression, starting with the basics of setting up the core infrastructure and implementing a naive agent, and progressing to more advanced topics such as observability, evals, and agent architecture.

Days 00-02 (Getting Started & Naive Agent): Setup of core infrastructure (Next. js, PostgreSQL, Redis), connecting an LLM via AI SDK, implementing a basic search tool, and saving conversations.

Days 03-05 (Observability & Evals): Integrating Langfuse for tracing LLM calls, setting up Evalite for objective testing, defining success criteria, and implementing LLM-as-a-Judge with custom datasets.

Days 06-07 (Agent Architecture – Task Decomposition): Refactoring the agent to handle complexity by breaking down tasks, designing a Next Action Picker, and implementing a processing loop.

Days 08-09 (Advanced Patterns – Evaluator-Optimizer): Differentiating Agents from Workflows, optimizing the search/crawl process, and building an Evaluator component for more reliable outputs, including showing sources and guardrails.

This progressive approach ensures that participants gradually build their knowledge and skills, allowing them to master the complexities of building production-grade LLM-powered applications.

Conclusion

The “Build DeepSearch in TypeScript” course, as highlighted, emphasizes the transition from basic LLM interactions to robust, production-ready AI applications. Key to this transition is embracing software engineering best practices, including rigorous testing with efficient tools like Vitest globals and Evalite, implementing scalable architectures with Drizzle ORM and Redis, and prioritizing observability through Langfuse. Rate limiting using solutions like ratelimiter create further ensures API protection, while structured agent architectures and iterative improvement based on user feedback contribute to building reliable and maintainable AI systems in a rapidly evolving technological landscape.

Sales Page:_https://www.aihero.dev/cohorts/build-deepsearch-in-typescript