Running a self-serve Proof of Concept for Knowledge Agents

👥

Access Required

Knowledge Agents are available on Guru’s All-in-One or Enterprise plans.

Only Admins can create Knowledge Agents. Admins may assign a Knowledge Agent Owner to manage settings and permissions.

Setting up your Knowledge Agent (Phase 1)

1. Create a New Knowledge Agent

• Go to the Knowledge Agents tab in Guru and click New Agent.
• If you’re new to Knowledge Agents, start with the Creating Knowledge Agents guide.

2. Name and Brand Your Knowledge Agent

• Choose a clear, descriptive name (e.g., “Product Help” or “IT Assistant”).
• Optionally upload an image to personalize the Knowledge Agent.

3. Connect Your Sources

• Select the relevant knowledge Sources, such as Confluence, Zendesk, or Salesforce.
• If no Sources are connected, follow the Sources Overview to get started.

💡

Tip

Only connect verified and up-to-date Sources to ensure high-quality answers.

4. Set User Permissions

• Start in private mode to limit access to yourself or a small group for testing.
• Share with broader teams only once you’re confident in the Knowledge Agent’s performance.

5. Customize the Prompt

• Use a clear and structured prompt to guide the Knowledge Agent’s behavior.
• Use the Prompt Architect Tool to help write your prompt.

Example:

“You are a helpful assistant trained on internal HR policies. Respond concisely and include Source links.”

6. Enable in Workflow

• Optionally connect the Knowledge Agent to:
  • Slack
  • Chrome Extension
  • Web App homepage

7. Save and Activate

• Review your setup, then click Save & Activate.
• Your Knowledge Agent is now live and ready for testing.

Testing, iterating, and validating the Knowledge Agent (Phase 2)

1. Test the Knowledge Agent

• Use the preview tool or test live in private mode.
• Ask a mix of questions:
  • Edge cases
  • Common workflows
  • Known problem areas

2. Ask Real Questions

• Use Slack, Teams, the web app, or the browser extension.
• Vary question phrasing and tone (casual vs. formal) to test versatility.

3. Provide Feedback

• Use thumbs-up/down on Knowledge Agent responses to rate accuracy.
• Share qualitative feedback with your team (e.g., “too vague,” “missing link”).
• Review feedback and results in the AI Agent Center.

4. Train the AI

• Refine your prompt based on response quality.
• Adjust Sources: remove irrelevant ones or add helpful content.
• Update knowledge by creating or editing Guru Cards to address content gaps.

5. Use Slack Suggested Answers

• Automatically surface answers to recurring questions in Slack.
• Enable Suggested Answers in your Slack channel settings (requires an active Knowledge Agent).

See: Searching Guru in Slack

📘

Best Practices

Optimizing performance

• Limit Sources to the most relevant and actively maintained collections.
• Keep prompts focused but adaptable—start small and iterate.
• Test by persona to ensure answers are relevant to different roles (e.g., sales, support, engineering).
• Use the Chrome Extension for contextual, in-app testing.
• Schedule weekly check-ins during rollout to review usage and feedback.

---

Repeating and improving

Testing is an ongoing process. After each change:

• Re-test the Knowledge Agent.
• Track improvements (e.g., fewer thumbs-down, faster resolution times).
• Celebrate milestones such as:
  • First successful Slack answer
  • 10+ correctly answered questions

Wrapping up and evaluating results

Final evaluation checklist:

• Is the Knowledge Agent accurately answering key questions?
• Should you expand visibility to more users or teams?
• What can you iterate next—prompt, Sources, or new department-specific Knowledge Agents?