How to Use Billy to Investigate Packages and Draft Policies

Beginner 15 minutes All Personas AI Assistant

Ask Billy natural language questions about your package consumption, vulnerabilities, and policy gaps, then review and approve Billy's policy proposals.

Overview

Billy is Chainsaw’s AI-powered supply chain assistant. You can ask Billy questions in plain English about your packages, policies, vulnerabilities, and events. Billy can also draft new policies for your review — it never creates policies automatically; you always approve before anything takes effect.

Prerequisites

  • Billy must be enabled for your organization (feature-flagged)
  • An OpenRouter API key configured by your administrator
  • Any role can chat with Billy (access to the Billy page)

Step 1: Open Billy

Click Billy in the sidebar navigation. The chat interface opens.

Billy chat interface
Billy's chat interface — type your question in natural language

Step 2: Ask Questions About Your Supply Chain

Billy can query your organization’s data using read-only SQL under the hood. Here are example questions organized by use case:

Package Investigation

"What npm packages have we installed in the last 7 days?"

"Show me all packages with a trust score below 50"

"Which packages have confirmed malware status?"
Billy answering a package question
Billy queries the package_metadata table to answer your questions

Vulnerability Analysis

"What are the most critical vulnerabilities in our supply chain?"

"Show me packages with CVSS score above 9.0"

"How many vulnerable packages did we download last month?"

Policy Review

"What policies do we currently have enabled?"

"Which policy is blocking the most packages?"

"Are there any gaps in our policy coverage?"

Event Analysis

"What packages were blocked yesterday?"

"Show me the top 10 most downloaded packages this week"

"Which clients have the most failed requests?"
Billy understands context from your conversation. You can ask follow-up questions like “Filter that to only npm packages” or “Show me the details for the first one.”

Step 3: Ask Billy to Draft a Policy

Billy can propose new policies based on conversational intent. It never creates policies directly — it presents a proposal for your review.

Example: “Block all packages with known malware”

Billy proposing a policy
Billy presents a policy proposal card with full details

Billy will show you a policy approval card with:

  • Policy name
  • Action (block, quarantine, allow)
  • Conditions (what triggers the policy)
  • Scope (which repositories/clients it applies to)

Step 4: Review and Approve Policies

When Billy proposes a policy, review the details carefully:

  1. Check the conditions match your intent
  2. Verify the scope is correct (not too broad or narrow)
  3. Consider the precedence relative to existing policies

If you’re satisfied, click Approve on the policy card.

Approving a Billy-drafted policy
Click Approve to create the policy — Billy never creates policies without your explicit approval
Always review Billy’s policy proposals before approving. Billy makes its best guess based on your request, but you understand your organization’s context better.

If you want changes, tell Billy:

"Make that policy only apply to the npm repository"

"Change the action from block to quarantine"

"Add an EPSS condition of 0.5 or higher"

Step 5: Common Workflows with Billy

Morning Security Check

"Summarize supply chain events from the last 24 hours"
"Were there any new violations?"
"Show me any packages with suspected typosquats"

Policy Gap Analysis

"What policies do we have for npm?"
"Are we checking for typosquats?"
"Do we have a freshness guard policy?"
"Draft a policy to block npm packages less than 14 days old"

Incident Investigation

"Show me all events for package lodash in the last 30 days"
"Which clients installed lodash@4.17.20?"
"What's the trust score for that version?"
Billy helping investigate a package
Billy helps trace package consumption and assess risk

Billy’s Capabilities and Limits

What Billy Can Do

  • Query the events, policies, and package_metadata tables
  • Draft new policies for your approval
  • Answer questions about your supply chain data
  • Provide explanations and recommendations

What Billy Cannot Do

  • Modify, update, or delete any data
  • Create policies without your approval
  • Access data outside your organization
  • Execute arbitrary code or commands
  • Run queries with JOINs, subqueries, or CTEs (for safety)

Safety Guardrails

  • All queries are scoped to your organization
  • Maximum 50 results per query
  • Maximum 5 tool iterations per conversation
  • Messages limited to 2,000 characters
  • Conversation history limited to 50 messages

Next Steps