Any automation you add without measuring might be helping or quietly making things worse. Three simple metrics tell the real story within a month.
Metric One: First-Post Rate
What percentage of new members post within 14 days of joining? This is the single best health signal in any community. A first-post rate above 40% usually means onboarding is working. Under 25% means the welcome experience isn’t landing — either it’s too generic, too delayed, or too cold. Welcome agents should push this number up within weeks. If they don’t, the inputs need work.
Think of it like a restaurant counting how many customers come back for a second visit. One of the few single numbers worth watching.
Metric Two: Weekly Active Members
Out of your total membership, how many posted, commented, or reacted in the last 7 days? This is the pulse of the community. Healthy campuses often run 30–50% weekly active. Dropping below 20% is a drift signal. Posting agents and engagement agents should keep this number stable or rising. If you add agents and it drops, the quality of posts is likely down — members feel the drift even if they can’t name it.
Metric Three: Reply Speed
How long does the average community question go unanswered? Under 4 hours is great. Under 24 is acceptable. Over 24 and members start concluding nobody’s home. Reply agents should pull this number down by at least 50% in the first month of deployment. If they don’t, either the drafts aren’t good enough to approve quickly or the approval workflow is broken.
What This Means for Educators
You want to know if agents are actually helping — not just making you feel busy. These three numbers separate theater from real improvement. Check them monthly. If they’re trending up, double down on what’s working. If they’re flat or dropping, the fix is usually simpler inputs to existing agents, not adding more agents. Layering complexity on unclear problems is how campuses end up over-automated.