We’re officially living in the Golden Age of TestFlight. It happens like clockwork — a buzzy new social app emerges in closed beta, and early users post screenshots on Twitter to generate FOMO. VCs clamor to score invites, and may even send in term sheets before the product publicly launches.
Don’t get us wrong — we love TestFlight! It’s invaluable to get real users on your app to find bugs, provide feedback, and hone in on killer features (or UI issues) before the scrutiny of a public launch. However, we caution founders not to over-index on TestFlight data when making decisions about product, strategy, and fundraising. Why? The engagement and retention metrics on TestFlight don’t always carry over to the public app.
We know founders who believe they’ve found product-market fit based on a TestFlight, and raise money to spend on marketing— only to discover they have a major churn problem. On the flip side, we’ve also seen founders disappointed by their TestFlight data who scrap their product — but the true issue may have been that the TestFlight users were a poor fit for the app.
After using dozens of TestFlights ourselves & meeting many founders raising off a TestFlight, we’ve developed a list of five questions investors may have about your TestFlight data — and how to respond to them!
1.To what extent is the hype around early access boosting your app’s engagement & retention? We’ve all seen tweets from people begging for invites to a hot TestFlight app. If your product is getting a lot of hype, your beta users will be more motivated to open the app, post, and engage with content. They might feel obligated to be a “good” beta tester, or want to show off the fact they have access. Investors may wonder if your engagement and retention metrics will fall off when the app is publicly available.
How to respond: Hype cycles die fast in social! Even a week of usage data can start to show a drop in excitement (or hopefully a lack thereof). Cohort data on DAUs, # sessions/day, session time, etc. on a daily or weekly basis is hugely helpful.
2. Are your users benefitting from the “big fish, little pond” effect? On a similar note, TestFlight engagement might be artificially high if users sense a temporary opportunity to gain influence. Everyone wants to claim their @firstname handle and start accumulating a following, which is easy to do when there’s not much content to compete with. This incentivizes users to post on your app more frequently, as they get an outsized reward compared to their posts on other social apps. Will this dynamic hold when the app is flooded with users and attention is less concentrated on the few early “stars”?
How to respond: In any social app, they’ll be some users who primarily “lean back” and consume content vs. creating it. Try to slice your data to look at them! If the engagement data for these consumers looks good, it suggests that there’s something working in the product beyond the allure of building an audience.
3. Are your TestFlight users representative of your target user base? Investors want to know that you’re testing the product with the same people who would be using it “in the wild.” If you’re building an app for teen girls and your TestFlight users are all VCs, you have a problem. VCs and other people in the startup ecosystem are particularly troublesome because we want new products to work, so we act accordingly. We have more time and patience for new apps than almost any other demographic of consumers — you probably shouldn’t trust our engagement data!
How to respond: This is harder to fix retroactively. If you have data on who your beta testers are and how they found you, that’s helpful to include in your pitch. We recommend being purposeful about onboarding users who represent your target demographic — which will likely take work, as your average consumer is much less likely to convert from a waitlist to TestFlight than someone in tech/VC.
4. Can your users bring their friends — and if not, how will their usage change when they can? Social networks aren’t single-player experiences. In some cases (e.g. Reddit), it doesn’t really matter if we know the people we’re interacting with. In other cases (e.g. Snap), we typically need to know other users to be interested in their content. If your app falls in the latter case, TestFlight is tricky — it’s tough for users to add their IRL friends. As a result, your engagement and retention metrics may end up looking artificially low.
How to respond: If your product relies on a user’s existing social graph, think about how you can onboard a community to the TestFlight (e.g. a high school or college campus). You can also think about segmenting your data — do engagement and retention differ for users who have 5+ contacts on the app?
5. Are the social dynamics on your TestFlight scalable to a larger user base? An ex-director of eng at a major social network once told us that for every 100 new users, there will be at least one particularly toxic troll. It’s easy to prevent scammy or inappropriate behavior when you’ve vetted and onboarded a small group, but this gets complex at scale. It’s worth thinking about how you’ll deal with bad actors once they emerge — as well as how to preserve the special energy of the early community as more people join.
How to respond: This is somewhat a question of who you hire and what content moderation systems you put in place, but the learnings of your TestFlight users can be instructive. How can you design an onboarding process that encourages the kind of behavior and norms that “hooked” your early users?
We’d love to hear what we’re missing from any founders that have utilized TestFlight, or consumer “frequent flyers” of TestFlight betas (like us!). Comment below, or tweet us @venturetwins.
If you’re interested in reading more from us, you can also subscribe to our weekly newsletter, Accelerated 🚀