Daily newsletter testing
In which I keep you updated on the testing and implementation of my daily newsletter, Authoresque.
LAUNCH PARTY: If you've been following along with our podcast, The Six Figure Author Experiment, then you might know that the first book in this experiment, Magic, launches on Monday.
To celebrate, we're having a very cool Q and A on Tuesday, Jul 9, 2024 12pm PT/3pm ET where Lee, Monica, and Russell hang out and answer questions about book launches.
Russell is also answer questions about the series, but he doesn't expect much to come up (though it's a fascinating story about how we got to this point).
Joining us is completely free and it’s gonna be fun on a bun. I don’t know exactly what’s gonna happen, but the three of us are a good time, and even better together.
Hi friends,
This post continues the saga of trying to figure out how to make a sustainable lead generation machine for The Author Stack using ads. These are the two previous articles I’ve written about this topic.
In the previous two articles, I explain why I started using ads again this year, and in the second I talk about how those initial tests ended in spectacularly terrible fashion.
From there, I began testing a daily newsletter in the style of Morning Brew, 1440, the Skimm, but for indie authors. I’ve been studying these newsletters for several months, and feel like I have a pretty decent template to start moving forward with a full test.
I wrote this article in real time as I was going through the tests, which might take a bit to get used to, but it gave me the more visceral reaction to the events transpiring. I thought that was important.
This article also has a paywall in the middle. I gave enough information so you can see what I’m doing and hopefully how to do it yourself, but not enough to see all the juicy bits about the success of failure of the subsequent tests.
Beehiiv catch up
I previously tested multiple options through Beehiiv with decent results. I tested two different options, and these were the stats for the winning test.
My main list has relatively similar results, so I did some simple math and decides that if these carried through, I should see roughly 4x the results, which would put me right around 15,000 opens and 300 unique clicks. You can see that the top ad for this set is the ad I ran, which meant I could expect roughly 200 clicks total and 80-100 total clicks, which would put my ad costs right around $300-$500, which would be a great result.
Convertkit test 1
If you’ve followed along the previous articles in this saga, you can probably guess this is not what happened. If you want to see the first test, here is a link to check it out.
I heard great things from people who reached out to tell me what they thought, but here are the results after the first day.
That’s 9,490 opens, well below my benchmarks. Remember, I was expecting closer to 15,000. To give you some context, this is what an email looks like after 48 hours on The Author Stack.
You can see that despite being sent to 8,000 fewer people, this email was viewed 15.1k times, or by 50% more people. Now, you might think that my Convertkit emails get worse open rates, but when I ran tests to each segment of my life, they performed mainly as I would expect.
As you can see, that’s about over 15,000 opens and over 350 clicks on a post that didn’t have many places to click. To go from these results to what happened on this test is an absolute disaster.
A 35% open rate is not very good in this game, but a 23% open rate is a awful. On top of this, the ad break only got 32 clicks, which means that instead of charging $300-$500, an ad break would cost $100-$200.
That takes the daily from the realm money generating to barely breaking even, especially if I bring in help.
So, what happened? I wasn’t sure. Maybe the headline was bad, though I don’t think it would have gotten a 50% reduction in open rate. Perhaps there were too many images, though I don’t think they had more than the initial test I ran on Beehiiv. Perhaps it was just too long to reach inboxes.
The only thing you can do when something like this happens is do some testing, so I decided to resend to unopens. Here are the results.
I tested a much hookier headline, and shorter, which didn’t seem to do anything to help deliverability. After a week, these were the results.
32% isn’t terrible, but when you send a daily you don’t really have a chance to resend to unopens.
If you’d like to play with Convertkit yourself, they have recently opened a free 10,000 subscriber plan, so you can get in there and see if it works for you.
Convertkit test 2
For the first test, I ran a 5x5, which is five articles and five bonus bits. For my next test, I tested a 3x3, with little or no imagery to see if that helps. I had such high hopes before this test as I had every indication this would go really, really well. Then, we ran it. You can read this email here.
The first of these emails went worse than the first one, even though I was running less graphics and fewer links by a wide margin. The resend went okay, but only compared to the last one.
Overall, after a couple of days, here’s how this one stands.
You might think this was normal, but I also run my Wannabe Press emails through Convertkit, and this is what our resend looked like over there.
That’s a 43% open rate, which way better than on Flodesk, and then with our next email we hit 44% without even a resend.
This second test went out to 41,000 people, and so after two sends I have to assume that a significant percentage of people just don’t want to read it, which is understandable. This is a whole new product and I’m basically asked other people to care about it from jump.
I’ve identified 12,000 people who haven’t opened or shown any interest. These include close to 5,000 people who haven’t opened an Author Stack email in a long time, and 7,000 people people who are only on my Wannabe Press email list, not my combined Wannabe Press/Author Stack list. So, there is no reason to believe that they would even care about what I have to say.
Additionally, holding a bunch of dead emails could seriously affect deliverability as well.
I created a cold email flow that should allow us to not deal with this once we have a good baseline. However, for these first tests I want to be more involved to manually see where we stand.
Next test we’ll isolate those people and remove them from the delivery segment. So, we’ll be sending to 28,000 people instead of 41,000, and see if we can get a 40%+ open rate from that. If I can get the same 12,000-15,000 opens then it should be possible.
Depending on how that goes, I’ll have to go back to my sponsorship team and tell them we actually only have 28,000 people on that list, not 41,000, which will probably force us to lower ad rates. I think what’s important to advertisers are opens, though, not total reach.
I have a lot more to share, but only for paid members. If you want to keep following my journey (and I have some juicy bits still to share) then I hope you’ll consider becoming a paid member, or at least start a free trial to access over 850 member-only posts, interviews, courses, and more.
The Author Stack open rate interlude
While going through these numbers, I started to look into the open rate of The Author Stack. At our peak we were getting a 40%+ open rate and being read by 20,000 subscribers. However, once the Google updates went into place at the beginning of April 2024, our opens went from 16-20k down to 12-16k, which was our baseline in November 2023.
This revelation gutted me, because it meant that the $40k I had spent on advertising and everything we’ve done this year to built The Author Stack meant nothing. Google basically wiped out all our gains with the snap of their finger.
Convertkit test 3 & 4
I originally thought we wre really close to what I think will be the final form now with the third test. I removed the header image which seemed to be bouncing me to inboxes, and I also segmented everything into three buckets.
The first segment above is all cold emails that haven’t opened in several months. Then, the middle segment are the people who are subscribed to the Author Stack, or have opened one of the previous two emails, and are engaged. Then, the third segment are people on the Wannabe Press email list who haven’t engaged with either of the first two emails.
I actually really like how Convertkit thinks about segments. Here is my segment for The Author Stack, which contains a couple dozen tags.
Then, when I segment my list to send an email, I use a combination of segments, tags, and sequences to narrow-band who gets what email. So, a final segment will look something like this, and the segments have already been grouped based on tags. If I add more relevant tags, then I update the segment and it populates the rest.
This week, I decided to send an email on one day followed by a resend the next day, followed by a new email the third day and a resend on the fourth day. This ramps up the sends a little bit from last week, but it’s still not quite daily yet.
I’m also testing including a second programmic ad and not including one to see if that affects deliverability at all. It seems like just focusing this on people who opened has already significantly helped deliverability.
After this, I will take all the emails who opened from segment 1 & 3 above and add them to segment 2. Following that, I will stop sending to the Wannabe Press folks who haven’t opened (segment 3), and add the people remaining in segment 1 through a cold email sequence to try and recover as many emails as possible before unsubscribing them from Convertkit.
Once that is done, I will keep sending to segment 2, which will then become the only segment. We officially booked our first sponsor this week, which is good news, but I’m still very concerned that we don’t have enough sponsors to make this work past the summer.
My plan is to official start sending daily on July 1st, but I’d like to book a few more sponsors before then. Otherwise, I might push it back a week to keep testing. My hope is that by the time we’re ready to launch, we have a lean, hungry audience ready for it.
Convertkit test 5
I really thought I had it after those last two, but then I sent a final test, or what I thought would be a final test, to a 31,000 email segment, that I would would have at least a 30% open on the first send. Yet, when I sent, this happened.
That’s a 25% open rate on the first send and a 10% on the second send. Yes, that’s a 32% open rate overall, but it’s down from a 39% open rate from the previous send. A 32% open rate isn’t terrible for a first send, but 25% is god awful for a first send.
So, now I’m back to testing. I’ve created a segment of everyone who has opened either an Author Stack or Authoresque email in the past 10 days, and built a 30,444 email segment of who should be the most active humans.
Once my excludes are added in, this should give me a 25,000-ish email segment. If this doesn’t work, then I’ll have to go back to the drawing board. My plan is still to start dailies on July 1st, but only if this testing gives good results by then. I need to consistently get at least 10,000 opens on a first send if I want to charge $300-$400 to advertisers. This first send only had 8,200 opens, which is not enough. Maybe I could still get advertisers for that price, but I don’t think there would be a lot of renewals, which are the lifeblood of an ad-based model.
Convertkit test 6
We’re still not where we need to be, but I just have to start at this point. What keeps happening is that I tighten the audience and the open rate stays the same, which means I’m getting seen by fewer and fewer people with each successive test.
Here is my latest test, sent to 26,000+ people, down from 41,000 people, and my open rate is almost the same as those first emails.
What’s supposed to happen is that the open rate goes up as you show it to more and more engaged people. Since I’ve only been emailing people who have opened at least one Authoresque email, I don’t understand how the open rate has been going down, except to say that it is summer and summer is always a bad time to be in the email game.
I think the testing phase has to be done though, and we’ll have to make decisions and move forward to build up from somewhere.
One of the hardest things about doing this work is that you can’t do something completely new but you can’t do the same thing either. People don’t actually know what they want. They know what feeling you have given them before and want to feel it again. Sometimes, the new thing will do that and sometimes a repeat of an old thing will, and nobody knows. You just have to keep trying things and doing the translation work between what people say they want and how they behave.
Programmic ads
One thing that Convertkit allows for are programmic ads, and I’ve been using them for most of these test, including when I can’t find sponsors. I haven’t really been trying for sponsors at this point, especially with the wild fluxes in open rates. From these results, you can see it’s not super encouraging. I haven’t even made $25 on them yet.
Convertkit allows for two other types of ads, and I’ve joined both the sponsorship and PPC programs which I’ll be checking out throughout this first month.
What do you think?
Does this interest you at all?
Are you running your own tests?
Why is this so hard?
Let me know in the comments.
If you saw value here, I hope you’ll consider becoming a paid member to help foster more of this type of thing. As a member, you’ll get access to over 850 exclusive posts, including books, courses, lessons, lectures, fiction books, and more, or you can give us a one-time tip to show your support.
Thanks for the detailed breakdown. My feedback is two things: Simplify and "everything in moderation".
Find what has worked consistently and focus on those. Advertising is no less iffy now than before.