How does one go about defining success in building a userbase when the target audience occupies a specific niche?

Niche is the sweet spot when it comes to defining success.

First ask yourself 2 questions:
Who are your customers?
Where are your customers (online, in real life, wtc)?

Answering those is success PT 1.

Next, define the messaging. What will these customers resonate with given who they are and where they are?

Finally, distribute your message with a unique position. This is done via marketing channels, some of which can be reverse-engineered since you know who and where your customwrs are.

Success, then, is when a customer profile you identified in step 1 (who/where) converts from a defined message distributed through a specific channel.