A random 300ish will be imported in 15mins, and the rest will be put on hold,
Yeah, basically, but it gets a little tricky/nuanced - there may be some semblance of order to things behind totally random. There’s just not a strict order, because the job system is multi-threaded. If it can bring in ~50 activities at a time, then there will be some vague representation of order on the scale of every 50-100 activities … put another way, I wouldn’t expect the first enqueued activity to suddenly somehow swap order with the last enqueued activity.
not sure if these are actually 150 every 15mins at 2api calls per activity multiplied out to 30,000 a day
new activities - those alerted by webhooks ‘cost’ 2 API calls each. The historic activities ‘cost’ 1 API call per 100 activities for a user and then 1 API call for each activity. So a batch of 100 activities for a user would ‘cost’ 101 API calls.
I assume the 200,000 activities are a mixture of supports, non, regular activities and new user imports?
Yeah, that’s a count of everyone.
If so is it possible to see how many other activities are in the same group as regular supporters?
These counts are expensive and I’m noticing performance issues as the queue sizes increase. That’s not to say it can’t be done - it’s just not heading down that path as things are written right now.
Do you have any plan to implement a system whereby the activities of supports are imported in chronological order as opposed to the current system?
No. The current system is wildly efficient - I push through 20-40 million jobs per day. I’m not expecting any different system to outperform this. This problem is not with my background job system, this problem resides solely on Strava having a very low API limit for my 12,379 current Strava-connected users with a growth rate of 55-227 new Strava connections per day over the last couple weeks.
Further, this system is “chronological enough” for me. The only place that this breaks down is when API limits are grossly exceeded (the past two weeks).
the actual activity numbers waiting are no longer listed on the page.
I can no longer display this information.
I’ve revamped the historic syncing from a single job that would attempt to sync 100 activities for a user in one go (which basically limited that syncing to 3 people each 15min period) … to one where the metadata for all the activities for the user are collected from Strava & enqueued in my job system. This increases the activity queue load to hundreds of thousands of activities, but makes it much more likely that all people will slowly be synchronized together (as opposed to one person at a time).
I’ve also de-prioritized this historical syncing so that subscriber activities first priority, then any new activity is prioritized, then the historical sync is worked on if there are more API calls left.
Strava has been completely unresponsive to my requests for an API rate limit increase. I’ve explained to them that I’m currently very close to the ceiling (12379 * 2 API calls per activity = 24758 API calls per day; leaving 5242 API calls for historical syncing) and that I’m roughly 60 days away from reaching that and being fully saturated.
Perhaps it’s time for those of you affected by this to submit support requests: https://support.strava.com/ (“Submit a request” in the top right), or publicly mentioning this to them on Twitter: https://twitter.com/stravasupport (Start your message with something like “Hi” and then include the at-mention, so that it’s a public tweet and not a reply that’s only shown to them). I don’t know what else to do at this point.