Follower/ing viewer now uses the i endpoint (which returns more users at once) as well as caching (less requests needed) so its response time has decreased significantly (to 1/3 in my case)