Your site isn't accessible when it hasn't loaded, but is it accessible once it has? Are you thrashing the main UI thread, impeding user interaction? Did you disable zooming, making your site illegible to many? To make your clients happy, your site must be accessible: it has to make it onto to device quickly, has to work once there, and has to have good usability no matter the current physical and cognitive status of your user.
The goal of web performance is to make your customers happy. If your customers can't access your site because it's too slow, they'll leave your site and become someone else’s customers instead of yours. Similarly, even if your site does load quickly but is not accessible, it will result in your visitors being unhappy, potentially losing them as potential customer for life.
The common statistic is 19% of the US population has a disability (pdf). When it comes to web accessibility, this statistic is irrelevant. Some of the disabilities making up that 19% don’t impact web usage ability at all, whereas traits that aren't considered a "disability" by research respondents impact how users interact with their connected devices. We are all at times differently abled.
The 19% includes Americans who are paraplegic, visually impaired, hard of hearing, etc., but does not include users who temporarily can't use a mouse, see the screen, use audio or completely focus on the task at hand. The 81% of us who are not "disabled" are only temporarily abled. Almost everyone will experience some difficulty experiencing the web in some way, even just temporarily: whether it's needing reading glasses, experiencing carpal tunnel pain, or even just breaking a headset and having to go without audio.
It has taken a few years to convince developers, but performance has finally become a priority. Why? Because performance has been proven to impact the bottom line: it impacts the corporate pocket book. You know what else impacts the corporate pocketbook? A site that is not usable because it isn't accessible.
We've convinced the CTO that the dev team needs to care about how fast the site loads: why are we having so much difficulty convincing the CTO to also care about customers with disabilities? When your site has bad UX because it's too slow, because it's inaccessible or because it simply has bad user experience, your customers will become your competitors customers. Don't let that happen.