A Conservative minister is facing a backlash for suggesting people with mobility and mental health problems should do “their duty” by working from home.