Robot companions are starting to become more common and people are becoming more familiar with devices such as Google Home, Alexa or Pepper, one must wonder what is the optimum way for people to control their devices? This paper presents an investigation into how much direct control people want to have of their robot companion and how dependent this is on the criticality of the tasks the robot performs. A live experiment was conducted in the University of Hertfordshire Robot House, with a robot companion performing four different type of tasks. The four tasks were: booking a doctor’s appointment, helping the user to build a Lego character, doing a dance with the user, and carrying biscuits for the user. The selection of these tasks was based on our previous research to define tasks which were relatively high and low in criticality. The main goal of the study was to find what level of direct control over their robot participants have, and if this was dependent on the criticality of the task performed by the robot. Fifty people took part in the study, and each experienced every task in a random order. Overall, it was found that participants’ perception of control was higher when the robot was performing a task in a semi-autonomous mode. However, for the task “carrying biscuits”, although participants perceived to be more in control with the robot performing the task in a semi-autonomous mode, they actually preferred to have the robot performing the task automatically (where they felt less in control). The results also show that, for the task “booking a doctor’s appointment”, considered to be the most critical of all four tasks, participants did not prefer that the robot chose the date of the appointment as they felt infantilised.