The GATE Grind

GATE 2025 DA – Question 48

Machine Learning · Multi-layer perceptron and feed-forward neural networks · 2 marks · Multiple select

Which of the following statements is/are correct about the rectified linear unit (ReLU) activation function defined as $\text{ReLU}(x) = \max(x, 0)$, where $x \in \mathbb{R}$?

  1. ReLU is continuous everywhere
  2. ReLU is differentiable everywhere
  3. ReLU is not differentiable at $x = 0$
  4. $\text{ReLU}(x) = \text{ReLU}(ax)$, for all $a \in \mathbb{R}$

Practise this question in The GATE Grind →

Show answer and explanation

Correct answer: (A) ReLU is continuous everywhere; (C) ReLU is not differentiable at $x = 0$

Explanation

ReLU is 0 for $x \le 0$ and $x$ for $x \ge 0$, so the two pieces join at 0 and it is continuous everywhere. The slope is 0 on the left of 0 and 1 on the right, so there is a corner at $x = 0$ and the function is not differentiable there. $\text{ReLU}(ax) = \text{ReLU}(x)$ fails in general, for example $a = 2$ and $x = 1$ give 2 and 1.