What is the output of this SQL Query?

You will get nothing in return.

Yes this query will return no record.

If you can see inner query is returning the maximum marks that is highest marks from student table. (select max(marks) from students)
and
The outer query is written as we want output from the student table that student has more than the highest marks in the students table.
(select * from students where marks > (select max(marks) from students))

So the conclusion is this query will not return any records. as there is no records in the table which has marks greater than maximum marks in student table.