Yes, NOTATION is a Scrabble word!

Letter values in Scrabble US: N 1, O 1, T 1, A 1, T 1, I 1, O 1, N 1.

Is it a valid word?

NOTATION is a valid word in the official Scrabble dictionary (US), the UK Scrabble Dictionary and Words with Friends (ENABLE).

a technical system of symbols used to represent special things
a comment or instruction (usually added)
the activity of representing something by a special system of marks or characters

Also: notational system.