date 'a.transaction_date' is trying to mix a date literal with a variable column value, but then also putting the column name inside single quotes, You end up essesntially doing:
to_date('a.transaction_date', 'YYYY-MM-DD')
which also gets ORA-01841, for perhaps clearer reasons.
If transaction_date is already a date then just refer to that:
SELECT DISTINCT
a.transaction_date,
to_char(a.transaction_date,'DAY') as Day
FROM transactions_table a
If it's a string - which it should not be! - then convert that value to a date properly:
SELECT DISTINCT
a.transaction_date,
to_char(to_date(a.transaction_date, 'YYYY-MM-DD'),'DAY') as Day
FROM transactions_table a
... or using whatever format mask is appropriate.
Also be aware that DAY (and the abbreviated DY, or Dy, which might be what you actually want) output depends on your NLS date language, but you can override that if you want it to always be in English, for instance:
SELECT DISTINCT
a.transaction_date,
to_char(a.transaction_date,'DY','NLS_DATE_LANGUAGE=ENGLISH') as Day
FROM transactions_table a
Quick demo using a CTE:
-- CTE for dummy date
with transactions_table(transaction_date) as (
select date '2019-01-01' from dual
union all
select date '2019-09-01' from dual
)
-- actual query
SELECT DISTINCT
a.transaction_date,
to_char(a.transaction_date,'Dy','NLS_DATE_LANGUAGE=ENGLISH') as Day
FROM transactions_table a
/
TRANSACTIO DAY
---------- ------------
2019-01-01 Tue
2019-09-01 Sun