Kurashiki secrets

https://preview.redd.it/01daqpt9uwoh1.png?width=3356&format=png&auto=webp&s=3a732e62b6355aa2657d330f6c8842d389e99560

so i gave the old man 5 peaches and got the book but apparently that didnt count as one of the secrets in Kurashiki? can anyone help me on how to get them or do i need to progress the game first?