I think you should do that, japan is a very family-friendly country!
I would recommend you to go in this time of the year, it can be extremely warm in june-august!