Authorities in the central city of Nagoya on Tuesday advised 400,000 residents to evacuate because of worries that rivers might burst their banks, Jiji Press news agency said.
The evacuation is not compulsory and it is not known how many people have heeded the advice.
Typhoon Roke, packing winds of up to 144 kilometres (89 miles) an hour near its centre, was churning towards southwestern Japan at midday Tuesday and on course to hit the main island of Honshu on Wednesday, the weather agency said.
The storm has already dumped 400 millimetres (16 inches) of rain over the past 24 hours on the southern province of Miyazaki.
The agency warned of downpours over a wide area of the country on Wednesday, saying as much as 50 millimetres of rain could fall in an hour.
Footage broadcast by NHK showed roads in the western city of Kobe under water.
The broadcaster said a further 23,000 people in western Japan have been advised to seek shelter away from possible flood and landslide areas.
Japan was hit by Typhoon Talas earlier this month, leaving around 100 people dead or missing, mostly in the west of the country.