Williams controlled the ball inside the box and rifled it pass John Sullivan after substitute Wayne Brown had halved the home side`s arrears with a 66th-minute effort.
Charlton were cruising through an eighth-minute header from Paul Benson and a 46th-minute strike from Kyel Reid before having two men sent off in the space of six minutes.
Reid was dismissed for a late challenge that saw James Tunnicliffe leave the field on a stretcher and skipper Jose Semedo followed on the hour for a second bookable offence.
Charlton boss Chris Powell was furious at the second red card and even more angry when appealing for handball against Williams before he buried Rovers' equaliser.
Top scorer Will Hoskins wasted a glorious chance to win it for Rovers in the dying moments, firing his shot into the ground and seeing the ball bounce up over the bar with only Sullivan to beat.
Source: PA
Source: PA