In a medium saucepan, combine sugars, butter, graham cracker crumbs and milk. Bring to a boil over medium-high heat and boil for 5 minutes, stirring constantly.
In a double boiler, heat chocolate chips, butterscotch chips and peanut butter until smooth.
Pour sugar mixture over the crackers. Top with another layer of crackers. Pour chocolate mixture over that. Allow to cool.
