The checkbox of the <paper-checkbox> has an ID of "checkbox", so you could imperatively get a reference to it with Automatic node finding syntax (this.$.myPaperCheckbox.$.checkbox) to style the inner checkbox's border radius (checkbox.style.borderRadius):
attached: function() {
// <paper-checkbox id="myPaperCheckbox">
this.$.myPaperCheckbox.$.checkbox.style.borderRadius = '50%';
}
Result:

demo
You might also consider switching to <paper-radio-button>, which already uses a circular checkbox (but with a different/circular checkmark).