There is not a way to delete or remove unused custom fields, but there is a way for them to become "hidden" and not populate when segmenting or exporting contact data information.
1) Go to the Contacts module
2) Click on Configuration (found on the left panel)
3) Click on Custom Fields
4) Hover over whichever field you do not want to populate within the UI.
5) Click on Edit (tab on the right side). The Edit Custom Field will populate.
6) Check the box for "Hide this field from account admins" and click OK
This will hide the custom field from segments and from any operation that allows you export values. For example, if you export contacts from the Contacts module, the hidden field will not populate from the list of fields to export.
Should you ever need to have a hidden field become visible, go back to the custom field, Edit, and uncheck the box for "Hide this field from account admins".