If you have a cat, have it declawed.
Sewing binding over the edge would be the best looking but would require a upholstery sewing machine or some hand stitching with big needles.
The unraveling of the unfinished edges CAN ALSO be stopped by applying liquid rubber. Lowes and Home Depot carry liquid rubber in cans and aerosol in various colors to include black, red, white, and clear. If the carpeting is not glued to the dog house, you can also apply liquid rubber to the back of it. The liquid rubber will seal the carpet fibers together. The face of the carpet can be taped and liquid rubber can be applied to the edges and back.