Cosmetically it's probably fine. The downsides all have to do with predictability and the ability to reason about what the wall is made of in the future.

A person who goes to eg hang a picture frame or shelf there will encounter a different material with different load bearing properties than expected. Pushing into the center of that area with EG a drill bit will not have the same physical response or give, and depending on how it was braced/integrated with the surrounding wall, the patch itself may be pushed or pulled out of place. Similar for anyone that leans on that area if it's at such a height.