If you are creating a Custom API and if it is an unbound API to perform any common or global action, and if you selected “Yes” for (Enabled For Workflow), you will get the following errors when you try adding input (Custom API Request Parameter) or output (Custom API Response Property) parameters of type Entity Reference or Entity.
Exception Message: Type “EntityReference” is not supported for workflow-enabled unbound custom API
Exception Message: Type “Entity” is not supported for workflow-enabled custom API
When I got these errors, googled the error message and did not find any link that described the reason for the error but I finally found the doc page that details the limitations of Custom API enable for Workflow. Please see the screenshot from the doc page below
As you can see from the screenshot, you can only use Entity Reference parameter in a bound Custom API when its Enabled for Workflow. And Entity, EntityCollection and String array parameters cannot be used in a Custom API enabled for Workflow, whether it is bound or unbound.