Hi,
Please elaborate for what type of document you are posting through the said BAPI .
You can create your own logic and fetch the database table based on that. For eg let assume that the reference(XBLNR) field will be unique for all the document ( document types you are posting ) then you can fetch the Table BKPF and check of that Reference (XBLNR) is already given for any other document or not.
Please try and make the logic on header Table (BKPF) as it would be easy to fetch .
Hope this helps.