Deduplication service

Deduplication service

1. Introduction
2. Request example
3. Location and version
4. Limitations
   1. Request method
   2. Request size
5. Bugs and feedback

Introduction

The dedupe method of the API offers a way of detecting several types of duplicate records in primary biodiversity data sets. Apart from proper record duplicates, this method also provides a way of detecting, flagging and/or removing potential or partial duplicates -- i.e, records that have the same information in certain key fields.

This method works asynchronously, meaning that the API returns almost as soon as the file has been uploaded to the server and lets the hard work for background threads. Notifications of results are later delivered via email to the user.

Request example

This is a typical request sending a small file via cURL.

curl -X POST -H "Content-Type: text/csv" --data-binary @file.csv http://<service_url>/api/<version>/dedupe?action=remove&duplicates=strict&email=foo@bar.baz&id=occurrenceid

Location and version

See the documentation home page for information on the current URL to access the service.

Limitations

Request method

Only POST methods are allowed. GET isn't designed to send large content, and any other method doesn't really make sense in this context.

Request size

According to the Google App Engine docs, there is a 32Mb limit on request size. So, files larger than 32Mb will fail.

Bugs and feedback

Asynchronous tasks are a bit tricky in the sense that the user cannot know if the process is running smoothly or if something wrong happened. The service has been designed to inform promptly of any failure that could happen during the de-duplication process, both to the user and the service administrator, but it is very difficult to foresee all that can potentially go wrong, and the service may break without notice.

If, for some reason, you don't receive a notification of your file being ready after 24h, please create a new issue in the code repository http://www.github.com/vertnet/dedupe/issues and/or send us an email at vertnetinfo@vertnet.org