From a15fae8a2e73818a8560eecae8c39ac3be92457f Mon Sep 17 00:00:00 2001 From: Abhishek Pokhriyal Date: Tue, 17 Feb 2026 10:49:34 +0530 Subject: [PATCH 1/3] docs: add Dalgo integration entity classification (Step 1) Classifies all CiviCRM entities (Collection Camp, Dropping Center, Events, Activities, Volunteers, GCOC, Mailing, Dispatch, Feedback, etc.) as native or custom (ECK), with API calls, DB tables, and timestamp field availability to support Dalgo integration. Co-Authored-By: Claude Opus 4.6 --- .../step-1-entity-classification.md | 191 ++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100644 docs/dalgo-integration/step-1-entity-classification.md diff --git a/docs/dalgo-integration/step-1-entity-classification.md b/docs/dalgo-integration/step-1-entity-classification.md new file mode 100644 index 000000000..c541d1247 --- /dev/null +++ b/docs/dalgo-integration/step-1-entity-classification.md @@ -0,0 +1,191 @@ +# Dalgo Integration - Entity Classification + +This document classifies CiviCRM entities in the Goonj CRM as **native** (built-in CiviCRM with APIv3/v4) or **custom** (defined by extensions via the ECK framework), to support integration with Dalgo. + +## Architecture Overview + +The Goonj CRM uses two mechanisms to model its domain entities: + +1. **Native CiviCRM entities** - built-in entities accessed via `\Civi\Api4\{EntityName}` (e.g., `Event`, `Activity`, `Contact`, `Contribution`) +2. **ECK (Entity Construction Kit) entities** - custom entities created via the `de.systopia.eck` extension, accessed via API entity name `Eck_{EntityTypeName}`. All ECK entities are automatically API-exposed. + +ECK entities share base fields defined in `CRM/Eck/DAO/Entity.php`: `id`, `title`, `subtype`, `created_id`, `modified_id`, `created_date`, `modified_date`. + +## Summary Table + +| # | Entity Name | Classification | API Entity Name | `created_date` | `modified_date` | +|---|-------------|---------------|----------------|----------------|-----------------| +| 1 | Collection Camp | Custom (ECK) | `Eck_Collection_Camp` | Yes | Yes | +| 2 | Dropping Center | Custom (ECK) | `Eck_Collection_Camp` | Yes | Yes | +| 3 | Events | Native | `Event` | Yes | No | +| 4 | Activities | Native | `Activity` | Yes | Yes | +| 5 | Volunteer Info | Native (Contact subtype) | `Contact` | Yes | Yes | +| 6 | GCOC | Mixed | `Eck_Institution_Visit`, `Contribution`, `Activity` | Varies | Varies | +| 7 | Mailing Info | Native | `Mailing` | Yes | Yes | +| 8 | Dispatch | Custom (ECK) | `Eck_Collection_Source_Vehicle_Dispatch` | Yes | Yes | +| 9 | Dispatch Ack | Custom field group on Dispatch | `Eck_Collection_Source_Vehicle_Dispatch` | Inherited | Inherited | +| 10 | Logistics | Custom field group on Collection Camp | `Eck_Collection_Camp` | Inherited | Inherited | +| 11 | Outcome | Custom field group on Collection Camp | `Eck_Collection_Camp` | Inherited | Inherited | +| 12 | Feedback | Custom (ECK) | `Eck_Collection_Source_Feedback` | Yes | Yes | + +## Detailed Classification + +### 1. Collection Camp (Individual/Institution) + +- **Type:** Custom (ECK entity) +- **ECK Entity Type:** `Collection_Camp` +- **ECK Subtypes:** `Collection_Camp` (Individual), `Institution_Collection_Camp` (Institution) +- **API call:** `Eck_Collection_Camp.get` with `subtype:name = 'Collection_Camp'` or `'Institution_Collection_Camp'` +- **DB table:** `civicrm_eck_collection_camp` +- **Custom field groups:** `Collection_Camp_Core_Details`, `Collection_Camp_Intent_Details`, `Collection_Camp_QR_Code`, `Logistics_Coordination`, `Camp_Outcome`, `Volunteer_Camp_Feedback` +- **`created_date`:** Yes +- **`modified_date`:** Yes + +Both Individual and Institution variants are subtypes of the same ECK entity type `Collection_Camp`. + +### 2. Dropping Center (Individual/Institution) + +- **Type:** Custom (ECK entity - subtype of `Collection_Camp`) +- **ECK Entity Type:** `Collection_Camp` (same entity type as Collection Camp) +- **ECK Subtypes:** `Dropping_Center` (Individual), `Institution_Dropping_Center` (Institution) +- **API call:** `Eck_Collection_Camp.get` with `subtype:name = 'Dropping_Center'` or `'Institution_Dropping_Center'` +- **DB table:** `civicrm_eck_collection_camp` (same table as Collection Camp) +- **Custom field groups:** `Dropping_Centre`, `Collection_Camp_Core_Details`, `Collection_Camp_QR_Code` +- **Companion entity:** `Eck_Dropping_Center_Meta` for monthly status tracking +- **`created_date`:** Yes +- **`modified_date`:** Yes + +### 3. Events + +- **Type:** Native CiviCRM +- **API call:** `Event.get` +- **DB table:** `civicrm_event` +- **`created_date`:** Yes +- **`modified_date`:** No (CiviCRM Events lack a native `modified_date`) + +Used for Goonj-initiated events (Rural Planned Visits, Book Fairs, Chaupals, etc.) with event type codes defined in `config/eventCode.php`. + +### 4. Activities + +- **Type:** Native CiviCRM +- **API call:** `Activity.get` +- **DB table:** `civicrm_activity` +- **`created_date`:** Yes +- **`modified_date`:** Yes + +Used for Induction, Volunteering, and Material Contribution tracking. Note: there is also a separate custom ECK entity `Eck_Collection_Camp_Activity` for camp-specific activity records. + +### 5. Volunteer Information + +- **Type:** Native CiviCRM (Contact with subtype) +- **API call:** `Contact.get` with `contact_sub_type:name = 'Volunteer'` +- **DB table:** `civicrm_contact` +- **Custom field group:** `Volunteer_fields` +- **`created_date`:** Yes +- **`modified_date`:** Yes + +Volunteer status (registered, inducted, active, inactive) is tracked via custom fields and Induction activity status in `InductionService.php`. + +### 6. GCOC (Material, Monetary, Visit tracking) + +GCOC is a business concept spanning multiple entities, not a single entity. + +| Sub-entity | Type | API call | `created_date` | `modified_date` | +|------------|------|----------|----------------|-----------------| +| Visit tracking | Custom (ECK) | `Eck_Institution_Visit.get` with `subtype:name = 'Urban_Visit'` | Yes | Yes | +| Material contribution | Native | `Activity.get` (with material contribution activity type) | Yes | Yes | +| Monetary contribution | Native | `Contribution.get` | No (`receive_date` instead) | No | + +Event code mapping: `'Urban Visit' => 'GCOC'` in `config/eventCode.php`. + +### 7. Mailing Information + +- **Type:** Native CiviCRM +- **API call:** `Mailing.get` +- **DB table:** `civicrm_mailing` +- **`created_date`:** Yes +- **`modified_date`:** Yes + +Note: The codebase primarily uses `\CRM_Utils_Mail::send()` for transactional emails (logistics notifications, feedback reminders), which is separate from the CiviMail `Mailing` entity. + +### 8. Dispatch + +- **Type:** Custom (ECK entity) +- **ECK Entity Type:** `Collection_Source_Vehicle_Dispatch` +- **API call:** `Eck_Collection_Source_Vehicle_Dispatch.get` +- **DB table:** `civicrm_eck_collection_source_vehicle_dispatch` +- **Custom field group:** `Camp_Vehicle_Dispatch` (Collection Camp, Date/Time, Bags, Weight, Vehicle Category, PU Center, etc.) +- **`created_date`:** Yes +- **`modified_date`:** Yes + +### 9. Dispatch Acknowledgement + +- **Type:** Not a separate entity - custom field group on the Dispatch entity +- **API call:** `Eck_Collection_Source_Vehicle_Dispatch.get` (same as Dispatch, reading `Acknowledgement_For_Logistics.*` fields) +- **Custom field group:** `Acknowledgement_For_Logistics` (fields: `No_of_bags_received_at_PU_Office`, `Ack_Email_Sent`, `Verified_By`) +- **`created_date`:** Inherited from parent Dispatch entity +- **`modified_date`:** Inherited from parent Dispatch entity + +### 10. Logistics + +- **Type:** Not a separate entity - custom field group on `Collection_Camp` +- **API call:** `Eck_Collection_Camp.get` (reading `Logistics_Coordination.*` fields) +- **Custom field group:** `Logistics_Coordination` (fields: `Camp_to_be_attended_by`, `Email_Sent`, `Self_Managed_By_Camp_Organiser`, `Feedback_Email_Sent`) +- **`created_date`:** Inherited from parent Collection Camp entity +- **`modified_date`:** Inherited from parent Collection Camp entity + +### 11. Outcome + +- **Type:** Not a separate entity - custom field group on `Collection_Camp` +- **API call:** `Eck_Collection_Camp.get` (reading `Camp_Outcome.*` fields) +- **Custom field group:** `Camp_Outcome` (fields: `Rate_the_camp`, `Camp_Status_Completion_Date`, `Filled_By`, `Last_Reminder_Sent`) +- **`created_date`:** Inherited from parent Collection Camp entity +- **`modified_date`:** Inherited from parent Collection Camp entity + +### 12. Feedback + +- **Type:** Custom (ECK entity) +- **ECK Entity Type:** `Collection_Source_Feedback` +- **API call:** `Eck_Collection_Source_Feedback.get` +- **DB table:** `civicrm_eck_collection_source_feedback` +- **Custom field group:** `Collection_Source_Feedback` (camp code, address, ratings, difficulties, images) +- **`created_date`:** Yes +- **`modified_date`:** Yes + +## All ECK Entity Types + +| ECK Entity Type | API Entity Name | DB Table | +|----------------|----------------|----------| +| `Collection_Camp` | `Eck_Collection_Camp` | `civicrm_eck_collection_camp` | +| `Collection_Camp_Activity` | `Eck_Collection_Camp_Activity` | `civicrm_eck_collection_camp_activity` | +| `Collection_Source_Feedback` | `Eck_Collection_Source_Feedback` | `civicrm_eck_collection_source_feedback` | +| `Collection_Source_Vehicle_Dispatch` | `Eck_Collection_Source_Vehicle_Dispatch` | `civicrm_eck_collection_source_vehicle_dispatch` | +| `Dropping_Center_Meta` | `Eck_Dropping_Center_Meta` | `civicrm_eck_dropping_center_meta` | +| `Institution_Visit` | `Eck_Institution_Visit` | `civicrm_eck_institution_visit` | + +## All Collection_Camp Subtypes + +| Subtype | Description | +|---------|-------------| +| `Collection_Camp` | Individual Collection Camp | +| `Dropping_Center` | Individual Dropping Center | +| `Institution_Collection_Camp` | Institutional Collection Camp | +| `Institution_Dropping_Center` | Institutional Dropping Center | +| `Goonj_Activities` | Individual Goonj Activities | +| `Institution_Goonj_Activities` | Institutional Goonj Activities | + +## Verification Commands + +```bash +# List all registered ECK entity types +cv api4 EckEntityType.get +s name + +# Confirm API exposure and list all fields for an ECK entity +cv api4 Eck_.getFields + +# Example: List Collection Camp fields +cv api4 Eck_Collection_Camp.getFields + +# Example: Get Collection Camps with subtype filter +cv api4 Eck_Collection_Camp.get +w subtype:name=Collection_Camp +l 5 +``` From dca07c09c67f015f394604fa0acb2d93db283367 Mon Sep 17 00:00:00 2001 From: Abhishek Pokhriyal Date: Tue, 17 Feb 2026 10:53:53 +0530 Subject: [PATCH 2/3] docs: add Dalgo integration action plan Co-Authored-By: Claude Opus 4.6 --- docs/dalgo-integration/README.md | 111 +++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 docs/dalgo-integration/README.md diff --git a/docs/dalgo-integration/README.md b/docs/dalgo-integration/README.md new file mode 100644 index 000000000..79d6576d5 --- /dev/null +++ b/docs/dalgo-integration/README.md @@ -0,0 +1,111 @@ +# ColoredCow Action Plan: CiviCRM API Audit for Dalgo + +## Step 1: Categorize entities by API type + +You built these extensions as CiviCRM plugins. So you already know which are native CiviCRM entities and which are Goonj-custom. Classify each: + +**Likely native CiviCRM APIs (APIv3/v4 out of the box):** +- Events → `Event` API +- Activities → `Activity` API +- Volunteer Information → `Contact` API (filtered by contact type/group/tag) +- Mailing Information → `Mailing` + `MailingJob` + `MailingEventBounce/Delivered/Opened` APIs + +**Goonj-custom entities (built as CiviCRM plugins — need verification):** +- Collection Camp (Individual/Institution) +- Dropping Center (Individual/Institution) +- Dispatch +- Dispatch Acknowledgement +- Logistics +- Outcome +- Feedback +- GCOC + +For each custom entity — confirm whether CiviCRM auto-exposes an API (it usually does for entities registered via `hook_civicrm_entityTypes`), or if a custom API wrapper is needed. + +## Step 2: For each of the 14 entities, produce this + +Hit the Goonj CiviCRM instance and document: + +| Check | What to capture | +|-------|----------------| +| **API endpoint** | Exact entity name and action (e.g., `CollectionCamp.get`) | +| **Fields returned** | Full field list from the API response — then cross-check against what the CSV asks for | +| **Missing fields** | Fields the CSV requires but the API doesn't return | +| **Filters available** | Can you filter by `is_active`, date range, geography, contact type? | +| **Incremental sync** | Does the response include `created_date` and `modified_date`? Can you sort by `modified_date`? | +| **Pagination** | Does `limit` + `offset` work? What's the max page size? | +| **PII concern** | For Collection Camp, Dropping Center, Volunteers — the CSV says "avoid personal information like email, phone, address". Confirm these fields can be excluded in the API call or need to be stripped. | +| **Aggregation** | For Contributors (Collection Camp, Dropping Center, Events) — does the API return counts, or do you get raw records that Dalgo will aggregate? | + +## Step 3: Test incrementals specifically + +Dalgo was explicit — they need incremental sync to work. For each entity: + +1. Call the API with `sort=modified_date ASC` and `options.limit=10` +2. Confirm `modified_date` updates when a record is edited +3. Confirm `created_date` exists and is reliable +4. If either is missing on a custom entity — that's a code fix ColoredCow needs to make before handing off + +## Step 4: Handle the Contacts/Volunteers problem + +The CSV asks for Volunteer info — name, contact ID, status (registered, inducted, active, inactive). But there are 3 lakh contacts. + +- Check if volunteers are in a specific CiviCRM **Group** or tagged with a **Tag** that separates them from general contacts +- Test the `Contact.get` API with that group/tag filter + `is_active` filter +- Report the count — how many volunteers come back? If it's manageable (say <50k), Dalgo can sync them. If it's still huge, flag it. + +## Step 5: Map the entity relationships + +This is the part Goonj didn't document but you already know from building the system. Write down: + +``` +Collection Camp + ├── has Dispatch(es) + │ └── has Dispatch Acknowledgement + ├── has Logistics + ├── has Outcome + ├── has Feedback + ├── has Volunteers (contacts) + └── has Contributors (contacts, need count only) + +Dropping Center + └── (same sub-entity structure as Collection Camp?) + +Event + ├── has Participants (registered/attended) + └── has Contributions (monetary) + +GCOC + └── (what links to what?) +``` + +Document the **foreign key** for each relationship — what field in Dispatch links back to Collection Camp? Is it `case_id`? `activity_id`? A custom field? + +This is critical for Dalgo to build joins in the warehouse. You don't need Shivangi for this — you built the system. + +## Step 6: Compile the deliverable + +One document (or spreadsheet) with: + +| Entity | API Endpoint | Fields Available | Fields Missing | Supports Incremental | Pagination | Related Entities & FK | Notes | +|--------|-------------|-----------------|----------------|---------------------|------------|----------------------|-------| +| | | | | | | | | + +Plus the entity relationship map from Step 5. + +## Step 7: Share with Dalgo + +Hand off the document. At this point Dalgo has everything they need to build the Airbyte connector. The only open items should be: +- Any missing fields that need a code fix (with your timeline) +- Any entities where incremental sync needs to be added (with your timeline) + +## Timeline + +| Day | Work | +|-----|------| +| Day 1 | Steps 1–2: Categorize entities + hit the APIs, document fields | +| Day 2 | Steps 3–4: Test incrementals + figure out the contacts/volunteers filtering | +| Day 3 | Steps 5–6: Map relationships with foreign keys + compile the deliverable | +| Day 4 | Step 7: Share with Dalgo + fix any gaps found | + +Four working days. No dependency on Goonj for any of this — you have the CiviCRM instance, you built the plugins, you have the CSV. From b5e08c3ad084d1cc07f0b734e1fdcbc38da86a71 Mon Sep 17 00:00:00 2001 From: Abhishek Pokhriyal Date: Tue, 17 Feb 2026 13:25:16 +0530 Subject: [PATCH 3/3] docs: add Dalgo integration API audit (Step 2) Document REST API endpoint testing results against staging, including field lists, incremental sync support, pagination, authentication (authx), entity relationships with foreign keys, and PII handling for all 11 entities. Co-Authored-By: Claude Opus 4.6 --- docs/dalgo-integration/step-2-api-audit.md | 363 +++++++++++++++++++++ 1 file changed, 363 insertions(+) create mode 100644 docs/dalgo-integration/step-2-api-audit.md diff --git a/docs/dalgo-integration/step-2-api-audit.md b/docs/dalgo-integration/step-2-api-audit.md new file mode 100644 index 000000000..bd2a749db --- /dev/null +++ b/docs/dalgo-integration/step-2-api-audit.md @@ -0,0 +1,363 @@ +# Dalgo Integration — Step 2: API Audit + +This document captures the results of testing CiviCRM's REST API on the staging environment (`https://staging-crm.goonj.org`) for all entities identified in Step 1. + +## REST API Access + +### Endpoint + +``` +POST https://staging-crm.goonj.org/civicrm/ajax/api4/{Entity}/{action} +``` + +Required headers: +``` +Content-Type: application/json +X-Requested-With: XMLHttpRequest +``` + +### Authentication + +- **`authx` extension is active.** Both `Authorization: Bearer ` and `X-Civi-Auth: Bearer ` return `HTTP 401 Invalid credential` with a bad key (confirms the auth mechanism works — just needs a valid key). +- **Without auth:** ECK entity base fields and Event records are accessible. Contact, Contribution, and custom fields require authentication. +- **For Dalgo's Airbyte connector:** Generate an API key for an admin CiviCRM contact. The connector will use `Authorization: Bearer ` on every request. + +### API Key Generation (run on staging server) + +```bash +# Find admin contact ID +cv api4 Contact.get +w 'email=' +s id + +# Set an API key +cv api4 Contact.update +w 'id=' +v 'api_key=' +``` + +--- + +## Entity API Audit + +### 1. Collection Camp (Individual / Institution) + +| Check | Result | +|-------|--------| +| **API endpoint** | `Eck_Collection_Camp.get` | +| **Subtype filter** | `where: [["subtype:name","=","Collection_Camp"]]` (Individual) or `"Institution_Collection_Camp"` (Institution) | +| **Base fields** | `id`, `title`, `subtype`, `created_id`, `modified_id`, `created_date`, `modified_date` | +| **Custom field groups** | `Collection_Camp_Intent_Details` (26 fields), `Collection_Camp_QR_Code` (2), `Collection_Camp_Core_Details` (6), `Logistics_Coordination` (14), `Camp_Outcome` (20), `Volunteer_Camp_Feedback` (11), `Dropping_Centre` (19), `Dropping_Center_Outcome` (5), `Institution_Dropping_Center_Intent` (16), `Institution_Collection_Camp_Intent` (18), `Institution_collection_camp_Review` (11), `Institution_Collection_Camp_Logistics` (11), `Institution_Dropping_Center_Review` (5), `Institution_Dropping_Center_Logistics` (2), `Goonj_Activities` (24), `Goonj_Activities_Outcome` (12), `Institution_Goonj_Activities` (23), `Institution_Goonj_Activities_Outcome` (12), `Institution_Goonj_Activities_Logistics` (5), `Core_Contribution_Details` (5) | +| **Incremental sync** | `created_date`: Yes, `modified_date`: Yes. Sorting by `modified_date` confirmed working. | +| **Pagination** | `limit` + `offset` confirmed working | +| **PII fields to exclude** | `Collection_Camp_Intent_Details.Name`, `Collection_Camp_Intent_Details.Contact_Number`, `Dropping_Centre.Name`, `Dropping_Centre.Contact_Number` | +| **FK fields** | `Collection_Camp_Core_Details.Contact_Id` → Contact ID of the organizer | +| **Notes** | All subtypes (Collection_Camp, Dropping_Center, Institution_Collection_Camp, Institution_Dropping_Center, Goonj_Activities, Institution_Goonj_Activities) share this single ECK entity type. Filter by `subtype:name`. | + +### 2. Dropping Center (Individual / Institution) + +| Check | Result | +|-------|--------| +| **API endpoint** | `Eck_Collection_Camp.get` (same entity as Collection Camp) | +| **Subtype filter** | `where: [["subtype:name","=","Dropping_Center"]]` or `"Institution_Dropping_Center"` | +| **Custom field groups** | `Dropping_Centre` (main fields), `Dropping_Center_Outcome`, `Institution_Dropping_Center_Intent`, `Institution_Dropping_Center_Review`, `Institution_Dropping_Center_Logistics` | +| **Companion entity** | `Eck_Dropping_Center_Meta` for monthly status/visit/donation tracking | +| **Incremental sync** | Yes (`created_date`, `modified_date`) | +| **Pagination** | Yes | + +### 3. Dropping Center Meta + +| Check | Result | +|-------|--------| +| **API endpoint** | `Eck_Dropping_Center_Meta.get` | +| **Custom field groups** | `Dropping_Center_Meta` (FK fields), `Visit` (date, visited_by, feedback), `Donation` (box numbers, amounts, dates), `Logistics` (materials, dates), `Status` (status, closing reason, dates) | +| **FK fields** | `Dropping_Center_Meta.Dropping_Center` → `Eck_Collection_Camp.id` (Individual DC), `Dropping_Center_Meta.Institution_Dropping_Center` → `Eck_Collection_Camp.id` (Institution DC) | +| **Incremental sync** | Yes | +| **Pagination** | Yes | + +### 4. Dispatch + +| Check | Result | +|-------|--------| +| **API endpoint** | `Eck_Collection_Source_Vehicle_Dispatch.get` | +| **Custom field groups** | `Camp_Vehicle_Dispatch` (27 fields), `Acknowledgement_For_Logistics` (7 fields — Dispatch Ack), `Camp_Institution_Data` (4 fields) | +| **FK fields** | `Camp_Vehicle_Dispatch.Collection_Camp` → `Eck_Collection_Camp.id`, `Camp_Vehicle_Dispatch.Dropping_Center` → `Eck_Collection_Camp.id`, `Camp_Vehicle_Dispatch.Institution_Collection_Camp` → `Eck_Collection_Camp.id`, `Camp_Vehicle_Dispatch.Institution_Dropping_Center` → `Eck_Collection_Camp.id`, `Camp_Vehicle_Dispatch.To_which_PU_Center_material_is_being_sent` → Processing Unit (OptionValue) | +| **Incremental sync** | Yes | +| **Pagination** | Yes | +| **Notes** | Dispatch Acknowledgement is NOT a separate entity — it's the `Acknowledgement_For_Logistics` custom field group on this same entity. | + +### 5. Feedback + +| Check | Result | +|-------|--------| +| **API endpoint** | `Eck_Collection_Source_Feedback.get` | +| **Custom field group** | `Collection_Source_Feedback` (12 fields: camp code, address, ratings, difficulties, images) | +| **FK fields** | `Collection_Source_Feedback.Collection_Camp_Code` → `Eck_Collection_Camp.id`, `Collection_Source_Feedback.Filled_By` → `Contact.id` | +| **Incremental sync** | Yes | +| **Pagination** | Yes | +| **Record count (staging)** | 24 | + +### 6. Institution Visit (GCOC) + +| Check | Result | +|-------|--------| +| **API endpoint** | `Eck_Institution_Visit.get` | +| **Subtype filter** | `where: [["subtype:name","=","Urban_Visit"]]` | +| **Custom field groups** | `Urban_Planned_Visit` (38 fields), `Visit_Feedback` (6 fields) | +| **FK fields** | `Urban_Planned_Visit.Institution` → `Contact.id` (Organization), `Urban_Planned_Visit.Select_Individual` → `Contact.id`, `Urban_Planned_Visit.Institution_POC` → `Contact.id`, `Urban_Planned_Visit.Which_Goonj_Processing_Center_do_you_wish_to_visit_` → OptionValue | +| **Incremental sync** | Yes | +| **Pagination** | Yes | +| **Record count (staging)** | 35 | + +### 7. Camp Activity + +| Check | Result | +|-------|--------| +| **API endpoint** | `Eck_Collection_Camp_Activity.get` | +| **Custom field group** | `Collection_Camp_Activity` (10 fields) | +| **FK fields** | `Collection_Camp_Activity.Collection_Camp_Id` → `Eck_Collection_Camp.id`, `Collection_Camp_Activity.Organizing_Person` → `Contact.id`, `Collection_Camp_Activity.Attending_Goonj_PoC` → `Contact.id` | +| **Incremental sync** | Yes | +| **Pagination** | Yes | + +### 8. Event + +| Check | Result | +|-------|--------| +| **API endpoint** | `Event.get` | +| **Base fields** | 70+ native CiviCRM fields (see full list below) | +| **Custom field groups** | `Goonj_Events` (5 fields), `Goonj_Events_Outcome` (16 fields), `Event_QR` (2), `Goonj_Events_Feedback` (2), `Rural_Planned_Visit` (10), `Rural_Planned_Visit_Outcome` (13), `Rural_Planned_Visit_Logistics` (1) | +| **Incremental sync** | `created_date`: Yes. **`modified_date`: NO — CiviCRM Events lack a native `modified_date`.** | +| **Pagination** | Yes | +| **PII concern** | None on Event itself (PII is on Participant records) | +| **BLOCKER** | No `modified_date` means incremental sync requires a workaround. Options: (1) Add a `hook_civicrm_post` to stamp a custom `modified_date` field, (2) Use full-refresh sync for Events only, (3) Track changes via Activity log. | + +### 9. Activity + +| Check | Result | +|-------|--------| +| **API endpoint** | `Activity.get` | +| **Base fields** | `id`, `source_record_id`, `activity_type_id`, `subject`, `activity_date_time`, `status_id`, `created_date`, `modified_date`, etc. | +| **Custom field groups** | `Induction_Fields` (6), `Material_Contribution` (17), `Office_Visit` (10), `Volunteering_Activity` (5), `Coordinator` (2), `Institution_Material_Contribution` (12), `Event_Feedbacks` (6), `Rural_Planned_Visit_Feedback` (9), `Events_Feedback` (5), `Meeting_Activity` (11) | +| **FK fields** | `Material_Contribution.Collection_Camp` → `Eck_Collection_Camp.id`, `Material_Contribution.Dropping_Center` → `Eck_Collection_Camp.id`, `Material_Contribution.Event` → `Event.id`, `Volunteering_Activity.Collection_Camp` → `Eck_Collection_Camp.id`, `Volunteering_Activity.Urban_Planned_Visit` → `Eck_Institution_Visit.id`, `source_contact_id` → `Contact.id`, `target_contact_id` → `Contact.id[]`, `assignee_contact_id` → `Contact.id[]` | +| **Incremental sync** | Yes (`created_date`, `modified_date`) | +| **Pagination** | Yes | +| **Notes** | Activities are polymorphic — filter by `activity_type_id:name` to get specific types (e.g., `Material Contribution`, `Induction`, `Volunteering Activity`). Requires auth. | + +### 10. Contact (Volunteers) + +| Check | Result | +|-------|--------| +| **API endpoint** | `Contact.get` | +| **Subtype filter** | `where: [["contact_sub_type","CONTAINS","Volunteer"]]` | +| **Base fields** | `id`, `display_name`, `contact_type`, `contact_sub_type`, `first_name`, `last_name`, `created_date`, `modified_date`, etc. | +| **Custom field groups** | `Individual_fields` (15), `Contact_QR_Code` (1), `Source_Tracking` (1), `Review` (8), `Volunteer_fields` (2+) | +| **Incremental sync** | Yes (`created_date`, `modified_date`) | +| **Pagination** | Yes | +| **PII fields to EXCLUDE** | `email` (via Email entity), `phone` (via Phone entity), `street_address` (via Address entity), `first_name`, `last_name` — select only: `id`, `display_name`, `contact_sub_type`, `Volunteer_fields.*`, `Individual_fields.Status`, `created_date`, `modified_date` | +| **Auth required** | Yes — returns empty without auth. | +| **Count** | Not testable without auth. Needs server-side `cv api4 Contact.getCount +w 'contact_sub_type:name=Volunteer'` | + +### 11. Contribution (GCOC Monetary) + +| Check | Result | +|-------|--------| +| **API endpoint** | `Contribution.get` | +| **Base fields** | `id`, `contact_id`, `total_amount`, `receive_date`, `contribution_status_id`, `payment_instrument_id`, `currency`, `source`, etc. | +| **Custom field groups** | `Cheque_Number` (2), `Wire_Transfer` (2), `Contribution_Details` (12 — includes `Source`, `PU_Source`, `Events` FK) | +| **FK fields** | `contact_id` → `Contact.id`, `Contribution_Details.Source` → OptionValue (PU Office), `Contribution_Details.Events` → `Event.id` | +| **Incremental sync** | **`modified_date`: NO. `created_date`: NO.** Has `receive_date`, `receipt_date`, `thankyou_date`, `cancel_date` as date proxies. | +| **Pagination** | Yes | +| **Auth required** | Yes — returns 403 without auth. | +| **BLOCKER** | No `created_date` or `modified_date`. For incremental sync, options: (1) Use `receive_date` as creation proxy, (2) Query by `id > last_synced_id`, (3) Add timestamps via a hook. | + +### 12. Mailing + +| Check | Result | +|-------|--------| +| **API endpoint** | `Mailing.get` | +| **Base fields** | `id`, `name`, `subject`, `from_name`, `from_email`, `status`, `created_date`, `modified_date`, `scheduled_date`, `start_date`, `end_date`, etc. | +| **Stats fields** | `stats_intended_recipients`, `stats_successful`, `stats_opens_total`, `stats_opens_unique`, `stats_clicks_total`, `stats_bounces`, `stats_unsubscribes`, etc. | +| **Incremental sync** | Yes (`created_date`, `modified_date`) | +| **Pagination** | Yes | + +--- + +## Incremental Sync Summary + +| Entity | `created_date` | `modified_date` | Incremental Ready | +|--------|---------------|-----------------|-------------------| +| Eck_Collection_Camp | Yes | Yes | **Yes** | +| Eck_Collection_Source_Vehicle_Dispatch | Yes | Yes | **Yes** | +| Eck_Collection_Source_Feedback | Yes | Yes | **Yes** | +| Eck_Institution_Visit | Yes | Yes | **Yes** | +| Eck_Collection_Camp_Activity | Yes | Yes | **Yes** | +| Eck_Dropping_Center_Meta | Yes | Yes | **Yes** | +| Event | Yes | **No** | **No — needs fix** | +| Activity | Yes | Yes | **Yes** | +| Contact | Yes | Yes | **Yes** | +| Contribution | **No** | **No** | **No — needs fix** | +| Mailing | Yes | Yes | **Yes** | + +### Blockers Requiring Code Fixes + +1. **Event: No `modified_date`** — CiviCRM's `civicrm_event` table doesn't have a `modified_date` column. Fix: Add a `hook_civicrm_post` in `goonjcustom` to stamp a custom field on Event update. Alternatively, accept full-refresh sync for Events (lower volume entity). + +2. **Contribution: No `created_date` or `modified_date`** — CiviCRM's `civicrm_contribution` table has `receive_date` but not proper audit timestamps. Fix: Use `receive_date` as a creation proxy and `id > last_synced_id` for incremental, or add a post-hook to stamp custom timestamps. + +--- + +## Entity Relationship Map (Foreign Keys) + +``` +Collection Camp (Eck_Collection_Camp, subtype: Collection_Camp) + │ + ├── Dispatch (Eck_Collection_Source_Vehicle_Dispatch) + │ FK: Camp_Vehicle_Dispatch.Collection_Camp → Eck_Collection_Camp.id + │ └── Dispatch Ack (same entity, Acknowledgement_For_Logistics.* fields) + │ + ├── Feedback (Eck_Collection_Source_Feedback) + │ FK: Collection_Source_Feedback.Collection_Camp_Code → Eck_Collection_Camp.id + │ + ├── Camp Activity (Eck_Collection_Camp_Activity) + │ FK: Collection_Camp_Activity.Collection_Camp_Id → Eck_Collection_Camp.id + │ + ├── Logistics (same entity, Logistics_Coordination.* fields) + │ + ├── Outcome (same entity, Camp_Outcome.* fields) + │ + ├── Volunteer Feedback (same entity, Volunteer_Camp_Feedback.* fields) + │ + ├── Material Contribution (Activity, type: Material Contribution) + │ FK: Material_Contribution.Collection_Camp → Eck_Collection_Camp.id + │ + ├── Volunteering Activity (Activity, type: Volunteering Activity) + │ FK: Volunteering_Activity.Collection_Camp → Eck_Collection_Camp.id + │ + └── Organizer Contact + FK: Collection_Camp_Core_Details.Contact_Id → Contact.id + + +Dropping Center (Eck_Collection_Camp, subtype: Dropping_Center) + │ + ├── Dispatch + │ FK: Camp_Vehicle_Dispatch.Dropping_Center → Eck_Collection_Camp.id + │ + ├── Dropping Center Meta (Eck_Dropping_Center_Meta) + │ FK: Dropping_Center_Meta.Dropping_Center → Eck_Collection_Camp.id + │ (monthly status tracking: Visit, Donation, Logistics, Status) + │ + ├── Material Contribution (Activity) + │ FK: Material_Contribution.Dropping_Center → Eck_Collection_Camp.id + │ + └── Outcome (same entity, Dropping_Center_Outcome.* fields) + + +Institution Collection Camp (Eck_Collection_Camp, subtype: Institution_Collection_Camp) + │ + ├── Dispatch + │ FK: Camp_Vehicle_Dispatch.Institution_Collection_Camp → Eck_Collection_Camp.id + │ + └── (same sub-entities as Collection Camp, via Institution_* custom field groups) + + +Institution Dropping Center (Eck_Collection_Camp, subtype: Institution_Dropping_Center) + │ + ├── Dispatch + │ FK: Camp_Vehicle_Dispatch.Institution_Dropping_Center → Eck_Collection_Camp.id + │ + ├── Dropping Center Meta + │ FK: Dropping_Center_Meta.Institution_Dropping_Center → Eck_Collection_Camp.id + │ + └── (same sub-entities as Dropping Center, via Institution_* custom field groups) + + +Event (native CiviCRM) + │ + ├── Participants → Participant.get (FK: event_id → Event.id) + │ + ├── Contributions + │ FK: Contribution_Details.Events → Event.id + │ + ├── Material Contribution (Activity) + │ FK: Material_Contribution.Event → Event.id + │ + └── Outcome (same entity, Goonj_Events_Outcome.* fields) + + +GCOC / Institution Visit (Eck_Institution_Visit, subtype: Urban_Visit) + │ + ├── Institution + │ FK: Urban_Planned_Visit.Institution → Contact.id (Organization) + │ + ├── Individual + │ FK: Urban_Planned_Visit.Select_Individual → Contact.id + │ + ├── Institution POC + │ FK: Urban_Planned_Visit.Institution_POC → Contact.id + │ + ├── Visit Feedback (same entity, Visit_Feedback.* fields) + │ + ├── Volunteering Activity (Activity) + │ FK: Volunteering_Activity.Urban_Planned_Visit → Eck_Institution_Visit.id + │ + └── Processing Center + FK: Urban_Planned_Visit.Which_Goonj_Processing_Center_do_you_wish_to_visit_ → OptionValue + + +Contact (Volunteer) + │ + ├── Induction (Activity, type: Induction) + │ FK: source_contact_id → Contact.id + │ + ├── Volunteering Activity (Activity) + │ FK: target_contact_id → Contact.id + │ + └── Material Contribution (Activity) + FK: target_contact_id → Contact.id +``` + +--- + +## Pagination + +Confirmed working for all entities: +```json +{ + "select": ["id", "title", "modified_date"], + "orderBy": {"modified_date": "ASC"}, + "limit": 100, + "offset": 0 +} +``` + +Increment `offset` by `limit` for each page. No observed max page size limit. + +--- + +## PII Handling + +For entities where Dalgo's CSV says "avoid personal information": + +| Entity | PII Fields to Exclude from `select` | +|--------|--------------------------------------| +| Collection Camp | `Collection_Camp_Intent_Details.Name`, `Collection_Camp_Intent_Details.Contact_Number` | +| Dropping Center | `Dropping_Centre.Name`, `Dropping_Centre.Contact_Number` | +| Institution DC | `Camp_Institution_Data.Email`, `Camp_Institution_Data.Contact_Number` | +| Contact (Volunteer) | `first_name`, `last_name`, `email`, `phone`, `street_address` — use `id`, `display_name` (or anonymize), `contact_sub_type`, status fields only | + +CiviCRM API4 supports explicit `select` — only requested fields are returned. PII can be excluded by not selecting those fields. + +--- + +## Security Concern + +**ECK entity base fields and Event records are accessible without authentication** via the AJAX endpoint with `X-Requested-With: XMLHttpRequest`. While custom fields (containing sensitive data) require auth, this is still an exposure risk. Consider: +1. Restricting anonymous CiviCRM API access via WordPress permissions +2. Or ensuring the staging/production server has IP-level restrictions + +--- + +## Next Steps + +1. **Generate API key** on staging for authenticated testing +2. **Verify custom field data** returns correctly with auth (especially FK fields) +3. **Test Contact/Volunteer count** with auth to determine volume +4. **Test Contribution** with auth +5. **Decide on Event/Contribution incremental sync** approach (code fix vs. full refresh) +6. **Share this document + Step 1** with Dalgo team