diff --git a/.claude/rules/grill-with-docs-for-new-features.md b/.claude/rules/grill-with-docs-for-new-features.md new file mode 100644 index 00000000..8b9775c9 --- /dev/null +++ b/.claude/rules/grill-with-docs-for-new-features.md @@ -0,0 +1,10 @@ +# Use /grill-with-docs For Development + +When a user asks to develop this codebase, use the `/grill-with-docs` skill before implementation. + +## Required behavior + +- Start a grilling session to resolve terminology, scope, and trade-offs. +- Ask one question at a time and wait for user feedback before continuing. +- If a question can be answered from the codebase or docs, check there first. +- Proceed to implementation only after the development intent is clear. diff --git a/.claude/rules/keep-system-architecture-up-to-date.md b/.claude/rules/keep-system-architecture-up-to-date.md new file mode 100644 index 00000000..4244659f --- /dev/null +++ b/.claude/rules/keep-system-architecture-up-to-date.md @@ -0,0 +1,18 @@ +# Keep System Architecture Doc Up To Date + +## Policy + +`system-architecture.md` is the source of truth for the repository architecture. + +## Required behavior + +When a task changes the system architecture (components, boundaries, data flow, key abstractions, integrations, or deployment topology), update `system-architecture.md` in the same task. + +If no architecture impact exists, explicitly confirm that no update is needed. + +## Verification checklist + +- Architecture-impacting code changes include corresponding `system-architecture.md` updates. +- New or changed architectural terms are reflected consistently. +- Mermaid diagrams remain accurate when affected. +- No secrets are introduced in documentation. diff --git a/.claude/rules/mirror-providers.md b/.claude/rules/mirror-providers.md new file mode 100644 index 00000000..a289117e --- /dev/null +++ b/.claude/rules/mirror-providers.md @@ -0,0 +1,23 @@ +# Mirror Providers Rule + +## Policy + +Keep both provider folders aligned: + +- skills are mirrored between `.claude/skills/` and `.cursor/skills/` +- rules are mirrored between `.claude/rules/` and `.cursor/rules/` + +## Required behavior + +When creating, updating, renaming, or deleting a skill or rule in one provider folder, apply the equivalent change to the other provider folder in the same task. + +## Allowed differences + +Only vendor-required differences are allowed (format or loader specifics). Core guidance and intent must remain equivalent. + +## Verification checklist + +- Both provider folders contain equivalent skill directories and rule files. +- Names map consistently across providers. +- Referenced local docs/scripts resolve in each provider folder. +- No secrets are introduced. diff --git a/.claude/skills/grill-with-docs/ADR-FORMAT.md b/.claude/skills/grill-with-docs/ADR-FORMAT.md new file mode 100644 index 00000000..da7e78ec --- /dev/null +++ b/.claude/skills/grill-with-docs/ADR-FORMAT.md @@ -0,0 +1,47 @@ +# ADR Format + +ADRs live in `docs/adr/` and use sequential numbering: `0001-slug.md`, `0002-slug.md`, etc. + +Create the `docs/adr/` directory lazily — only when the first ADR is needed. + +## Template + +```md +# {Short title of the decision} + +{1-3 sentences: what's the context, what did we decide, and why.} +``` + +That's it. An ADR can be a single paragraph. The value is in recording *that* a decision was made and *why* — not in filling out sections. + +## Optional sections + +Only include these when they add genuine value. Most ADRs won't need them. + +- **Status** frontmatter (`proposed | accepted | deprecated | superseded by ADR-NNNN`) — useful when decisions are revisited +- **Considered Options** — only when the rejected alternatives are worth remembering +- **Consequences** — only when non-obvious downstream effects need to be called out + +## Numbering + +Scan `docs/adr/` for the highest existing number and increment by one. + +## When to offer an ADR + +All three of these must be true: + +1. **Hard to reverse** — the cost of changing your mind later is meaningful +2. **Surprising without context** — a future reader will look at the code and wonder "why on earth did they do it this way?" +3. **The result of a real trade-off** — there were genuine alternatives and you picked one for specific reasons + +If a decision is easy to reverse, skip it — you'll just reverse it. If it's not surprising, nobody will wonder why. If there was no real alternative, there's nothing to record beyond "we did the obvious thing." + +### What qualifies + +- **Architectural shape.** "We're using a monorepo." "The write model is event-sourced, the read model is projected into Postgres." +- **Integration patterns between contexts.** "Ordering and Billing communicate via domain events, not synchronous HTTP." +- **Technology choices that carry lock-in.** Database, message bus, auth provider, deployment target. Not every library — just the ones that would take a quarter to swap out. +- **Boundary and scope decisions.** "Customer data is owned by the Customer context; other contexts reference it by ID only." The explicit no-s are as valuable as the yes-s. +- **Deliberate deviations from the obvious path.** "We're using manual SQL instead of an ORM because X." Anything where a reasonable reader would assume the opposite. These stop the next engineer from "fixing" something that was deliberate. +- **Constraints not visible in the code.** "We can't use AWS because of compliance requirements." "Response times must be under 200ms because of the partner API contract." +- **Rejected alternatives when the rejection is non-obvious.** If you considered GraphQL and picked REST for subtle reasons, record it — otherwise someone will suggest GraphQL again in six months. diff --git a/.claude/skills/grill-with-docs/CONTEXT-FORMAT.md b/.claude/skills/grill-with-docs/CONTEXT-FORMAT.md new file mode 100644 index 00000000..ddfa247c --- /dev/null +++ b/.claude/skills/grill-with-docs/CONTEXT-FORMAT.md @@ -0,0 +1,77 @@ +# CONTEXT.md Format + +## Structure + +```md +# {Context Name} + +{One or two sentence description of what this context is and why it exists.} + +## Language + +**Order**: +{A concise description of the term} +_Avoid_: Purchase, transaction + +**Invoice**: +A request for payment sent to a customer after delivery. +_Avoid_: Bill, payment request + +**Customer**: +A person or organization that places orders. +_Avoid_: Client, buyer, account + +## Relationships + +- An **Order** produces one or more **Invoices** +- An **Invoice** belongs to exactly one **Customer** + +## Example dialogue + +> **Dev:** "When a **Customer** places an **Order**, do we create the **Invoice** immediately?" +> **Domain expert:** "No — an **Invoice** is only generated once a **Fulfillment** is confirmed." + +## Flagged ambiguities + +- "account" was used to mean both **Customer** and **User** — resolved: these are distinct concepts. +``` + +## Rules + +- **Be opinionated.** When multiple words exist for the same concept, pick the best one and list the others as aliases to avoid. +- **Flag conflicts explicitly.** If a term is used ambiguously, call it out in "Flagged ambiguities" with a clear resolution. +- **Keep definitions tight.** One sentence max. Define what it IS, not what it does. +- **Show relationships.** Use bold term names and express cardinality where obvious. +- **Only include terms specific to this project's context.** General programming concepts (timeouts, error types, utility patterns) don't belong even if the project uses them extensively. Before adding a term, ask: is this a concept unique to this context, or a general programming concept? Only the former belongs. +- **Group terms under subheadings** when natural clusters emerge. If all terms belong to a single cohesive area, a flat list is fine. +- **Write an example dialogue.** A conversation between a dev and a domain expert that demonstrates how the terms interact naturally and clarifies boundaries between related concepts. + +## Single vs multi-context repos + +**Single context (most repos):** One `CONTEXT.md` at the repo root. + +**Multiple contexts:** A `CONTEXT-MAP.md` at the repo root lists the contexts, where they live, and how they relate to each other: + +```md +# Context Map + +## Contexts + +- [Ordering](./src/ordering/CONTEXT.md) — receives and tracks customer orders +- [Billing](./src/billing/CONTEXT.md) — generates invoices and processes payments +- [Fulfillment](./src/fulfillment/CONTEXT.md) — manages warehouse picking and shipping + +## Relationships + +- **Ordering → Fulfillment**: Ordering emits `OrderPlaced` events; Fulfillment consumes them to start picking +- **Fulfillment → Billing**: Fulfillment emits `ShipmentDispatched` events; Billing consumes them to generate invoices +- **Ordering ↔ Billing**: Shared types for `CustomerId` and `Money` +``` + +The skill infers which structure applies: + +- If `CONTEXT-MAP.md` exists, read it to find contexts +- If only a root `CONTEXT.md` exists, single context +- If neither exists, create a root `CONTEXT.md` lazily when the first term is resolved + +When multiple contexts exist, infer which one the current topic relates to. If unclear, ask. diff --git a/.claude/skills/grill-with-docs/SKILL.md b/.claude/skills/grill-with-docs/SKILL.md new file mode 100644 index 00000000..5ea0aa91 --- /dev/null +++ b/.claude/skills/grill-with-docs/SKILL.md @@ -0,0 +1,88 @@ +--- +name: grill-with-docs +description: Grilling session that challenges your plan against the existing domain model, sharpens terminology, and updates documentation (CONTEXT.md, ADRs) inline as decisions crystallise. Use when user wants to stress-test a plan against their project's language and documented decisions. +--- + + + +Interview me relentlessly about every aspect of this plan until we reach a shared understanding. Walk down each branch of the design tree, resolving dependencies between decisions one-by-one. For each question, provide your recommended answer. + +Ask the questions one at a time, waiting for feedback on each question before continuing. + +If a question can be answered by exploring the codebase, explore the codebase instead. + + + + + +## Domain awareness + +During codebase exploration, also look for existing documentation: + +### File structure + +Most repos have a single context: + +``` +/ +├── CONTEXT.md +├── docs/ +│ └── adr/ +│ ├── 0001-event-sourced-orders.md +│ └── 0002-postgres-for-write-model.md +└── src/ +``` + +If a `CONTEXT-MAP.md` exists at the root, the repo has multiple contexts. The map points to where each one lives: + +``` +/ +├── CONTEXT-MAP.md +├── docs/ +│ └── adr/ ← system-wide decisions +├── src/ +│ ├── ordering/ +│ │ ├── CONTEXT.md +│ │ └── docs/adr/ ← context-specific decisions +│ └── billing/ +│ ├── CONTEXT.md +│ └── docs/adr/ +``` + +Create files lazily — only when you have something to write. If no `CONTEXT.md` exists, create one when the first term is resolved. If no `docs/adr/` exists, create it when the first ADR is needed. + +## During the session + +### Challenge against the glossary + +When the user uses a term that conflicts with the existing language in `CONTEXT.md`, call it out immediately. "Your glossary defines 'cancellation' as X, but you seem to mean Y — which is it?" + +### Sharpen fuzzy language + +When the user uses vague or overloaded terms, propose a precise canonical term. "You're saying 'account' — do you mean the Customer or the User? Those are different things." + +### Discuss concrete scenarios + +When domain relationships are being discussed, stress-test them with specific scenarios. Invent scenarios that probe edge cases and force the user to be precise about the boundaries between concepts. + +### Cross-reference with code + +When the user states how something works, check whether the code agrees. If you find a contradiction, surface it: "Your code cancels entire Orders, but you just said partial cancellation is possible — which is right?" + +### Update CONTEXT.md inline + +When a term is resolved, update `CONTEXT.md` right there. Don't batch these up — capture them as they happen. Use the format in [CONTEXT-FORMAT.md](./CONTEXT-FORMAT.md). + +`CONTEXT.md` should be totally devoid of implementation details. Do not treat `CONTEXT.md` as a spec, a scratch pad, or a repository for implementation decisions. It is a glossary and nothing else. + +### Offer ADRs sparingly + +Only offer to create an ADR when all three are true: + +1. **Hard to reverse** — the cost of changing your mind later is meaningful +2. **Surprising without context** — a future reader will wonder "why did they do it this way?" +3. **The result of a real trade-off** — there were genuine alternatives and you picked one for specific reasons + +If any of the three is missing, skip the ADR. Use the format in [ADR-FORMAT.md](./ADR-FORMAT.md). + + diff --git a/.cursor/rules/grill-with-docs-for-new-features.mdc b/.cursor/rules/grill-with-docs-for-new-features.mdc new file mode 100644 index 00000000..c6d92e36 --- /dev/null +++ b/.cursor/rules/grill-with-docs-for-new-features.mdc @@ -0,0 +1,15 @@ +--- +description: Use grill-with-docs for development work +alwaysApply: true +--- + +# Use /grill-with-docs For Development + +When a user asks to develop this codebase, use the `/grill-with-docs` skill before implementation. + +## Required behavior + +- Start a grilling session to resolve terminology, scope, and trade-offs. +- Ask one question at a time and wait for user feedback before continuing. +- If a question can be answered from the codebase or docs, check there first. +- Proceed to implementation only after the development intent is clear. diff --git a/.cursor/rules/keep-system-architecture-up-to-date.mdc b/.cursor/rules/keep-system-architecture-up-to-date.mdc new file mode 100644 index 00000000..4244659f --- /dev/null +++ b/.cursor/rules/keep-system-architecture-up-to-date.mdc @@ -0,0 +1,18 @@ +# Keep System Architecture Doc Up To Date + +## Policy + +`system-architecture.md` is the source of truth for the repository architecture. + +## Required behavior + +When a task changes the system architecture (components, boundaries, data flow, key abstractions, integrations, or deployment topology), update `system-architecture.md` in the same task. + +If no architecture impact exists, explicitly confirm that no update is needed. + +## Verification checklist + +- Architecture-impacting code changes include corresponding `system-architecture.md` updates. +- New or changed architectural terms are reflected consistently. +- Mermaid diagrams remain accurate when affected. +- No secrets are introduced in documentation. diff --git a/.cursor/rules/mirror-providers.mdc b/.cursor/rules/mirror-providers.mdc new file mode 100644 index 00000000..3b67f533 --- /dev/null +++ b/.cursor/rules/mirror-providers.mdc @@ -0,0 +1,23 @@ +# Mirror Providers Rule + +## Policy + +Keep both provider folders aligned: + +- skills are mirrored between `.cursor/skills/` and `.claude/skills/` +- rules are mirrored between `.cursor/rules/` and `.claude/rules/` + +## Required behavior + +When creating, updating, renaming, or deleting a skill or rule in one provider folder, apply the equivalent change to the other provider folder in the same task. + +## Allowed differences + +Only vendor-required differences are allowed (format or loader specifics). Core guidance and intent must remain equivalent. + +## Verification checklist + +- Both provider folders contain equivalent skill directories and rule files. +- Names map consistently across providers. +- Referenced local docs/scripts resolve in each provider folder. +- No secrets are introduced. diff --git a/.cursor/skills/grill-with-docs/ADR-FORMAT.md b/.cursor/skills/grill-with-docs/ADR-FORMAT.md new file mode 100644 index 00000000..da7e78ec --- /dev/null +++ b/.cursor/skills/grill-with-docs/ADR-FORMAT.md @@ -0,0 +1,47 @@ +# ADR Format + +ADRs live in `docs/adr/` and use sequential numbering: `0001-slug.md`, `0002-slug.md`, etc. + +Create the `docs/adr/` directory lazily — only when the first ADR is needed. + +## Template + +```md +# {Short title of the decision} + +{1-3 sentences: what's the context, what did we decide, and why.} +``` + +That's it. An ADR can be a single paragraph. The value is in recording *that* a decision was made and *why* — not in filling out sections. + +## Optional sections + +Only include these when they add genuine value. Most ADRs won't need them. + +- **Status** frontmatter (`proposed | accepted | deprecated | superseded by ADR-NNNN`) — useful when decisions are revisited +- **Considered Options** — only when the rejected alternatives are worth remembering +- **Consequences** — only when non-obvious downstream effects need to be called out + +## Numbering + +Scan `docs/adr/` for the highest existing number and increment by one. + +## When to offer an ADR + +All three of these must be true: + +1. **Hard to reverse** — the cost of changing your mind later is meaningful +2. **Surprising without context** — a future reader will look at the code and wonder "why on earth did they do it this way?" +3. **The result of a real trade-off** — there were genuine alternatives and you picked one for specific reasons + +If a decision is easy to reverse, skip it — you'll just reverse it. If it's not surprising, nobody will wonder why. If there was no real alternative, there's nothing to record beyond "we did the obvious thing." + +### What qualifies + +- **Architectural shape.** "We're using a monorepo." "The write model is event-sourced, the read model is projected into Postgres." +- **Integration patterns between contexts.** "Ordering and Billing communicate via domain events, not synchronous HTTP." +- **Technology choices that carry lock-in.** Database, message bus, auth provider, deployment target. Not every library — just the ones that would take a quarter to swap out. +- **Boundary and scope decisions.** "Customer data is owned by the Customer context; other contexts reference it by ID only." The explicit no-s are as valuable as the yes-s. +- **Deliberate deviations from the obvious path.** "We're using manual SQL instead of an ORM because X." Anything where a reasonable reader would assume the opposite. These stop the next engineer from "fixing" something that was deliberate. +- **Constraints not visible in the code.** "We can't use AWS because of compliance requirements." "Response times must be under 200ms because of the partner API contract." +- **Rejected alternatives when the rejection is non-obvious.** If you considered GraphQL and picked REST for subtle reasons, record it — otherwise someone will suggest GraphQL again in six months. diff --git a/.cursor/skills/grill-with-docs/CONTEXT-FORMAT.md b/.cursor/skills/grill-with-docs/CONTEXT-FORMAT.md new file mode 100644 index 00000000..ddfa247c --- /dev/null +++ b/.cursor/skills/grill-with-docs/CONTEXT-FORMAT.md @@ -0,0 +1,77 @@ +# CONTEXT.md Format + +## Structure + +```md +# {Context Name} + +{One or two sentence description of what this context is and why it exists.} + +## Language + +**Order**: +{A concise description of the term} +_Avoid_: Purchase, transaction + +**Invoice**: +A request for payment sent to a customer after delivery. +_Avoid_: Bill, payment request + +**Customer**: +A person or organization that places orders. +_Avoid_: Client, buyer, account + +## Relationships + +- An **Order** produces one or more **Invoices** +- An **Invoice** belongs to exactly one **Customer** + +## Example dialogue + +> **Dev:** "When a **Customer** places an **Order**, do we create the **Invoice** immediately?" +> **Domain expert:** "No — an **Invoice** is only generated once a **Fulfillment** is confirmed." + +## Flagged ambiguities + +- "account" was used to mean both **Customer** and **User** — resolved: these are distinct concepts. +``` + +## Rules + +- **Be opinionated.** When multiple words exist for the same concept, pick the best one and list the others as aliases to avoid. +- **Flag conflicts explicitly.** If a term is used ambiguously, call it out in "Flagged ambiguities" with a clear resolution. +- **Keep definitions tight.** One sentence max. Define what it IS, not what it does. +- **Show relationships.** Use bold term names and express cardinality where obvious. +- **Only include terms specific to this project's context.** General programming concepts (timeouts, error types, utility patterns) don't belong even if the project uses them extensively. Before adding a term, ask: is this a concept unique to this context, or a general programming concept? Only the former belongs. +- **Group terms under subheadings** when natural clusters emerge. If all terms belong to a single cohesive area, a flat list is fine. +- **Write an example dialogue.** A conversation between a dev and a domain expert that demonstrates how the terms interact naturally and clarifies boundaries between related concepts. + +## Single vs multi-context repos + +**Single context (most repos):** One `CONTEXT.md` at the repo root. + +**Multiple contexts:** A `CONTEXT-MAP.md` at the repo root lists the contexts, where they live, and how they relate to each other: + +```md +# Context Map + +## Contexts + +- [Ordering](./src/ordering/CONTEXT.md) — receives and tracks customer orders +- [Billing](./src/billing/CONTEXT.md) — generates invoices and processes payments +- [Fulfillment](./src/fulfillment/CONTEXT.md) — manages warehouse picking and shipping + +## Relationships + +- **Ordering → Fulfillment**: Ordering emits `OrderPlaced` events; Fulfillment consumes them to start picking +- **Fulfillment → Billing**: Fulfillment emits `ShipmentDispatched` events; Billing consumes them to generate invoices +- **Ordering ↔ Billing**: Shared types for `CustomerId` and `Money` +``` + +The skill infers which structure applies: + +- If `CONTEXT-MAP.md` exists, read it to find contexts +- If only a root `CONTEXT.md` exists, single context +- If neither exists, create a root `CONTEXT.md` lazily when the first term is resolved + +When multiple contexts exist, infer which one the current topic relates to. If unclear, ask. diff --git a/.cursor/skills/grill-with-docs/SKILL.md b/.cursor/skills/grill-with-docs/SKILL.md new file mode 100644 index 00000000..5ea0aa91 --- /dev/null +++ b/.cursor/skills/grill-with-docs/SKILL.md @@ -0,0 +1,88 @@ +--- +name: grill-with-docs +description: Grilling session that challenges your plan against the existing domain model, sharpens terminology, and updates documentation (CONTEXT.md, ADRs) inline as decisions crystallise. Use when user wants to stress-test a plan against their project's language and documented decisions. +--- + + + +Interview me relentlessly about every aspect of this plan until we reach a shared understanding. Walk down each branch of the design tree, resolving dependencies between decisions one-by-one. For each question, provide your recommended answer. + +Ask the questions one at a time, waiting for feedback on each question before continuing. + +If a question can be answered by exploring the codebase, explore the codebase instead. + + + + + +## Domain awareness + +During codebase exploration, also look for existing documentation: + +### File structure + +Most repos have a single context: + +``` +/ +├── CONTEXT.md +├── docs/ +│ └── adr/ +│ ├── 0001-event-sourced-orders.md +│ └── 0002-postgres-for-write-model.md +└── src/ +``` + +If a `CONTEXT-MAP.md` exists at the root, the repo has multiple contexts. The map points to where each one lives: + +``` +/ +├── CONTEXT-MAP.md +├── docs/ +│ └── adr/ ← system-wide decisions +├── src/ +│ ├── ordering/ +│ │ ├── CONTEXT.md +│ │ └── docs/adr/ ← context-specific decisions +│ └── billing/ +│ ├── CONTEXT.md +│ └── docs/adr/ +``` + +Create files lazily — only when you have something to write. If no `CONTEXT.md` exists, create one when the first term is resolved. If no `docs/adr/` exists, create it when the first ADR is needed. + +## During the session + +### Challenge against the glossary + +When the user uses a term that conflicts with the existing language in `CONTEXT.md`, call it out immediately. "Your glossary defines 'cancellation' as X, but you seem to mean Y — which is it?" + +### Sharpen fuzzy language + +When the user uses vague or overloaded terms, propose a precise canonical term. "You're saying 'account' — do you mean the Customer or the User? Those are different things." + +### Discuss concrete scenarios + +When domain relationships are being discussed, stress-test them with specific scenarios. Invent scenarios that probe edge cases and force the user to be precise about the boundaries between concepts. + +### Cross-reference with code + +When the user states how something works, check whether the code agrees. If you find a contradiction, surface it: "Your code cancels entire Orders, but you just said partial cancellation is possible — which is right?" + +### Update CONTEXT.md inline + +When a term is resolved, update `CONTEXT.md` right there. Don't batch these up — capture them as they happen. Use the format in [CONTEXT-FORMAT.md](./CONTEXT-FORMAT.md). + +`CONTEXT.md` should be totally devoid of implementation details. Do not treat `CONTEXT.md` as a spec, a scratch pad, or a repository for implementation decisions. It is a glossary and nothing else. + +### Offer ADRs sparingly + +Only offer to create an ADR when all three are true: + +1. **Hard to reverse** — the cost of changing your mind later is meaningful +2. **Surprising without context** — a future reader will wonder "why did they do it this way?" +3. **The result of a real trade-off** — there were genuine alternatives and you picked one for specific reasons + +If any of the three is missing, skip the ADR. Use the format in [ADR-FORMAT.md](./ADR-FORMAT.md). + + diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 011ff6df..f053f4d0 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -1,10 +1,9 @@ --- name: Bug report about: Create a report to help us improve -title: '' +title: "" labels: bug -assignees: '' - +assignees: "" --- @@ -14,6 +13,7 @@ A clear and concise description of what the bug is. **To Reproduce** Steps to reproduce the behavior: + 1. Go to '...' 2. Click on '....' 3. Scroll down to '....' @@ -26,9 +26,10 @@ A clear and concise description of what you expected to happen. If applicable, add screenshots to help explain your problem. **Desktop (please complete the following information):** - - OS: [e.g. iOS] - - Browser [e.g. chrome, safari] - - Version [e.g. 22] + +- OS: [e.g. iOS] +- Browser [e.g. chrome, safari] +- Version [e.g. 22] - PaperMemory version [e.g. 0.3.3] **Idea for a fix** (Can you think of a solution? Can you contribute a PR if you do?) diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md index 59094e26..b97aa9ad 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.md +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -1,10 +1,9 @@ --- name: Feature request about: Suggest an idea for this project -title: '' +title: "" labels: enhancement -assignees: '' - +assignees: "" --- **Is your feature request related to a problem? Please describe.** diff --git a/.github/ISSUE_TEMPLATE/other.md b/.github/ISSUE_TEMPLATE/other.md index 03f75317..00f59ac6 100644 --- a/.github/ISSUE_TEMPLATE/other.md +++ b/.github/ISSUE_TEMPLATE/other.md @@ -1,10 +1,7 @@ --- name: Other about: Something else to ask/suggest/flag? -title: '' -labels: '' -assignees: '' - +title: "" +labels: "" +assignees: "" --- - - diff --git a/.github/workflows/_build.yml b/.github/workflows/_build.yml new file mode 100644 index 00000000..82bf13ce --- /dev/null +++ b/.github/workflows/_build.yml @@ -0,0 +1,24 @@ +name: Build (reusable) + +on: + workflow_call: + +permissions: + contents: read + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: "22.x" + cache: "npm" + + - name: Install dependencies + run: npm ci + + - name: Build Chrome & Firefox + run: npm run build diff --git a/.github/workflows/_test-matrix.yml b/.github/workflows/_test-matrix.yml new file mode 100644 index 00000000..8dfc8609 --- /dev/null +++ b/.github/workflows/_test-matrix.yml @@ -0,0 +1,59 @@ +name: Test matrix (reusable) + +on: + workflow_call: + secrets: + VICT0RSCH_GITHUB_PAT: + required: true + +permissions: + contents: read + +jobs: + test-matrix: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + include: + - name: Utils Tests + files: test/test-utils.js + - name: Extension Loading + files: test/test-extension-loading.js + - name: Popup UI + files: test/test-popup-search.js test/test-popup-paper-ui.js + - name: Memory UI + files: test/test-memory-item-actions.js test/test-memory-table-ui.js + - name: Menu + files: test/test-menu.js + - name: Sync + files: test/test-sync.js + - name: Duplicates + files: test/test-duplicates.js + - name: All test scripts are listed in a workflow + files: test/test-meta.js + + name: ${{ matrix.name }} + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: "22.x" + cache: "npm" + + - name: Install Dependencies + run: npm ci + + - name: Build Extension + run: npm run build:chrome + + - uses: nick-fields/retry@ce71cc2ab81d554ebbe88c79ab5975992d79ba08 # v3.0.2 + name: Run Test + env: + github_pat: ${{ secrets.VICT0RSCH_GITHUB_PAT }} + with: + timeout_minutes: 20 + max_attempts: 2 + retry_on: error + command: npm run test:file ${{ matrix.files }} diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index a0723649..f15dfd9e 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -1,18 +1,9 @@ -name: automerge +name: Build CI on: [push, pull_request] -jobs: - build: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: - node-version: "22.x" +permissions: + contents: read - - name: Install Yarn - run: npm install --global yarn - - name: Install dependencies - run: yarn install - - name: Build - run: gulp build +jobs: + build: + uses: ./.github/workflows/_build.yml diff --git a/.github/workflows/ci.yaml b/.github/workflows/docs.yaml similarity index 98% rename from .github/workflows/ci.yaml rename to .github/workflows/docs.yaml index bf1e63d5..43773f0b 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/docs.yaml @@ -1,4 +1,4 @@ -name: ci +name: Docs CI on: push: branches: diff --git a/.github/workflows/submit.yml b/.github/workflows/submit.yml index 8a693282..8ced52bb 100644 --- a/.github/workflows/submit.yml +++ b/.github/workflows/submit.yml @@ -1,23 +1,66 @@ -name: Submit +name: Submit to stores on: - workflow_dispatch: - inputs: - tag: - description: "Release tag to submit, i.e 0.4.3" - required: true + workflow_dispatch: + inputs: + target: + description: "Which store(s) to submit to" + type: choice + options: [both, chrome, firefox] + default: both + dry_run: + description: "Dry run (validate credentials without submitting)" + type: boolean + default: true + +concurrency: + group: submit-stores + cancel-in-progress: false + +permissions: + contents: read jobs: - submit: - runs-on: ubuntu-latest - steps: - - name: Download Github Release Assets - uses: plasmo-corp/download-release-asset@v1.0.0 - with: - files: Archive-* - tag: ${{ github.event.inputs.tag }} - - name: Browser Plugin Publish - uses: plasmo-corp/bpp@v1 - with: - artifact: "Archive-${{ github.event.inputs.tag }}.zip" - keys: ${{ secrets.SUBMIT_KEYS }} + test-matrix: + uses: ./.github/workflows/_test-matrix.yml + secrets: + VICT0RSCH_GITHUB_PAT: ${{ secrets.VICT0RSCH_GITHUB_PAT }} + + submit: + needs: test-matrix + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: "22.x" + cache: "npm" + + - run: npm ci + + - run: npm run zip + + - name: Submit to Chrome Web Store + if: github.event.inputs.target == 'both' || github.event.inputs.target == 'chrome' + run: | + npx wxt submit \ + ${{ github.event.inputs.dry_run == 'true' && '--dry-run' || '' }} \ + --chrome-zip "dist/*-chrome.zip" + env: + CHROME_EXTENSION_ID: ${{ secrets.CHROME_EXTENSION_ID }} + CHROME_CLIENT_ID: ${{ secrets.CHROME_CLIENT_ID }} + CHROME_CLIENT_SECRET: ${{ secrets.CHROME_CLIENT_SECRET }} + CHROME_REFRESH_TOKEN: ${{ secrets.CHROME_REFRESH_TOKEN }} + + - name: Submit to Firefox Add-ons + if: github.event.inputs.target == 'both' || github.event.inputs.target == 'firefox' + run: | + npx wxt submit \ + ${{ github.event.inputs.dry_run == 'true' && '--dry-run' || '' }} \ + --firefox-zip "dist/*-firefox.zip" \ + --firefox-sources-zip "dist/*-sources.zip" + env: + FIREFOX_EXTENSION_ID: ${{ secrets.FIREFOX_EXTENSION_ID }} + FIREFOX_JWT_ISSUER: ${{ secrets.FIREFOX_JWT_ISSUER }} + FIREFOX_JWT_SECRET: ${{ secrets.FIREFOX_JWT_SECRET }} diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml new file mode 100644 index 00000000..2d2c8238 --- /dev/null +++ b/.github/workflows/test.yml @@ -0,0 +1,45 @@ +name: Tests CI +on: push + +permissions: + contents: read + +jobs: + test-matrix: + uses: ./.github/workflows/_test-matrix.yml + secrets: + VICT0RSCH_GITHUB_PAT: ${{ secrets.VICT0RSCH_GITHUB_PAT }} + + test-storage: + runs-on: ubuntu-latest + name: Storage Tests + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: "22.x" + cache: "npm" + + - name: Install Dependencies + run: npm ci + + - name: Build Extension + run: npm run build:chrome + + - uses: nick-fields/retry@ce71cc2ab81d554ebbe88c79ab5975992d79ba08 # v3.0.2 + name: Run Storage Test + with: + timeout_minutes: 20 + max_attempts: 2 + retry_on: error + command: npm run test:file test/test-storage.js + + - uses: actions/upload-artifact@v4 + if: always() # Upload screenshots even on failure for debugging + with: + name: test-storage-screenshots + path: ./tmp/*.jpg + if-no-files-found: ignore + retention-days: 7 + compression-level: 3 diff --git a/.gitignore b/.gitignore index b9ecc1ee..e07b7252 100644 --- a/.gitignore +++ b/.gitignore @@ -4,9 +4,17 @@ extra/archives/ node_modules/ keys.json +.env +.env.submit +.env.* test/tmp/ .nyc_output/ coverage .cache .todo site/ +__pycache__ +tmp/ +dist/ +.wxt/ +*.log \ No newline at end of file diff --git a/.prettierignore b/.prettierignore new file mode 100644 index 00000000..ee0caf5e --- /dev/null +++ b/.prettierignore @@ -0,0 +1,6 @@ +*.min.* +**/dist/ +**/build/ +**/bundle/ +**/bundles/ +**.bundle.** diff --git a/.prettierrc b/.prettierrc index 998b722a..c4d3a39e 100644 --- a/.prettierrc +++ b/.prettierrc @@ -1,5 +1,12 @@ { - "tabWidth": 4, - "useTabs": false, - "printWidth": 88 -} \ No newline at end of file + "tabWidth": 4, + "printWidth": 88, + "overrides": [ + { + "files": ["*.yaml", "*.yml"], + "options": { + "tabWidth": 2 + } + } + ] +} diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..fba70d90 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,83 @@ +# 1. Think Before Coding + +**Don't assume. Don't hide confusion. Surface tradeoffs.** + +Before implementing: +- State your assumptions explicitly. If uncertain, ask. +- If multiple interpretations exist, present them - don't pick silently. +- If a simpler approach exists, say so. Push back when warranted. +- If something is unclear, stop. Name what's confusing. Ask. +- Read `system-architecture.md` first to understand repository structure and boundaries before broad grepping/searching across the entire codebase. +- If the user's request is too vague or short or unclear, use the "grill-with-docs" skill to clarify the request. +- Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification. + +# 2. Simplicity First + +**Minimum code that solves the problem. Nothing speculative.** + +- No features beyond what was asked. +- No abstractions for single-use code. +- No "flexibility" or "configurability" that wasn't requested. +- No error handling for impossible scenarios. +- If you write 200 lines and it could be 50, rewrite it. + +Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify. + +# 3. Surgical Changes + +**Touch only what you must. Clean up only your own mess.** + +When editing existing code: +- Don't "improve" adjacent code, comments, or formatting. +- Don't refactor things that aren't broken. +- Match existing style, even if you'd do it differently. +- If you notice unrelated dead code, mention it - don't delete it. + +When your changes create orphans: +- Remove imports/variables/functions that YOUR changes made unused. +- Don't remove pre-existing dead code unless asked. + +The test: Every changed line should trace directly to the user's request. + +# 4. Plans Must Be Executable By Small Models + +**Every plan must be clear, exhaustive, imperative, and verifiable.** + +When writing a plan: +- Use numbered imperative steps (`Do X`, `Then do Y`), never vague goals. +- Make the plan exhaustive for the requested scope: no hidden steps, no implicit work. +- State assumptions and prerequisites explicitly before execution steps. +- For each step, define an objective verification check (test, command, observable output, or diff condition). +- Define concrete completion criteria so an agent can truthfully decide "done" vs "not done." +- Use language that a small model can follow literally without guessing intent. + +Hard rule: +- If a small-capability agent could not implement the plan truthfully from the plan text alone, the plan is invalid and must be rewritten. + +# 5. Agent Entry Point (Shared Across Providers) + +Provider-specific rule files are the source of truth for cross-provider mirroring: + +- `.cursor/rules/mirror-providers.mdc` +- `.claude/rules/mirror-providers.md` + +# 6. Be safe, be consistent + +**Security is a hard gate. A task is not done until this section is satisfied.** + +Always perform a security pass while reading and writing code: +- Treat secrets exposure as critical (API keys, tokens, passwords, credentials, private keys, webhook URLs, `.env` contents). +- Never hardcode secrets in code, tests, docs, examples, commits, PR text, logs, or terminal snippets. +- Do not move secrets into "safe-looking" files (fixtures, sample configs, scripts). If a value is sensitive, keep it out. +- If you detect a leak or high-risk pattern, stop and warn clearly. Do this even when it is outside the original request. + +Before declaring completion, run a critical consistency review of your own work: +- Verify the result is consistent with sections 1-4: explicit assumptions, minimal scope, and surgical edits only. +- Confirm every changed line traces directly to the user's request (no speculative features, no side-quest refactors). +- Check that your naming, structure, and behavior align with existing project language and conventions. +- Check consistency with documentation produced by the `/grill-with-docs` process (`CONTEXT.md`, `CONTEXT-MAP.md`, and relevant ADRs). If code and docs disagree, flag it explicitly. +- Challenge your own decisions: "Where did I overreach?" and "What part is inconsistent with the stated principles?" +- If inconsistencies remain, report them explicitly and propose the smallest concrete fix. + +Completion rule: +- Do not present the task as complete until all checks are done: (1) secret/leak/security scan, (2) critical consistency self-review, (3) consistency check against `/grill-with-docs` artifacts (`CONTEXT.md`, `CONTEXT-MAP.md`, relevant ADRs). \ No newline at end of file diff --git a/Readme.md b/Readme.md index 8d361570..6a4bdd10 100644 --- a/Readme.md +++ b/Readme.md @@ -73,37 +73,37 @@ Share ideas 💡 in [issues](https://github.com/vict0rsch/PaperMemory/issues) an ## Supported venues -- Arxiv - - PaperMemory will try to find if a pre-print has been published and create a corresponding `note` to the paper (see [preprints](#preprints)) - - Also detects and matches papers from [huggingface.co/papers](https://huggingface.co/papers), [AlphaXiv](https://alphaxiv.org), [ar5iv.org](https://ar5iv.org) and [scirate.com/](https://scirate.com/) -- BioRxiv -- NeurIPS -- Open Review (ICLR etc.) -- Computer Vision Foundation (I/ECCV, CVPR etc.) -- Proceedings of Machine Learning Research (PMLR) (AISTATS, ICML, CoRL, CoLT, ALT, UAI etc.) -- Association for Computational Linguistics (ACL) (EMNLP, ACL, CoNLL, NAACL etc.) -- Proceedings of the National Academy of Sciences (PNAS) -- SciRate -- Nature (Nature, Nature Communications, Nature Machine Intelligence etc.) -- American Chemical Society (ACS) -- IOPscience -- PubMed Central -- International Joint Conferences on Artificial Intelligence (IJCAI) -- Association for Computing Machinery (ACM) -- IEEE -- Springer (books, chapters and, of course, articles) -- American Physical Society (APS) -- Wiley (Advanced Materials, InfoMat etc.) -- Science Direct -- Science (Science, Science Immunology, Science Robotics etc.) -- FrontiersIn (Frontiers in Neuroscience, Frontiers in Neuroscience, Frontiers in Microbiology etc.) -- PLOS -- MDPI -- Oxford University Press -- HAL Archives ouvertes -- Royal Society of Chemistry -- [Sci-Hub](https://papermemory.org/faq/#can-i-reference-my-pdf-in-papermemory) -- [Add more](https://github.com/vict0rsch/PaperMemory/issues/13) +- Arxiv + - PaperMemory will try to find if a pre-print has been published and create a corresponding `note` to the paper (see [preprints](#preprints)) + - Also detects and matches papers from [huggingface.co/papers](https://huggingface.co/papers), [AlphaXiv](https://alphaxiv.org), [ar5iv.org](https://ar5iv.org) and [scirate.com/](https://scirate.com/) +- BioRxiv +- NeurIPS +- Open Review (ICLR etc.) +- Computer Vision Foundation (I/ECCV, CVPR etc.) +- Proceedings of Machine Learning Research (PMLR) (AISTATS, ICML, CoRL, CoLT, ALT, UAI etc.) +- Association for Computational Linguistics (ACL) (EMNLP, ACL, CoNLL, NAACL etc.) +- Proceedings of the National Academy of Sciences (PNAS) +- SciRate +- Nature (Nature, Nature Communications, Nature Machine Intelligence etc.) +- American Chemical Society (ACS) +- IOPscience +- PubMed Central +- International Joint Conferences on Artificial Intelligence (IJCAI) +- Association for Computing Machinery (ACM) +- IEEE +- Springer (books, chapters and, of course, articles) +- American Physical Society (APS) +- Wiley (Advanced Materials, InfoMat etc.) +- Science Direct +- Science (Science, Science Immunology, Science Robotics etc.) +- FrontiersIn (Frontiers in Neuroscience, Frontiers in Neuroscience, Frontiers in Microbiology etc.) +- PLOS +- MDPI +- Oxford University Press +- HAL Archives ouvertes +- Royal Society of Chemistry +- [Sci-Hub](https://papermemory.org/faq/#can-i-reference-my-pdf-in-papermemory) +- [Add more](https://github.com/vict0rsch/PaperMemory/issues/13) [📑—About finding published papers from preprints](https://papermemory.org/features/#preprint-matching) @@ -143,11 +143,11 @@ Checkout [📑—All configuration options](https://papermemory.org/configuratio In the extension's [📑—`options`](https://papermemory.org/configuration/#advanced-options) (right click on the icon or in the popup's menu) you will find advanced customization features: -- **Auto-tagging**: add tags to papers based on regexs matched on authors and titles -- **Source filtering**: filter out some paper sources you don't want to record papers from -- **Custom title function**: provide Javascript code to generate your own web page titles and pdf filenames based on a paper's attributes -- **Data management**: export/load your memory data and export the bibliography as a `.bib` file -- **Online Synchronization**: use Github Gists to sync your papers across devices +- **Auto-tagging**: add tags to papers based on regexs matched on authors and titles +- **Source filtering**: filter out some paper sources you don't want to record papers from +- **Custom title function**: provide Javascript code to generate your own web page titles and pdf filenames based on a paper's attributes +- **Data management**: export/load your memory data and export the bibliography as a `.bib` file +- **Online Synchronization**: use Github Gists to sync your papers across devices

@@ -195,5 +195,5 @@ See [📑—how it works](https://papermemory.org/features/#code-repositories). ## Todo -- [ ] Improve `Contributing.md` -- [ ] Write many more tests! **Help is wanted** (it's not so hard to write unittests 😄) (see `Contributing.md`) +- [ ] Improve `Contributing.md` +- [ ] Write many more tests! **Help is wanted** (it's not so hard to write unittests 😄) (see `Contributing.md`) diff --git a/SOURCE_CODE_REVIEW.md b/SOURCE_CODE_REVIEW.md new file mode 100644 index 00000000..b82c63af --- /dev/null +++ b/SOURCE_CODE_REVIEW.md @@ -0,0 +1,8 @@ +# Build from source + +```bash +npm install +npm run zip:firefox +``` + +Output: `dist/papermemory--firefox.zip` diff --git a/contributing.md b/contributing.md index 9a3601ad..9921818d 100644 --- a/contributing.md +++ b/contributing.md @@ -4,187 +4,353 @@ PaperMemory is pure JS+HTML with minimal dependencies: no framework, (almost) no external dependencies so it's easy to help :) -The only external deps. are [`select2.js`](https://select2.org/) which requires `JQuery` and some of the latter here and there (but I'm working on getting rid of it, replacing it with a simple set of helper functions in `src/shared/utils/miniquery.js`). +The only external deps are [`tom-select`](https://tom-select.js.org/) (tag inputs) and [`jQuery`](https://jquery.com/) (content-script page UI only) — both installed via npm. -`npm` and `gulp` are here to make the dev+release lifecycle easier, if you don't want to set it up there still are a lot of things you can help with in the raw source code (just don't bother with the `min` files) +The project uses modern ES modules and [WXT](https://wxt.dev/) for bundling, dev server, manifest generation, and cross-browser packaging. + +### Related documentation + +**This guide covers how to set up, build, and contribute** (commands, conventions, step-by-step how-tos, tests, release). For **how the system works** — the mental model, data flow, core abstractions, component responsibilities, and design rationale — see [`system-architecture.md`](system-architecture.md) (e.g. its [Technical Stack](system-architecture.md#technical-stack) for the full dependency and tooling list). ## Set-up -1. [Install `yarn`](https://classic.yarnpkg.com/lang/en/docs/install): Node's package manager -2. [Install `gulp`](https://gulpjs.com/): a build tool -3. Install dependencies: from the root of this repo `$ yarn install` -4. Watch file changes: `$ gulp watch` -5. Edit files! - -`gulp` mainly runs the concatenation of files into a single one (especially for css and js) and its minification. - -In `popup.html` you will notice: - -```html - - - - - - - - - - - - - +1. [Install `npm`](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) (Node.js 22+ recommended) +2. Install dependencies: `npm install` +3. Start the dev server: `npm run dev` +4. Load the extension in Chrome from `dist/chrome-mv3/` (see [Loading the extension](#loading-the-extension)) +5. Edit files — WXT auto-reloads! + +## What is WXT? + +[WXT](https://wxt.dev/) is a framework for building browser extensions: + +- **Bundling**: Uses Vite under the hood to bundle ES modules +- **Manifest generation**: Writes `manifest.json` automatically from `wxt.config.js` (Chrome MV3 and Firefox MV2) +- **Dev server**: Hot module reloading with `npm run dev` +- **Zip packaging**: `npm run zip` produces ready-to-submit archives for both browsers + +You do **not** need to know WXT internals to contribute. The key thing to know is that WXT discovers entry points from `src/entrypoints/` and builds them into `dist/`. For how WXT (and these entry points) fit the overall architecture, see [`system-architecture.md` → System Architecture](system-architecture.md#system-architecture). + +## Build commands + +```bash +npm run dev # Dev server for Chrome (HMR, auto-reload) +npm run dev:firefox # Dev server for Firefox + +npm run build # Production build for Chrome + Firefox +npm run build:chrome # Production build for Chrome only +npm run build:firefox # Production build for Firefox only + +npm run zip # Build + zip for both browsers +npm run zip:chrome # Build + zip for Chrome only +npm run zip:firefox # Build + zip for Firefox only ``` -Those `@` commands are meant for `gulp` (using the `preprocess` package) to choose whether to use raw, un-minified files for development (`$ gulp watch`) or concatenated and minified ones for production (`$ gulp build`) +Build output goes to `dist/chrome-mv3/` and `dist/firefox-mv2/`. + +## Loading the extension + +### Chrome + +1. Run `npm run dev` (or `npm run build:chrome`) +2. Open `chrome://extensions/`, enable "Developer mode" +3. Click "Load unpacked" and select the `dist/chrome-mv3/` directory +4. The extension auto-reloads on file changes when using `npm run dev` + +### Firefox + +1. Run `npm run dev:firefox` +2. Open `about:debugging#/runtime/this-firefox` +3. Click "Load Temporary Add-on" and select any file inside `dist/firefox-mv2/` + +More info: https://extensionworkshop.com/documentation/develop/temporary-installation-in-firefox/ + +## Refreshing the extension -Note: the file loaded in the popup is `src/popup/min/popup.min.html`, _not_ `src/popup/popup.html` so you _have_ to use `gulp watch` to see changes you make to `popup.html` reflected in the actual popup. +When using `npm run dev`: -### Refreshing the extension +- **Popup / options / fullMemory changes**: Auto-reload via HMR +- **Background script changes**: WXT reloads the service worker automatically +- **Content script changes**: Require refreshing the web page where the content script runs -Once you load the local extension as an unpackaged extension, changes that affect the popup will directly take effect, no need to refresh anything. +### Debugging utilities (`PMDebug`) -**Content scripts** however, are loaded and not binded to the source so you _have to_ refresh the extension in the settings (and then any web page you want to see changes on) for those to be taken into account. +A global `PMDebug` object is available in all HTML pages (popup, options, fullMemory, bibMatcher) for inspecting internal state from the browser console: -## Loading in Firefox +```javascript +PMDebug.config.state; // Global state (papers, prefs, etc.) +PMDebug.config.state.papers; // All stored papers +PMDebug.data.getStorage(); // Raw chrome.storage access +PMDebug.urls.parseIdFromUrl("https://arxiv.org/abs/2301.12345"); +PMDebug.paper.isPaper("https://arxiv.org/abs/2301.12345"); +// Per-source handlers (parse, urlToId, toAbs, toPDF, displayId, …) +PMDebug.sources.getSource("arxiv").toAbs({ + pdfLink: "https://arxiv.org/pdf/1234.56789v2.pdf", + source: "arxiv", +}); +PMDebug.listAllFunctions(); // Discover everything available +``` + +Available modules: `config`, `functions`, `miniquery`, `data`, `paper`, `bibtexParser`, `sync`, `state`, `urls`, `files`, `parsers`, `sources`, `preprintMatching`, `templates`, `handlers`, `memory`. -https://extensionworkshop.com/documentation/develop/temporary-installation-in-firefox/ +Paper-specific logic lives under `PMDebug.sources`, not on `PMDebug.parsers` (e.g. there is no `PMDebug.parsers.makeArxivPaper`). + +The debug system is implemented in `src/debug/debug.js` and imported by each entry point's `main.js`. ## Conventions ### File structure -(slightly deprecated since some files have moved but names are unique enough for you to still understand I hope) - -```tree -├── jsconfig.json ➤➤➤ vscode js config -├── manifest.json ➤➤➤ the extension's configuration file for the browser -└── src ➤➤➤ actual code - ├── background ➤➤➤ background code - │   └── background.js ➤➤➤ background.js can interact with some browser apis content_scripts can't use - ├── content_scripts - │   ├── content_script.css ➤➤➤ styling for pages modified by the content_script - │   ├── content_script.js ➤➤➤ the code run at the opening of a page matched in manifest.json - ├── popup - │   ├── css ➤➤➤ The popup's style - │   │   ├── dark.css ➤➤➤ Dark mode style sheet - │   │   ├── options.css ➤➤➤ Sliding checkboxes style - │   │   ├── popup.css ➤➤➤ Main style - │   │   ├── select2.min.css ➤➤➤ Style for the select2 dropdowns (don't modify) - │   ├── js ➤➤➤ The javascript files specific to the popup, minified together in this order into js.min.js - │   │   ├── handlers.js ➤➤➤ Event handlers - │   │   ├── memory.js ➤➤➤ Memory-specific functions - │   │   ├── popup.js ➤➤➤ Main execution - │   │   ├── theme.js ➤➤➤ The first thing that is executed when the popup is opened: selecting dark/light theme based on user preferences - │   │   ├── select2.min.js ➤➤➤ JQuery-based tagging lib for paper tags - │   │   └── templates.js ➤➤➤ HTML string templates: memory items and paper popup - │ ├── min - │   │   └── minified scripts - │   ├── popup.html ➤➤➤ Main HTML file - └── shared - ├── jquery.min.js ➤➤➤ JQuery lib. Do not modify. - ├── utils ➤➤➤ Shared utility functions minified together in this order into utils.min.js - │   ├── bibtexParser.js ➤➤➤ Class to parse bibtex strings into objects - │   ├── config.js ➤➤➤ Constants / State variables used throughout out the code - │   ├── data.js ➤➤➤ Data/Memory manipulation (migrations, paper validation, overwrite etc.) - │   ├── functions.js ➤➤➤ Utility functions, relying on config.js - │   ├── logTrace.js ➤➤➤ Single var script to include the log stack trace in dev (gulp watch) - │   ├── miniquery.js ➤➤➤ Custom vanilla js replacement for JQuery (working towards removing that dependency) - │   ├── parsers.js ➤➤➤ Parsing functions to create papers - │   ├── paper.js ➤➤➤ Single-paper-related functions (isPaper, paperToAbs, paperToPDF) - │   └── state.js ➤➤➤ State-related functions (init, custom title function, addOrUpdatePaper etc.) - ├── utils.min.js ➤➤➤ Concatenation and minification of all files in src/shared/utils/ - └── loader.css ➤➤➤ the style for the loader before the BibTex entry is displayed on arxiv.org/abs/* ``` +├── wxt.config.js ➤ WXT + Vite config (manifest, aliases, HTML include plugin) +├── jsconfig.json ➤ VS Code config with path aliases (@pm, @pmu) +├── public/ ➤ Static assets copied verbatim to build output +│ ├── theme.js ➤ Dark mode detection (runs before modules, loaded via - - - - - - - - - - - -

-
-
-
-

- Paste the content of your .bib file and PaperMemory will automatically match the Arxiv - entries - to publications by - fetching information from DBLP, Semantic Scholar, CrossRef and Google Scholar. -

- -

- The matching procedure is identical to the one you can trigger to match your PaperMemory ArXiv - entries in the - - Advanced Options - -

- -

Warning: the matching procedure produces an output - which - does - NOT maintain comments. Select - matched - entries one by one if you want to keep your initial comment structure. -

-
-
- -
-
-
-
-
-    - -
-
-
When a pre-print is matched to a publication, the - latter may have a - standard citation key. You can either update the entry's citation key or use the new one. - Note that if you use the new one, your existing citations will be broken.
-
-
-
-    - -
-
-
Data providers protect their APIs by restricting - the - number of queries per seconds you can make. PaperMemory's BibMatcher may iterate too fast - over your entries, reaching the API rate limits. Adding a timeout between requests slows - down the matching process but increases your matching rate.
-
- - -
-
- -
- - -
-
- - - - - -
- -
- -
- - - - - - - - - \ No newline at end of file diff --git a/src/bibMatcher/bibMatcher.js b/src/bibMatcher/bibMatcher.js index b92236e6..b7eda1cb 100644 --- a/src/bibMatcher/bibMatcher.js +++ b/src/bibMatcher/bibMatcher.js @@ -1,3 +1,25 @@ +// ES Module imports +import { + addListener, + findEl, + setHTML, + dispatch, + val, + showId, + hideId, + querySelector, +} from "@pmu/miniquery.js"; +import { copyTextToClipboard, escapeHtml } from "@pmu/functions.js"; +import { BibtexParser, bibtexToObject, bibtexToString } from "@pmu/bibtexParser.js"; +import { + tryDBLP, + trySemanticScholar, + tryCrossRef, + tryUnpaywall, + tryGoogleScholar, +} from "@pmu/preprintMatching.js"; +import { sleep } from "@pmu/sync.js"; + var STOPMATCH = false; var DISABLE_MATCH = {}; @@ -63,18 +85,18 @@ const setListeners = () => { arxivs.length ? setHTML( "n-arxivs", - `Matching ${arxivs.length} arXiv entries, out of ${parsed.length} total entries:` + `Matching ${arxivs.length} arXiv entries, out of ${parsed.length} total entries:`, ) : setHTML( "n-arxivs", - `No arXiv entries found in ${parsed.length} total entries.` + `No arXiv entries found in ${parsed.length} total entries.`, ); const matched = arxivs.length ? await matchItems(arxivs) : []; matched.length && setTimeout(() => { - findEl({ element: "papers-successfully-matched" })?.scrollIntoView( - true - ); + findEl({ + element: "papers-successfully-matched", + })?.scrollIntoView(true); }, 250); showBibliography(parsed, matched, arxivIndices); addListener("show-only-matches", "change", () => { @@ -110,7 +132,7 @@ const updateStatusInfo = () => { let reasons = Object.entries(DISABLE_MATCH) .map( ([key, value]) => - `Disabling ${display[key]} for this matching process because the server returned a status of ${value}` + `Disabling ${display[key]} for this matching process because the server returned a status of ${value}`, ) .join("
"); if (reasons) { @@ -271,7 +293,7 @@ const matchItems = async (papersToMatch) => {
Looking for publications on
- ` + `, ); const progressbar = querySelector("#matching-progress-bar"); @@ -291,7 +313,7 @@ const matchItems = async (papersToMatch) => { setHTML("matching-status-index", idx + 1); setHTML( "matching-status-title", - paper.title.replaceAll("{", "").replaceAll("}", "") + escapeHtml(paper.title.replaceAll("{", "").replaceAll("}", "")), ); changeProgress(parseInt((idx / papersToMatch.length) * 100)); @@ -337,17 +359,17 @@ const updateMatchedTitles = (matchedBibtexStrs, sources, venues) => { if (entries.length) { const keys = entries.map((e) => e.citationKey); const titles = entries.map((e) => - e.title.replaceAll("{", "").replaceAll("}", "") + e.title.replaceAll("{", "").replaceAll("}", ""), ); htmls.push(""); for (const [idx, title] of titles.entries()) { htmls.push( ` - - - - - ` + + + + + `, ); } htmls.push("
${keys[idx]}${title}${venues[idx]}${sources[idx]}
${escapeHtml(keys[idx])}${escapeHtml(title)}${escapeHtml(venues[idx])}${escapeHtml(sources[idx])}
"); @@ -355,7 +377,7 @@ const updateMatchedTitles = (matchedBibtexStrs, sources, venues) => { setHTML( "matched-list", `

Papers successfully matched: ${entries.length}

` + - htmls.join("") + htmls.join(""), ); }; diff --git a/src/content_scripts/content_script.js b/src/content_scripts/content_script.js index 629b5415..fdb941e0 100644 --- a/src/content_scripts/content_script.js +++ b/src/content_scripts/content_script.js @@ -1,3 +1,61 @@ +// ES Module imports +import { + addListener, + setHTML, + findEl, + style, + querySelector, + addClass, + hasClass, + removeClass, + queryAll, +} from "@pmu/miniquery.js"; +import { + copyTextToClipboard, + isPdfUrl, + warn, + log, + info, + sendMessageToBackground, + downloadURI, + dummyEvent, + escapeHtml, +} from "@pmu/functions.js"; +import { bibtexToString, bibtexToObject } from "@pmu/bibtexParser.js"; +import { + getStorage, + deletePaperInStorage, + getDefaultKeyboardAction, +} from "@pmu/data.js"; +import { isPaper, addOrUpdatePaper } from "@pmu/paper.js"; +import { state } from "@pmu/config.js"; +import { getSource } from "@pmu/sources/index.js"; +import { initSyncAndState, sleep } from "@pmu/sync.js"; +import { + handleOpenItemAr5iv, + handleCopyBibtex, + handleCopyPDFLink, + handleOpenItemLink, + handleCopyMarkdownLink, + handleOpenItemHuggingface, + handleOpenItemAlphaxiv, + handleOpenItemScirate, + handleCopyHyperLink, +} from "@pm/popup/js/handlers.js"; +import { stateTitleFunction } from "@pmu/state.js"; +import { parseIdFromUrl, isSourceURL, isArxivAbstractUrl } from "@pmu/urls.js"; + +// Notification object for feedback system +var notif = { + element: null, + timeout: null, + prevent: false, + isLoading: false, + displayDuration: 5000, + showSpeed: 500, + hideSpeed: 200, +}; + /* * jQuery Easing v1.3 - http://gsgd.co.uk/sandbox/jquery/easing/ * @@ -78,8 +136,6 @@ $.extend($.easing, { }, }); -var PDF_TITLE_ITERS = 0; - /** * Centralizes HTML svg codes * @param {string} name svg type @@ -197,8 +253,8 @@ const makePaperMemoryHTMLDiv = (paper) => { style="display: flex; justify-content: center; align-items: center;" id="pm-venue" > - ${paper.venue} - ${bibtexToObject(paper.bibtex).year} + ${escapeHtml(paper.venue)} + ${escapeHtml(bibtexToObject(paper.bibtex).year)}

{ } if (!id) return; const e = dummyEvent(id); - const paper = global.state.papers[id]; + const paper = state.papers[id]; let text; if (!paper) return; switch (action) { @@ -341,7 +397,7 @@ const contentScriptMain = async ({ tryArxivDisplay({ url }); await remoteReadyPromise; } - const prefs = global.state.prefs; + const prefs = state.prefs; let is = await isPaper(url, true); @@ -366,17 +422,17 @@ const contentScriptMain = async ({ } else { if (ignorePaper(is, ignoreSources)) { warn( - "Paper is being ignored because its source has been disabled in the Advanced Options." + "Paper is being ignored because its source has been disabled in the Advanced Options.", ); } else if (prefs.checkPdfOnly && !isPdfUrl(url)) { warn( `Paper is being ignored because you have checked the PDF-Only option ` + - `and the current URL (${url}) is not that of a pdf's.` + `and the current URL (${url}) is not that of a pdf's.`, ); } else if (prefs.checkNoAuto && !manualTrigger) { warn( "Paper is being ignored because you disabled automatic parsing" + - " in the menu." + " in the menu.", ); } } @@ -388,15 +444,17 @@ const contentScriptMain = async ({ if (id && prefs.checkPdfTitle) { const makeTitle = async (id) => { - if (!global.state.papers.hasOwnProperty(id)) return; - const paper = global.state.papers[id]; + if (!state.papers.hasOwnProperty(id)) return; + const paper = state.papers[id]; const maxWait = 60 * 1000; - while (1) { - const waitTime = Math.min(maxWait, 250 * 2 ** PDF_TITLE_ITERS); + const maxIters = 20; + let pdfTitleIters = 0; + while (pdfTitleIters < maxIters) { + const waitTime = Math.min(maxWait, 250 * 2 ** pdfTitleIters); await sleep(waitTime); document.title = ""; document.title = paper.title; - PDF_TITLE_ITERS++; + pdfTitleIters++; } }; makeTitle(id); @@ -404,60 +462,60 @@ const contentScriptMain = async ({ }; const makeNotif = () => { - if (global.notif.element) return; - const notif = /*html*/ `

`; - document.body.insertAdjacentHTML("beforeend", notif); - global.notif.element = $("#feedback-notif"); + if (notif.element) return; + const notifHtml = /*html*/ `
`; + document.body.insertAdjacentHTML("beforeend", notifHtml); + notif.element = $("#feedback-notif"); style("feedback-notif", "padding", "0px"); }; const hideNotif = () => new Promise(async (resolve) => { const end = ({ dontWait = false } = {}) => { - global.notif.prevent = false; - global.notif.isLoading = false; + notif.prevent = false; + notif.isLoading = false; querySelector("#feedback-notif")?.classList.remove("notif-small"); setTimeout(resolve, dontWait ? 0 : 150); }; - if (!global.notif.element) { - warn("[hideNotif] Notif element not found"); + if (!notif.element) { + // warn("[hideNotif] Notif element not found"); end({ dontWait: true }); return; } - global.notif.element.animate( + notif.element.animate( { right: "-200px", opacity: "0" }, - global.notif.hideSpeed, + notif.hideSpeed, "easeInOutBack", - end + end, ); // sometimes animate does not call the callback - setTimeout(end, global.notif.hideSpeed + 50); + setTimeout(end, notif.hideSpeed + 50); }); const setNotifContent = (text) => { - if (!global.notif.element) { + if (!notif.element) { warn("[setNotifContent] Notif element not found"); return; } - global.notif.element.html(text); + notif.element.html(text); }; const showNotif = () => new Promise((resolve) => { - if (!global.notif.element) { + if (!notif.element) { console.warn("[PM][showNotif] Notif element not found"); makeNotif(); } - global.notif.element.animate( + notif.element.animate( { right: "64px", opacity: "1", }, - global.notif.showSpeed, + notif.showSpeed, "easeInOutBack", - resolve + resolve, ); }); @@ -470,7 +528,7 @@ const showNotif = () => const feedback = async ({ text, paper = null, - displayDuration = global.notif.displayDuration, + displayDuration = notif.displayDuration, loading = false, }) => { if (document.readyState === "loading") { @@ -478,18 +536,18 @@ const feedback = async ({ return; } makeNotif(); - if (global.notif.prevent && !global.notif.isLoading) { + if (notif.prevent && !notif.isLoading) { setTimeout(() => feedback({ text, paper, displayDuration, loading }), 100); return; } try { - clearTimeout(global.notif.timeout); + clearTimeout(notif.timeout); await hideNotif(); - global.notif.prevent = true; + notif.prevent = true; } catch (error) {} let content = ""; - global.notif.isLoading = false; + notif.isLoading = false; if (paper) { content = /*html*/ `
@@ -499,7 +557,7 @@ const feedback = async ({ ${svg("notif-cancel")}
`; } else if (loading) { - global.notif.isLoading = true; + notif.isLoading = true; querySelector("#feedback-notif")?.classList.add("notif-small"); content = /*html*/ `
`; } else { @@ -510,18 +568,18 @@ const feedback = async ({ setNotifContent(content); await showNotif(); - global.notif.timeout = setTimeout(hideNotif, displayDuration); + notif.timeout = setTimeout(hideNotif, displayDuration); paper && addListener("notif-cancel", "click", async () => { - clearTimeout(global.notif.timeout); - await deletePaperInStorage(paper.id, global.state.papers); - if (!global.state.deleted) { - global.state.deleted = {}; + clearTimeout(notif.timeout); + await deletePaperInStorage(paper.id, state.papers); + if (!state.deleted) { + state.deleted = {}; } - global.state.deleted[paper.id] = true; - setTimeout(() => delete global.state.deleted[paper.id], 30 * 1000); - global.notif.timeout = setTimeout(hideNotif, displayDuration); + state.deleted[paper.id] = true; + setTimeout(() => delete state.deleted[paper.id], 30 * 1000); + notif.timeout = setTimeout(hideNotif, displayDuration); setHTML("notif-text", "
Removed from memory
"); }); }; @@ -551,14 +609,14 @@ const displayPaperVenue = (paper) => { const venueDiv = /*html*/ `
- ${paper.venue} - ${bibtexToObject(paper.bibtex).year} + ${escapeHtml(paper.venue)} + ${escapeHtml(bibtexToObject(paper.bibtex).year)}
`; findEl({ element: "pm-publication-wrapper" })?.remove(); findEl({ element: "pm-header-content" })?.insertAdjacentHTML( "afterbegin", - venueDiv + venueDiv, ); }; @@ -570,9 +628,13 @@ const displayPaperCode = (paper) => { if (!paper.codeLink) { return; } + const safeLink = /^https?:\/\//.test(paper.codeLink) + ? escapeHtml(paper.codeLink) + : ""; + if (!safeLink) return; const code = /*html*/ `
-

Code:

${paper.codeLink} +

Code:

${safeLink}
`; findEl({ element: "pm-code" })?.remove(); @@ -591,14 +653,15 @@ const huggingfacePapers = (paper, url) => { const venueDiv = /*html*/ `
- ${paper.venue} - ${bibtexToObject(paper.bibtex).year} + ${escapeHtml(paper.venue)} + ${escapeHtml(bibtexToObject(paper.bibtex).year)} (PaperMemory)
`; - abstractH2 = queryAll("h2").find((h) => h.innerText.trim() === "Abstract"); + const abstractH2 = queryAll("h2").find((h) => h.innerText.trim() === "Abstract"); if (!abstractH2) { log("Missing 'Abstract' h2 title on HuggingFace paper page."); + return; } const authorDiv = abstractH2.parentElement.previousElementSibling; log("Adding venue to HuggingFace paper page."); @@ -632,7 +695,7 @@ const arxiv = async (checks) => {
-
` +
`, ); let pdfUrlButton = await queryOrWait({ query: ".abs-button.download-pdf" }); let pdfUrl = pdfUrlButton?.href; @@ -651,7 +714,7 @@ const arxiv = async (checks) => { findEl({ element: "pm-arxiv-direct-download" })?.remove(); (await queryOrWait({ query: "#pm-header-content" }))?.insertAdjacentHTML( "beforeend", - button + button, ); var downloadTimeout; addListener("arxiv-button", "click", async () => { @@ -667,13 +730,13 @@ const arxiv = async (checks) => { } if (!pdfUrl) { console.error( - "Could not parse the PDF URL from HTML element: `.abs-button.download-pdf`" + "Could not parse the PDF URL from HTML element: `.abs-button.download-pdf`", ); return; } - if (!global.state.papers.hasOwnProperty(id)) { + if (!state.papers.hasOwnProperty(id)) { const title = await fetch( - `https://export.arxiv.org/api/query?id_list=${id.split("-")[1]}` + `https://export.arxiv.org/api/query?id_list=${id.split("-")[1]}`, ).then((data) => { return $($(data).find("entry title")[0]).text(); }); @@ -697,9 +760,9 @@ const arxiv = async (checks) => { // ----- Markdown Link ----- // --------------------------- - let paper = global.state.papers.hasOwnProperty(id) - ? global.state.papers[id] - : await makeArxivPaper(url); + let paper = state.papers.hasOwnProperty(id) + ? state.papers[id] + : await getSource("arxiv").parse(url, null, {}); if (paper.venue) { displayPaperVenue(paper); @@ -710,8 +773,8 @@ const arxiv = async (checks) => { } if (checkMd) { - const mdTitle = global.state.papers.hasOwnProperty(id) - ? global.state.papers[id].title + const mdTitle = state.papers.hasOwnProperty(id) + ? state.papers[id].title : document.title; const mdContent = `[${mdTitle}](${pdfUrl})`; const mdHtml = /*html*/ ` @@ -746,7 +809,7 @@ const arxiv = async (checks) => { ${svg("clipboard-default")} ${svg("clipboard-default-ok")}
${bibtexToString( - paper.bibtex + paper.bibtex, ).replaceAll("\t", " ")}
`; @@ -777,13 +840,30 @@ const arxiv = async (checks) => { }); }); copyTextToClipboard( - findEl({ element: "markdown-link" }).innerText.replaceAll("\n", "") + findEl({ element: "markdown-link" }).innerText.replaceAll("\n", ""), ); feedback({ text: "Markdown Link Copied!" }); }); } }; +// Because Puppeteer does not receive content script logs as "console" events, +// we need another way to signal parsing completion to the testing script. +const updateCompleteSecretHTML = (paper) => { + let intervalId = null; + intervalId = setInterval(() => { + if (document?.querySelector("head")?.insertAdjacentHTML) { + clearInterval(intervalId); + document + .querySelector("head") + .insertAdjacentHTML( + "beforeend", + /*html*/ ``, + ); + } + }, 50); +}; + const tryArxivDisplay = async ({ url = null, paper = null, @@ -792,13 +872,13 @@ const tryArxivDisplay = async ({ // a paper was parsed // user preferences - const prefs = global.state.prefs; + const prefs = state.prefs; // paper.source may not be "arxiv" even on arxiv.org // because of the existing paper matching mechanism let is = await isPaper(url, true); - if (is.arxiv && !isPdfUrl(url)) { + if (is.arxiv && !isPdfUrl(url) && isArxivAbstractUrl(url)) { // larger arxiv column adjustArxivColWidth(); @@ -811,16 +891,16 @@ const tryArxivDisplay = async ({ paper = await preprintsPromise; } else { const id = await parseIdFromUrl(url); - const paperExists = global.state.papers.hasOwnProperty(id); + const paperExists = state.papers.hasOwnProperty(id); if (!paperExists) return; - paper = global.state.papers[id]; + paper = state.papers[id]; } // update bibtex if (prefs.checkBib) { if (findEl({ element: "pm-bibtex-textarea" })) { findEl({ element: "pm-bibtex-textarea" }).innerHTML = bibtexToString( - paper.bibtex + paper.bibtex, ).replaceAll("\t", " "); } } @@ -833,7 +913,8 @@ const tryArxivDisplay = async ({ } }; -(async () => { +export async function initContentScript() { + log("Running PaperMemory's content script"); var prefs, paper; var paperPromise, preprintsPromise, paperResolve, preprintsResolve; const url = window.location.href; @@ -843,7 +924,7 @@ const tryArxivDisplay = async ({ let stateIsReady = false; if (url.startsWith("file://")) { await initSyncAndState({ isContentScript: true }); - prefs = global.state.prefs; + prefs = state.prefs; stateIsReady = true; } @@ -865,6 +946,8 @@ const tryArxivDisplay = async ({ paperUpdateDoneCallbacks: { update: paperResolve, preprints: preprintsResolve, + done: updateCompleteSecretHTML, + feedback, }, }); } else if (request.message === "manualParsing") { @@ -876,6 +959,8 @@ const tryArxivDisplay = async ({ paperUpdateDoneCallbacks: { update: paperResolve, preprints: preprintsResolve, + done: updateCompleteSecretHTML, + feedback, }, }); } else if (request.message === "defaultAction") { @@ -900,6 +985,8 @@ const tryArxivDisplay = async ({ paperUpdateDoneCallbacks: { update: paperResolve, preprints: preprintsResolve, + done: updateCompleteSecretHTML, + feedback, }, }); } else { @@ -933,4 +1020,4 @@ const tryArxivDisplay = async ({ } } await hideNotif(); -})(); +} // end initContentScript diff --git a/src/debug/debug.js b/src/debug/debug.js new file mode 100644 index 00000000..74494759 --- /dev/null +++ b/src/debug/debug.js @@ -0,0 +1,120 @@ +// Debug bundle - exports all utility functions for development debugging +// This file is only built and loaded in development mode + +// Import all utility modules +import * as config from "@pmu/config.js"; +import * as functions from "@pmu/functions.js"; +import * as miniquery from "@pmu/miniquery.js"; +import * as data from "@pmu/data.js"; +import * as paper from "@pmu/paper.js"; +import * as bibtexParser from "@pmu/bibtexParser.js"; +import * as sync from "@pmu/sync.js"; +import * as state from "@pmu/state.js"; +import * as urls from "@pmu/urls.js"; +import * as files from "@pmu/files.js"; +import * as parsers from "@pmu/parsers.js"; +import * as sources from "@pmu/sources/index.js"; +import * as preprintMatching from "@pmu/preprintMatching.js"; +// Import popup-specific modules (when available) +// Important: these modules themselves import popup.js which contains +// immediately-invoked functions that should not be called twice. This is why +// we use a global variable to track whether the popup has been initialized. + +// If in the future we need to import other non-utils modules, we should +// check that this double-import issue is addressed. +import * as templates from "@pm/popup/js/templates.js"; +import * as handlers from "@pm/popup/js/handlers.js"; +import * as memory from "@pm/popup/js/memory.js"; + +// Create the debug object +const PMDebug = { + // Utility modules + config, + functions, + miniquery, + data, + paper, + bibtexParser, + sync, + state, + urls, + files, + parsers, + sources, + preprintMatching, + // Popup modules + templates, + handlers, + memory, + + // Helper to access commonly used functions directly + get getStorage() { + return data.getStorage; + }, + get setStorage() { + return data.setStorage; + }, + get getPrefs() { + return data.getPrefs; + }, + get log() { + return functions.log; + }, + get info() { + return functions.info; + }, + get findEl() { + return miniquery.findEl; + }, + get setHTML() { + return miniquery.setHTML; + }, + + get getPapers() { + return () => config.state.papers; + }, + + // Utility to list all available functions + listAllFunctions() { + const modules = [ + "config", + "functions", + "miniquery", + "data", + "paper", + "bibtexParser", + "sync", + "state", + "urls", + "files", + "templates", + "handlers", + "parsers", + "sources", + "preprintMatching", + "memory", + ]; + modules.forEach((moduleName) => { + if (this[moduleName]) { + console.group(`PMDebug.${moduleName}`); + Object.keys(this[moduleName]).forEach((key) => { + if (typeof this[moduleName][key] === "function") { + console.log(`• ${key}()`); + } else { + console.log(`• ${key}`); + } + }); + console.groupEnd(); + } + }); + }, +}; + +// Make it globally available +if (typeof window !== "undefined") { + window.PMDebug = PMDebug; +} else if (typeof global !== "undefined") { + global.PMDebug = PMDebug; +} + +export default PMDebug; diff --git a/src/entrypoints/background.js b/src/entrypoints/background.js new file mode 100755 index 00000000..02379202 --- /dev/null +++ b/src/entrypoints/background.js @@ -0,0 +1,5 @@ +import { initBackground } from "@pm/background/background.js"; + +export default defineBackground(() => { + initBackground(); +}); diff --git a/src/entrypoints/bibMatcher/index.html b/src/entrypoints/bibMatcher/index.html new file mode 100644 index 00000000..ec21d235 --- /dev/null +++ b/src/entrypoints/bibMatcher/index.html @@ -0,0 +1,204 @@ + + + + PaperMemory BibMatcher + + + + + + + + + + + +
+
+
+

+ Paste the content of your .bib file and PaperMemory + will automatically match the Arxiv entries to publications by + fetching information from DBLP, Semantic Scholar, CrossRef and + Google Scholar. +

+ +

+ The matching procedure is identical to the one you can trigger + to match your PaperMemory ArXiv entries in the + + Advanced Options + +

+ +

+ Warning: + the matching procedure produces an output which does + NOT maintain comments. Select matched entries + one by one if you want to keep your initial comment structure. +

+
+
+ +
+
+
+
+
+    + +
+
+
+ When a pre-print is matched to a publication, the latter + may have a standard citation key. You can either update + the entry's citation key or use the new one. Note that + if you use the new one, your existing citations will be + broken. +
+
+
+
+    + +
+
+
+ Data providers protect their APIs by restricting the + number of queries per seconds you can make. + PaperMemory's BibMatcher may iterate too fast over your + entries, reaching the API rate limits. Adding a timeout + between requests slows down the matching process but + increases your matching rate. +
+
+ +
+
+ +
+ + +
+
+ + + + +
+
+ + + + + diff --git a/src/entrypoints/bibMatcher/main.js b/src/entrypoints/bibMatcher/main.js new file mode 100644 index 00000000..3e593e3d --- /dev/null +++ b/src/entrypoints/bibMatcher/main.js @@ -0,0 +1,2 @@ +import "@pm/bibMatcher/bibMatcher.js"; +import "@pm/debug/debug.js"; diff --git a/src/entrypoints/content.js b/src/entrypoints/content.js new file mode 100644 index 00000000..d8819d01 --- /dev/null +++ b/src/entrypoints/content.js @@ -0,0 +1,14 @@ +import "@pm/shared/js/jquery-setup.js"; +import "@pm/shared/css/loader.css"; +import "@pm/content_scripts/content_script.css"; +import { initContentScript } from "@pm/content_scripts/content_script.js"; + +export default defineContentScript({ + matches: [""], + runAt: "document_start", + cssInjectionMode: "manifest", + + async main() { + await initContentScript(); + }, +}); // end defineContentScript diff --git a/src/entrypoints/fullMemory/index.html b/src/entrypoints/fullMemory/index.html new file mode 100644 index 00000000..867b230a --- /dev/null +++ b/src/entrypoints/fullMemory/index.html @@ -0,0 +1,136 @@ + + + + PaperMemory Page + + + + + + + + + + + + + +
+
+ + +
+ + × +
+ + + + + + + + + + + + + + + +
+
+
+ + + + diff --git a/src/entrypoints/fullMemory/main.js b/src/entrypoints/fullMemory/main.js new file mode 100644 index 00000000..ab8950c1 --- /dev/null +++ b/src/entrypoints/fullMemory/main.js @@ -0,0 +1,2 @@ +import "@pm/fullMemory/fullMemory.js"; +import "@pm/debug/debug.js"; diff --git a/src/entrypoints/options/index.html b/src/entrypoints/options/index.html new file mode 100644 index 00000000..a008659e --- /dev/null +++ b/src/entrypoints/options/index.html @@ -0,0 +1,945 @@ + + + + PaperMemory Options + + + + + + + + + + + + + +
+

+ These customization options are complementary to those available in + the popup such as Dark Theme or Notifications. You can also access + your full-screen memory + here. +

+

+ Warning: features which alter your Memory such as + publication matching, importing papers or overwriting your Memory + should not be triggered while you are browsing papers. This would + have unpredictable effects as you would concurrently write to the + same database. + +

+ +
+ +
+ +
+ +
+
+ +
+ +
+

PapersWithCode preferences

+ +
+ [1] Only store official code repositories: + + + + +
+
+ [2] Preferred implementation framework: + + + +
+ +
About
+

+ If you select option [1] and PapersWithCode reports no official + implementation for the paper then no repository will be stored, even + if there are non-official implementations. Otherwise, official + implementations will always be preferred. +

+

+ If a framework is selected in [2] and such an implementation is + available, all other implementations will be ignored. +

+

+ If multiple code repositories are available at this point, + PaperMemory will store the one with most stars. +

+
+ +
+ +
+

Auto-tagging

+

+ Provide Javascript regular expressions to automatically add tags to + papers based on their titles and authors. The two columns represent + an AND. Use two different entries for an + OR. Matching is not case-sensitive. An + empty input matches everything, it is equivalent to .*. +

+ +

+ The Authors RegEx will be matched against a string which joins + authors in a BibTex fashion, e.g. + Abc Def and Ghi Jkl and Mno Pqr. +

+ +

+ For instance, to match all papers containing + "GAN" the corresponding Title RegEx could be + .*gan.* (note this would match any title + containing those 3 letters subsequently). +

+ +
+
+
Title RegEx
+
Authors RegEx
+
+ Tags to use (coma-seprated) +
+
+ +
+
+
+
+
+ +
+
+ +
+
+ +
+
+
+
+ + + + +
+
+
+
+
+
+ +
+ +
+

Preprint matching

+ +

+ In this section, you can manually trigger the + preprint matching procedure + to discover publications from Arxiv pre-prints and code for your + papers from paperswithcode.com. +

+ +

+ A paper's note will only be updated to + Accepted @ venue (year) -- [source] + if  you don't have a custom note already. +

+ +

+ You currently have + papers missing a publication + venue. +

+ + + + +
+
+ +
+ +
+

Select Sources

+

+ If you don't want to track papers from all the sources + PaperMemory can handle, you can disable them here. +

+

+ Note that you can only disable per + source not exact venue. For instance you cannot + distinguish between Nature Communications and Nature Climate Change + (both nature venues) or between CVPR and ICCV (both + cvf venues). +

+

Sources to parse papers from:

+
+
+ +
+ +
+

Data Management

+ +

+ You can export your Memory as a json file or as a + bibliography. +

+

+ To perform advanced data manipulation (paper merges, batch tagging + etc.) you can process the exported Memory file and then load it into + the extension, overwriting your current data. +

+ +

You can also share a list of papers for others in your team!

+
+
Full Memory Exports
+ +
+
+

+ > Export the Memory as a BibTex + bibliography: +

+
+
+ +
+
+ +
+
+ +
+
+

+ > Export your full Memory data: +

+
+
+ +
+
+ +
+
+

+ > Load a json memory file (this will + overwrite + your Memory): +

+
+
+ +
+
+ +
+
+
+ +
+ + +
+ +
+
Export by tags
+ +

+ Select papers to export by providing a list of tags. If you select + the AND operator, papers will be required to have all + those tags. If you select the OR operator, papers will + be required to have at least one of those tags. +

+ +

+ You can export to .bib for bibliographies, or to + .json. In the latter case you can choose to either just + export URLs, or include title, + codeLink and tags fields (in order to + match the import format, and + title is just for the sake of human readability) +

+ +
+
+ +
+
+ +
+
+ + +
+
+ +
+
+ +
+
+ +
Import papers
+ +

+ PaperMemory lets you add papers from a list of URLs in the form of a + .json file. +

+ +
+ The file must be structured as follows: a list of entries where each + entry is either a string pointing to a paper's + url (PDF or abstract) or an object with a + mandatory "url": string field and optionally : +
    +
  • + a "codeLink": string field pointing a to code + repository +
    • +
  • + a "tags": Array<string> field with tags. +
    • +
+
+ +

+ Note    : all URLs must come from sources known to PaperMemory +

+

+ Note 2 : If you have enabled online synchronization through Github + Gists, be aware that synchronization will only happen after + all imported papers have been parsed. +

+ +

Example valid .json file:

+ + [ "https://arxiv.org/pdf/1901.06500.pdf", + "https://arxiv.org/abs/2110.02871", { "url": + "https://arxiv.org/abs/1811.12833", "codeLink": + "https://github.com/valeoai/ADVENT" }, { "url": + "https://openreview.net/forum?id=xQUe1pOKPam", "tags": ["graphs", + "3d", "molecules"], "codeLink": + "https://github.com/chao1224/graphmvp" } ] + + +
+
+ + +
+
+ +
+
+ + +
+
    +
    +
    +
    + +
    + +
    +

    Online synchronization

    + +

    How it works

    + +
      +
    • + Your Memory data gets written to a + .json file as a gist on your Github account (read about gists) +
      • +
    • + Although it is non-discoverable, your gist data is still + accessible to someone you'd give the link to +
      • +
    • + The synchronization happens in 2 stages: +
        +
      • + Push : when you close the Popup or when a + paper gets added to the memory, the remote gist gets + overwritten +
        • +
      • + Pull   : when you open the popup + or before a paper gets added, the remote gist is + downloaded and overwrites your local Memory +
        • +
      +
      • +
    • This is all very alpha!
      • +
    + +

    + Syncing is meant for sequential use across multiple + devices. +

    + +

    + Syncing does NOT turn PaperMemory into a multi-user + tool: in its current state, PaperMemory will NOT handle conflicts or + concurrent writes to the remote data. Any conflict will result in + potential loss of data. +

    + +

    Syncing

    + +
    + PaperMemory relies on Github's + Personal Access Tokens which you can easily generate from + your Github account. Make sure +
      +
    1. It has no expiration date
      2. +
    2. The gist scope is selected
      3. +
    + You can revoke the token later at any time. Click + here + to generate a new token. +
    + +
    + + + +
    +
    + Note: your token is saved locally and does not leave your + computer. +
    +
    + +

    +
    + + +
    +
    + +
    + + + + +
    + + + + + diff --git a/src/entrypoints/options/main.js b/src/entrypoints/options/main.js new file mode 100644 index 00000000..de63dc14 --- /dev/null +++ b/src/entrypoints/options/main.js @@ -0,0 +1,2 @@ +import "@pm/options/options.js"; +import "@pm/debug/debug.js"; diff --git a/src/popup/html/popup.html b/src/entrypoints/popup/index.html similarity index 77% rename from src/popup/html/popup.html rename to src/entrypoints/popup/index.html index 14dc80be..f348c4b0 100755 --- a/src/popup/html/popup.html +++ b/src/entrypoints/popup/index.html @@ -1,64 +1,31 @@ - + - - + - + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - +