Add author index - #9179
Conversation
|
I'm setting this up. Grobid seems pretty good: https://aclanthology.org/2010.amta-srw.1/ Existing data: {
"anthology": {
"abstract": "English-Manipuri language pair is one of the rarely investigated with restricted bilingual resources. The development of a factored Statistical Machine Translation (SMT) system between English as source and Manipuri, a morphologically rich language as target is reported. The role of the suffixes and dependency relations on the source side and case markers on the target side are identified as important translation factors. The morphology and dependency relations play important roles to improve the translation quality. A parallel corpus of 10350 sentences from news domain is used for training and the system is tested with 500 sentences. Using the proposed translation factors, the output of the translation quality is improved as indicated by the BLEU score and subjective evaluation.",
"authors": [
{
"first": "Thoudam Doren",
"id": "thoudam-doren-singh",
"index": 0,
"last": "Singh",
"name": "Thoudam Doren Singh"
},
{
"first": "Savaji",
"index": 1,
"last": "Bandyopadhyay",
"name": "Savaji Bandyopadhyay"
}
],
"bibkey": "singh-bandyopadhyay-2010-statistical",
"month": "October 31-November 4",
"paper_id": "2010.amta-srw.1",
"paper_type": "paper",
"publisher": "Association for Machine Translation in the Americas",
"publisher_address": "Denver, Colorado, USA",
"title": "Statistical Machine Translation of English-Manipuri using Morpho-syntactic and Semantic Information",
"venue_ids": [
"amta"
],
"volume_id": "2010.amta-srw",
"volume_title": "Proceedings of the 9th Conference of the Association for Machine Translation in the Americas: Student Research Workshop",
"web_url": "https://aclanthology.org/2010.amta-srw.1/",
"year": "2010"
},
"extractor": {
"name": "GROBID processHeaderDocument",
"options": {
"consolidateHeader": "0",
"includeRawAffiliations": "1",
"includeRawCopyrights": "1"
},
"service_url": "http://localhost:8070",
"version": "{\"version\":\"0.9.0\",\"revision\":\"0.9.0\"}"
},Extracted: "grobid": {
"abstract": "English-Manipuri language pair is one of the rarely investigated with restricted bilingual resources. The development of a factored Statistical Machine Translation (SMT) system between English as source and Manipuri, a morphologically rich language as target is reported. The role of the suffixes and dependency relations on the source side and case markers on the target side are identified as important translation factors. The morphology and dependency relations play important roles to improve the translation quality. A parallel corpus of 10350 sentences from news domain is used for training and the system is tested with 500 sentences. Using the proposed translation factors, the output of the translation quality is improved as indicated by the BLEU score and subjective evaluation.",
"affiliations": [
{
"address": {
"country": {
"key": "IN",
"text": "India"
},
"post_code": "700032",
"settlement": "Kolkata"
},
"attributes": {
"key": "aff0"
},
"id": "aff0",
"organizations": [
{
"name": "Department of Computer Science and Engineering",
"type": "department"
},
{
"name": "Jadavpur University",
"type": "institution"
}
],
"raw": "Department of Computer Science and Engineering Jadavpur University Kolkata - 700032 , India",
"text": "Department of Computer Science and Engineering Jadavpur University Kolkata - 700032 , India Department of Computer Science and Engineering Jadavpur University 700032 Kolkata India"
},
{
"address": {
"country": {
"key": "IN",
"text": "India"
},
"post_code": "700032",
"settlement": "Kolkata"
},
"attributes": {
"key": "aff1"
},
"id": "aff1",
"organizations": [
{
"name": "Department of Computer Science and Engineering",
"type": "department"
},
{
"name": "Jadavpur University",
"type": "institution"
}
],
"raw": "Department of Computer Science and Engineering Jadavpur University Kolkata - 700032 , India",
"text": "Department of Computer Science and Engineering Jadavpur University Kolkata - 700032 , India Department of Computer Science and Engineering Jadavpur University 700032 Kolkata India"
}
],
"authors": [
{
"affiliation_ids": [
"aff0"
],
"attributes": {
"role": "corresp"
},
"emails": [
"thoudam.doren@gmail.com"
],
"forenames": [
{
"text": "Doren",
"type": "first"
}
],
"index": 0,
"name": "DorenThoudam",
"surnames": [
"Thoudam"
]
},
{
"affiliation_ids": [
"aff0"
],
"index": 1,
"name": "Singh",
"surnames": [
"Singh"
]
},
{
"affiliation_ids": [
"aff1"
],
"forenames": [
{
"text": "Sivaji",
"type": "first"
}
],
"index": 2,
"name": "SivajiBandyopadhyay",
"surnames": [
"Bandyopadhyay"
]
}
],
"grobid_application": {
"ident": "GROBID",
"version": "0.9.0",
"when": "2026-07-18T14:12+0000"
},
"identifiers": [
{
"attributes": {
"type": "MD5"
},
"text": "9FE7EB56204B11CE4AA0A14E40A4D6F7"
}
],
"title": "Statistical Machine Translation of English -Manipuri using Morpho-syntactic and Semantic Information",
"titles": [
{
"attributes": {
"level": "a",
"type": "main"
},
"text": "Statistical Machine Translation of English -Manipuri using Morpho-syntactic and Semantic Information"
}
]
},
"paper_id": "2010.amta-srw.1",
"schema_version": 1,
"source": {
"checksum": "6093d80b",
"reference": "2010.amta-srw.1",
"sha256": "61de6a82fe18c118535746235b25f55b51f596c8d7e084280f2ca139864bf433",
"size": 696037,
"url": "https://aclanthology.org/2010.amta-srw.1.pdf"
},
"status": "success"
} |
|
Would be good to evaluate against the known affiliations. |
|
For pre-2020 papers, datasets with GROBID extractions from the entire Anthology already exist, it seems unnecessary to reinvent the wheel here. I have a tar here with GROBID data for the entire Anthology from 2022 and earlier that I downloaded in May, though I forgot where I got it from, I think it may have been from the ACL OCL? https://github.com/shauryr/ACL-anthology-corpus |
|
Thanks, I was unaware of this! I'll have to pick this project up again later. What I'd like to do is generate volume-level extractions at ingestion time, and then store them on the server as side files to the PDFs (e.g., |
That seems useful! In the meantime, it seems like the Google Drive link from the ACL-OCL repo is no longer working, so if you want the data (or want me to do something with the data), let me know. I’ll try to look at the rest of this PR too at some point, I haven’t gotten around to that yet. :) |
Thanks, will do!
We figured you're still catching up. For the new, non-urgent stuff, my triage list is:
|
|
Small touch: the example author is now chosen uniformly at random from the Anthology database: 
|
|
Since this also contains the unmerged #9158 it’s a bit hard to review the diff against Functionality-wise, I do like the look & feel of the author index page on first impression (without having tested it much), but would note that it’s design language also completely diverges from the rest of the Anthology. For example, the buttons on the right ("Verification explained", "Use your ORCID ID", ...) are nice, but one would question why this design with these colors & icons doesn’t show up elsewhere on the Anthology where it would be relevant (on the author pages proper, for example). Of course, one solution might be to transfer it to other pages too if we like it, but this in general is an annoying consequence of AI-generating such layouts, IMO. Also, the first button ("Get verified") does exactly the same thing as the button right above it ("Verify your author page"), which seems superfluous. |
|
Sure, this makes sense. I agree about the design language, and in fact, I'd prefer to keep our own. I'll note that I really like this search box. When I navigate to an author page, my normal m.o. is to just type the URL, hoping I guess their slug. I now find myself using the search box in this preview. |
I mean, yeah, it’s exactly the kind of thing I wanted to try using Pagefind for since forever, but to essentially search in everything, paper titles and abstracts included. |
I looked into this a little bit. Extending this to searching over papers as well would take some work. From what I've read, PageFind works great on up to tens of thousands of pages, but we have hundreds of thousands. I think there might be a confluence with #5324. I am thinking that we should run OCR + text extraction on all papers and create a sidecar file for every PDF (e.g., W18-6319.json). This could have, for example, a grobid extraction dump. We could run this in the background and it would automatically just notice new PDFs, kind of like the way that the thumbnails work now (so there is precedent for this, as well as administrative savings, since this isn't a new type of thing, just extended work for an existing process). Those stats could then be folded into the library and used to build a Pagefind index, statistics, and so on. But I think this would all be a separate project. Maybe something for the annual Christmas break push. |
Note that this assumes full-content web pages; if we were to do this with, say, just paper titles + author lists, this seems manageable (and would already be infinitely more useful than the annoying and broken Google search we have now). Anyway, remember this old issue? :) #165 (comment) |
New approach searches both authors and papers and returns matches in the union, noting which portion of the Venn diagram each result matched. Records are sharded by three-character prefixes of title words Index is ~106 MiB compressed; queries are 0.35–1.3 seconds locally (faster once cached)
Wow, that is a deep dive into old work. I did add an implementation that searches over authors and paper titles simultaneously. It'd be nice to have this mimic the form you had earlier, where results are returned and can be selected by author or title. But for the top search bar, I'd really prefer to also be able to search at least abstracts and ideally also full-text. |
|
When you say “Index is ~106 MiB compressed” (in the commit message), is that the amount of data that needs to be transferred to the client before search works? If so, that seems like something I probably wouldn’t want to deploy. What about mobile users checking the Anthology from an expensive data plan at their conference destination and getting blasted with that? How big is just the author index in comparison? In any case, that’s exactly why stuff like Pagefind exists, and I’d probably hold back on this functionality until we had time to integrate that. |
I added an author count to the first-page stats, and then, on a whim, realized it'd be nice to have an author index.
No need to review yet, I haven't even looked through it yet.
Closes #4603.