ErgativeGPT

Published

July 2, 2025

Overall Performance

  • Progress: 5/9 verbs completed with confirmed high-quality results (close, dry, dissolve, fill, freeze)
  • Overall Accuracy: 94.8-97.0% across all confirmed, high-quality experiments.
  • Processing Efficiency: 100% parsing success rate
  • Robust Performance: Consistent results across different verb types
  • Cost Effective: Using o4-mini model for optimal cost/performance ratio

Methodology

Data Sources

  • COHA (Corpus of Historical American English) sentences
  • Target verbs: “close”, “dry”, “dissolve” and “fill” (completed), with 5 remaining verbs
  • Sample size: 180 sentences per verb
  • Ground truth: Expert linguistic annotations

Prompt Engineering Impact

Critical Finding: The detailed prompt (prompt_2025-03-07.md) significantly outperforms a simpler version:

  • Linguistic definitions: Clear examples and contextual guidance
  • Structured examples: Multiple exemplars for each category
  • Robust instructions: Detailed classification criteria
  • JSON schema: Structured output validation

Decision: All remaining verbs will use the detailed prompt for optimal performance.

Classification Variables

Core Linguistic Variables (evaluated for accuracy):

  1. Transitivity: transitive vs intransitive
  2. Causativity: causative vs anticausative
  3. Subject Role: agent vs patient
  4. Subject Animacy: animate vs inanimate

Quality Control Variable (filtering only):

  1. Part of Speech: verb vs other - used to exclude non-verbal uses of target words from linguistic analysis. Only sentences classified as “verb” are included in accuracy calculations for the four core variables above.

Model Configuration

  • Model: OpenAI o4-mini
  • Structured Output: JSON schema validation
  • Retry Logic: Up to 3 attempts per sentence

Results overview

Performance overview across verbs

Verb Total Sentences Overall Accuracy F1
close 180 94.8% 93.7%
dissolve 180 94.8% 94.6%
dry 180 95.7% 95.6%
fill 180 97.0% 96.4%
freeze 180 94.9% 94.8%

Metrics visualizations for each verb

close

Close Performance Summary

dissolve

Dissolve Performance Summary

dry

Dry Performance Summary

fill

Fill Performance Summary

freeze

Freeze Performance Summary

Cross-verb performance analysis

Performance by variable and verb

Verb Variable Accuracy F1
close Causativity 92.7% 89.7%
close Subject Animacy 91.9% 91.2%
close Subject Role 96.8% 96.3%
close Transitivity 97.6% 97.5%
dissolve Causativity 90.6% 90.5%
dissolve Subject Animacy 92.5% 91.9%
dissolve Subject Role 98.8% 98.6%
dissolve Transitivity 97.5% 97.2%
dry Causativity 94.4% 94.3%
dry Subject Animacy 93.8% 93.7%
dry Subject Role 95.1% 95.1%
dry Transitivity 99.3% 99.3%
fill Causativity 100.0% 100.0%
fill Subject Animacy 93.5% 93.5%
fill Subject Role 98.4% 97.7%
fill Transitivity 96.0% 94.3%
freeze Causativity 92.4% 92.4%
freeze Subject Animacy 95.8% 95.8%
freeze Subject Role 94.1% 93.8%
freeze Transitivity 97.5% 97.1%

Linguistic variables by difficulty

Linguistic Variable Average F1 Score
Subject Animacy 93.2%
Causativity 93.4%
Subject Role 96.3%
Transitivity 97.1%

Confusion matrices for each verb

Error analysis

Errors in Transitivity Classification

Verb Sentence Context Ground Truth GPT Prediction
close cent at least have the same idea I have? that the schools must be closed. I want it blazoned to the world that Chicago's school-teachers have been forced intransitive transitive
close having sent Fles orders to sell. A day or two later the exchanges were closed and, shortly after, the banks. Business came to a practical standstill. intransitive transitive
dissolve men. Under such a trend, the government could still reserve the right to dissolve. in civil proceedings, cooperative monopolies in gross restraint of trade, and in intransitive transitive
freeze . HALLORANI'll speak to you at the station. I've had your promotion frozen. Wear a uniform. Jack and Halloran stare at each other for a beat intransitive transitive
freeze aside. When the cream is cold, add the remaining cream, and partly freeze. intransitive transitive
fill an Administration supporter, Representative James W. Trimble of Arkansas, is expected to be filled by another Administration man. This would give Administration forces a more comfortable majority of intransitive transitive
fill little while, as it happened every day, the hours being too long to fill with talk of the places they came from or talk of their families who could intransitive transitive
fill plain Norman work. All the windows have, however, been enlarged, and filled with Decorated or Perpendicular tracery, chiefly with an internal cusping only, as is intransitive transitive
close system probably crisscrossed That entire valley down there. I guess after the tunnel was closed up by the earthquakes, The yavapai tribe just disappeared, Almost as if they intransitive transitive
freeze They freeze SCIENTIST #1 Impossible to think.... we, in this lab. intransitive transitive

Errors in Causativity Classification

Verb Sentence Context Ground Truth GPT Prediction
freeze Ezra Weeks fightin' out a game of checkers is like gettin' your feet froze in January and waitin' for spring to come and thaw' em out. causative anticausative
freeze They freeze SCIENTIST #1 Impossible to think.... we, in this lab. anticausative causative
close in the basic Kamerad position. You have walked through a trap which will now close around you for 30 days. It seems very logical, sir, under the anticausative causative
dissolve electrolyte where nascent hydrogen is evolved. But, whereas iron after this latter treatment dissolves' See MASINC, Trans., A.I.M.E. Institute of Metals Div., vol. 104 causative anticausative
freeze having read the letter that Cassy had written him, said that she could " freeze at the station " for all of him. Aunt Kate had said nothing then anticausative causative
dry on sale by Indian Head Mills, Inc. The sheets and matching pillow cases dry to a wrinkle-free finish in little more than half the time of ordinary sheets, causative anticausative
close their faces, and then up at -- THE BOARD where April Orange Juice has closed out at $0.29 a pound. BACK TO BILLY RAY AND WHIPPLE as it begins causative anticausative
dry will need when he's older. 2. Gauze diapers are more absorbent and dry more quickly than Birdseye. To save, buy irregulars, or seconds, from causative anticausative
dry of tea may be made from the same leaves, according to the treatment while drying. This is the source of the various kinds found in the market under the causative anticausative
dissolve reboil them with the fresh imitation coffee. Sugar is beige-colored crystals which never completely dissolve. Bread is grayish brown, heavy, hard, and contains splinters or straw causative anticausative

Errors in Subject Role Classification

Verb Sentence Context Ground Truth GPT Prediction
close she whispered promptly, " sometime. If you want me. " His hands closed convulsively over hers. " Make it soon, " he begged. " It patient agent
dissolve . " He turned and strode rapidly to his jeep. The retinue parted, dissolved, scrambled for the jeeps. Sirens cut the morning silence of the hill. agent patient
fill I can make so I can die with dignity. There's a form to fill out about how I want my life to end. I don't know what patient agent
close morning, and which she had not thought of once since. As her fingers closed on it a wave of new life seemed to flow through her body; she patient agent
dry texture of a fairy princess's. Its owner was apparently hanging it out to dry. But it kept coming! It must have been down to her waist, patient agent
freeze They freeze SCIENTIST #1 Impossible to think.... we, in this lab. patient agent
close in the basic Kamerad position. You have walked through a trap which will now close around you for 30 days. It seems very logical, sir, under the patient agent
freeze don't worry. I'm not gon na blow yer cover. (They freeze. Lights come down slowly.) END OF ACT ONE (More new patient agent
freeze believe in... (There is a knock on the door. They freeze. Another knock, angrier, more forceful. DOUG gets up and goes to patient agent
freeze see Ben holding onto the Driver. They raise their guns. # COP # FREEZE! PUT THE MAN AND THE SEAT DOWN! Ben looks at the cops. agent patient

Errors in Subject Animacy Classification

Verb Sentence Context Ground Truth GPT Prediction
dry the British shellac into the rotting wall. Within a year the mold had entirely dried up, and the painting remained. # Blots Bandages. Still, it was inanimate animate
dissolve No religious corporation may conduct an educational institution. 4) Monasteries and nunneries are dissolved. 5) No religious publication may publish any account of or comment on national inanimate animate
close . # One source familiar with department operations said that months after Stockton Developmental Center closed in 1995, the state's computerized client file showed more than 100 disabled people inanimate animate
dry either an autoclave, buck-press, iron, or steam-chamber; and (3) drying the crease-set garn'ient in a drying chamber. This spray solution concentrate has been licensed animate inanimate
close having sent Fles orders to sell. A day or two later the exchanges were closed and, shortly after, the banks. Business came to a practical standstill. inanimate animate
freeze at itself? " From the corner of my eye I saw the tiny fingers freeze. " I seem to feel a need to groom my fur, " it inanimate animate
dissolve .. " he whispered, even as he felt his own body absorbing, dissolving... " Be loved.... " Soft lips sought his inanimate animate
freeze They freeze SCIENTIST #1 Impossible to think.... we, in this lab. inanimate animate
fill WE can not linger to detail all the types of structure which Rome devised to fill the needs of her complication. The temples which contained her gods, the triumphal animate inanimate
close 5562158 How's the market? Not so good. International Pine Box closed at a new low. American Tombstone toppled from its high point... and now inanimate animate