Skip to content

gpsaggese/helpers / .claude/skills/testing.update_test_outcomes

testing.update_test_outcomes

Verified

Update the expected outcome of failing unit tests

Goal

  • The user passes a list of tests, or a repro script, with failing tests <FAILING_TESTS> whose actual and expected outputs mismatch

Important

  • For all code, follow .claude/skills/coding.rules.md
  • Follow the invariants and conventions for unit tests in .claude/skills/testing.rules.md

Workflow

Run the Failing Tests

  • Run the tests one at a time, the way the user suggested, to confirm they fail due to mismatching outputs

Classify the Assertion Method

  • For each failing test method, check whether the result is checked with

    • self.check_string()
    • self.assert_equal() or a similar assertion (e.g., hunteuti.assert_sys_calls(), self.assertEqual())
  • Print a table with the following info: path of test | fail / pass | check_string or assert_equal

    | Path | Test | Status | Assertion Method |
    |------|------|--------|------------------|
    | dev_scripts_helpers/documentation/test/test_notes_to_pdf.py | Test_notes_to_pdf_latex_options::test1 | FAIL | self.assert_equal() with fuzzy_match=True |
    | dev_scripts_helpers/documentation/test/test_notes_to_pdf.py | Test_notes_to_pdf_latex_options::test2 | FAIL | self.assert_equal() with fuzzy_match=True |
    | dev_scripts_helpers/documentation/test/test_notes_to_pdf.py | Test_notes_to_pdf_output_types::test1 | FAIL | self.assert_equal() with fuzzy_match=True |
    
  • Save this to tmp.update_test_outcomes.failing_tests.md

Update the Expected Output

  • For each failing test method:

    • If the result is checked with self.check_string()
      • Run the test with the pytest option --update_golden to refresh the output
    • If the result is checked with self.assert_equal() or a similar assertion (e.g., hunteuti.assert_sys_calls()), read the file <TEST_DIR>/tmp.initial.actual.txt (e.g., dev_scripts_helpers/documentation/test/outcomes/Test_run_pandoc_to_html.test1) that holds the actual variable, and update the expected variable
      • E.g.,
        expected = """
        [{'args': (f'jupytext --to ipynb --update {py_file}',),
          'function': 'hsystem.system',
          'kwargs': {}},
         {'args': (f'jupytext --sync {py_file}',),
          'function': 'hsystem.system',
          'kwargs': {}}]
        """
        
  • Follow .claude/skills/testing.rules.md:

    • ## Code Formatting in Tests
    • # Test Input and Output Handling
  • Do not change how results are checked

    • E.g., do not replace self.assert*() with self.check_string() or vice versa
    • E.g., do not replace self.assert*() with self.assertIn()

Verification

  • Run all the tests to make sure they are passing
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 319kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 319kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 319kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 196kUpdated Claude CodeCodexVerifiedInstall