test(unit_test): check for broken links #22

2022-01-27T08:57:17Z

jon_nfc commented

2022-01-27 08:57:17 +00:00

(Migrated from gitlab.com)

📝 Notes

Ref: see https://stackoverflow.com/a/50533154

break this code up so that the fetching of hyperlinks is part of the data collection in conftest.py.

The unit test should just use the data.

def test_dead_link_check(self):

  good_http_status = [200, 302]

  for file_check in html_pages:

    with self.subTest(f"{check_file.replace('./build/','')}"):

      check_url = 'file://' + suffux_path + check_file.replace('./build','')

      self.driver.get(check_url)

      links =  driver.find_elements_by_css_selector("a")

        for link in links:

          with self.subTest(f"{link.get_attribute('href')}"):
            r = r equests.head(link.get_attribute('href'))

            self.assertTrue(r.status_code in good_http_status, 
                                      msg=f"page {} has a link to {link.get_attribute('href')} that returned http {r.status_code}")


          # print(link.get_attribute('href'), r.status_code)

✅ test page load all resources
- ✅ external page load resources - approved
all internal links - valid and work

test data structure from note in !15

data  
  |--page_load_resource_links  
        |--{hash, protocol, domain, path, http_status, source_files[]}  
  |--source_files[]  
  |--hyperlinks
       |--{id, protocol, domain, path, http_status, source_file line_number}

found a working one ~~load external request links: [link for key in data.page_load_resource_links for link in key['links']] if link['protocol'][0:4] == "http"~~

~~load all links [link for key in data.page_load_resource_links for link in key['links']]~~

🔗 References

Related #3
Closes #7
child of nofusscomputing/ops#41

👷 Tasks

Dead Link Check internal and external (Task from #7)

all links from page must be checked to ensure they work
#7 closes in commit message
add a deploy stage after build stage and before test stage where a gitlab pages job runs to deploy the site.
- only deploys when not on master branch
- needs added to deploy job and test job so they are chained
update sitemap template to use the git updated date if module installed.

### :memo: Notes - Ref: see https://stackoverflow.com/a/50533154 break this code up so that the fetching of hyperlinks is part of the data collection in `conftest.py`. The unit test should just use the data. ```python def test_dead_link_check(self): good_http_status = [200, 302] for file_check in html_pages: with self.subTest(f"{check_file.replace('./build/','')}"): check_url = 'file://' + suffux_path + check_file.replace('./build','') self.driver.get(check_url) links = driver.find_elements_by_css_selector("a") for link in links: with self.subTest(f"{link.get_attribute('href')}"): r = r equests.head(link.get_attribute('href')) self.assertTrue(r.status_code in good_http_status, msg=f"page {} has a link to {link.get_attribute('href')} that returned http {r.status_code}") # print(link.get_attribute('href'), r.status_code) ``` - :white_check_mark: test page load all resources - :white_check_mark: external page load resources - approved - all internal links - valid and work test data structure from note in !15 ``` data |--page_load_resource_links |--{hash, protocol, domain, path, http_status, source_files[]} |--source_files[] |--hyperlinks |--{id, protocol, domain, path, http_status, source_file line_number} ``` found a working one ~~load external request links: `[link for key in data.page_load_resource_links for link in key['links']] if link['protocol'][0:4] == "http"`~~ ~~load all links `[link for key in data.page_load_resource_links for link in key['links']]`~~ ### :link: References - Related #3 - Closes #7 - child of nofusscomputing/ops#41 ### :construction_worker: Tasks - [ ] Dead Link Check internal and external _(Task from #7)_ all links from page must be checked to ensure they work - [x] #7 closes in commit message - [x] add a `deploy` stage after `build` stage and before `test` stage where a gitlab `pages` job runs to deploy the site. - [x] only deploys when not on `master` branch - [x] `needs` added to `deploy` job and `test` job so they are chained - [x] update sitemap [template](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/templates/sitemap.xml) to use the git updated date if module installed.

jon_nfc (Migrated from gitlab.com) approved these changes 2022-01-27 08:57:17 +00:00

jon_nfc commented

2022-01-27 08:57:18 +00:00

(Migrated from gitlab.com)

requested review from @jon_nfc

jon_nfc commented

2022-01-27 08:57:18 +00:00

(Migrated from gitlab.com)

assigned to @jon_nfc

jon_nfc (Migrated from gitlab.com) changed target branch from master to development

2022-01-27 08:57:47 +00:00

jon_nfc commented

2022-01-27 09:17:30 +00:00

(Migrated from gitlab.com)

changed the description

jon_nfc commented

2022-01-27 09:18:39 +00:00

(Migrated from gitlab.com)

changed the description

jon_nfc commented

2022-01-27 09:26:15 +00:00

(Migrated from gitlab.com)

changed the description

jon_nfc commented

2022-01-27 09:54:41 +00:00

(Migrated from gitlab.com)

changed the description

jon_nfc commented

2022-01-28 02:59:19 +00:00

(Migrated from gitlab.com)

added 10 commits

76c51109 - test(data): created a function to parse url to dict
d13cc5bd - test(data): add fetching of all hyperlinks
ebc17c12 - test(unit_test_data): chrome driver should not be a class object.
8fd0e8ae - test(unit_test_data): include the url in url dict
26af6031 - test(unit_test_data): web driver to be class obj
23dc5894 - test(privacy): clear the class data
b9f56c21 - test(privacy): print the test data to stdout
276d1219 - test(dead_link_check): test for dead hyperlinks
fb7555d6 - ci(unit_test): capture stdout in JUnit Report
618feea0 - ci(unit_test): include requests in test pipfile

Compare with previous version

added 10 commits <ul><li>76c51109 - test(data): created a function to parse url to dict</li><li>d13cc5bd - test(data): add fetching of all hyperlinks</li><li>ebc17c12 - test(unit_test_data): chrome driver should not be a class object.</li><li>8fd0e8ae - test(unit_test_data): include the url in url dict</li><li>26af6031 - test(unit_test_data): web driver to be class obj</li><li>23dc5894 - test(privacy): clear the class data</li><li>b9f56c21 - test(privacy): print the test data to stdout</li><li>276d1219 - test(dead_link_check): test for dead hyperlinks</li><li>fb7555d6 - ci(unit_test): capture stdout in JUnit Report</li><li>618feea0 - ci(unit_test): include requests in test pipfile</li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320204411&start_sha=94c2762571adc42259b70f863f30da25f1c7a8d3)

jon_nfc commented

2022-01-28 03:00:25 +00:00

(Migrated from gitlab.com)

added 1 commit

0f802b4b - ci(unit_test): include requests in test pipfile

Compare with previous version

added 1 commit <ul><li>0f802b4b - ci(unit_test): include requests in test pipfile</li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320204762&start_sha=618feea0f77455267a81fd489e6072b23b5df65c)

jon_nfc commented

2022-01-28 03:03:35 +00:00

(Migrated from gitlab.com)

marked the checklist item #7 closes in commit message as completed

marked the checklist item **#7 closes in commit message** as completed

jon_nfc commented

2022-01-28 03:19:19 +00:00

(Migrated from gitlab.com)

added 1 commit

2fde2fb5 - test(ssl_hyperlinks_only): Test added

Compare with previous version

added 1 commit <ul><li>2fde2fb5 - test(ssl_hyperlinks_only): Test added</li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320211508&start_sha=0f802b4b936699bcdd47b35202fc64e508154ce8)

jon_nfc commented

2022-01-28 03:38:58 +00:00

(Migrated from gitlab.com)

added 2 commits

7fcc3caf - ci(stages): deploy job to occur after build
960f2630 - ci(tests): rijigged the job order

Compare with previous version

added 2 commits <ul><li>7fcc3caf - ci(stages): deploy job to occur after build</li><li>960f2630 - ci(tests): rijigged the job order</li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320217085&start_sha=2fde2fb5cec6de1c166410868aeec8e1157e1c7a)

jon_nfc commented

2022-01-28 03:44:22 +00:00

(Migrated from gitlab.com)

added 1 commit

de81c19b - ci(static_pages): dont run build if MD lint fails

Compare with previous version

added 1 commit <ul><li>de81c19b - ci(static_pages): dont run build if MD lint fails</li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320218809&start_sha=960f26306d601d787a7eb39f4b5c5cc254eabf44)

jon_nfc commented

2022-01-28 03:49:07 +00:00

(Migrated from gitlab.com)

added 1 commit

c634b122 - ci: ensure placeholder test runs and passes

Compare with previous version

added 1 commit <ul><li>c634b122 - ci: ensure placeholder test runs and passes</li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320219910&start_sha=de81c19b7a9ef94ebec112fe69ea6c515eecd609)

jon_nfc commented

2022-01-28 04:02:26 +00:00

(Migrated from gitlab.com)

added 1 commit

fb940138 - ci: command only required for unit test

Compare with previous version

added 1 commit <ul><li>fb940138 - ci: command only required for unit test</li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320223528&start_sha=c634b122591373214627410208c7b808effebc98)

jon_nfc commented

2022-01-28 04:05:01 +00:00

(Migrated from gitlab.com)

marked the checklist item add a deploy stage after build stage and before test stage where a gitlab pages job runs to deploy the site. as completed

marked the checklist item **add a `deploy` stage after `build` stage and before `test` stage where a gitlab `pages` job runs to deploy the site.** as completed

jon_nfc commented

2022-01-28 04:05:03 +00:00

(Migrated from gitlab.com)

marked the checklist item only deploys when not on master branch as completed

marked the checklist item **only deploys when not on `master` branch** as completed

jon_nfc commented

2022-01-28 04:05:07 +00:00

(Migrated from gitlab.com)

marked the checklist item needs added to deploy job and test job so they are chained as completed

marked the checklist item **`needs` added to `deploy` job and `test` job so they are chained** as completed

jon_nfc commented

2022-01-28 05:22:22 +00:00

(Migrated from gitlab.com)

added 3 commits

e6d6cc99 - build(requirements): added wheel for module build
0c8318da - feat(vscode): added build task
568c560d - feat(vscode): added pytest settings

Compare with previous version

added 3 commits <ul><li>e6d6cc99 - build(requirements): added wheel for module build</li><li>0c8318da - feat(vscode): added build task</li><li>568c560d - feat(vscode): added pytest settings</li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320248515&start_sha=fb940138cbb81970f251453cb2c0150931b8645c)

jon_nfc commented

2022-01-28 05:36:01 +00:00

(Migrated from gitlab.com)

added 1 commit

538aeb42 - fix(vscode): use correct params for tests

Compare with previous version

added 1 commit <ul><li>538aeb42 - fix(vscode): use correct params for tests</li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320252225&start_sha=568c560df8d178fab69556367a46345705d0012a)

jon_nfc commented

2022-01-28 05:57:29 +00:00

(Migrated from gitlab.com)

added 1 commit

aa91e357 - feat(sitemap): git_revision_date for page change

Compare with previous version

added 1 commit <ul><li>aa91e357 - feat(sitemap): git_revision_date for page change</li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320258085&start_sha=538aeb4299545cdb612f4eb6add9b979cbf7d2d3)

jon_nfc commented

2022-01-28 06:02:29 +00:00

(Migrated from gitlab.com)

marked the checklist item update sitemap template to use the git updated date if module installed. as completed

marked the checklist item **update sitemap [template](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/templates/sitemap.xml) to use the git updated date if module installed.** as completed

jon_nfc commented

2022-01-28 06:05:24 +00:00

(Migrated from gitlab.com)

added 1 commit

edee74f9 - test: named test to refelect its function.

Compare with previous version

added 1 commit <ul><li>edee74f9 - test: named test to refelect its function.</li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320260537&start_sha=aa91e35765ea2d89ae88ced288581623653d2c3c)

jon_nfc commented

2022-01-28 06:15:20 +00:00

(Migrated from gitlab.com)

      - "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/$CI_JOB_NAME/*"
    reports:
      junit:
        - "*.junit.xml"

```suggestion:-0+0 - "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/$CI_JOB_NAME/*" reports: junit: - "*.junit.xml" ```

jon_nfc commented

2022-01-28 06:15:58 +00:00

(Migrated from gitlab.com)

resolved all threads

jon_nfc commented

2022-01-28 06:15:59 +00:00

(Migrated from gitlab.com)

added 1 commit

9f8ae06d - ci(pytest): fix to collect junit report

Compare with previous version

added 1 commit <ul><li>9f8ae06d - ci(pytest): fix to collect junit report </li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320264377&start_sha=edee74f9c2fac186d10390b91b2f773abf65d9e9)

jon_nfc commented

2022-01-28 06:25:50 +00:00

(Migrated from gitlab.com)

    - cp *.junit.xml "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/$CI_JOB_NAME/"

```suggestion:-0+0 - cp *.junit.xml "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/$CI_JOB_NAME/" ```

jon_nfc commented

2022-01-28 06:26:24 +00:00

(Migrated from gitlab.com)

resolved all threads

jon_nfc commented

2022-01-28 06:26:25 +00:00

(Migrated from gitlab.com)

changed this line in version 14 of the diff

changed this line in [version 14 of the diff](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320268297&start_sha=9f8ae06d3efd78b2b953773f8bc9e868962aa3e0#587d266bb27a4dc3022bbed44dfa19849df3044c_83_83)

jon_nfc commented

2022-01-28 06:26:25 +00:00

(Migrated from gitlab.com)

added 1 commit

4e2def5b - ci(unit_test): copy report to right location

Compare with previous version

added 1 commit <ul><li>4e2def5b - ci(unit_test): copy report to right location</li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320268297&start_sha=9f8ae06d3efd78b2b953773f8bc9e868962aa3e0)

jon_nfc commented

2022-01-28 06:37:15 +00:00

(Migrated from gitlab.com)

#Integration Tests:
#  extends: .Pytest_template
#  needs: 
#    - pages
#    - 'Unit Tests'

```suggestion:-4+0 #Integration Tests: # extends: .Pytest_template # needs: # - pages # - 'Unit Tests' ```

jon_nfc commented

2022-01-28 06:37:15 +00:00

(Migrated from gitlab.com)

#   - echo "placeholder job for integration tests" > "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/$CI_JOB_NAME/DETEMEME.txt"

```suggestion:-0+0 # - echo "placeholder job for integration tests" > "$CI_PROJECT_DIR/artifacts/$CI_JOB_STAGE/$CI_JOB_NAME/DETEMEME.txt" ```

jon_nfc commented

2022-01-28 06:38:07 +00:00

(Migrated from gitlab.com)

changed this line in version 15 of the diff

changed this line in [version 15 of the diff](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320272947&start_sha=4e2def5b717f41c7003d613d4048754c8e957ff6#587d266bb27a4dc3022bbed44dfa19849df3044c_91_87)

jon_nfc commented

2022-01-28 06:38:07 +00:00

(Migrated from gitlab.com)

changed this line in version 15 of the diff

changed this line in [version 15 of the diff](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320272947&start_sha=4e2def5b717f41c7003d613d4048754c8e957ff6#587d266bb27a4dc3022bbed44dfa19849df3044c_93_93)

jon_nfc commented

2022-01-28 06:38:07 +00:00

(Migrated from gitlab.com)

resolved all threads

jon_nfc commented

2022-01-28 06:38:08 +00:00

(Migrated from gitlab.com)

added 1 commit

998fc911 - ci(integration_test): disable job until developed

Compare with previous version

added 1 commit <ul><li>998fc911 - ci(integration_test): disable job until developed</li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320272947&start_sha=4e2def5b717f41c7003d613d4048754c8e957ff6)

jon_nfc commented

2022-01-28 06:43:22 +00:00

(Migrated from gitlab.com)

#  script:

```suggestion:-0+0 # script: ```

jon_nfc commented

2022-01-28 06:43:47 +00:00

(Migrated from gitlab.com)

resolved all threads

jon_nfc commented

2022-01-28 06:43:48 +00:00

(Migrated from gitlab.com)

changed this line in version 16 of the diff

changed this line in [version 16 of the diff](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320275185&start_sha=998fc911b3c4222f28eab53ee48cc1ed5e6e1517#587d266bb27a4dc3022bbed44dfa19849df3044c_92_92)

jon_nfc commented

2022-01-28 06:43:48 +00:00

(Migrated from gitlab.com)

added 1 commit

a436fa5c - ci: amend to last commit

Compare with previous version

added 1 commit <ul><li>a436fa5c - ci: amend to last commit</li></ul> [Compare with previous version](/nofusscomputing/infrastructure/website/-/merge_requests/4/diffs?diff_id=320275185&start_sha=998fc911b3c4222f28eab53ee48cc1ed5e6e1517)

jon_nfc commented

2022-01-28 06:45:45 +00:00

(Migrated from gitlab.com)

added 7h 15m of time spent

jon_nfc commented

2022-01-28 06:50:01 +00:00

(Migrated from gitlab.com)

task Dead Link Check internal and external (Task from #7) internal check done.

✅ external links done.

internal check will still have to be done.

idea

all hyperlinks that use protocol file, can be checked against the source files. reconstructing the file path from the url and checking if in the data[source_files]

task `Dead Link Check internal and external (Task from #7)` internal check done. :white_check_mark: external links done. internal check will still have to be done. - idea all hyperlinks that use protocol file, can be checked against the source files. reconstructing the file path from the url and checking if in the `data[source_files]`

jon_nfc commented

2022-01-28 06:50:19 +00:00

(Migrated from gitlab.com)

approved this merge request

jon_nfc commented

2022-01-28 06:51:02 +00:00

(Migrated from gitlab.com)

mentioned in commit 06372cf8f1

mentioned in commit 06372cf8f1f72036d6e830eb82ed66b5a9edb100

jon_nfc (Migrated from gitlab.com) merged commit 06372cf8f1 into development

2022-01-28 06:51:03 +00:00

jon_nfc commented

2022-01-28 06:53:15 +00:00

(Migrated from gitlab.com)

mentioned in issue #7

jon_nfc commented

2022-01-29 09:59:23 +00:00

(Migrated from gitlab.com)

mentioned in merge request !3

jon_nfc commented

2022-08-28 08:17:18 +00:00

(Migrated from gitlab.com)

changed time estimate to 7h 20m

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: nofusscomputing/website#22