Showing posts with label issue-severe. Show all posts
Showing posts with label issue-severe. Show all posts

Friday, September 13, 2024

*Updated* - Partial Internet Service Outage beginning 9/13/2024 @ 9:10pm

WHAT: Partial Internet Service Outage

WHEN: 9/13/2024 Duration: beginning at 9:10pm until resolved by ISP

WHERE: Main Campus

Current Status: Resolved

Description: 
*Update #2*
The part finally arrived and was installed mid-afternoon on Wednesday, September 18th.  The connection has remained stable for 24 hours, so this issue appears to be resolved.

*Update #1*
Our ISP identified a failed component during onsite troubleshooting around 3am Saturday morning.  Unfortunately, replacement parts are not available nearby and will need to be flown in from the east coast.  Current ETA is arrival no earlier that sometime Monday, September 16th with scheduling of installation to follow after confirmation of receipt.  At this time our other ISP continues to provide all of our Internet service to campus.

*Original Post*
At approximately 9:10pm our primary Internet Service Provider (ISP) began experiencing a service outage that affected the campus network access to the Internet and access from off-campus to on-campus resources.  We have failed our ISP service over to entirely rely on our secondary ISP which should sustain service to campus until the faulty ISP restores their service.  At this time we do not have information from our ISP as to the cause of the issue or the time to repair, but they are aware of the issue and have dispatched technicians to investigate and resolve the cause.

Service(s) Impacted: Campus Network

Workarounds: We have failed our ISP service over to entirely rely on our secondary ISP which should sustain service to campus until the faulty ISP restores their service.

Next Update: Upon receipt of updated information from our ISP

Root Cause: Pending component replacement, the service outage appears to be the result of a transceiver component failure.

Check https://www.plu.edu/status/ for periodic updates.

Tuesday, July 25, 2023

**RESOLVED** Banner Service Outage

WHAT: Banner Service Outage

WHEN: 7/25/23 at 2:15PM

Current Status: Resolved

Description: We are investigating an unplanned outage of the Banner Student Information System. We will provide more information on the status when available.

***Update*** 7/25/23 at 3:30pm:  The issue with accessing Banner has been resolved. Please report any ongoing issues by calling the I&TS Help Desk at 253-535-7525 or by entering in a ticket at helpdesk.plu.edu. 

Service(s) Impacted: Banner Self-Service, Banner Administrative Pages


Check https://www.plu.edu/status/ for periodic updates.


Thursday, May 26, 2022

*RESOLVED* Unplanned Campus Telephone Service Disruption beginning Thursday, May 25th - 10:30pm

WHAT: Unplanned Campus Telephone Service Disruption

WHEN: 05/25/2022  Duration: Beginning around 10:30pm ending approximately 8:15am

WHERE: Outbound calling from Campus telephones

Current Status: Resolved

Description: 
Around 8:15am this morning fiber repairs by the carrier brought PLU's services back online.  The carrier confirmed completion of the repairs for all customers a couple of hours later and PLU's services have been stable throughout the morning.  At this time no ongoing impact from this outage is anticipated.

*Original Post*
Around 10:30pm one of our primary telecommunications providers began experiencing some type of service failure which is affecting services to the PLU campus.  Telephone service out of campus is offline due to this outage.  Inbound calling to campus telephones continues to function using a backup service path.  Also, at this time, our Internet connection is being handled by a single ISP as our secondary service is offline.

I&TS has been in contact with our telecommunications carrier and they have technicians actively working to isolate and repair the issue, which is suspected to be a fiber-optic cable cut somewhere between Tacoma and Tukwila.  Updates to this status page will be provided when/if we get new information from the carrier.

Service(s) Impacted: Campus Telephone Services, Campus Network

Workarounds: Cell phones, Zoom, & other non-landline telephone based services

Estimated Time to Resolution or Next Update: Unknown, update provided as information is available.

Root Cause: Telecommunications carrier owned fiber cable damaged by impact from a projectile.

Check https://www.plu.edu/status/ for periodic updates.

Monday, October 18, 2021

**8:15am 10/19/2021 Update** Unplanned Campus Telephone Service Disruption beginning Monday, October 18th - 10:00am

WHAT: Unplanned Campus Telephone Service Disruption

WHEN: 10/18/2021  Duration: Beginning around 10:00am

WHERE: In/Out from Campus

Current Status: Resolved

Description: 
**8:15am 10/19 Update**
Telephone service to campus appears to be restored and functioning properly, but we have not received official word from our provider.  Our secondary ISP connection has yet to be tested, but will likely be brought back online tonight in order to minimize further disruption.  The repair team continues to work on the remainder of the damaged cable with at least two-thirds of the cable remaining to repair.  During the repair process it is possible, though unlikely, for additional issues to be identified.  Also, the temporary cable is apparently in an active construction zone, so until a permanent repair is completed we anticipate some continued risk of further disruption.

**7:30am 10/19 Update**
The telecommunications carrier has had a team of (4) fiber splicers working on splicing the fiber for a temporary replacement cable since approximately 1:00am this morning.  Unfortunately, even after 6 hours of splicing there is still no official ETR for this issue.  I&TS continues to actively monitor the progress and will provide updates as more information is available.

**11:30pm Update**
Our telecommunications carrier provider is still actively working to dig and gain access to the damaged cable and hopes to fully assess the damage soon.  The fiber in the damaged location is a high count fiber cable which means if the repair requires replacement of a section of the cable, they will have to splice both ends of the new cable.  This is likely to be a time consuming process.  Based on this updated information there is no ETR currently available.

**9:30pm Update**
Updated information from our telecommunications providers indicates the outage is due to fiber damaged by construction work in the Tacoma area.  Current Estimated Time to Repair (ETR) is approximately midnight, but this is not a firm time commitment as additional issues may be identified during the course of replacing the damaged fiber.

**Original Post**
Around 10:00am this morning one of our primary telecommunications providers began experiencing some type of service failure which is affecting services to the PLU campus.  Telephone service in/out of campus is offline due to this outage.  Also, at this time, our Internet connection is being handled by a single ISP as our secondary service is offline.

Service(s) Impacted: Campus Telephone Services, Campus Network

Workarounds: Cell phones, Zoom, & other non-landline telephone based services

Estimated Time to Resolution or Next Update: Unknown, update provided as information is available.

Root Cause / Continuous Improvement: To be determined.

Check https://www.plu.edu/status/ for periodic updates.

Friday, August 28, 2020

*Resolved* Campus Wireless Network Disruption (3 buildings - Hauge, Xavier, & Morken) -- Friday, August 28, 2020 1:00PM

WHAT: Campus Wireless Network Disruption (3 buildings)

WHEN: 08/28/2020  Duration: Beginning approx. 1:00 PM

WHERE: Hauge, Xavier, & Morken

Current Status: Resolved

Description: A faulty firmware upgrade process from our network vendor caused the newly deployed network equipment in 3 building (Hauge, Xavier, & Morken) to go offline around 1:00PM this afternoon.  After reaching the vendor's tech support and identifying the problem and resolution I&TS began the process of restoring service around 2:45PM.  Unfortunately the restoration process requires manual configuration changes on each wireless access point in the new network which slows the resolution process, but full restoration is anticipated before 5:00 PM.

Service(s) Impacted: Campus Wireless Network

Workarounds: Wired (ethernet) network is functioning

Estimated Time to Resolution or Next Update: Full resolution anticipated by 5:00 PM.  Resolution will be completed first for Hauge, then Xavier, then Morken.

Root Cause / Continuous Improvement: Faulty firmware upgrade process from network vendor.

Check https://www.plu.edu/status/ for periodic updates.

Thursday, January 9, 2020

RESOLVED: Banner Administrative Pages Unavailable

WHAT: Banner Administrative Pages Unavailable

WHEN: Thursday, January 9, 2020

WHERE: On Campus

Current Status: Open

Description: UPDATE:  Services to Banner Administrative Pages has been restored.  Please notify the Help Desk should you continue to experience issues.

Original Message:  Beginning at 8:40 am on Thursday, January 9th, I&TS began receiving calls that Banner Administrative Pages is unavailable.  We are currently researching the issue and hope to have it resolved soon.  Please check this post for updates.


Service(s) Impacted: Banner Administrative Pages




Check https://www.plu.edu/status/ for periodic updates.

Thursday, November 14, 2019

RESOLVED: Banner Self-Service Currently Unavailable

WHAT: Banner Self-Service

WHEN: Thursday, November 14th, 1:00pm

WHERE: On & Off Campus

Current Status: Resolved

Description: UPDATE:  Access to Banner Self-Service has been restored.  The root cause of this issue has not been determined however I&TS staff will continue to monitor and research the situation.

At approximately 1:00 pm today, Thursday, November 14th, I&TS began to receive notification that Banner Self-Service is unavailable.  Members of the Information Systems team are currently working to identify the issue and hope to have a resolution soon.  This message will be updated as soon as more information is available or when the issue has been resolved.

Service(s) Impacted: Banner Self-Service

Workarounds: None




Check https://www.plu.edu/status/ for periodic updates.



Wednesday, April 10, 2019

*Resolved* Softdocs Etrieve Service Disruption - Tuesday, April 9, 4:40pm

WHAT: Softdocs Etrieve Service Disruption

WHEN: 4/9/2016 Duration: Beginning about 4:40pm

Current Status: Resolved

Description:
*Update*
Etrieve service was restored this morning around 8:00am after identifying the missing configuration element.

*Original Post*
Late this afternoon, the Etrieve service went offline.  I&TS staff has been investigating the cause of the outage, but has yet to identify a cause or determine a path to service restoration.  We will continue to troubleshoot and work with our support resources to restore service as soon as possible.

Service(s) Impacted: Softdocs Etrieve

Workarounds: None at this time

Next Update: Wednesday, April 10, 9:00am

Root Cause / Continuous Improvement: To be determined.

Check https://www.plu.edu/status/ for periodic updates.

Friday, February 15, 2019

Sakai Turnitin Unplanned Service Disruption

*** UPDATE *** Feb. 15, 4:50pm - Turnitin is still experiencing disruption. For periodic updates, please visit the Turnitin Status page.

WHAT:  Sakai Turnitin Unplanned Service Disruption

WHEN:  02/15/19 8:45am

Description:  We are currently experiencing an unexpected disruption with the Sakai Turnitin Service. Turnitin support staff are currently working to resolve this issue.

Service(s) Impacted:  Sakai Turnitin

Workarounds:  None

Estimated Time to Resolution or Next Update:  February 15, 2019 4:00pm

Check https://www.plu.edu/status/ for periodic updates.

Sunday, May 27, 2018

Unplanned ePass Authentication Disruption - Updated 7:30pm Tuesday, May 29th

WHAT: Unplanned ePass Authentication Disruption

WHEN: 5/28/2018-current Duration: intermittent

Current Status: Closed

Description:
*7:30pm, Tuesday, May 29th Update*
Unfortunately, what appeared to be a one-time failure has now occurred 3 times since Saturday (twice today).  I&TS is investigating further, but has yet to determine a root cause.  Restoration of service is a fairly rapid process, but requires manual intervention by an I&TS System Administrator after we are notified of an issue.  We apologize for the inconvenience.

*5:00pm, Saturday, May 27th Update*
After a restart of a key service login services appear to be restored.  I&TS will continue to monitor and investigate the root cause of this issue which is still undetermined at this time.

*Original Post*
Around 4:30pm Sunday afternoon I&TS was notified that a number of users were having trouble logging into various services using their ePass credentials.  With the error message "Login Failure: Pool is empty and connection creation failed" appearing for most users.  I&TS staff is working actively to resolve this issue promptly.

Service(s) Impacted: ePass login services impacting Sakai, Banner, Gmail, Google Apps, Campus Calendar, Web Help Desk, Campus Network, etc.

Workarounds: None at this time.

Estimated Time to Resolution or Next Update: Next Update 10:00am Wednesday, May 30th

Root Cause / Continuous Improvement: To be determined.

Check https://www.plu.edu/status/ for periodic updates.

Wednesday, August 30, 2017

Inbound Campus Telephone Service Disruption - Wednesday, August 30, 2017

WHAT: Inbound Campus Telephone Service Disruption - Wednesday, August 30, 2017

WHEN: 8/30/2017 Duration: beginning approximately 12:00pm

WHERE: PLU Campus

Current Status: Open

Description: Our primary telephone carrier is experiencing an equipment failure that is interfering with their ability to complete various phone calls.  This is primarily manifesting itself by calls coming to campus telephones not completing.  The caller may hear dead air, an "all circuits are busy" message, or a busy signal depending on where they are calling from.

Service(s) Impacted: Campus telephone System

Workarounds: E-mail, or an outbound call from campus to the caller should both work normally.

Estimated Time to Resolution or Next Update: As information becomes available from our carrier.

Root Cause: Carrier equipment failure.

Check https://www.plu.edu/status/ for periodic updates.

Tuesday, June 6, 2017

RESOLVED: Banner Service Disruption - Tuesday, June 6th 2017

WHAT: Banner

WHEN: Beginning 6/5/17, 11:45 pm

Duration: Unknown

WHERE: On & Off campus

Current Status: Resolved

Description:  UPDATE:  As of 6;45 am the issue with the server has been resolved and all Banner Services (INB and Self-Serivce) are once again available.  

Original Post:
We are currently experiencing an issue with the primary database server for Banner.  As a result, services reliant on Banner including Banweb (aka Banner Self-Service), INB, & various reporting systems, are offline or unable to complete some operations.

I&TS is actively working on a resolution and will provide an update as soon as possible.

Service(s) Impacted: All Banner connected services including, Internet Native Banner & Banner Self-Service

Workarounds: None at this time

Estimated Time to Resolution or Next Update: Unknown

Friday, October 21, 2016

*Resolved* Nationwide Internet Service Disruptions - Friday, October 21, 2016

WHAT: Nationwide Internet Service Disruptions due to DDoS attacks

WHEN: 10/21/2016 Duration: Beginning early morning

WHERE: Many websites across the country

Current Status: Closed

Description: A leading DNS (Domain Name Service) provider "Dyn" is under attack from a Distributed Denial of Service (DDoS) attack that has effectively taken their services offline multiple times today.  DNS is the service that works like a phonebook and directs users to the correct server based on the server name typed into a browser.  If DNS does not respond or provide an accurate answer the access to the service is effectively blocked.  Due to Dyn's position as a leading provider of DNS services, their outage is causing wide-spread issues connecting to many other leading web sites including:

  • Ancesty.com
  • BBC
  • Box
  • Business Insider
  • CNN
  • Dailynews.com
  • Disqus
  • Donorschoose.org
  • Etsy
  • Eventbrite
  • Fox News
  • Github
  • Guardian.co.uk
  • HBO Now
  • Iheart.com (iHeartRadio)
  • Imgur
  • Mashable
  • NHL.com
  • Netflix
  • Netflix
  • New York Times
  • Okta
  • PayPal
  • People.com
  • PeopleAdmin
  • Pinterest
  • Playstation Network
  • Qualtrics
  • Recode
  • Reddit
  • Reeher
  • SBNation
  • Shopify
  • Soundcloud
  • Spotify
  • Squarespace Customer Sites
  • Starbucks rewards/gift cards
  • The Verge
  • Time.com
  • Twitter
  • Visa
  • WSJ.com
  • Weather.com
  • Wikia
  • Wired.com
  • Xbox.com
  • Yelp
  • .... and many more.

Service(s) Impacted: Reeher & PeopleAdmin (employment.plu.edu) appears to be the only PLU services currently affected, but many non-PLU services are impacted from both on-campus as well as off-campus.

Workarounds: None available at this time.

Estimated Time to Resolution or Next Update: None available at this time.

Root Cause / Continuous Improvement: DDoS

Check http://www.plu.edu/status/ for periodic updates.

Monday, August 24, 2015

*Resolved* Long Distance Calling Issues - Monday, August 24, 2015 - Update 12:41pm

WHAT: Long Distance Calling Issues

WHEN: 8/24/2015 Duration: Beginning around 7:00am

WHERE: PLU Campus

Current Status: Resolved

Description:
**Update**  Monday, August 24th 12:40pm
Our telephone carrier notified us that this problem should be resolved and our testing confirms this is the case.  No further issues are anticipated and no further action is required.

**Original**
Information & Technology Services began receiving reports of issues making outbound long distance telephone calls first thing this morning.  Upon contacting our primary telephone carrier we were informed they are having issues resulting from a power outage at one of the central offices.  This outage is causing issues with nearly all outbound long distance calls (toll free appears unaffected), though random calling attempts appear to go through.

At this time it appears inbound calls of all types are unaffected by this issue.  Our carrier does not have an ETA for restoration of service, but we will continue to monitor the issue and relay their status updates whenever available.

Service(s) Impacted: Campus Long Distance Telephone Services

Workarounds: Using a cell phone to make calls or asking the other party to call into campus.

Root Cause: Power outage at telephone carrier CO facility.

Check http://www.plu.edu/status/ for periodic updates.

Friday, August 21, 2015

***Update- Tues., August 25th*** community.plu.edu Web Services Offline - Wednesday, August 19th

WHAT: community.plu.edu Web Services Offline

WHEN: Beginning 8/18/15; Update August 25th

Current Status: Closed

Description:
**Update (8/25)**
The I&TS Systems team has nearly completed a migration to a replacement system following the security issue that required taking the previous system offline.  The team is working on the final touches, but access was restored to this system Tuesday afternoon.  If you identify an issue at this time please contact the I&TS Helpdesk (253-535-7525, helpdesk@plu.edu, or https://helpdesk.plu.edu/ )

**Original Post**
Due to a security issue Information & Technology Services found it necessary to take the server responsible for the community.plu.edu web pages offline to protect critical PLU resources.  For websites accessed through community.plu.edu (also known previously as www.plu.edu/~ ), the server providing hosting and web page redirection services will not be available until further notice.  We are working to restore this service as soon as possible.

Service(s) Impacted: community.plu.edu based web sites

Workarounds: For web sites hosted on the Google Sites portion of this service, they can still be access and managed through https://sites.google.com/a/plu.edu  All other sites hosted through the on-campus server (aka allspice.plu.edu), will not be accessible until service restoration is complete.

Estimated Time to Resolution or Next Update: For user facing services on this platform resolution should be complete now.  Final configuration testing & verification is underway and should be complete before the end of the week.  No impact on end-users is anticipated with the remaining work.

Root Cause / Continuous Improvement: Review pending.

Check http://www.plu.edu/status/ for periodic updates.

Wednesday, April 15, 2015

***RESOLVED*** EBSCOHost outage

WHAT: EBSCOHost outage

WHEN: 04/15/15 Duration: unknown


Current Status: Open

Description:

RESOLVED: EBSCOHost appears to be back online. Please contact williagr@plu.edu if issues persist.

The EBSCOHost database platform is experiencing problems at their end. EBSCO is aware of the problem and working on a fix. No ETA at this time.

Service(s) Impacted: Library

Workarounds: Other databases, including ProQuest and JSTOR, are unaffected, as is the library catalog.

Estimated Time to Resolution or Next Update: unknown

Root Cause / Continuous Improvement: To be determined.



Check http://www.plu.edu/status/ for periodic updates.

Thursday, September 19, 2013

***Resolved*** PLU Primary Internet Connection Down - 9/19/13 12:15am

WHAT: PLU Internet Connection Down

WHEN: 9/19/13 12:15am  Duration: approximately 1 hour

Current Status: Closed

Description: Shortly after 1am our connection was restored by our ISP. The root cause is under investigation, but currently underdetermined so a reoccurrence is still possible though not expected.

Original Post:  Around midnight PLU began experiencing connectivity issues through our primary Internet provider.  We are in the process of contacting our ISP now to notify them and, hopefully, determine when service will be restored.  The root cause of the problem is currently undetermined so estimated time to repair will not be available until the cause is identified.

Service(s) Impacted: Internet connectivity

Next Update:  As needed

Root Cause / Continuous Improvement:  A device issue experienced by our ISP resulted in a configuration problem.  The end result was connectivity issues for PLU, which have since been resolved.  We do not anticipate a recurrence of this issue at this time.

Check http://www.plu.edu/status/ for periodic updates.

Saturday, April 13, 2013

***Resolved*** No Access to Banner INB and Banner Web

WHAT: Banner Offline

WHEN: 04/13/2013

Current Status: Resolved

Description: Update at 9:20am:  Access to Banner has been restored.

Original Message: We received word that Banner (INB & BanWeb) is offline this morning following the normal downtime for backups on Friday night/Saturday morning.  We are actively working to restore service and hope to have Banner online soon.  No official estimated time to repair is available yet.

Service(s) Impacted: Banner

Next Update: 10:30am or sooner if resolved

Root Cause / Continuous Improvement:  Investigation into the root cause will be conducted shortly.

Check http://www.plu.edu/status/ for periodic updates.

Wednesday, January 23, 2013

***Resolved*** Some Web Applications Offline 6:00-8:00 AM on January 23, 2013

WHAT: Some Web applications will be offline from 6:00-8:00 AM on January 23, 2013

WHEN: 01/23/2013 6:00 AM Duration: 2 hours

Current Status: Closed

Description: On January 23, 2013, from 6:00-8:00 AM, the following Web-based applications will be offline while maintenance is performed:

Service(s) Impacted: Online Search & Directory, Sakai Self-Service and Course Request forms, Vehicle Registration, PLU Driver's Certification



Update - 8:00 AM: Maintenance has been completed, and the above services have been restored.


Check http://www.plu.edu/status/ for periodic updates.

Wednesday, November 28, 2012

Testing Labels

Magna scelerisque aliquam porta facilisis, tempor placerat velit scelerisque, placerat, porta pulvinar auctor tincidunt facilisis! Massa, ac, adipiscing elementum? Rhoncus a mattis urna rhoncus? Rhoncus urna! Augue! Dolor phasellus magna velit elementum scelerisque, rhoncus nunc cum duis aenean pellentesque. Ac augue. Et elit nisi habitasse nisi mid, et adipiscing nec scelerisque nascetur magna, dignissim vut eros ac nec purus! Lacus lacus, quis turpis nunc porta? Turpis in augue, eros amet ut? Porttitor vel parturient? 

Est. Cum cras massa natoque, mattis augue, lectus odio aliquet eros adipiscing porta adipiscing porttitor! Adipiscing auctor facilisis sit lorem tincidunt! Proin elementum pellentesque vel, tincidunt. Eu nec a pulvinar, in eros urna odio mus, nec et, pulvinar odio, quis platea ac integer, auctor mus.