Access issues to some services

Uptime Impact: 5 hours, 16 minutes, and 34 seconds
Resolved
Resolved

We've now resolved the incident. Thanks for your patience.

What happened? As part of our commitment to keeping your data in secure Canadian facilities, we use specialized Canadian datacenters that meet our strict privacy standards. One of these datacenters was affected by a distributed denial of service attack which greatly slowed your access to PayEvo services.

While PayEvo services were slowed, your account, data and information were not compromised.

The sophisticated attack was highly distributed and on many orders of magnitude greater than what has been seen in most Canadian facilities. Our team worked to mitigate the impact, but the volume of traffic still meant that many clients could not access their accounts today.

We apologize and want you to know we are committed to ensuring your beloved PayEvo service is available when you need it. For anyone affected by the processing delays, our team will work with you to ensure your staff are still paid on time. Please get in touch with support@paymentevolution.com and we'll make it right.

We have already taken steps to mitigate against further attacks. Some of this is already visible (notice the new PayEvo website?) but much of this is the redundant infrastructure behind the scenes.

We are humbled by the offers of assistance, kind messages to our engineering teams and above all, the patience you have shown as we have worked to bring the affected services back. Thank you!

Updated

As we bring identity subsystems back online you may see automated reports that we are back up. We will provide a written confirmation once we have confirmed there are no lingering issues. Thank you again for your patience.

Updated

Thank you for your continued patience - we are working to restore access to the identity management subsystem for those who need access to credential-backed services. Next update 7pmET

Identified

All services with the exception of identity / authentication services are functioning. This means some clients will have access issues for services that require login or credentials. We continue to work on a fix. Next update 5pmET

Updated

We continue to work on fixing the access issues some clients are still having in connecting to a subset of PayEvo services. While many of the the affected services are now accessible, some core services are still having performance issues. Next update 4pmET.

Updated

Services are being restored. Access to payroll, benefits management and B2B payments will be intermittent. Next update 3pmET.

Updated

Networking issues are still affecting some services. Access may be intermittent for some services. Work continues on the resolution. Next update 12:30pmET

Updated

Work continues to remediate the underlying networking issue at one of our datacenters. Some services still have degraded performance and access may be intermittent. Next update NoonET.

Recovering

The affected datacenter has identified the issue and is working to resolve the performance degradation. Some PayEvo services will be intermittently available. Next update 11:30ET

Identified

One of our datacenters is experiencing issues - access to PayEvo services may be limited. We are working to resolve the issue and will post updates on our progress.

Investigating

Our monitoring systems say we're experiencing degraded performance. The team is taking a closer look.

Began at:

Affected components
  • Core Services
    • Identity Services