In light of the escalating COVID‐19 pandemic, and the overwhelming recommendations from the CDC, the state of Arizona, and the Federal Government, our company will be taking action to ensure the safety and well-being of its employees.
While our employees are not part of the population that would be most adversely affected by contracting the virus, many other people are, and stopping the spread is the best way to protect the community at large. Arizona is not one of the hardest hit regions of the country; however, we all want to do our part to help ensure the spread of the virus slows as much as possible. Our hearts go out to everyone affected by the coronavirus COVID‐19 pandemic, and we want to do what we can to help.
With that in mind, all of our employees will be working from home and following the best practices for self quarantining where possible, regardless of symptoms or risks of past exposure. Prior to this, our employees already worked from home for the majority of their employment, so this is not a significant change in our business operations.
We do not anticipate any impact on our hosting or technical support.
On behalf of the entire Crucial Hosting family, I want to thank you for putting your trust in us and supporting our business throughout this pandemic. We are committed to working through this unprecedented challenge in such a way that ensures the best outcome for our customers and the community at large.
If you have any questions about this, please feel free to reach out to us.
General summary:
Unfortunately, our datacenter, PhoenixNAP, experienced very serious power failures throughout the evening of 11-13-18. These issues have been resolved by the datacenter. While we don't own this datacenter, these issues were widespread and affected our cage and equipment.
General summary:
At 9:15 PM (Arizona Local Time -700), the primary power at our datacenter experienced a catastrophic failure, but it quickly recovered in 5 minutes. Our router went down as a result of this, though none of our servers went down. After investigating this remotely, and seeing that such a critical device went down to what appeared to only be a failure on our "A Power" (we have both A and B power for our equipment), we went to the datacenter to see what happened in person.
While at the datacenter, the same circuits experienced much more damaging failure, and we were able to determine that the redundancy for the router had failed under the increased load, and after replacing the appropriate equipment, we brought our network back online. This outage lasted 22 minutes, starting at 11:20 PM (-700). While we could have swapped that part out relatively quickly, the primary power affected the "man traps" in the datacenter, so the datacenter security had to bring the many frustrated customers through a back door.
The datacenter continued to experience very severe power issues; however, all of our physical servers maintained at least one working PSU during these power issues. While the power issues lasted close to 3 hours, we were fortunate to mitigate that down to 27 minutes (the combined outage for the two incidents mentioned above). We've added additional redundancy in the network in place so that if this sort of issue does occur again, we shouldn't have any network downtime.
While the power issues damaged a lot of equipment (power supplies in particular), at this time we don't expect any further downtime. We will be replacing many power supplies that were destroyed during the datacenter's power failures, as well as other affected redundant equipment.
The datacenter itself will be providing an official "RFO" detailing the cause of these issues. If you're interested in that, we can pass it along once they have made it available. They will be doing an investigation on their end, but hopefully they will have that within several weeks.
In addition, we relied on our datacenter's IP services for our support system. Unfortunately we had issues with their network during these power failures as well. This caused our support system to be unreachable intermittently during the power failure event. In the short term, we're going to be moving that system to our redundant network, so that this sort of outage won't affect that system. Soon after that we'll setup a high-availability solution, to ensure those tools for communication with us are always available.
We're very sorry for these issues. We've never had this severe of a power issue at this particular datacenter in the 6 years we've been there. While things could have been worse, we always want perfection for our customers.
More info: PhoenixNap Status AlertsStarting at roughly 11:30AM Eastern Time, a Russian Network began exploiting a zero-day vulnerability in the Litespeed Web service. This allowed the attackers to cause the web service become unresponsive to requests; however, there is no evidence to suggest the vulnerability had any implications beyond creating downtime for customer websites.
Litespeed full disclosure:
- http://blog.litespeedtech.com/2017/02/17/ddos-attacks-taken-offline-by-litespeed-enterprise-5-1-13/
Crucial Hosting worked directly with Litespeed to provide feed-back and help identify the vulnerability, as well as assist in identifying the offending networks. We provided this information to other companies on the internet, so they could also mitigate this denial of service attack. We were the first company to publicly identify the attacking networks, and we began preventing those networks from continuing their attack on our servers almost immediately.
Since this was a zero-day attack, it affected all known version of Litespeed. The only solution to the problem was ultimately waiting for Litespeed to patch the code which allowed the denial of service to take place.
Some customers may have experienced intermittent access to their websites, between 11:30 AM Eastern Time and roughly 2:15PM Eastern Time. Most traffic continued to work fine, though some sites, in particular, were targeted especially for this denial of service. We were actively working to stop this malicious activity throughout the process, and as a result of that, we mitigated the vast majority of downtime.
We have continued to monitor all of our servers for downtime, and while we have not seen this issue since patching all of our systems, we will continue to monitor for issues going forward.
- Crucial AdministrationThe datacenter we lease equipment from will be performing a required power maintenance in the early morning on April 3rd between 1AM and 4AM EDT. They expect the maintenance to last roughly 20 to 30 minutes, and during that time our support desk and ns3.crucialwebhost.com will be unavailable.
While we do our best to avoid any outages affecting important systems of ours, physical server migrations are a reality of web hosting and running a datacenter.
While almost all of our infrastructure is in our own physical control at our Phoenix datacenter, allowing us to mitigate downtime to just a few minutes, our support desk and nameservers are strategically on different networks from our main one. This is intended to diversify our availability in the event of a network outage.
We expect very little impact from this maintenance, as most browsers will automatically use one of our other two available nameservers. NS1 and NS2 will not be impacted by this maintenance.
We apologize for any inconvenience this may cause.
Upgrade Your Magento 2.x Site Now
Today, Magento has made a new upgrade available that improves the security and functionality of Magento 2.x sites. The new release, Magento 2.0.3, is available for both Magento Enterprise Edition and Community Edition, and contains several security improvements, including:
In addition, Magento 2.0.3 includes performance improvements and functional enhancements to the Orders API, Google Tag Manager, permissions, and other areas. Full details on the functional enhancements are included in the release notes. More information on the security updates can be found on the Magento Security Center.
All Magento 2.x users are strongly encouraged to update immediately.
UPDATE
Magento has released Magento Enterprise Edition and Community Edition 2.0.4 to address a packaging issue with yesterday's release. If you have already installed Magento Enterprise Edition or Community Edition 2.0.3, you must replace it with the new version to ensure that you receive all security enhancements.
In our continuing effort to improve the Crucial Hosting network infrastructure and further reduce chance of an unplanned outage, we have scheduled the following Network Maintenance windows. During this maintenance there will be a short service outage as the maintenance is performed which we expect to last 5-20 minutes. Servers will be gracefully powered down and rebooted during the maintenance.
Network Technicians will be on site as this maintenance is performed to ensure that downtime is limited to the shortest amount of time possible.
If you are experiencing an outage during this maintenance window your services will be available in a short time as the maintenance is completed.
Thank you for your patience as this mandatory maintenance is performed.
Crucial Administration
=================================================== Scheduled Network Maintenance Window 1 Date: Friday, March 25, 2016 (03/25/2016) Start Time: 08:00 PM EST End Time: 02:00 AM EST (03/26/2016) Overflow: 03/26/2016, 8PM EST - 02:00AM (03/27/2016) Duration: 5 - 20 minutes Hosts affected: phx-1310.split-dedicated.com phx-1344.split-dedicated.com phx-1348.split-dedicated.com phx-1349.split-dedicated.com phx-1350.split-dedicated.com phx-1351.split-dedicated.com phx-1352.split-dedicated.com phx-1354.split-dedicated.com phx-1355.split-dedicated.com phx-1356.split-dedicated.com phx-1357.split-dedicated.com phx-1358.split-dedicated.com phx-1360.split-dedicated.com phx-1362.split-dedicated.com phx-1365.split-dedicated.com phx-1375.split-dedicated.com phx-1389.split-dedicated.com phx-1430.split-dedicated.com =================================================== Scheduled Network Maintenance Window 2 Date: Saturday, March 26, 2016 (03/26/2016) Start Time: 08:00 PM EST End Time: 02:00 AM EST (03/27/2016) Overflow: 03/27/2016, 8PM EST - 02:00AM (03/28/2016) Duration: 5 - 20 minutes Hosts affected: phx-1239.split-dedicated.com phx-1266.split-dedicated.com phx-1267.split-dedicated.com phx-1299.split-dedicated.com phx-1316.split-dedicated.com phx-1320.split-dedicated.com phx-1321.split-dedicated.com phx-1322.split-dedicated.com phx-1323.split-dedicated.com phx-1324.split-dedicated.com phx-1326.split-dedicated.com phx-1327.split-dedicated.com phx-1328.split-dedicated.com phx-1329.split-dedicated.com phx-1330.split-dedicated.com phx-1331.split-dedicated.com phx-1332.split-dedicated.com phx-1335.split-dedicated.com phx-1336.split-dedicated.com phx-1337.split-dedicated.com phx-1338.split-dedicated.com phx-1339.split-dedicated.com phx-1340.split-dedicated.com phx-1342.split-dedicated.com phx-1345.split-dedicated.com phx-1346.split-dedicated.com phx-1347.split-dedicated.com phx-1359.split-dedicated.com phx-1378.split-dedicated.com phx-1382.split-dedicated.com phx-1384.split-dedicated.com phx-1396.split-dedicated.com phx-1404.split-dedicated.com phx-1405.split-dedicated.com phx-1407.split-dedicated.com phx-1409.split-dedicated.com ===================================================
In our continuing effort to improve the Crucial Hosting network infrastructure and further reduce chance of an unplanned outage, we have scheduled the following Network Maintenance windows. During this maintenance there will be a short service outage as the maintenance is performed which we expect to last 5-20 minutes. Servers will be gracefully powered down and rebooted during the maintenance.
Network Technicians will be on site as this maintenance is performed to ensure that downtime is limited to the shortest amount of time possible.
If you are experiencing an outage during this maintenance window your services will be available in a short time as the maintenance is completed.
Thank you for your patience as this mandatory maintenance is performed.
Crucial Administration
===================================================
Scheduled Network Maintenance Window 1
Date: Friday, March 18, 2016 (03/18/2016)
Start Time: 08:00 PM EST
End Time: 02:00 AM EST (03/19/2016)
Overflow: 03/19/2016, 8PM EST - 02:00AM (03/20/2016)
Duration: 5 - 20 minutes
Hosts affected:
===================================================
===================================================
Scheduled Network Maintenance Window 2
Date: Saturday, March 19, 2016 (03/19/2016)
Start Time: 08:00 PM EST
End Time: 02:00 AM EST (03/20/2016)
Overflow: 03/20/2016, 8PM EST - 02:00AM (03/21/2016)
Duration: 5 - 20 minutes
Hosts affected:
===================================================
In our continuing effort to improve the Crucial Hosting network infrastructure and further reduce chance of an unplanned outage, we have scheduled the following Network Maintenance windows. During this maintenance there will be a short service outage as the maintenance is performed which we expect to last 5-20 minutes. Servers will be gracefully powered down and rebooted during the maintenance.
Network Technicians will be on site as this maintenance is performed to ensure that downtime is limited to the shortest amount of time possible.
If you are experiencing an outage during this maintenance window your services will be available in a short time as the maintenance is completed.
Thank you for your patience as this mandatory maintenance is performed.
Crucial Administration
===================================================
[COMPLETED] Scheduled Network Maintenance Window 1
Date: Friday, February 19, 2016 (02/19/2016)
Start Time: 08:00 PM EST
End Time: 01:00 AM EST (02/20/2016)
Overflow: 02/20/2016, 8PM EST - 01:00AM (02/21/2016)
Duration: 5 - 20 minutes
Hosts affected:
===================================================
===================================================
[COMPLETED] Scheduled Network Maintenance Window 2
Date: Saturday, February 20, 2016 (02/20/2016)
Start Time: 08:00 PM EST
End Time: 01:00 AM EST (02/21/2016)
Overflow: 02/21/2016, 8PM EST - 01:00AM (02/22/2016)
Duration: 5 - 20 minutes
Hosts affected:
===================================================
Magento has made new releases and patches available to improve the security and functionality of all Magento sites. While there are no confirmed attacks related to the security issues, certain vulnerabilities can potentially be exploited to access customer information or take over administrator sessions.
The security issues vary across products and all versions of Magento are affected. Full articles about the Magento 1.x and Magento 2.x issues are posted in the Magento Security Center. Additionally, all new releases and a separate USPS patch support recent USPS changes.
The Magento Community Edition 2.0.1 release also contains several important functional updates, including official support for PHP7.0.2, which provides dramatic performance improvements, drastically reduces memory consumption, and supports brand-new PHP language features. These updates are detailed in the release notes.
Ransomware is one of the most insidious types of malware that one can come across. These infections will encrypt all files on the target account. Magento e-commerce sites seem to be a big target of this new website ransomware, but it is by no means limited to them.
The Security Specialists at Sucuri.net have released an article on a new type of exploit that is taking special aim at vulnerable Magento eCommerce installations.
The malware was first discovered last November by Russian security company Dr.Web. Unsurprisingly, vulnerabilities in outdated software seem to be the main method of distribution. Remember web admins: Update, update update!
Ransomware Now Targeting Websites
Usually websites are used to spread PC-based ransomware; visiting the infected website will launch the payload onto the end-user machine, encrypting its contents. Earlier this year we documented a widespread campaign affecting WordPress sites that redirected visitors to exploit kits distributing ransomware. Itââ¬â¢s also commonly distributed through tech support scams and bogus e-mails, but now weââ¬â¢ve seen website admins fall victim as well. Magento e-commerce sites seem to be a big target of this new website ransomware, but it is by no means limited to them.
You can read the full article here. There is no 'fix' for this one other than to restore from backup - Make sure your backup strategy is in place or check out the Crucial Enterprise Backup Product available in your cPanel.